Entity Stamping Precision Judgment Guidelines

1 Task Overview
The goal of this task is to evaluate if an entity is one of the primary entities for a web page. A “primary entity” is the entity that is one
of the main topics for the web page. There could be multiple primary entities per page.

Once you log in to UHRS and have launched the HitApp, you will see a screen similar to this example below:

The HitApp shows two windows side-by-side:

• On the left is a web page. With link, Title and Main body.
• On the right is the entity pane with an information card for entity (People, Place, Organization, Event, etc.)
You need to judge if the entity in the right window is one of the primary entities of the web page on the left.

2 Judgment Instructions
2.1 Process Steps

1. Read the web page on the left.

2. Read the entity card on the right.
3. Determine whether the right entity is one of the main topics for the web page on the left.
4. Choose an option from below.
a. Primary main subject: any of three conditions (i, ii or iii below) are met
i. If the webpage is clearly a homepage of the entity displayed or dedicated to it.
ii. If using one sentence to summarize the webpage and the entity is one of the main subjects of this sentence;
iii. If the main content of the webpage is a forum/discussion thread, the opening post of the thread should be
primarily about the entity. If the main content is a Q&A thread, either the question post or the best answer
post (if exists) should be primarily about the entity.
b. Mention: any of the following conditions are met
i. The entity is mentioned outside of its main content area like navigation menu, sidebar, footer, or
advertisement OR
ii. The entity is mentioned in the left web page in the main content area BUT the entity is not the primary
main subject as explained in (a)
c. Related: any of the following conditions are met
i. The entity is not mentioned in the web page but the web page mentions some topic related to this entity;
ii. The entity is not mentioned in the web page but the web page mentions some other entities which are
highly related to the entity, for example: family members, competing companies, author and their book,
singer and their song, CEO and their company, company and its product;
d. Not Related: The content on the web page is totally not related to the entity.
e. Web Page Not readable: The following conditions are met:
i. If the left web page content is not loaded/displayed (in HitApp OR via click) OR
ii. If the entity information card is not loaded/displayed (in HitApp OR via click) OR
iii. The page is in a foreign language that you don’t understand;

5. Click “Submit”.

2.2 Note
 If the URL links in entity card do not load for any reason, please follow the URL link to view the webpage.
 Some webpages refer to an entity by an alternative name that is different from the one presented in the entity information
card. For example, http://www.screenjunkies.com/tag/the-rock-106/ refers to Dwayne Johnson by his ring name “The Rock”;
http://www.yelp.com/biz/the-bush-man refers to David Johnson (http://en.wikipedia.org/wiki/World_Famous_Bushman)
as “The Bush Man”. They are considered as the same entity. http://www.yelp.com/biz/the-bush-man-san-francisco

3 Rating Examples

Rating Description Examples

Entity Information Left Web Page (content URL)

Card
(a) Web page is Skyfall https://www.youtube.com/watch?
Primary primarily about or "Skyfall" is the theme song of the 2012 James Bond film v=q-gLRp5bSpw or
dedicated to the Skyfall, performed by British singer Adele. It was co- http://www.007. com/adeleskyfall-
entity. written by Adele and producer Paul Epworth and features themesong-is-live/
orchestration by J. A. C. Redford. Film company Eon
Productions invited the singer to work on the theme song
in early 2011, a task that Adele accepted after reading the
film's script. While composing the song, Adele and
Epworth aimed to capture the mood and style of the
other Bond themes, including dark and moody lyrics
descriptive of the film's plot....
http://www.lyrics.com/skyfall-lyrics-adele.html

Adele http://www.adele.tv/ (Adele’s official

Adele Laurie Blue Adkins MBE is an English singer- website)
songwriter. After graduating from the BRIT School for http://www.metr olyrics.com/adel e-
Performing Arts and Technology in 2006, Adele was given lyrics.html (A page dedicated to Adele
a recording contract by XL Recordings after a friend with the lyrics of her songs)
posted her demo on Myspace the same year. In 2007, she
received the Brit Awards "Critics' Choice" award and won
the BBC Sound of 2008 poll. Her debut album, 19, was
released in 2008 to commercial and critical success. It is
certified seven times platinum in the UK, and three times
platinum in the US. The album contains her first song,
"Hometown Glory", written when she was 16, which i...
https://en.wikipedia.org/wiki/Adele

Danny Sullivan http://www.linke din.com/in/dann

Danny Sullivan is the Chief Content Officer at Third Door ysullivan
Media, and co-founder of Search Engine Land, an industry http://twitter.co m/dannysullivan (main
publication that covers news and information about twitter page of Danny Sullivan)
search engines, and search marketing, SEO and SEM http://daggle.com/ (Blog of Danny
topics. Third Door Media also produces Marketing Land, a Sullivan)
sister website that covers broader digital marketing topics
including social media, display advertising, email
marketing, analytics, mobile, and marketing technology.
Search Engine Land and Marketing Land are owned by
Third Door Media, of which Danny Sullivan is partner and
chief content officer....
http://en.wikipe dia.org/wiki/Dan ny_Sullivan_(tec
hnologist)

(b) Danny Sullivan http://daggle.com/my-fall-of-the-berlin-

Mentio The entity is Danny Sullivan is the Chief Content Officer at Third Door wall-story-3509 (Danny Sullivan’s blog
n mentioned in Media, and co-founder of Search Engine Land, an industry post about berlin wall
)
 the left web page publication that covers news and information about
main content but search engines, and search marketing, SEO and SEM
the page is not topics. Third Door Media also produces Marketing Land, a
primarily about sister website that covers broader digital marketing topics
the entity OR the including social media, display advertising, email
entity is
marketing, analytics, mobile, and marketing technology.
mentioned
Search Engine Land and Marketing Land are owned by
outside of its
main content Third Door Media, of which Danny Sullivan is partner and
area (such as chief content officer....
navigation http://en.wikipe dia.org/wiki/Dan ny_Sullivan_(tec
menu, hnologist)
sidebar, footer, or
advertisement)

(C) The entity is not Microsoft
Related mentioned in the Microsoft Corporation is an American multinational
web page. But it corporation headquartered in Redmond,
highly related to Washington, that develops, manufactures, licenses,
the primary entity
supports and sells computer software, consumer
of this web page
electronics and personal computers and services. Its
best known software products are the Microsoft
Windows line of operating systems, Microsoft Office
office suite, and Internet Explorer web browser. Its
flagship hardware products are Xbox game console
and the Microsoft Surface series of tablets.
Apple confirms it now uses Google
Cloud for iCloud services
Apple has confirmed that it uses
Google’s public cloud to store data
for its iCloud services in its latest
version of the iOS Security Guide last
month, as spotted by CNBC. Reports
that Apple relied on Google’s cloud
services surfaced in 2016 but were
previously never confirmed.
https://www.theverge.com/2018/2/
26/17053496/apple-google-cloud-
platform-icloud-confirmation

Donald Trump Engine failure forces Ivanka, Jared's

Donald John Trump is the 45th and current
President of the United States, in office since
January 20, 2017. Before entering politics, he was a
businessman and television personality.
helicopter to return to airport
(CNN) A helicopter carrying Ivanka
Trump and Jared Kushner had to
return to an airport in Washington on
Thursday after one of its engines
failed, two law enforcement sources
told CNN. President Trump's
daughter and son-in-law were flying
from Washington to New York on
Thursday afternoon in a two-engine
helicopter when one engine failed,
causing the chopper to return to
Washington. The helicopter safely
made it back to Ronald Reagan
National Airport and the couple
 scrambled to get on a commercial
flight instead. The sources could not
say why the couple were flying to
New York via a helicopter instead of a
plane. The aircraft was a Sikorsky
helicopter owned by the Trump
Organization, according to
information recorded by the aviation
website LiveATC.net. Before the
engine failure, the plan was for the
chopper to take the couple and a
member of their security detail to a
Manhattan helipad, one of the
sources said. CNN's Aaron Cooper
and Susannah Cullinane contributed
to this report.
https://www.cnn.com/2018/03/18/
politics/ivanka-jared-
helicopter/index.html

The entity is NOT Tim Tebow

related to the Timothy Richard Tebow is a former professional American
web page at all. football player and current professional baseball
outfielder in the New York Mets organization. He played
college football for the University of Florida, winning the
Heisman Trophy in 2007 and appearing on BCS National
Championship-winning teams during the 2006 and 2008
http://www.linkedin.com/pub/tim-
seasons. Tebow was selected by the Denver Broncos in
tebow/13/28/b50 (A different person)
the first round of the 2010 NFL Draft and spent two
seasons with the team. He also played for the New York
Jets in 2012. Additionally, he had preseason stints with
the New England Patriots and the Philadelphia Eagles in
http://en.wikipedia.org/wi
ki/Tim_Tebow

Movies
"Movies" is a song by Alien Ant Farm, released as the first
single from their album Anthology in 2001, then re-
released to a larger audience after the success of
"Smooth Criminal". Though it peaked only at No. 18 on http://www.mov ies.com/ (A different
the US Modern Rock chart, it remained on the chart for entity)
(d) thirty-two weeks, five weeks longer than "Smooth
Not Criminal" which hit No.
Related http://en.wikipedia.org/wiki/Movies_(song)
 Michael Jordan
Michael Jeffrey Jordan, also known by his initials, MJ, is
an American retired professional basketball player,
businessman, and principal owner and chairman of the
Charlotte Hornets. Jordan played 15 seasons in the
National Basketball Association for the Chicago Bulls and
Washington Wizards. His biography on the NBA website http://www.cs.b erkeley.edu/~jor dan/
states: "By acclamation, Michael Jordan is the greatest (A different person)
basketball player of all time." Jordan was one of the most
effectively marketed athletes of his generation and was
considered instrumental in popularizing the NBA around
the world in the 1980s and 1990s....
http://en.wikipedia.org/wiki/Michael_Jordan

UPS Airlines
https://www.ups.com/
UPS Airlines is an American cargo airline owned by United
(note that the page shown has
Parcel Service Inc.. The company is headquartered in
interaction needed to see actual page, so
Louisville, Kentucky. Its international all-points air hub,
you are expected to click the url in
Worldport, is based at Louisville International Airport. The
HitApp and judge that.).
pilots are represented by the Independent Pilots
In this case the entity is specific but page
Association....
is general and does not mentioned the
http://en.wikipedia.org/wiki/UPS_Airlines
entity in topic area.

4 FAQ
a) Q: The page sometimes fails to load or content is blocked with a popup/overlay.

IF the left web page fails to load or content is blocked with popup/overlay, please open the page via content link given and
continue judging.

b) Q: I am using Internet Explorer 11 and I see the following message pop up when I open the HitApp, what should I do?

A: Click “Options for this site” button, and then click “Always allow” in the drop-down menu.

If see the “Are you sure you want to leave this page” dialog afterwards, click “Leave this page”, and the page will be
refreshed.
 c) Q: I am using chrome and I see the alert icon in the address bar, what should I do?

A: Click the red alert button, and then click “Load unsafe script” in the message pop up.

--You are ready judge now---