Ads NPNC Relevance Rating Guidelines v3: Workflow Goals + Context [PDF]

Zitiervorschau

Ads NPNC Relevance Rating Guidelines v3 Revision Date 12/11/2019

Workflow Goals + Context The purpose of this workflow is to identify relevant and high-quality ads for search queries on Facebook SERP (All search tab). For each ad to be rated, raters will see a search query containing the terms used to search and a sponsored post. This sponsored post includes the name of the advertiser, visuals such as images or video, and text commentary. Then, based on the content, raters will rate the ad for relevance if there is no defined product intent. The convention of {query} → [the main topic of result] will be used to provide examples in these guidelines. Definitions o Query is the text used to perform a search for specific results. The word(s) or phrase entered into the search engine (e.g. “cat videos,” “pancake recipes,” “Michelle Obama”.) o Result is an item returned by the search engine as an outcome of the search. Landing page is considered to be a part of result for all Ads workflows. o Product is the item that is the subject of the resulting post. Note that for our purposes a “product” can encompass multiple items for sale in the same result. o Searcher is a supposed person who conducts a search on Facebook All (SERP workflow) or Marketplace (MP workflow) tabs using the given query and receives the given result. o Searcher's intent is what the searcher intended or wanted to find when they typed their query into search bar considering the query itself, the intention behind those words and the context. The most traditional query intent classification includes informational (to learn about something), navigational (to find a specific site) and transactional (i.e. to buy a car or to download an app) intents. o SERP (Search Engine Result Page) is the page of search results that a search engine (e.g. Google or Facebook) returns in response to a query, i.e. Facebook All Tab SERP or Facebook Marketplace SERP.

o Facebook Marketplace is a digital marketplace where users can arrange to buy, sell, and trade items with other people in their area. o Workflow refers to the way we group together a particular type of search results, depending on what kind of search process we are interested in examining. For example, in the Ads SERP “workflow,” we look at only Ads showing in response for queries on Facebook All Tab SERP, and assess them for relevance. o NPNC: no people no commerce.

Search Ads NPNC Labels Skip/Do Not Skip • 0 - Skip o Gibberish Query o Foreign Language o Rendering Issue o No Landing Page o Brand Policy Violation • 1 - Rate Product Intent/ No Defined Product Intent • 0 - Product Intent • 1 - No Defined Product Intent Query-Product Relevance • 1 - Off-Topic o Poor Quality o Not Topically Related • 2 - Somewhat Relevant o Not the main topic o Location sensitive • 3 - Relevant o Complements o Substitutes o Meet most needs

0 - Skip SKIP is used when the job cannot be evaluated in Ads Rating workflows because of one of the following reasons: • The job cannot be rated consistently with other jobs, i.e. because of technical issues. • The job does not belong to Ads Rating workflows, i.e. the query or the result is not compliant with Facebook policies. A B C 1 Skip sub-label Brief explanation

2

Job cannot be rated consistently with other jobs

3 4 5 6 7

Job does not belong to Ads Rating workflows

Gibberish

Rater cannot understand the query

Foreign Language Rendering Issues No Landing Page Other Tool Issues

If the result including the landing page contains foreign language Error message shows up in SRT instead of a job The result does not include a link to the landing page All the other tool issues preventing rater from understanding the query or the result in full The query or the result do not comply with Facebook policies

Brand Policy Violation

8 SKIP/Gibberish: Skip if you cannot understand what the query means after conducting a thorough side search and considering possible search intentions. This sub-label includes queries in a foreign language and queries that have a very broad interpretation in side search or are not understood by side search at all.

•

Examples: {m}, {123879324509y14}, {семантика}

SKIP/Rendering Issues: Sometimes an error message shows up in SRT instead of a job. It happens for various reasons; for example, when an ad has been removed by the advertiser after the job was created. Skipping is the only way to remove a job

from the workflow so it wouldn't show up again. If a job has both rendering issue and no landing page, please skip under Rendering Issues. Please note: It is important to copy the error to the text box when possible, so that we can track these errors.

SKIP/No Landing Page: Please SKIP the jobs that render but do not have links to a landing page or have non-functional links under No Landing Page sub-label. We take the landing page into consideration when rating, thus these jobs cannot be rated consistently with the others. SKIP/Foreign Language: If the result including the landing page contains foreign language, and it prevents you or can prevent your colleagues from understanding the result consistently and in full, please label the job as skip/foreign language, for example:

SKIP/Other Tool Issues: SKIP the job if you cannot rate it because you cannot understand the result from what you see in SRT, i.e. visuals of the result are cropped too badly and you cannot comprehend the text on it. If a tool issue can be classified as a Rendering Issue or No Landing page scenario, use the corresponding sub-labels. If it can't be, choose Other Tool Issues sub-label and explain your selection in the text box.

SKIP/No landing page: SKIP the job if the link doesn’t load when you click on it so you are not able to rate the job. Also, if the external link does not have a working website or present a 404 error or non functional landing page it should be SKIP.

@

SKIP/Brand Policy Violation: Skip the jobs where either query is asking for Prohibited Content/Sensitive results, or the result can be classified as one of those, or if you discover it at any moment of further rating. Prohibited Content: Query asks for products from the list below or the resulting post falls under one of the Prohibited Content categories. • Prohibited Products: o Tobacco Products o Drugs & Drug-Related Products o Personal Health items o Unsafe Supplements o Unauthorized Streaming Devices o Surveillance Equipment o Spyware or Malware o Circumventing Systems o Counterfeit Documents o Weapons, including parts, accessories and related uncommon products o Any Illegal Products and Services • Prohibited Types of Content in the Ad: o Profanity or Bad Grammar

o Sensational Content, including shocking, sensational, disrespectful or excessively violent content o Misleading or False Content: deceptive, false, or misleading content, including deceptive claims, offers, or methods o Controversial Content: content that exploits controversial political or social issues for commercial purposes. o Third-Party Infringement: content that infringes upon or violates the rights of any third party, including copyright, trademark, privacy, publicity, or other personal or proprietary rights. o Personal Attributes and Discriminatory Practices: content that discriminates or encourages discrimination against people based on personal attributes such as race, ethnicity, color, national origin, religion, age, sex, sexual orientation, gender identity, family status, disability, medical or genetic condition or content that asserts or implies personal attributes. • Financial: o Payday Loans, Cash Advances and Bail Bonds o Multilevel Marketing o Penny Auctions o Prohibited Financial Products and Services such as initial coin offerings and binary options • Sensitive Content: Query is related to one of the topics from the list below or the topic is present in the resulting post: o Violence or Calls to Violence: any weapons, including weapon parts and accessories, display/antique weapons (e.g. display/antique swords) o Sexual Content and/or Nudity o Child Exploitation and/or Nudity o Human Trafficking o Acts of Terrorism o Self-Injury o Suicidal Content o Hate Speech o Bullying and Harassment Please note, if the category is contains Sensitive Content, escalate the Job ID to your lead immediately. The full list of Prohibited Content items and detailed explanations can be found here: https://www.facebook.com/policies/ads/prohibited_content. Note: #15 of the advertising policy (non-functional landing page) should be rated as SKIP/No Landing Page

Product Intent vs. No Defined Product Intent In order to determine if the query has no product intent (or vague product intent, VPI, as opposed to defined product intent, DPI), please conduct the side search and check products on Google Shopping Tab or bing.com/shop. Relevance rating will require a deeper understanding of the query and side search in “All” Tab, but possible product intents are the key to determine if the query is VPI or DPI. product intent NPNC workflow doesn’t rate Product Intent jobs. A good way to determine if a query has product intent is to ask yourself if any searcher would look for the results displayed in the side search using the given query. If the answer is “Yes”, and the side search reveals an intent for specific product or product category, please label the job as “0 - Defined Product Intent”. No further relevance rating is required for DPI queries. Here are some examples:

Product query {iPhone XS} has product intent for [iPhone XS], search query {285 75 15} has strong product intent for [tires.] If all the products in the side search belong to 1 or 2 products or product categories, the query has product intent for these product(s)/product categories. If the query mentions features or characteristics of the product, i.e. {iPhone XS 128GB}, it should be taken into consideration, and it also narrows the intent. Chances that [iPhone] is an expected result for query {iPhone} are very high. #productintent

Brand queries like {Apple}, {because cosmetics} can be matched with the products and product categories associated with the queried brand and have product intent. Side search results make sense and are expected considering the query. #productintent

Query for specialized retailer {footlocker} has product intent for [active shoes]. Side search results are focused on one product category and this product category is expected considering the query. There are other search terms to look for shoes, but at least some searchers can use this query to get these results. #productintent no defined product intent If you can easily think of much better queries to search for the results that you see in the side search and the side search results are not focused on a specific product or product category, rate the job as “1 - No Defined Product Intent” and proceed to relevance rating. For example, the query is {Seattle}, the result is [a t-shirt with depiction of the Space Needle]. The most reasonable product query to find this result is {space needle t-shirt}, and no product result including [a t-shirt with depiction of the Space Needle] cannot fully meet the searcher’s intent.

Broad retailers typically carry a wide range of products and product categories, and cannot be clearly associated with only one product category. Query for a broad retailer {walmart} has navigational intent for [www.walmart.com] or [walmart page/group on Facebook.] Chances that any searcher would intentionally search for a [tent] or an [Apple iPhone 6] using the query {walmart} are very slim. #nodefinedproductintent

Search query {Chicago} primarily has informational intent to learn more about the city. Product interpretation of this query is vague and side search returns topically related souvenir products at the top of shopping results which typically is a good indicator of no product intent. #nodefinedproductintent

Similar to {Chicago}, query {nicki} primarily has informational intent to learn more about the rapper and no specific product intent. Side search returns tshirts. #nodefinedproductintent

Broad product categories: Queries {hair}, {Christmas}, {moving} can be associated with some products but do not make the expected product clear and may have other interpretations than the product intent one. A searcher might expect the products returned by the side search, but the query is not strongly focused on one of them, and there are much better search terms to look up any of the returned results (for example, “hair extensions”, “blond wig”, “dry hair treatment”, “hair regrowth treatment” etc.) #nodefinedproductintent

Food and drinks: query {Aztecan Mexican}, the query’s intent is for a restaurant with this title, likely located near the searcher. Food and drink establishments have no product intent, as they are informational queries. Food and drink establishment have no defined product intent, as they are informational queries. #nodefinedproductintent

Please note: Most queries in this workflow should not have defined product intent (DPI). If there is defined product intent, raters should select “0 - Defined Product Intent”, and the job is considered complete with no relevance decision needed. Also, note that if a company/brand offer both services and products, it should be considered as No Defined Product Intent.

Side Search for not defined product intent queries Please conduct a thorough side search on “All” Search Tab. It is important to conduct the side search with and without searcher’s location as the same search terms may refer to different businesses based on location. For example, • {golden boys} is primarily related to novel by Australian author Sonya Hartnett, • {golden boys Austin} refers to a musical band, • {golden boys San Francisco} refers to a pizza restaurant,

•

{golden boys Fitzgerald GA} refers to a food wholesale company

Relevance 1 - Off-Topic Off-Topic / poor quality First, please note that if the quality of the displayed job in SRT prevents you from understanding the result, SKIP this job under SKIP/Other Tool Issues or SKIP/Rendering Issues whichever seems to be more appropriate. Job can be rated as Off-Topic/Poor Quality based on poor visuals in the ad or because the ad may create misleading experience for the searcher. Each ad is accompanied by visuals like photos or videos. Some of the visuals may employ disruptive techniques like flashing screens in videos to attract more attention, which goes against Facebook policies and should labeled Off-Topic / Poor Quality. Most common factors creating poor viewing experience of photos include: • Lighting effect is too light or too dark • Has oppressive text, masks, excessive filters that distract from the viewing experience Most common factors creating poor viewing experience of videos include: • Poor stability and wildly moving video (unless the context of the video justifies it) • Subtitles are overwhelming and take up the entire screen • Video of a video The following factors are not considered to be Poor Quality indicators: • Photo: Resolution is grainy or blurry, • Photo: Aspect ratio is skewed causing distorted images, • Photo: Image is cropped badly and makes it visually jarring, • Video: Video does not have the correct sharpness, contrast, size, and aspect ratio for a good visual experience, • Video: Poor audio or static (unless the context of the video justifies it.) SRT is aimed to convey the main idea of the ad to rater, but it simplifies the imagery. Thus, these cases are really tool issues rather than bad quality ads. Please find some examples below:

Adfarm.mediaplex.com is used by brands to track where visits are coming from, but it is perceived as misleading AND does not return an expected page for this ad, hence this result should be rated Off-Topic / Poor Quality.

The paragraph states that 5 million people like this product, but in reality there are only 5 likes. This should be rated as Off-Topic / Poor Quality for containing misleading/false content. If the major topics of 1) the ad commentary, 2) ads visuals and 3) the landing page outside of Facebook are not quite coherent and do not correspond with each other, the ad may create misleading experience for the searcher.

Example of a misleading job, SRT UI screenshot. The post commentary and the visuals are about furniture, and at first the ad seems to be relevant to the query.

The landing page is completely on different topic. Considering searcher’s intent, this result can create bad searcher experience, and should be rated as Off-Topic / Poor Quality Job can be also considered misleading and rated as Off-Topic / Poor Quality if the ad visuals do not match the ad commentary. An unrealistic price or 50%+ price difference between the ad and the landing page are also indicators of Off-Topic / Poor Quality. If the price difference is less than 50%, disregard the price and rate normally. Kindly note that if the external link on SRT job leads to a landing page within Facebook domains (fb.me, facebook.com), in some cases you won’t be able to access the landing page content at the moment of rating. These jobs are not considered misleading. Off-topic / not related The ad in the result is not related to query’s primary intent and does not look like a natural or reasonable response to the query. The result may or may not match keywords, but if it does, the match is in a random or unintelligible way. In other

words, the result should be rated as Off-Topic when the result does not make sense considering the query and there are no or very slim chances that any searcher would be interested to learn more about this result. • Random word match: o {Fox News} → [domesticated foxes] • Random word match: o {land} → [informational article on land rover vehicles] • Based on the side search, the query has navigational intent for the popular page https://tasty.co/, example is not topically related: o {Tasty Videos} → [A mouth-watering video posted by someone of their gigantic burrito with the caption “tasty”] • Navigational intent for Facebook page https://www.facebook.com/AmandaEllersMissing/, not related: o {findamanda} → [MCO RodBuckle] • Category “Cities of the U.S.” matches, but there are no other common topics and there are no chances that any searcher’s need could be met: o {city of dallas georgia – government} → [promotion by ‘Visit Las Vegas!’ Facebook page] • The result is clearly not expected for given search query o {Housekeeping services} → [Hotel cart housekeeping sold by Walmart] Note: word match is not always necessary for topical relevance, especially for vague product intent queries. For example: • Topically related: o {bad credit} → [Get Your Next Used SUV Today Regardless of Credit Score. Warranty Included and a Carfax History Report]. While the product intent of the query {bad credit} is broad, the result is a meaningful response to the query and the searcher might be interested to learn more about it. “Bad credit” and “regardless of credit score” are not strictly synonyms, but “regardless of credit score” implies the idea of “bad credit” in given context.

2 - Somewhat Relevant Somewhat Relevant is used when the result is loosely related to the query, for example, the result may satisfy the same need, but we cannot assume it is a perfect match to the query. Somewhat Relevant result is of interest to the searcher considering the query, but does not fully meet searcher’s need, for example, it can possibly satisfy the same need with another product or service, or it can be helpful to the searcher in any other quality, but it might be not the exact match and not fully meet most searchers’ intent. Somewhat Relevant/Location sensitive If user is explicitly looking for a service (restaurant, grocery, cinema, etc.) in a

specific location and the ad is in a different location > 100 miles radius from the specified location, rate as Somewhat Relevant. If the ad is an online service, then location does not matter. o {sushi restaurants in Mountain View} → [Top 5 restaurants in New York] o {tugwell roofing California} → [roofing service in Texas] o {motes pools Arizona} → [Solstice pools Houston] Note: • If the user is looking for a service in a specific location but it’s a brand chain that has multiple branches and the ad is from a different location, in this case, the distance doesn’t matter, thus rate as relevant. o {Trader Joe’s San Jose} → [New product launch in Trader Joe’s San Diego] → relevant • If user is looking for a specific school but the ad is an online program of a similar education, in this case location should not matter (education category). o {MBA Berkeley} → [Online MBA in Columbia] → relevant somewhat relevant / Not the main topic The query and the result (including the landing page) have a topic in common, but this topic is not the main focus of the ad and/or the landing page. • Topic is not the main focus of the ad and/or the landing page. The query and the result (including the landing page) have a topic in common, but this topic is not the main focus of the ad and/or the landing page. Somewhat Relevant includes other search engines as focused on searcher’s intent clarification, pages that require sign in or sign up as focused on collecting data, and all the other scenarios when the queried topic or reasonable substitutes are not the main focus of the ad and the landing page. o {Las Vegas} → [Las Vegas Getaway] User is looking for Las Vegas, though it’s a vague query the landing page seems to be relevant to the query, but the searcher is required to introduce the contact information in order to see the result so this job is somewhat relevant/not the main topic. o {The Handmaid’s tale} → [Hulu- subscription free trial] In this example the user is looking for a specific TV show and the ad is a landing page to enter the contact information in order to try the free trial. This LP doesn’t allow us to rate the job but it’s topically related. •

•

Resulting ad’s topic is not the primary intent of the query: query and result have word match, but the result is not the most common product/service that user would search for with the given query, but could be of some interest. o {John Mayer} → [John Mayer electric guitar] Query and result are under the same category but it’s not exactly what the user is looking for but it’s still related.

o {Trader Joe’s} → [Hello fresh black Friday offer meal prep delivery] Query is for a public figure/celebrity, in this case a singer, songwriter. The primary intent would be information, articles, and videos of this celebrity. The resulting ad is an item (a guitar) from the celebrity’s product line, which would be a secondary intent and may not be exactly what a searcher is looking for, but could be of some interest. o {emily king} → [an ad for a concert subscription]: intent for an American singer. ▪ category is ‘Music & Audio’, there is topical connection and shared topic, but the topic of the ad is not directly related to Emily King. o {kopkey manure management} → [Manna Irrigation ad: “A new technology in the irrigation industry, Manna provides a software only, sensor-free irrigation recommendations solution that is based on satellite technology“]: ▪ Navigational intent for Facebook page Kopkey Manure Management, category is ‘Home & Garden’. ▪ The query and the result are acceptably topically related, but are not directly related to each other.

3 - Relevant The Relevant label is reserved for results that meet most searchers' need considering the query. There should be a high chance that the searcher would have continued interest in the result and learning more detail about ad's main topic. relevant / substitutes Query is for a specific service, result is other services that satisfy a similar need or desire. This category may include proper name/location mismatch, but still has the same general operation. o {dean smith photography} → [erickmarquez_photography] o {lion's tail brewing co.} - [McFleshman's Brewing Co.] o {joes pizza ann arbor} → [Guido's Pizza] o {national life group} - [Colonial Penn Life Insurance] Note: If the service location is specified in the query, ad shows substitute service more than 100 miles away, then rate as Somewhat Relevant as instructed in Somewhat Relevant section. relevant / COMPLEMENTS User’s query refers to a service and the shown product/service does not completely satisfy the user's need but can be used for that service. o {Dean Smith photography} → [picture frame] o {Yoga Classes} → [Yoga Mat]

o {Blue Horsehoe Tattoo Hampton} → [Revitalize Tattoo Aftercare] o {Ski Resort in Tahoe} → [Daily equipment rental to ski] relevant / mEET MOST SEARCHER’S NEED The Relevant label is reserved for results that meet most searchers' need considering the query. There should be a high chance that the searcher would have continued interest in the result and learning more detail about ad's main topic. • Result that responds directly to the query: o {disney} → [Disney cartoon theme event]: Category is ‘TV & Movies’, the results looks very natural considering the query. o {coffee} → Seattle's Best Coffee‘s post [We’re celebrating #NationalDonutDay on June 7 with free samples of Seattle’s Best Coffee® and Walmart Freshness Guaranteed donuts! Click to find events at your nearest Walmart]: category is ‘Food & Drink’, the result naturally responds to the query. • Common products reasonably related to the query can all be 4 - Relevant. o {hair} → [hair extension/shampoo/hair iron/hair color]: category is ‘Fashion & Style’, the result is very reasonable considering the query. o {garden} → [garden plants/soil/water hose/pruner]: category is ‘Home & Garden’, the result belongs to broad category pointed by the query. o {Nicki Minaj} → [Nicki Minaj music CD]: category is ‘Music & Audio’, the result is a natural response to the query and can be helpful for many searchers.

FAQ/Troubleshooting Q: I can't get into the queue! A: There are many reasons you might not be able to enqueue. First, check the group chat to make sure that your workflow is still using that queue. Try to enqueue a few times, and make sure your internet connection is steady. If you still can't get in, screenshot the error and send it to your lead. Q: I got a maxed out error. A: This might mean that either you've done all the necessary work OR there's an error with the queue. Screenshot the error message and send this to your lead to see what you can do next. Q: I accidentally submitted a job before I was done. Can I go back in SRT and fix it, or pick up where I left off?

A: Unfortunately, you can't go back and change your answer in SRT. The flow will also not pick up where you left off last, so it's best to finish the post you're on before closing the browser.