!

REPORT: Enterprise Crowdsourcing, A Flexible Solution Table of Contents

1. Introduction 2. Enterprise Crowdsourcing 3. A Flexible Alternative 4. Turnaround Time, Quality, Cost Savings 5. Ensuring Quality 6. Applications 7. Proof Points

Introduction
All businesses have large-scale information and data needs that require human annotation to be completed successfully. Whether it be collecting and correcting data, categorizing information, or even creating content, internal teams are often not scalable, while outsourcing poses management headaches and flexibility issues. For the first time, there is a fully managed, on-demand solution to these problems that elegantly handles fluctuations in volume and offers accurate results at an affordable price. That solution is Enterprise Crowdsourcing.

Enterprise Crowdsourcing
Enterprise Crowdsourcing matches the scale of crowdsourcing with the reliability and stability that enterprises require. The hallmark of enterprise crowdsourcing is thousands of individuals, distributed throughout the world, participating independently and in parallel. For enterprises, the most successful projects focus on simple, repetitive tasks (micro-tasks) that can be performed by anyone with internet access. With contributors in nearly every time zone, crowdsourcing offers true always-on capabilities. Enterprise Crowdsourcing promises multiple reviewers for every data point. This redundancy leads to more predictably accurate results than relying on the work of a single person. Contributors opt-in with automated management and oversight, so overhead is greatly reduced. ! On-going quality control and training in real-time ensure the highest levels of accuracy. 1!

Enterprise Crowdsourcing, A Flexible Solution

! Where Work Comes From: Contributor map from ONE Crowdsourcing Project

Crowdsourcing: A Flexible Alternative

Internal Teams Internal teams are often the first step of a data annotation project. What these teams lack in scale, they make up for in flexibility and minimal startup costs. Simply put, internal teams are often the fastest way to spin-up a project. However, as the demands on a project grow, internal teams have trouble scaling up. As well, internal resources are often tied-up doing repetitive tasks when they could be adding more value on higher-level projects. Outsourcing Outsourcing is a possible next step if a project outgrows its internal team. However, large-scale outsourcing engagements require precise planning to hire, train, and occupy workers. Depending on the stage of the project, these needs vary substantially. When starting a new project, the manager must invest considerable resources to find and train the right workers. Ongoing training is also a large expense.

For on-going projects, outsourcing vendors require a steady supply of work to keep their employees occupied. Attrition of outsourced personnel is a serious concern, and it is the most productive workers that are most likely to leave a project that doesnt promise steady employment. !

Enterprise Crowdsourcing, A Flexible Solution

2!

! Crowdsourcing Enterprise Crowdsourcing offers an appealing alternative to tying-up an internal team or engaging a traditional outsourcing vendor. With access to a pool of contributors of unprecedented size, crowdsourcing can scale up and down dynamically to meet an organizations needs. A two-week delay in data procurement no longer means having to find other projects to keep a team occupied. Additionally, crowdsourcing can handle spikes in volume without the need to contract additional employees.

Turnaround Time, Quality, Cost Savings

In addition to increased scalability and flexibility, crowdsourcing often offers improvements in at least one of these following additional characteristics: turnaround time, quality, and cost.

Turnaround Time Enterprise Crowdsourcing uses a massive team of contributors, so projects often complete more quickly. Additionally, with a team distributed across the globe, crowdsourcing offers a truly always-on workforce, capable of processing data around the clock.

Enterprise Crowdsourcing, A Flexible Solution

3!

! Quality Enterprise Crowdsourcing also uses redundant annotation, and so the impact of any individual bias is reduced, often improving accuracy. Furthermore, by controlling the entire interaction between data and annotator, crowdsourcing provides highly detailed information to track and predict overall accuracy.

Cost Savings And lastly, because Enterprise Crowdsourcing automates the management process, costs are often greatly reduced. Crowdsourcing also eliminates slack-time, by only engaging with contributors when theyre actively participating on a project.

Applications
In general, any data or content related projects that are highly repeatable with an objective, unambiguous correct response are good candidates for Enterprise Crowdsourcing. Additionally, tasks with high volume and on-going needs are better able to take advantage of the economies of scale that Enterprise Crowdsourcing offers.

eCommerce Solutions Boosting conversions by offering a well-organized and personalized shopping experience is the name of the game for eCommerce companies. To do this, eCommerce companies need quality product data classified appropriately in their product catalog.

Product Attribute Annotation Product Attribute Verification Product Image Moderation Product Disambiguation Product Categorization Product Search Relevance !

Find the UPC for this product Is this a 12.0 MP camera? Does this image match this product title? Are these two products equivalent? Is this a Camera, a Camera Accessory or something else? How relevant is this product to the query Xbox 360?

Enterprise Crowdsourcing, A Flexible Solution

4!

! SEO Content Creation and Verification Search rankings are vital for most online businesses. Media companies rely on SEO content to drive advertising and ecommerce companies rely on SEO content to boost conversions. Moreover, fresh content is often good for business regardless of search results, keeping users engaged and satisfied.

Generate content Analyze Content Edit Content

Write 3 sentences on what a 401(k) plan is Verify that the keyword 401(k) exists in this description Edit spelling and grammar for this product review.

Business Lead Verification and Generation Data Providers, Search Companies, and Group Buying sites all need highly accurate business listing information to gain an edge. Any large company with a massive customer list needs the same. Crowdsourcing offers the only real-time solution for accurate business listing information.

For existing lists this means quality assurance- ensuring leads meet minimum accuracy standards- and enrichment- adding additional data that may not be supplied. Furthermore, crowdsourcing can generate customized lists of leads that meet your business specific criteria.

Verify accuracy of leads Enrich existing leads Generate custom leads

Is the phone number for this business correct? What hours is this business open? Find 25 BYOB Restaurants in St. Louis

Enterprise Crowdsourcing, A Flexible Solution

5!

The Four Cs of Crowdsourcing

The above examples are by no means exhaustive. When considering projects to crowdsource, you can use the Four Cs. Collect Any objectively verifiable piece of data accessible on the internet can be collected by crowdsourcing. Any objectively verifiable piece of data accessible on the internet can be confirmed or corrected by crowdsourcing. Given specific business rules or an existing taxanomy, crowdsourcing can categorize any type of information. Crowdsourcing can generate or transcribe content successfully as long as it is no more than a few paragraphs in length.

Correct

Categorize

Create

Proof Points
Client Profile Fortune 50 eCommerce Company Problem, Previous Approach, and Challenges Needed millions of discrete product relevance annotations Using BPO and internal team and still needed more scale Fortune 50 eCommerce Company Needed a scalable, cost-effective way to annotate product info Outsourcing solution not flexible enough Large Group Buying Company Needed data to qualify and prioritize leads No cost-effective solution existed previously Large Media Company Needed to generate high quality answers to user-generated questions Looking for more cost-effective, scalable solution ! Within a week generated thousands of answers, between 100 and 1,000 characters in length 90% of results deemed excellent by client 6! 100x faster throughput Able to reassign outsourcing team to other projects Qualified and prioritized Tens of Thousands of Leads per day Improvement effectively unlimited overflow capacity

Enterprise Crowdsourcing, A Flexible Solution

! Large Financial Services Company Needed handwritten forms transcribed into digital format Internal teams could not provide scale Fortune 1000 Internet Services Company, Local Search Team Needed extremely accurate and enriched business listing data Purchased list could not provide accuracy or richness of data Large Educational Institution Classification of images Small internal team Couldnt scale Classifying 10s of images per week to classifying 1,000s of images per week Achieved <98% accuracy for core business data and additional attributes Transcribed 25,000 fields daily at 98% accuracy

CrowdFlower is the worlds largest enterprise crowdsourcing platform, with access to over a million on-demand,!contributors around the world. Crowdflowers platform combines statistical quality control technology and automated workflow management to ensure superior quality at unprecedented scale. Clients of all sizes, including numerous fortune 500 companies, enjoy increased flexibility, turnaround time and cost savings for their information and data needs.

FOR MORE INFORMATION

Contact a platform consultant today at solutions@crowdflower.com

Enterprise Crowdsourcing, A Flexible Solution

7!