CrowdWorx Berlin | Munich | Boston | Poznan team@crowdworx.

com

CrowdWorx White Paper Series

The Rise of Social Forecasting How the Crowd Learned to be smart

Abstract Social Forecasting is an Enterprise Crowdsourcing approach for the aggregation of distributed knowledge from employees and experts which is then converted into quantitative business KPIs for use by management. We explore how Social Forecasting has been used in corporations and which mechanisms are behind the excellent forecasting performance of these Social Business tools. We also take a look at the challenges companies face when introducing Social Forecasting and Crowdsourcing tools.

CrowdWorx, Analyx 2007-2011. All rights reserved.

The Rise of Social Forecasting

Introduction to Social Forecasting

What is Social Forecasting? Social Forecasting is a Crowdsourcing approach used in companies for forecasting sales, new products or strategic scenarios and KPIs. Social Forecasting is based on the wisdom of the crowds, i.e. your employees contribute their knowledge to the system. The approach has a set of truth-telling incentives which immensely increase the quality of the inputs, which in turn results in excellent predictions by the system. A key ingredient of Social Forecasting are Prediction Markets (PM), sometimes called Decision Markets or Information Markets. PMs allow employees to bet on outcomes which they forecast, e.g. the sales of product X in Q3. The closer a participant forecasts the actual outcome the more play-money he or she can win on the Prediction Market. This ensures a very high incentive for each participant to make the most accurate forecast. The system then combines these component forecasts into one crowd forecast which then in many cases is more accurate than traditional methods like statistical modelling or Business Intelligence (see examples below). By the way: Those participants who do forecast poorly are automatically weighted lower by the system. The earliest applications of modern Social Forecasting with prediction markets are from the 1980s at US universities who used small financial markets with a couple of hundred students to test markets as information gathering tools with spectacular results (see Plott and Sunder, 1982; Plott, 2000). In the early 1990s, the Iowa Electronic Markets (IEM), an ongoing university project, was launched as a web-based market focused on forecasting election results. The IEM proved remarkably accurate in forecasting election results as compared to traditional surveys and expert judgement (see Table 1).

gence of employees to forecast business KPIs such as sales (Chen and Plott, 2002), project deadlines (Ortner, 1996) and probabilities for important events, e.g. competitor market entry likelihood (Keller, 2007; Berg, 2007; Cowgill at al., 2009). Since then Corporate Prediction Markets have come a long way and have seen many implementations. In a recent McKinsey survey among 3,249 executives worldwide lists Prediction Markets as one of the top Web2.0 tools used in companies (Bughin and Chui, 2010).

Who is using Social Forecasting? The number of companies using Predictive Markets and other Social Forecasting tools has steadily grown with 2010 being a breakthrough year for Social Business tools. Nowadays, Web 2.0 concepts are being perceived as useful business tools by CEOs and business leaders. Table 2 lists some examples of corporate applications of Prediction Markets from both big and small firms from various industries.

Why Social Forecasting works

Social Forecasting is more accurate than other methods, but why is that so? The answer to this question is not a straightforward one because social mechanics are not like physics where B always follows from A. Prediction Markets are a mixture of concepts from Decision Theory (incentive system), Crowdsourcing (utilize the knowledge/skill of many), and Collective Intelligence (turn the knowledge/skill of many into results). Incentives increase quality of inputs The fact that participants can win something in a Prediction Market causes them to give more accurate and more truthful answers. This has been scientifically proven (see Luckner, 2007; Sprenger, 2007). As a result, the quality of the inputs to the forecast is much better when using Social Forecasting. In addition to that, participants can weight their input by staking more or less play-money. This additional weighting information further increases the quality of the resulting forecast.

1992 Number of elections Elections where poll was more accurate Social Forecasting more accurate 151 28%

1996 157 13%

2000 229 24%

Total 537 22%

72%

87%

76%

78%

Self-adjusting crowd wisdom Participants who have proven to be more accurate than others are automatically weighted higher by the social mechanism. People who forecast poorly are weighted down. That way the system maximizes the quality of the inputs and the accuracy of the forecasts.

Table 1: Comparison of accuracy of polls and Social Forecasting in U.S. election forecasting. (Source: Rietz et al, 2003)

After these promising scientific results, the late 1990s saw the first early adopters from the corporate world take Social Forecasting into business applications using the Collective Intelli-

CrowdWorx, Analyx 2007-2011. All rights reserved.

The Rise of Social Forecasting

Company CrowdWorx project: Deutsche Telekom CrowdWorx project: Henkel CrowdWorx project: Tchibo CrowdWorx project: MVS Zeppelin CrowdWorx project: Syngenta General Electric Ford Motor Company Siemens Illy Lilly Pfizer Abbott Laboratories Best Buy Hewlett Packard Google Microsoft ElectronicArts Touchstone, Simon & Schuster ArcelorMittal Munich Re Thomson Financial Nokia Motorola Intel Dentsu Corning FritoLay

Areas of application New product potential, strategic topics Sales forecasts of existing and new products Sales forecasting in retail Forecasting utilization, actual sales prices, marketing campaigns, impact of new technology, and strategic topics Market forecasts, demand forecasting, business planning Ideation / sourcing and assessment of new ideas. Higher quality of ideas than before. Sales forecasts, potential new car features, electrification and economic KPIs, commodity prices. Forecasting project deadlines. Two months advance warning works as an truth-telling early-warning system. Chances of pharmaceutical products and substances in pipeline Chances of pharmaceutical products and substances in pipeline Chances of pharmaceutical products and substances in pipeline, risk management Sales, new products, new shop openings, launch dates, strategic topics. Higher forecasting accuracy than internal forecasters. Printer sales. Significantly higher forecasting accuracy than other methods used by HP. Service utilization, demand forecasting, industry outlook, project deadlines. Project deadlines. Prediction Market correctly predicts project deadlines. Project management, launch dates, product quality. 50% smaller error than internal forecasts. Selection of titles for publication Industry outlook, demand forecasts. PM measures employee satisfaction for management. Successfully supported 1 year internal re-structuring phase. Assessment of investment opportunities. Increase CRM performance of 1,000 agents across cultures of 174 countries. Product innovation, rate new ideas Demand planning, 20% lower error than previous methods Advertising campaign optimization Market growth for LCD TV, price elasticity, predicting industry cycles. Predicting the success of new product platforms

Table 2: Examples of corporate use of Prediction Markets. (Sources: CrowdWorx projects; NewsFutures, 2010; Berg, 2007; Burnham, 2009; Chen and Plott, 2002; Cowgill et al., 2009; Farmetrics, 2009; Fogarty, 2007; Hengl et al., 2008; Keller, 2007; LaComb, 2007;, 2011; Needle, 2011)

CrowdWorx, Analyx 2007-2011. All rights reserved.

The Rise of Social Forecasting

Law of large numbers A law from mathematical statistics called Law of Large Numbers is also at work in Social Forecasting. The law says that the larger the sample, the closer it converges to the true value of the underlying target. In Social Forecasting this law means that errors of many people are cancelled out and due to the incentives (see above) they cancel out in the right direction, i.e. errors cancel out instead of accumulating. This law is well documented in group psychology where it has been shown that group discussion usually reduces the variance of the group members opinions (see Sprenger, 2007). Using modern web technologies Social Forecasting enables companies to collect the knowledge of large numbers of people at low cost, which maximizes the effect of the law of large numbers. No politics Social Forecasting is an effective way to eliminate common individual biases and group biases, such as groupthink, overconfidence, conformity and strategic behavior. In a series of experiments Sprenger et al. (2007) ran group discussions and Prediction Markets both consisting of biased individuals. In the group discussions, the biases remained while the Prediction Market completely eliminated the biases due to the more objective interaction of people via the market and the incentives to disclose information. Influential group members significantly biased the group decisions, while in PMs their influence was eliminated. Feedback loop Direct feedback is a major characteristic of discussions. But it is completely missing from surveys, which are traditional way of Crowdsourcing. Prediction Markets have a feedback mechanism, which goes via the prices of the market. But it is more subtle than in a discussion, thus eliminating the impact of opinion leaders, which can severely bias an otherwise smart group of people. Other characteristics

Topics Whether you are using Wikis, blogs, surveys or a Social Forecasting, it is important to have relevant topics. The topics must be relevant both for managers, who are the ultimate users of the social forecasting results, but also for the crowd itself. Luckily, topics which are important for a companys well-being, such as sales, marketing, strategy, new products etc., will also be of strong interest to both managers and participants because everybody is a stakeholder and there is always at least some involvement for normal employees, too. After all, it is about ones own company. Participants Having chosen the topics of interest, the next step is to select participants who might be knowledgeable about those topics. Participants and topics are very closely related. It is important to have a so called wise crowd for the topics at hand. In contrast, a representative sample is not relevant for Social Forecasting methods. How many participants are needed? This depends on the activity of people and the topics at hand. We can establish some good rules of thumb: 50 participants are needed for 30 forecasts per week (e.g. 10 products in 3 regions per week or month), 100 participants for 60 forecasts, 200 participants for 100 forecasts. Social Forecasting platform and system Even when having a well-chosen set of topics and a wellcomposed crowd that is knowledgeable about these topics, one still needs to make sure that this setup is not spoiled by utilizing an inferior platform. A good platform must be easy to use without destroying the incentives by oversimplifying. Most vendors spent many years in optimizing their user interfaces until these have matured to a level that will ensure that employees of every level can easily use the system without the need for dedicated trainings. Incentive design

Some more technical but nevertheless accurate explanations of Prediction Market power can be found in the Efficient Market Hypothesis and the theory of marginal traders. For some fascinating experiments with Prediction Markets please refer to Plott and Sunders, 1982, 1988 and Plott, 2000).

Best practices in Social Forecasting

To unlock the benefits of Social Forecasting organizations need to consider some basic ground rules. There are six broad success factors of Social Forecasting introduction projects.

This is one of the key design decisions of any Crowdsourcing initiative. A professional vendor should provide consulting in this area based on his practical experience in order to guide a client though the setup of a good incentive scheme. The impact of the incentive scheme should not be underestimated, as shown in a study by Luckner (2007) who demonstrates how the same Prediction Market can deliver different results based on the incentive schemes used. Monetary or material prizes are not the only way to incentivize people. Recognition and other social factors have proven to be strong incentives, too. While rankings are a good idea in theory, the modern way to incentivize people is by lotteries, levels and other concepts coming

CrowdWorx, Analyx 2007-2011. All rights reserved.

The Rise of Social Forecasting

from the world of games. This move towards individual incentive schemes is often referred to as gamification. Communication When a Crowdsourcing system is ready to be launched, one cannot simply switch it on like a machine. Since Social Forecasting is a social phenomenon, one needs to oil the social machine and in this case the oil is communication. One has to introduce the system to the target group with simple but welltargeted information, such as newsletters, mailings, blog entries etc. This may sound simple but again the tone and timing of the communication will be important for the people to accept the new initiative. Management support We all know that a corporate project cannot be run without a dedicated high-level sponsor. Here the new world of democratic social media intersects with the well-established rules of corporate organization. A sponsor must clearly see the business use and benefits of the initiative and not only support it because wikis and blogs are currently hot. Management often requires some measurable output and this is where Social Forecasting excels. Unlike wikis and blogs, Social Forecasting does not create endless amounts of text and data. Instead, it bundles the knowledge from participants and turns it into quantitative KPIs. The quality of these crowd-generated numbers can be easily assessed with traditional tools, e.g. forecasting error analysis, which gives a very clear measurement on the business impact of Social Forecasting.

Secondly: Sometimes when a Social Forecast is not dependent on an outside event, e.g. when comparing new products there is no objective event by which to determine which product would be must successful in the market. In these cases herding is controlled by other mechanism, e.g. limiting the number of inputs per user to, say, 3 or by hiding the collective forecast so that nobody can herd. Even when the collective forecast is hidden, participants will still have an incentive to make their best forecast since they have no other alternative strategy. Manipulation A very interesting objection to Prediction Market is that they can be easily manipulated, similar to stock exchanges. Even if there are people who try to manipulate a PM, the incentive scheme of a Prediction Market makes sure that there are many more people who will correct wrong stock price, because they know that they can make profits on overvalued or undervalued stocks once the actual outside outcome is published. Christiansen (2007) showed that manipulation attempts in Prediction Markets survive only for 5 to 20 minutes. After that the manipulation is completely corrected and the manipulators are discouraged due to the losses they incur.

If Social Forecasting works so well why is it not used all the time?

I still dont believe Social Forecasting works!

Social Forecasting is new. The first corporate applications are from the mid 1990ies and only after the year 2000 has Social Forecasting been able to make its first steps into the world of corporate decision support (see Table 2). Therefore, Social Forecasting is still a niche topic and not many executives are aware that this amazing Crowdsourcing method exists. A lack of trained consultants Widespread adoption also requires trained professionals who know how to run Enterprise 2.0 tools. Many are trained in survey design but few are trained in Social Forecasting. Even though both approaches have similarities, Social Forecasting require less knowledge about statistics but more skills in community management and incentive design. Anxiety against Social Forecasting Another challenge faced by Social Forecasting is the resistance it encounters from internal experts and forecasters. Sometimes a corporate forecaster will think that Social Forecasting is there to replace him. This is not true of course. Just like ERP systems have not replaced forecasters, Social Forecasting will not make

There are many potential objections to the concept of Social Forecasting, which need to be heard and responded to. The two major objections often voiced are Herding and Manipulation of results. Herding Herding refers to the phenomenon of people not following their own opinion but just following what the market does. In Social Forecasting herding is prevented by two mechanisms. First, when participants know there is an outside event, e.g. sales figures coming in next month, they know that the other market players are just as dependent on it as they are. Those who simply follow the market will lose play-money because herding will not have any impact on the outside event. Hence participants which herd are quickly weeded out by the system.

CrowdWorx, Analyx 2007-2011. All rights reserved.

The Rise of Social Forecasting

experts obsolete. Social results still have to be used by human experts who interpret, judge and decide what to do next. Openness Social Forecasting requires certain private information, e.g. past sales data, to be disclosed to employees (not to the public!) in order to enable them to make an informed forecast. Many companies are not ready for this and will therefore refrain from introducing Social Forecasting. However, as Web 2.0 tools are increasingly making inroads into companies, information sharing will become a normal part of business and those who dont follow will be suffering competitive disadvantages. As information sharing and Social Business become established practices, we can expected to also see the use of Social Forecasting to continue its rise in the very near future.

CrowdWorx, Analyx 2007-2011. All rights reserved.

The Rise of Social Forecasting

References
Berg et al. (2000): Results from a Dozen Years of Election Futures Markets Research, University of Iowa, Working Paper Berg, Henry (2007): Prediction Markets at Microsoft, Conference on Corporate Applications of Prediction Markets, Kansas City Bughin, Jacques; Michael Chui (2010): The rise of the networked enterprise: Web 2.0 finds its payday, McKinsey Quarterly Burnham, Kristin (2009): How Motorola Uses Prediction Markets to Choose Innovations, NetworkWorld, http://www.networkworld.com/news/2009/043009-howmotorola-uses-prediction-markets.html Chen, Kay-Yut; Charles R. Plott (2002): Information Aggregation Mechanisms: Concept, Design and Implementation for a Sales Forecasting Problem, Working paper 1131, Pasadena Cowgill, Bo; Justin Wolfers; Eric Zitzewitz (2009): Using Prediction Markets to Track Information Flows: Evidence from Google Christiansen, Jed D. (2007): Prediction Markets: Practical Experiments in Small Markets and Observed Behaviors, Journal of Prediction Markets, Vol. 1, No. 1, Buckingham University Press, Buckingham CrowdWorx (2008): Forecasting new products at European retailer Tchibo, CrowdWorx client case study. CrowdWorx (2009): Prediction Markets versus polls in the German Federal election 2009, internal research project. Ericsson, Anders K.; Jacqui Smith (1991): Toward a General Theory of Expertise: Prospects and Limits, Cambridge University Press, Cambridge Ericsson, Anders K. et al (2006): The Cambridge Handbook of Expertise and Expert Performance, Cambridge University Press, Cambridge Fogarty, Mat (2007): Expert Predictions, Conference on Corporate Applications of Prediction Markets, Kansas City Forsythe, R. T.R. Palfrey, and C.R. Plott (1984): Futures Markets and Informational Efficiency: A Laboratory Examination, Journal of Finance 39, 955-981. Graefe, Andreas; Christof Weinhardt (2008): Long-term forecasting with Prediction Markets A Field Experiment on Applicability and Expert Choice, Journal of Prediction Markets, Vol. 2, No. 2, Buckingham University Press, Buckingham

Hengl, Michael; Oliver Menz; Johannes Wiek (2008): Kollektiv handeln lernen, Harvard Businessmanager Keller, Dawn (2007): TagTrade Best Buys Prediction Market, Conference on Corporate Applications of Prediction Markets, Kansas City LaComb, Christina (2007): GE's Imagination Markets - Oct 2007, Technical Report, General Electric Company Malone, Thomas (2004): Bringing the Market Inside, Harvard Business Review MIT (2011): Prediction market perspective on collective intelligence, MIT Center for Collective Intelligence, http://scripts.mit.edu/~cci/HCI/index.php?title=Prediction_mar ket_perspective_on_collective_intelligence Needle, David (2011): Ford Taps Cloud-Based Prediction Market, http://www.internetnews.com/busnews/article.php/3925571/Ford-Taps-Cloud-Based-PredictionMarket.htm NewsFutures (2010): Various client projects published by NewsFutures Ortner, Gerhard (1997): Forecasting Markets An Industrial Application, Working Paper, Vienna University of Technology Ortner, Gerhard (1998): Forecasting Markets An Industrial Application Part II, Working Paper, Vienna University of Technology Plott, Charles (2000): Markets as Information Gathering Tools, Southern Economic Journal, Vol. 67, No. 1, Chattanooga Plott, Charles; Shyam Sunder (1982): Efficiency of experimental security markets with insider information: An application of rational-expectations models, Journal of Political Economy, Vol. 90 (2), S. 663-698. Plott, Charles; Shyam Sunder (1988): Rational expectations and the aggregation of diverse information in laboratory security markets. Econometrica 56, 10851118. Rietz et al. (2003): Accuracy and Forecast Standard Error of Prediction Markets, University of Iowa, Working Paper Skiera, Bernd, C. Slamka, W. Jank (2009): Second-Generation Prediction Markets for Information Aggregation: A Comparison of Payoff Mechanisms, available at SSRN: http://ssrn.com/abstract=1435316 Soukhoroukova, Arina; Spann, Martin (2005): New Product Development with Internet Based Information Markets: Theory and Empirical Application, ECIS 2005 Proceedings, Paper 133

CrowdWorx, Analyx 2007-2011. All rights reserved.

The Rise of Social Forecasting

Sprenger et al. (2007): Conditional Prediction Markets as Corporate Decision Support Systems, Journal of Prediction Markets, Vol. 1, No. 3, Buckingham University Press, Buckingham

CrowdWorx, Analyx 2007-2011. All rights reserved.

The Rise of Social Forecasting

Contact
The CrowdWorx Prediction Market is a product of global business analytics firm Analyx, headquartered in Poznan, West Poland. We serve clients in Europe and North America with the full range of Social Decision Support services based on Collective Intelligence and State-of-the-Art Enterprise 2.0 methods. For more information on CrowdWorx please write to team@crowdworx.com or call an office near you.

Berlin, Germany Rotherstr. 18 10245 Berlin Germany Tel: +49-30-51300-211 Fax: +49-30-41721-281 Munich, Germany Isartalstrae 30 82008 Munich (Unterhaching) Germany Tel: +49-89-559-485-92 Fax: +49-89-61-066-793

Boston, USA Analyx, Inc. 17 Stonecleve Rd. Wellesley (Boston), MA 02482 Tel: +1-617-763-8082 Fax: +1-781-489-5595 Poznan, Poland ul. Fredry 1 61-701 Poznan Poland Tel: +48-61-88577-00 Fax: +48-61-88577-15

CrowdWorx, Analyx 2007-2011. All rights reserved.