LSE - Small Logo
LSE - Small Logo

Lian Jian

Sha Yang

Sulin Ba

Li Lu

Crystal Li Jiang

September 25th, 2018

Getting the best out of your crowdsourcing contest

0 comments

Estimated reading time: 5 minutes

Lian Jian

Sha Yang

Sulin Ba

Li Lu

Crystal Li Jiang

September 25th, 2018

Getting the best out of your crowdsourcing contest

0 comments

Estimated reading time: 5 minutes

For a firm, big or small, the idea of outsourcing tasks, especially innovation tasks, that are either costly or risky is attractive. This is especially so when outsourcing is made easy by the Internet. Some firms use their own sites (e.g., Netflix or Dell), while others use well-established crowdsourcing platforms, such as InnoCentive for engineering problems, Kaggle for data sciences, and 99designs for creative designs. The upside is obvious – you reach a large crowd of potential talent with a few mouse clicks. Equally important, in a contest the firm only pays if a good solution is found, which makes the process practically risk-free.

However, from the participants’ perspective, the process is far from risk-free. People who participate in such contests battle various uncertainties to receive the rewards, and if such uncertainties are not well managed by the firm, ultimately, the outcome of the whole exercise can be compromised. If you think that by setting up a contest and promising a reward, you can just leave it alone to do the magic, chances are you are in for a big surprise, and most likely an unpleasant one.

We conducted a study of these uncertainties and offer our two cents on how to manage them effectively to attract healthy participation in online contests. In a nutshell, reduce the uncertainties. You can do so by either putting a guarantee on the rewards (accomplished by prepaying the winning dollar amount to the online platform) and/or by providing very frequent feedback to the participants, i.e., engage with them and work with them. Unless you have guaranteed your contest, your feedback can’t be too positive too early; otherwise, you’ll scare away potential talents by making them think that the winning solution has already been submitted. On the other hand, your feedback can’t be too negative either—this can make you seem like a very picky contest host, one that will be very hard to please, and that’ll scare away good talent, too. Below, we’ll walk you through the whys and the hows of setting up your own online contest.

By and large, the uncertainties faced by crowdsourcing participants fall into three categories. First, apart from some well-defined engineering problems (e.g., machine learning tasks), at the beginning, many tasks have ill-defined criteria for evaluating the solutions. This is mostly due to the nature of innovative/creative problems. Oftentimes, contest organizers don’t quite know what they’re looking for until they see it. And their criteria may be evolving during the contest too, which means the participants could be chasing a moving target.

Second, let’s face it: although most contest hosts have good intentions starting out, there are a few bad apples. Because setting up a contest is so easy (you don’t have to pay at the beginning), some might decide to try it out or may not even intend to pay at all. The worst could be those that refuse to pay but secretly steal the participants’ ideas and use them later.

And finally, at the end of the day, competition is inherently uncertain, to some degree. When a participant is facing an unknown number of competitors with unknown capabilities on the Internet, it is hard to predict who will turn out to be the winner (and get paid), even if the previous two uncertainties disappear. Undoubtedly, these uncertainties faced by the participants will affect their behaviours and likewise the contest’s outcome.

What can a contest host do to alleviate these uncertainties and make the best use of this novel platform for innovation and problem-solving? Our study examined data from a leading creative design crowdsourcing platform (their management demanded anonymity) and found that first of all, providing prize guarantees by pre-paying the promised reward amount to the platform is helpful. Although it means that the contest host assumes the risks of not receiving a good solution, it dramatically increases participant levels, which is, in general, a good starting point for receiving good solutions. Participants still face other uncertainties, for sure, but it does relieve a big part of their worry – that the contest host never intends to pay.

So, if you’re not willing to pay the prize upfront, the next best choice is to engage with participants throughout the contest. Usually, these contests last for a period of time, e.g., a week, a month, or even several months. As the submissions come in, it’s a good opportunity for the contest host to provide some feedback, such as numerical ratings that indicate how much the contest host likes it, or some textual comments. It turns out that such feedback itself is highly useful in reassuring the (existing or potential) participants that the contest host is there. He or she did not disappear and still cares about the outcome of the contest. Psychologically, it makes the participants more trusting that the contest host does intend to pay (although objectively such beliefs could be unfounded). Indeed, experienced participants of the platform have repeatedly expressed that when a contest host does not engage with the participants during the contest and has not prepaid the winning amount, it’s usually a telling sign of payment denial.

Furthermore, what ratings you give to the participants matter. This has something to do with how potential participants gauge the competition level in an existing contest. Not surprisingly, no one wants to enter a contest that already has a high-scoring solution. Given that hundreds of contests are posted on the platform every day, contest participants have plenty of other options. So, it’s not a good idea to give very high ratings to contest entries, at least not too early. Interestingly, though, if a contest does have a prize guarantee, the contest host appears to be able to somewhat get away with giving high ratings. In such cases, participation doesn’t decline as much. Perhaps participants are willing to tolerate higher competition intensity if the prize is assured.

On the flip side, you also don’t want to rate the solutions too negatively. For example, on most websites the rating scales are usually one to five stars, but they also offer the option of indicating that certain solutions are just “unacceptable” or to put it mildly, “not the right direction.” This latter type of rating appears to hurt participation quite a bit, especially in contests without guarantees. Maybe these nasty ratings are just too harsh and could potentially suggest that the contest host is too picky to honour payment later. As expected, if you then guarantee your contest, such extremely negative ratings don’t hurt the participation as much.

In sum, these nuanced analyses suggest that crowdsourcing contests are not black boxes. There is almost an art in managing the crowds to get them to do the best work for you. It’s not just about shifting the risks involved in innovation onto the crowds, or distributing them among a large number of people. It’s also about engaging with them like a community.

Notes:


Lian Jian is an assistant professor at the Annenberg School of Communication, University of Southern California. She earned her PhD from the School of Information, University of Michigan at Ann Arbor. Her research is focused on individual behaviours in electronic markets, online knowledge communities, and small groups. She is an interdisciplinary scholar whose work contributes to theories in economics, psychology, and communication, and has published in top journals in both management information systems (e.g., Management Science and MIS Quarterly) and communication (e.g., Journal of Computer-Mediated Communication and Communication Research).

Sha Yang is Ernest Hahn professor of marketing at the University of Southern California. She earned her PhD from the Ohio State University. Her research focuses on modelling consumer purchase behaviour and market competition. Her recent research interests include search engine advertising and social network. She is a leader in developing models with sophisticated interactions among consumers and firms, and is a pioneer in introducing Bayesian methods into the estimation of such interactions. Professor Yang has collaborated with many companies including Visa, Miller Brewing Company, HP, Pepsi Co., CBS, TNS, McCann Ericson, epinions.com, Taobao.com and China Mobile.

Sulin Ba is Treibick Family Endowed chair for the Connecticut Information Technology Institute, associate dean of academic and research support, and professor of operations and information management at the School of Business, University of Connecticut. She holds a PhD from the University of Texas at Austin. Her work on the institutional setup to help small businesses grow in the digital economy has been used as the basis for testimony before the Congressional House Committee on Small Business.

Li Lu is an assistant professor in the management department of the School of Business and Public Management at West Chester University. She received her M.S. from Cornell University and PhD from the University of Southern California. Her research focuses on group decision making and knowledge management.

Crystal Li Jiang is an associate professor in the department of communication at the City University of Hong Kong. Her research focuses on social media, health communication, and mediated perceptions. 

About the author

Lian Jian

Lian Jian is an asistant professor at the Annenberg School of Communication, University of Southern California. She earned her PhD from the School of Information, University of Michigan at Ann Arbor. Her research is focused on individual behaviors in electronic markets, online knowledge communities, and small groups. She is an interdisciplinary scholar whose work contributes to theories in economics, psychology, and communication, and has published in top journals in both management information systems (e.g., Management Science and MIS Quarterly) and communication (e.g., Journal of Computer-Mediated Communication and Communication

Sha Yang

Sha Yang is Ernest Hahn professor of marketing at the University of Southern California. She earned her PhD from the Ohio State University. Her research focuses on modeling consumer purchase behavior and market competition. Her recent research interests include search engine advertising and social network. She is a leader in developing models with sophisticated interactions among consumers and firms, and is a pioneer in introducing Bayesian methods into estimation of such interactions. Professor Yang has collaborated with many companies including Visa, Miller Brewing Company, HP, Pepsi Co., CBS, TNS, McCann Ericson, epinions.com, Taobao.com and China Mobile.

Sulin Ba

Sulin Ba is Treibick Family Endowed chair for the Connecticut Information Technology Institute, associate dean of academic and research support, and professor of operations and information management at the School of Business, University of Connecticut. She holds a PhD from the University of Texas at Austin. Her work on the institutional setup to help small businesses grow in the digital economy has been used as the basis for testimony before the Congressional House Committee on Small Business.

Li Lu

Li Lu is an assistant professor in the management department of the School of Business and Public Management at West Chester University. She received her M.S. from Cornell University and PhD from the University of Southern California. Her research focuses on group decision making and knowledge management.

Crystal Li Jiang

Crystal Li Jiang is an associate professor in the Department of Communication at the City University of Hong Kong. Her research focuses on social media, health communication, and mediated perceptions.

Posted In: Featured | Technology