LSE - Small Logo
LSE - Small Logo

Blog Admin

October 18th, 2016

Collecting data using crowdsourcing marketplaces raises ethical questions for academic researchers

6 comments | 1 shares

Estimated reading time: 5 minutes

Blog Admin

October 18th, 2016

Collecting data using crowdsourcing marketplaces raises ethical questions for academic researchers

6 comments | 1 shares

Estimated reading time: 5 minutes

gleibsThe need for larger sample sizes and ready access to a diverse group of participants has seen many researchers turn to crowdsourcing platforms such as MTurk for their data collections. However, Ilka Gleibs argues that the ethical implications of using crowdsourcing marketplaces demand further attention. To safeguard academic progress and public trust in research it is imperative that we treat participants as stakeholders in research and not as passive objects or merely a human resource.

“Treat your workers with respect and dignity. Workers are not numbers and statistics. Workers are not lab rats. Workers are people and should be treated with respect.”

turker ‘T’, a Turkopticon moderator

New technologies like large-scale social media sites (e.g. Facebook, Twitter) and crowdsourcing services (e.g. Amazon Mechanical Turk, Crowdflower, Clickworker) have had a significant impact on the social sciences and provide many new and interesting avenues for research. These crowdsourcing services have become increasingly important and popular tools for participant recruitment in the psychological sciences and beyond. For many social scientists in the USA and elsewhere, Amazon’s Mechanical Turk (MTurk) is a primary source for crowdsourced research and many empirical papers in leading journals are largely based on data collected on MTurk. In addition, the current replication crisis in psychology and related social sciences highlights the need for larger samples and ‘easy’ access to diverse participants. So, the use of crowdsourcing marketplaces for data collections seems to be a way of increasing access to sufficiently large samples while also overcoming the overreliance on undergraduate students.

However, in the last year there have been several media stories about the life of ‘Turkers’, their working conditions and the ethics of conducting research in ‘science factories’. In addition to media criticism, recent efforts of crowdworkers themselves highlighted poor academic practices on MTurk. These accounts demonstrate that an uncritical use of such research sites, together with the assumption that MTurk and other platforms are subject to the same types of ethical concerns and considerations as traditional forms of recruitment and data collection may well demand further scrutiny.

For a sizeable number of people, partaking in research on MTurk represents a way to increase or sustain an income; essentially these workers become ‘professional crowdworkers’. This is in contrast to the typical psychology research participant: usually a student or other volunteer, who is compensated for time and/or expenses but who should not expect a ‘wage’. We know that about 10% of MTurk’s US workers report a household income of below $15,000 per annum, with 25% on below $25,000 (for a real-time tracker of key demographics – i.e. age, gender, marital status, income, household size, and country – see the MTurk Tracker by Panos Ipeirotis). Given these data, we can assume that for some participants the money earned on MTurk is an important source of income.

ethicalImage credit: Ethical by NY. This work is licensed under a CC BY-SA 3.0 license.

In my recently published paper I argue that we must consider the ethical implications of making research participation a source of income; specifically the power differential created by an employer-contractor relationship and the question of whether certain workers can afford to reject tasks. Some legal scholars argue that MTurk has developed into an unregulated labor marketplace; one with very low wages, incomplete contracting, weak access to enforcement and a disciplining role of reputation, where workers are denied basic workplace rights and the community has no recourse to employer wrongdoing. In essence, the requester (and therefore researcher) engages in an employer-contractor relationship that shifts away from the common relationship between investigators and participants as a revenue-neutral experience to one in which the requester is a client and the participant a contractor. In addition, they hold more power than the workers in setting wages and withdrawing work. This in turn can violate one important pillar of human participant research, namely the respect for autonomy, which implies that we must protect the rights, freedom and dignity of our participants.

Thus – and despite arguments that in the age of the sharing economy and ‘big data’ hopes are high for gaining new insights through observation of social interactions on a larger scale than otherwise possible using offline sources (see Gary King for an intriguing perspective) – I argue that MTurk raises ethical concerns that demand our attention. As a social scientist, it is important to focus not only on the quality of the data and validity of results obtained from MTurk and other crowdsourced platforms but also on issues of working conditions and fair pay, and how users are treated when doing research with us. It is our responsibility as scholars to ensure our research methods and processes remain rooted in long-standing ethical practices. From my point of view, offline and online research fields are not entirely equivalent and crowdsourcing platforms in particular warrant special attention. Hence, issues of fair pay, ‘withdrawal-without-prejudice’ and a commitment to participants as active agents and stakeholders in research become more pressing and must be discussed in this context when dealing with MTurk and other commercial providers of large online panels.

Lack of attention to the ethical concerns arising from technological developments such as crowdsourcing platforms can hinder academic progress and diminish our standing and trustworthiness as a community. We ultimately need an earnest, innovative and creative discussion on how to implement ethical guidelines that first and foremost protect participants but also allow researchers to conduct sound research. I propose that we start to reconsider the social contract of ethical dos and don’ts between researcher and participants. For this we have to engage in a discussion on the ethical issues of conducting research using MTurk to ensure that we treat participants as stakeholders in research and not as passive objects or merely a human resource. Researchers, funders and ethics committees must reconsider issues of fair pay, engagement with research participants on crowdsourced marketplaces and the challenges posed by internet research. As a field we should make sure our work has social value that promises knowledge creation but also respects research participants; and that we are at the front line of setting standards for accessing and working with online sources that are in line with our ethical consciousness and research practice.

Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.

About the author

Dr Ilka Gleibs is an Assistant Professor in Social and Organisational Psychology at the Department of Psychological and Behavioural Science at the London School of Economics. Previously she worked as a Postdoctoral Fellow at the University of Exeter and as a Lecturer in Experimental Social Psychology at the University of Surrey. She obtained a PhD from the Friedrich-Schiller-University Jena (Germany), and has a range of interests in social psychology but focuses mainly on social identity dynamics. She also has an interest in the ethics of web-based research and is a member of the LSE Research Ethics Committee.

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: LSE comment | Research ethics

6 Comments