In debates on the validity of academic research findings, focus has been drawn to so-called questionable research practices, commonly understood to encompass a laundry list of behaviours that can increase the likelihood of statistically significant (and so more publishable) results. Anand Krishna and Sebastian M. Peter report on research examining attitudes to questionable research practices among students who have recently completed their theses. Although almost half had engaged in at least one questionable practice, the practices students most often admitted to were issues of reporting results, many of which can be solved by adopting open science standards of data sharing. Among the more important findings was that advisor attitudes matter: if students thought their thesis advisors endorsed questionable practices, they were more likely to admit engaging in such practices.
There has been much debate on the validity of psychological research findings lately, with both large–scale replication efforts and work focusing on individual effects and theories, as well as work from specific labs. Aside from engendering much discussion and follow-up research in the field, doubts about psychological research (especially social psychology) have spilled over into the general public, given additional fuel by recent high-profile cases of data fraud. In the current environment of widespread science skepticism, evidence of the unreliability of some findings is a major danger to the credibility of social psychology and possibly even the broader field. For this reason, it behooves those of us in the social sciences to consider the causes of such unreliability closely.
Beyond issues of publication bias (the tendency within peer-reviewed academia to publish statistically significant findings that provide evidence for an effect with greater probability than findings that might dispute an effect), the discussion of possible causes for the fragility of psychological findings has focused on so-called questionable research practices. Although they are often mentioned in the same breath as data falsification or plagiarism, questionable research practices are commonly understood to encompass a laundry list of behaviours that can increase the likelihood of getting statistically significant (and therefore publishable) results. For example, researchers might report only the measure that “worked” in their study, because reporting the five other measures where the effect failed to manifest would make the effect appear weaker; or they might decide to exclude participants whose data goes against their predictions, cherry-picking their results. Previous research has shown high incidences of such questionable practices in published research. Many psychologists admit to performing such practices at least once.
Unfortunately, applying questionable research practices leads to unreliable findings. These practices are designed to twist statistics that show us whether the results are likely to be simple chance. Researchers use them to systematically force those tests to say “no”, which directly leads to tests doing a worse job of identifying actual chance effects. This is why it is important to figure out how common these practices actually are and stop them from happening. Our recently published work has provided some insight into questionable research practices in a particularly under-researched group: psychology students.
Because psychology students encompass everyone who will ever have a degree in psychology, understanding their use of questionable research practices and what they think of these practices is important. Consider that these people will be not just future academics, but also counsellors, therapists, or human resource managers. These are roles that are relied upon in society to perform their tasks on a solid scientific foundation. Therefore, they simply have to be able to evaluate scientific work properly and judge what new findings are likely to be stable in order to keep on top of their field.
Image credit: Nik MacMillan, via Unsplash (licensed under a CC0 1.0 license).
We surveyed over 200 German students anonymously and asked them about their practices in their final theses (data collected in 2016). Although almost half of those responding said they had performed at least one of the relevant questionable practices we asked them about, most of these had limited it to one specific practice and less than 10% had performed more than two. Less than 3% reported engaging in the more egregious practices (such as data falsification or “tuning” the sample until the desired result was achieved). Although these numbers may still be disappointingly high given the far greater consciousness of these issues in the field today, the fact that very few thesis studies contain more than once questionable practice means that their effect on the studies’ validity is likely quite small. In addition, the majority of practices students most often admitted to were issues of reporting results, many of which can easily be solved by adopting open science standards of data sharing.
Moreover, the students correctly identified these questionable practices as problematic for science when we asked for their opinion. Students do not seem to believe that questionable research practices are anything other than questionable, although the extremity of this belief varies between unambiguous items like “falsifying data”, which almost every respondent thought was problematic, and more innocuous ones like “rounding of p-values”, which was placed between problematic and neutral on average.
Although students’ opinions of most questionable practices were strongly associated with their likelihood to admit to engaging in such practices, some other psychological variables also played a role. Strongly motivated students were less likely to indicate that they had engaged in most questionable practices. Another important factor was what students perceived their supervisors as thinking of questionable practices. If students thought their thesis advisors endorsed questionable practices, they were more likely to admit engaging in such practices. Part of this was due to advisors apparently exerting direct influence on the theses’ studies. However, for some of the questionable practices, supervisors’ opinions also seemed to have an indirect effect by shaping student opinions, which in turn impacted admitted questionable practice use.
So what is the takeaway from this? On the one hand, even though some are claiming that research practices are becoming better in psychology, this doesn’t seem to have impacted students much. They seem to be admitting to problematic practices at similar rates to those of their seniors during the time when the problem was first being explored. On the other hand, however, the renaissance of open science methodologies such as preregistration are well-suited to counter the kinds of problematic practices students engage in, so perhaps it’s just a matter of time until these kinds of methods become the norm in teaching. Furthermore, students are coming out of their courses with negative opinions of questionable practices on average, so the system isn’t promoting them.
However, the most important finding is probably the role of the students’ advisors. Advisor attitudes towards research practices matter. They are likely to shape students’ attitudes, either through teaching or through example (when advisors engage in problematic practices on students’ behalf). Although our study cannot rule out that students were projecting their own opinions onto their advisors, it certainly seems plausible that advisors need to be careful to set a good example. Motivating students well might also insulate them from problematic practices. It definitely seems like thesis advisors have an important role to play in shaping the scientifically grounded psychologists of the future.
This blog post is based on the authors’ article, “Questionable research practices in student final theses – Prevalence, attitudes, and the role of the supervisor’s perceived attitudes”, published in PLoS ONE (DOI: 10.1371/journal.pone.0203470), and on related work.
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
Anand Krishna is a postdoctoral researcher at the University of Würzburg. He participated in the major replication initiative of the ego depletion effect spearheaded by Professor Kathleen Vohs and maintains a lively interest in questions of open science, the evidentiary value of studies, and questionable research practices. His other work focuses on the impact of approach- and avoidance-oriented motivational states on basic and social cognitive functions, evaluations, and preferences, as well as on economic decision making. His ORCID ID is 0000-0003-3838-9210.
Sebastian M. Peter, BSc Psychology, is a Research Assistant at the Department of Social Psychology at the University of Wuerzburg.