The shift to ‘open’ working across the social sciences as a discipline group entails a welcome but demanding cultural change. Yet, Patrick Dunleavy argues that there have already been three false starts: focusing only on isolated bits of the open agenda in ways that don’t connect and so are not meaningful; loading researchers with off-putting, external bureaucratic requirements; and risking reopening ‘sectarian’ divides between quantitative and qualitative social scientists.
Open social science (OSS) is new, and like any beginner is still finding its way. However, to a large extent we are still operating in the shadow of open science (OS) in the STEMM disciplines. Nearly a decade ago an influential Royal Society report argued:
‘Open science is often effective in stimulating scientific discovery, [and] it may also help to deter, detect and stamp out bad science. Openness facilitates a systemic integrity that is conducive to early identification of error, malpractice and fraud, and therefore deters them. But this kind of transparency only works when openness meets standards of intelligibility and assessability – where there is intelligent openness’.
More recently, the Turing Way project defined open science far more broadly as a range of measures encouraging reproduceability, replication, robustness, and the generalisability of research. Alongside CIVICA researchers we have put forward an agenda for progressing open social science in line with these ambitions. Yet for open social science to take root it must develop an ‘intelligent’ concept of openness, one that is adapted to the wide range of concerns that our discipline group addresses, and is appropriate for the sharply varying conditions in which social research must be carried out.
This task has been made more difficult by a number of premature and partial efforts to ‘graft’ an ‘open science’ concept from STEMM disciplines onto the social sciences. Three false starts have already been made and have created misconceptions about open social science. Below, I want to show how each of the strategies may actually work to obstruct the wider development of open social science.
Bricolage – Reading across directly from STEMM
This approach sees open social science as just about picking up (not quite at random) the best-known or most discussed individual components of open science in STEMM disciplines – focusing on specific things like open access publishing, the FAIR principles for data management, replication studies, or the pre-registration of hypotheses. Important though they are, adoption of these open components doesn’t reveal any deep commitment to openness or innovative ways of working.
This is highlighted by Christensen et al’s fascinating 2020 research on American social science authors ‘open’ practices (Fig.1). They found that publishing at least something open access (once) is now near universal, and that depositing data at least once is also common in experimental fields, less so in quantitative work. Differences in posting the research instruments from studies online are sharper and pre-registering hypotheses has yet to take off much in non-experimental work. The results also unequivocally show substantial differences in their use of ‘open’ between authors in experimental fields (who came out top), those using non-experimental quantitative work (who were in the middle), and theoretical or qualitative fields (whose use of ‘open’ was lagging).
Figure 1: How Christensen et al. chart the development of three open science practices amongst U.S. social scientists over time
Notes: The authors explain: ‘The chart shows for a given year the proportion of Published Authors who had first completed an open science practice in that year or previously, categorized by the focus of their research. The classification is based on answers to the question “What methods do you use in your research? Please check all that apply.” If a scholar only selected “Qualitative” or “Theoretical”, they are classified as “Qualitative or Theoretical”; if they selected “Quantitative – Observational” or “Quantitative – Other” but not “Quantitative – Experimental”, they are classified as “Quantitative non-experimental”; if they selected “Quantitative – Experimental”, they are classified as “Experimental”. The sample is restricted to Published Authors who completed their PhDs by 2009’ (p.13).
The questionnaire only asked if the academics have done something ‘open’ at least once, so it may considerably overstate the extent at which open practices are being used at any one time. However, Christensen et al also asked all the authors (taken together) to give their perceptions of how much progress ‘open’ was making in their field, versus how much they had used open working themselves (their Figure 6). Respondents generally estimated open science progress in their subject field more pessimistically than changes in their own practices. Yet, the survey highlights how this can lead to isolated sites of open practice, especially when open methods are not situated within an overall approach to open science.
Of course, it is useful for social scientists to consider adopting particular practices that STEMM scientists have found useful and developed concrete protocols for, and in some fields (like cross-national questionnaire research) this approach is already almost inevitable. However, it may also be wise to remain somewhat sceptical about how far ‘open’ behaviours pledged by authors are actually carried through. Some recent studies in medicine and other subjects show that where data for an article is supposedly available ‘on demand’, the researchers involved overwhelmingly do not respond to requests for access.
‘Open’ as bureaucracy and neo-liberal surveillance
A second, more organised view misrepresents the cultural shift to ‘open’ as being primarily about more external surveillance of academia. ‘Open’ ways of working can easily come across as just another set of burdensome and extraneous bureaucratic hoops through which academics must jump. Especially when allied with the bricolage approach above, it is very easy for a ‘research briefing’ by well-intentioned university support staff to end up as just a lengthy Powerpoint presentation detailing the varying ‘open’ requirements of funding bodies, or extensive data management principles. This burden-boosting account is particularly off-putting in the social sciences, because only 9% of UK social scientists are full-time researchers with no teaching commitments, and thus able to specialise in particular aspects of grant seeking and meeting OS requirements. (By contrast, in the STEMM sciences 35% of staff are research-only). Busy researchers-plus-teachers have a lot on their plate, so more time-consuming bureaucracy is doubly unwelcome.
In addition, many university staff dislike recent trends towards greater use of metrics in research assessment and are understandably critical of further (allegedly ‘neo-liberal’) surveillance. Even amongst STEMM sciences some authors have extended the critique of metricization and ‘surveillance’ to apply to open science. For example, Philip Mirowsksi argues: ‘The [OA] agenda is effectively to re-engineer science along the lines of platform capitalism, under the misleading banner of opening up science to the masses’. While much of this commentary is wildly overstated, and ignores both the many autonomous sources of the impact agenda within academia and the research benefits of systematic search, this reaction is still widespread and deeply felt in parts of the social sciences.
These problems are made worse by justifications of open social science (OSS) that dwell on counteracting problematic practices like p-hacking (massaging model regression analyses over particular ‘statistical significance’ levels) to push work into high status journals, selective publication of positive results and non-publication of negative results, and ‘discoveries’ by one researcher that cannot be replicated. These are live problems, and many components of open social science may help to mitigate them, but this is not what OSS is centrally about. Social scientists cannot assume that they are studying an invariant entity (‘nature’), with unchanging law-like (if very complex) mechanisms at work. Instead, they must deal with a constantly changing and reflexive human and societal capability to absorb knowledge and do things differently as a result. Open social science is a cultural change that aims to do this more successfully and reliably than before, a goal that could not be more integral to the whole purpose and rationale of good research from its earliest beginnings through to its publication and discussion by a professional or wider audience.
Reawakening methodological sectarianism
Many social science disciplines have recently had (or still have) acute ‘methodology wars’ between exponents of formal theories and highly developed quantitative and mathematized research approaches on the one hand, and on the other hand social scientists using more qualitative methods. Furthermore, in all the social sciences a great deal of scholarship involves updating ‘first draft of history’ accounts of recent societal and economic changes, using more discursive and descriptive approaches.
These conflicts have declined from their peak intensity, notably in political science where the APSR editors recently welcomed robust qualitative work, a radical change of mood from the perestroika era conflicts at APSA’s 2000 conference in San Francisco. It would be all too easy, however, for those pushing a premature STEMM-only notion of ‘open social science’ to reopen past wounds and reawaken the past alarms of qualitative scholars about being ‘squeezed out’ of their disciplines. This is particularly notable, where an ‘open’ effort is over-linked to an alleged ‘replication crises’, or becomes associated with attempts to rewrite research integrity rules in crude ways that actually are only practicable in (some kinds of) quantitative work.
It would be all too easy, however, for those pushing a premature STEMM-only notion of ‘open social science’ to reopen past wounds and reawaken the past alarms of qualitative scholars about being ‘squeezed out’ of their disciplines.
For instance, the 2019 book, Transparent and Reproducible Social Science Research: How to Do Open Science, falls headlong into this trap. It claims to be ‘The first book to summarize and synthesize new approaches to combat false positives and non-reproducible findings in social science research, document the underlying problems in research practices, and teach a new generation of students and scholars how to overcome them’. Yet, in practice, this volume covers only disciplines and areas of work with the most developed quantitative methods, and stresses only the replication/research integrity arguments linked to quantitative methods. No effort is made to consider how open science approaches might be involved in movements towards an inclusive open social science, or how ‘open’ ways of working might be developed for other kinds of social science work. Indeed, remarkably, the book’s nearly 200 pages manage never to mention qualitative research at all. Searching the institutional website behind the book (Berkeley’s ITSS) reveals only a few scattered examples of any effort to address qualitative research as open social science.
Other US-based promoters of open science, like the Centre for Open Science, also cover mainly the quantitative disciplines in social sciences (like psychology and health studies). Yet COS’s somewhat disturbing strategy for accelerating cultural change towards ‘open’ starts with ‘Make it Easy’, goes through ‘Make it Normative (sic)’ and then ‘Make it Rewarding’, but ends up with ‘Make it Required’. No apparent explanation is given of how that last stage would work outside of laboratory or experimental work, some kinds of randomized control trials, or purely computational research. A careful analysis of many psychology journals’ policies by Prosser et al suggests that ‘open science risks becoming a closed door’ for qualitative researchers in the discipline.
Doing open social science right
Valuable as the insights about open science in STEMM disciplines are, a broader and more inclusive approach is needed if ‘open’ is to develop fully across the social sciences – in the process hopefully reshaping not just into crossover disciplines with the humanities (like law, social and political philosophy, contemporary history and the digital humanities), but also spilling over into a wider range of humanities subjects (like older history, philosophy and literature studies). Open social science is not about science-envy, or yet more research bureaucracy, or strengthening top-down surveillance within academia, or a new episode of past destructive and misguided methodology wars. It is instead about moving all social science research forward towards being more soundly based, more coherent and generalizable, and more open for citizens to access and understand.
CIVICA Research brings together researchers from eight leading European universities in the social sciences to contribute knowledge and solutions to the world’s most pressing challenges. The project aims to strengthen the research & innovation pillar of the European University alliance CIVICA. CIVICA Research is co-funded by the EU’s Horizon 2020 research and innovation programme.
The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.
Image Credit: Fig.1 Source: Figure 4 in Garret Christensen, Zenan Wang, Elizabeth Levy Paluck, Nicholas Swanson, David Birke, Edward Miguel, and Rebecca Littman. ‘Open Science Practices Are on the Rise: The State of Social Science (3S) Survey’, 13 January 2020. https://escholarship.org/uc/item/0hx0207r
Featured Image, Tasha Lyn via Unsplash.
I have tried to “do open social science” for 20-30 years, but this was in the area of quantitative social science and public health/health services/health policy. My key learning experience in this area was being involved in a major cross-country WHO-sponsored project in the 1970s where I realised that the project only worked to its full potential because we all shared and harmonised our research tools and outcomes. Since then, in my discipline of empirical and applied sociology, I have discovered that our initial and dominant paradigm of quantitative policy-oriented methodology has been almost completely overtaken by qualitative methods. As someone originally trained in history, I have no problem with such an approach, but it is a bit disconcerting that the UK/NZ/OZ tradition has gone pretty much 90% over to this methodological paradigm. More than that, because of the often confidential, in-depth nature of the data and the difficulty in agreeing on inter-investigator and inter-study standards and sharing in this paradigm, openness gets a lot harder to define and achieve. So, yes, I agree with this blog that any simplistic translation of an “open science” paradigm from STEMM to, say, sociology is questionable. But at the same time I have to say that the disciplines I learned and developed in my quantitative niche – data sharing, open and usually standardised and harmonised tools, models and code, all a “work in progress” I agree – should not be sacrificed just because our qualitatively-oriented colleagues are having difficulty coming up with an “open science” modality that does not do perceived harm to their paradigm.
Our journal is part of a major grouping of free open access journals across the sciences and SS&H, and we have had these debates often. SS&H needs to be published open access for sure [and not in the journals of the major commercial publishers preferably, unless you are going to compromise your ethics in support of large capitalist enterprises that charge us a lot – who has $3000 or more unless you have a large grant or a tolerant university/library system? Why not focus on these ethical issues in these posts? https://doi.org/10.21428/6ffd8432.5e24d46d%5D. But whether ‘datasets’ used need to be made available as well, is a source of great debate. For most qualitative work, interviewees and respondents would not like to see their utterances transcribed and reproduced verbatim in an anonymised dataset, and if parts would need redacting, which researchers really have the time for that and who could do it satisfactorily? A study on Indigenous knowledge for example, by a trusted researcher and beholden to a human ethics board protocol, would surely be exempt from all data being supplied as a file. Would these transcriptions potentially be mined for other studies and is this fair? The problem here is ‘machine readability’ that STEM disciplines like the sound of, for conducting meta-analysis based on all existing studies of a phenomenon, which requires access to original data [and xml versions for the text and figures of journal articles]. Scientists get very excited about this, and the possibilities it affords for picking out research findings missed by individual researchers. No such excitement exists in my discipline [strongly represented at LSE]. I just think the STEM approaches to publishing are just a whole other ballgame. Aspirations to move to full OA publishing is a shared goal, but issues of metadata and replicability only really apply to highly quantitative social science, which is only part of the whole vast field.