Researchers have long been encouraged to use Twitter. But does researchers’ presence on Twitter influence citations to their papers? José Luis Ortega explored to what extent the participation of scholars on Twitter can influence the tweeting of their articles and found that although the relationship between tweets and citations is poor, actively participating on Twitter is a powerful way of promoting and disseminating academic outputs, potentially indirectly influencing the scholarly impact and improving prospects of increased citations.
Many altmetric studies – meaning altmetrics in general rather than Altmetric.com specifically – have wrongly taken the conceptual framework of bibliometrics as a model to understand the meaning of alternative metrics. For example, it has been assumed that the mention of a research paper on Twitter is comparable to a bibliographic citation, taking for granted that a tweet can be an appreciation index and Twitter a kind of citation index (Eysenbach, 2011; Shuai et al., 2012). However, the processes that generate a citation are very different to a tweet. To have an article cited one must first conduct a study, write a paper and publish it in an indexed journal; whereas the mention of a paper on Twitter requires one only to write a short message. This difference could cause the greater intervention of authors in the mentions of their own papers, at least relative to the case of citations (i.e. instances of self-citation).
Starting from that hypothesis, my recent research explores to what extent the participation of scholars on Twitter can influence the tweeting of their articles, and, by extension, the likelihood of those articles being cited. To that end, 4,166 articles from 76 Twitter users and 124 non-Twitter users were analysed. These data were crawled using PlumX Analytics, which counts the number of tweets a document receives, whereas Scopus was used to extract citation numbers. Finally, a manual search was done to distinguish authors with a handle on Twitter from others not registered to that social network.
Image credit: twitter-sparrow 092 by Josey. This work is licensed under a CC BY 2.0 license.
Results showed that papers from Twitter users could be on average 33% more tweeted than documents of non-Twitter users (Twitter users mean = 2.33 tweets per paper; non-Twitter users mean = 1.75). Obviously, this result is important not because participating on Twitter improves the mention of academic papers, but because the number of mentions could be greatly influenced by the authors of those papers. This introduces the possibility of manipulation of tweets metrics and puts in doubt the suitability of Twitter as source for research evaluation.
Logically, merely having a Twitter account is not enough. So, specifically, what author activity on Twitter most influences the mention of their papers? To answer that question, number of tweets, followers and followings of 76 Twitter users were extracted and time-normalized. Next, a linear regression analysis was performed to detect the variable that most influences the mention of articles. Results showed that the number of followers explains 34% (R2=.341) of the tweets received by their publications, claiming that a 1% increase in followers would generate 0.31% of new tweets. This result demonstrates that the number of followers is an important factor for the spreading of messages on Twitter. However, the effect of this variable is small because an author needs three times more followers to gain only one mention more.
Nevertheless, the most interesting question in this study is to clarify the relationship between tweets and citations. Does authors’ presence on Twitter have some influence on the next citations to their papers? Results show that there is no statistical difference (p value = 0.144), and therefore to be or not to be on Twitter does not affect the number of citations (non-Twitter users mean = 1.77; Twitter users mean = 2.00). This once more evidences the poor relationship between tweets and citations and puts in doubt suggestions that mentions in social networks can be considered an early proxy of research impact. However, when the regression was applied to detect if some activity parameters on Twitter are related to citations, a weak yet significant relationship was found between followers and citations.
Thus, followers explain only 17% (R2=.171) of the number of citations, meaning a 1% increase of followers could produce 0.24% of citations. Although this result could suggest that participation on Twitter has some influence on the citation of articles, it should be interpreted in a different way, introducing the concept of dissemination. As we have seen, Twitter followers act as information speakers by retweeting articles. Authors with a large number of followers can reach a much wider audience, increasing the likelihood that their papers are cited in future. This new interpretation highlights the importance of dissemination in the citation of articles and suggests that part of research impact could be explained by the intensity in which a paper is spread. Likewise, the more media used to promote an academic result, the greater the likelihood of it being read and later cited.
In conclusion, actively participating on Twitter is a powerful way of promoting and diffusing our academic outputs. This allows us to maintain a wide network of followers that amplify our message and reach a larger audience. Indirectly, such broad dissemination could influence the scholarly impact, slightly improving the prospect of increased citations. Therefore, Twitter cannot be viewed as a citation index, but as an information-spreading network; and the tweets of articles should not be considered an impact indicator, but a measure of research dissemination. However, these conclusions set out a disturbing fact about the use of citations for research evaluation. If dissemination could be a factor for citation success, then to what extent are citations a reflection of research quality and novelty? Resolving this doubt would be an appealing challenge for those studying bibliometrics.
This blog post is based on the author’s article, ‘To be or not to be on Twitter, and its relationship with the tweeting and citation of research papers’, published in Scientometrics (DOI: 10.1007/s11192-016-2113-0).
Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the author
José Luis Ortega is a web researcher partner of the Cybermetrics Lab at the Spanish National Research Council (CSIC). He has published more than 40 research papers about web metrics (link analysis, altmetrics, etc.), information consumption, web usage mining and academic search engines (Google Scholar, Microsoft Academic Search). Recently, he has released the monograph Social Network Sites for Scientists: A Quantitative Survey where he analyses the most relevant academic social networks (ResearchGate, Academia.edu, Mendeley, etc.) using webometric techniques. His ORCID iD is 0000-0001-9857-1511.
This is a flawed study, because the new scholarship paradigm is article, then blogost, then tweet. This author has missed key blogging stage.
This is a good attempt to begin to model the relationships between Twitter usage and citations. It leaves,out a few key concepts:
1. Of course the relationship between # of twitter followers and citations is “week but significant.” Those more famous within a discipline are simultaneously likely to have more followers, AND are more likely to be cited regardless of Twitter. I suspect that using something like H-index for each author as a covariate, to assess not just Twitter follower size, but also long-term citations of all articles from the author, would be useful (actually there are a number of other metrics…h is just a suggestion).
2. Citations always take time to build up for a paper. If I tweet a paper today, and another scholar notices it, they may decide to cite it. Even if that other scholar decides to insert the citation in a manuscript the same day as they saw it on Twitter, AND they submit the manuscript IMMEDIATELY, AND. he paper gets accepted in a traditional joiurnal, I have just described what is a many months – to – many years process. In fact there is almost certainly lag between becoming aware of an article, and its use as a reference in a formal paper…and lag at every step along the way.
3. Another confounded that may mitigate the differences NOT seen between citations for papers by those on Twitter vs. those who are not – again, think of H-index or something similar, it is possible that those on Twitter are an entirely different population, and one of the assortive features could be generational (and manu of the more highly cited scholars have longer careers, because they are older). Some of the most well-cited people I know are barely active on Twitter, or not on at all. Conversely, I know lots of scholars who are trying to push their papers out into the world precisely because they need to build their audience,
3. So, there a easily fixable missing-variable problems…but I also wonder if, even with the right predictors in the model, the problem is the asssumption of linearity.
4. Final thought – this column (and associated study) seem to assume that the point of Tweeting an article is to drive up citations, I,e. That the alt-metric is designed to predict citations, which are a true measure of impact, others have proposed, however, that Tweeting and other forms of dissemination (such as …ahem….blogging about one’s paper), gain a new and different audience, than it does simply by being published and waiting for citations to roll in. In other words – alt-metrics aren’t attempting to be predictive impact as measured by citation; rather, they are measures of a different kind of impact. So, of course the weak relationships observed here exist – the underlying assumption that the intended relationship is tweet –> citation may be flawed.
However, great work! Clearly, the blog post got me thinking, regardless,of,whether I cite your original paper. In something I write and publish. Hence my point…your.blog work had an impact on me that may not lead to a citation of your paper very quickly, or perhaps ever (mostly because I don’t generally publish on bibliometrics).
Interesting that having a twitter account translates to less than one additional tweet (mean of 2.33 vs 1.75) – implying that account holders are not even tweeting their own papers an average of once!
In addition more famous scientists are likely to have larger twitter followings and also to be cited more. This would give a correlation with followers and citations but doe not imply causation.
I am interested in how the methodology accounts for the considerable time lag between someone discovering a paper and a paper being published in which that first paper is cited. This can be several years.
I would assume that Twitter may improve discoverability (the likelihood that your paper is discovered and read) over just hoping the right people find it when they are searching for relevant literature. And perhaps even get some people to read things that they may not think would be relevant and reconsider their approach (less common). But there are a lot of steps between there and actually using the article in a way that warrants direct citation, and getting that new article through the publication process.
How has this been accounted for? Do we even know, statistically, the average time from publication to citation and the modal time?
Is it only the “soft” social scientists who understand the problem here?