LSE - Small Logo
LSE - Small Logo

Blog Admin

June 28th, 2018

How to compare apples with oranges: using interdisciplinary “exchange rates” to evaluate publications across disciplines

5 comments | 20 shares

Estimated reading time: 5 minutes

Blog Admin

June 28th, 2018

How to compare apples with oranges: using interdisciplinary “exchange rates” to evaluate publications across disciplines

5 comments | 20 shares

Estimated reading time: 5 minutes

Academic research performance is typically assessed on the basis of scientific productivity. While the number of publications may provide an accurate and useful metric of research performance within one discipline, interdisciplinary comparisons of publication counts prove much more problematic. To solve this problem, Timo Korkeamäki, Jukka Sihvonen, and Sami Vähämaa introduce interdisciplinary “exchange rates”, which can be used to convert the publication records of individuals or institutions to a common scale. Adopting such an approach can increase the objectivity of cross-disciplinary comparisons by eliminating disparities in publishing potential across disciplines.

Deans, promotion and recruiting committees, administrators, and funding agencies are constantly faced with the challenge of evaluating research performance across disciplines. These comparisons are far from straightforward as different disciplines have different publishing standards which ultimately dictate the rate of publishing and publication volumes. As a result, researchers in some disciplines are inherently less likely to produce a high number of publications, and so find themselves at a competitive disadvantage relative to scholars from disciplines where high publication volumes are the norm.

It is important that faculty members from different disciplines are treated, evaluated, and incentivised fairly and objectively. Publications in highly regarded peer-reviewed journals are key to hiring, promotion, and tenure decisions, and also influence salaries and teaching loads at many universities. Therefore, any perceived inequities in cross-disciplinary performance assessments are likely to lead to poor motivation among faculty members who feel that their disciplines are unfairly treated. Moreover, publication records are used by administrators, governments, and funding agencies as the primary criterion for allocating resources and funding to universities, and between faculties, departments, and individual scholars within universities. Given the pivotal role of interdisciplinary comparisons of publication records, it is surprising how little research attention the relative valuation of top-tier publications in business disciplines has received.

Constructing the publication “exchange rates”

In our recent article, we propose an objective method for comparing the value of publications across business disciplines. We use publication data from the leading peer-reviewed journals in accounting, economics, finance, management, and marketing to construct intradisciplinary author rankings that we then employ to estimate the empirical association between the number of publications and author rankings in each discipline. Based on the estimated effort required for improving an individual’s ranking within his or her own discipline, we can deduce the marginal value of a single-authored publication in each discipline. We convert these marginal values into “exchange rates” to compare the interdisciplinary value of publications.

As an example, consider the association between the number of publications and author rankings in accounting and finance (Figure 1). As is the case for all business disciplines, the relationships are linear-logarithmic but differ in slope. A steeper slope means that an individual needs more publications to improve her standing in the intradisciplinary ranking. For example, it takes approximately three articles for a finance scholar to advance from top-10 per cent group to top-1 per cent group. In accounting, the corresponding improvement requires two articles, which indicates that the marginal value of a publication is higher in accounting than in finance. Since publishing three articles in finance leads to similar ranking improvement than two articles in accounting, the publication “exchange rate” from accounting to finance becomes approximately 1.5.

Figure 1: The association between the number of publications and corresponding author ranking in accounting (a) and finance (b). The author ranking on the horizontal axis is measured in terms of logarithmic percentiles, a smaller percentile indicating higher author ranking. This figure is taken from “Evaluating publications across business disciplines: Inferring interdisciplinary ‘exchange rates’ from intradisciplinary author rankings”, Journal of Business Research (2018), and is published under a CC BY-NC-ND 4.0 license.

The underlying premise in the proposed approach is that the marginal value-add of an article reflects the significance and value of a top-tier publication in a competitive scholarly environment. While we illustrate the interdisciplinary “exchange rate” approach within business disciplines, the proposed methodology provides a generic approach for comparative assessments of research performance across any scientific disciplines in which the number of top-tier publications is viewed as an indicator of scientific impact.

The empirical estimation of publication “exchange rates”

We base our empirical estimations on a vast dataset which includes the authors of each article published 2005–2015 in journals classified as “Journals of Distinction” (category 4*) in the Chartered Association of Business Schools’ Academic Journal Guide. The 24 top-ranked journals published a total of 15,610 articles by 18,154 authors during this period. Using these publication data, we estimate the marginal effect of an additional single-authored publication in a top journal on the individual’s ranking within his or her own discipline. Since the relationship between the number of publications and author rankings is linear-logarithmic in all disciplines, we use standard OLS regression to estimate the marginal effect. The estimation results presented in Table 1 demonstrate that there are substantial differences between the disciplines in b, the implied effort required to improve one’s intradisciplinary author ranking. In particular, we find that the value of a single publication in a top-tier journal is highest in accounting and lowest in marketing. After rescaling the results vis-à-vis accounting, our estimates suggest that a single-authored article in a leading accounting journal corresponds to approximately two marketing articles and 1.3 to 1.4 top-ranked economics, finance, and management articles.

Table 1: The empirical association between the number of publications and author ranking in various business disciplines.

Verifying the validity of the publication “exchange rates”

The domination of marketing scholars and the absence of accounting scholars in the unadjusted top-50 author ranking presented in Figure 2a reflect our empirical observation that publishing in the leading marketing journals appears to differ significantly from publishing in leading accounting journals. Consequently, we propose publication “exchange rates” to be used to convert publications into equivalent units. Such adjustment would increase the objectivity of cross-disciplinary comparisons by eliminating the influence of discipline-specific publishing standards. The advantage of applying the “exchange rates” is evident in Figure 2b; all business disciplines seem to be almost equally represented in the interdisciplinary top-50 author ranking once the authors’ outputs are converted into equivalent units by using the estimates in Table 1.

Figure 2: The disciplines of top-50 most prolific authors in business and economics before (a) and after (b) adjusting the number of publications with the publication “exchange rates”.

Overall, the results of our research indicate that the use of publication “exchange rates” for converting publications into equivalent units increases the objectivity of cross-disciplinary comparisons by eliminating the disparity in publishing potential across disciplines.

In the light of our findings, we would suggest tenure and promotion committees should review the candidates in the context of their own disciplines, or, equivalently, transform the number of their publications into equivalent units using proposed publication “exchange rates”.

This blog post is based on the authors’ article, “Evaluating publications across business disciplines: Inferring interdisciplinary “exchange rates” from intradisciplinary author rankings”, published in the Journal of Business Research (DOI: 10.1016/j.jbusres.2017.11.024).

Featured image: Daniel Fazio, via Unsplash (licensed under a CC0 1.0 license).

Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.

About the authors

Timo Korkeamäki is the Dean of Research and the Pentti Family Professor of Finance at Hanken School of Economics.

 

 

Jukka Sihvonen is an Assistant Professor of Accounting at Aalto University School of Business.

 

 

Sami Vähämaa is a Professor of Accounting and Finance at the University of Vaasa.

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: Academic publishing | Citations | Higher education | Research evaluation

5 Comments