MSc Sociology graduand, Mira Buerger, offers a summary of the research she conducted for her dissertation exploring the role algorithms play in alternative credit scoring.

Algorithms: Neither makers nor mirrors – How basically everything can be a sociological topic

Far too often technology and finance are seen as topics of mere numbers and neutrality, free from any social touch. Why should sociology deal with these? Why not leave them to the IT and economy professionals? Put simply, because technology and finance are genuinely social and political. In my postgrad dissertation I explored some of these social and political dynamics by investigating the role of algorithms in alternative credit scoring and their relation to human actors.

Algorithms & alternative credit scoring

Algorithms surround us, with their application areas ranging from health and dating to airspace business and policing. To put it bluntly, “we’re already halfway towards a world where algorithms run nearly everything” (Hickman, 2013). Yet, their work remains for many a mysterious black box and is often actively silenced (Citron & Pasquale, 2014: 5). Sociology offers some tools for not only opening these black boxes, but also for empirically investigating the production of their walls and boundaries.

Credit scores can severely affect people’s lives, limited not only to the decision for and the conditions of receiving a loan or car financing, but also increasingly affecting employment opportunities (e.g. Bornstein, 2014; Konczal, 2011). However, algorithmic credit scoring has not been broadly researched in sociology and completely unresearched are the so-called ‘alternative’ credit scoring services, provided by ‘fintech’ (financial technology) startups. These startups develop scoring systems which differ from traditional services primarily in the types of data used, ranging from social media information to device data.

Reputation paradox

Both algorithms and credit scores are often thought of as objective, suggesting that they neutrally reflect reality. On the other hand, however, there have been examples in the news where algorithms are portrayed as powerful shapers of everyday life and creators of inequality; for instance, causing African-Americans to be tagged in picture recognition software as apes (Curtis, 2015; Nieva, 2015), and that high paying executive jobs are primarily offered to male internet users (Datta, Tschantz & Datta, 2015).

This reputation paradox of algorithms – as either impartially reflecting reality or autonomously producing it – is the puzzle I investigate. I argue that algorithmic credit scoring is neither mirroring reality nor producing it independently. By employing a Science and Technology Studies (STS) perspective, I show that both human and algorithm-based decisions are at stake when alternative credit scores are developed.

Human-based and algorithm-based inequality in alternative credit scoring

In order to demonstrate the extent of algorithmic and human action and the intertwined politics in credit scoring, I outline how human and algorithm-based decisions may lead to inequality.

Firstly, the human decisions in the development of scores include several points where inequality could arise. Already the selection of data types can have unequal effects, e.g. when wage payment history is taken into account and unpaid forms of labour, such as care or household work, often associated with female workers, may be neglected. Furthermore, the high dependence on online data may discriminate against those with low internet usage, potentially accelerating a “digital divide” (Norris, 2001). Nonetheless, it should be emphasised that alternative credit scoring may actually tackle such problems by taking into account diverse data types. Beyond the selection of data types, human assumptions can lead to selecting categories and variables that implicitly favour some population groups over others (Pasquale, 2015: 35; Barocas & Selbst, 2015: 8).

Secondly, algorithmic processes can lead to inequality. Algorithmically developed correlations might negatively evaluate variables that may be especially prevalent in minority groups, such as low-paying service jobs (Citron and Pasquale, 2014: 14). Furthermore, historical data can carry old discriminatory elements due to previous human assessment of creditworthiness. This may lead algorithms to detect ‘patterns’ that are based on previous disadvantaging treatment of minority groups (Barocas & Selbst, 2015: 1).

The relation between algorithms and humans: assemblages of power

These examples of potential inequality firstly emphasise that it is essential to consider the human agency in the shaping of algorithms and scores. However, they also demonstrate how powerful the algorithmic identification of correlations can be in its social effects. They thus underscore that algorithms are neither makers nor mirrors, but that they are intertwined with human influence. So how can we imagine the connections between algorithms and humans in the shaping of credit scoring?

I suggest that it is useful to see their relationship in credit scoring as ‘assemblages’, following Deleuze and Guattari (1987: 7). An assemblage presents a “multiplicity” of connections that can be imagined as a “rhizome” (ibid.). Similarly, algorithms and humans are complexly connected to each other, with decisions by humans affecting algorithms and vice versa. In such a multiplicity, power is executed in “power centres”, which present points “where flows are converted into segments” and thus function as “exchangers, converters, oscillators” (ibid.: 226). For algorithmic credit scoring this suggests that power is executed in practices of turning consumer information – such as social media data, online activity or device characteristics – into a score. According to Deleuze and Guattari (1987: 224), these power centres include micro-level competitions for dominating the conversion practices. This concept pictures what I observed in my research: only in some occasions is algorithmic processing the dominant form of conversion; in other cases, the human ability to convert information with the help of expertise and experience knowledge is superior and can trump algorithmic decisions.

To conclude, although it is analytically necessary to separate algorithms and humans in their ‘doings’ in order to investigate the details of decision-making processes, I argue that it is also important to see their interconnectedness as assemblages of power that can replace simplistic notions of algorithms being either neutral mirrors or determining makers. The theoretical framing as assemblage enables to understand the power dynamics in credit scoring and thus presents a first step towards tackling digital dimensions of inequality.

References

Bornstein, D. (2014, October 2). ‘Invisible’ credit? The New York Times. Retrieved from: <http://opinionator.blogs.nytimes.com/2014/10/02/invisible-credit-read-this-now/?_r=0> [Accessed August 20, 2015].

Curtis, S. (2015, July 1). Google photos labels black people as ‘gorillas’. Telegraph. Retrieved from: <http://www.telegraph.co.uk/technology/google/11710136/Google-Photos-assigns-gorilla-tag-to-photos-of-black-people.html> [Accessed August 20, 2015].

Citron, D. K., & Pasquale, F. A. (2014). The scored society: Due process for automated predictions. Rochester, NY: Social Science Research Network. Retrieved from: <http://papers.ssrn.com/abstract=2376209> [Accessed August 20, 2015].

Datta, A., Tschantz, M. C., & Datta, A. (2015). Automated experiments on ad privacy settings. Proceedings on Privacy Enhancing Technologies, 2015(1), 92–112. http://doi.org/10.1515/popets-2015-0007.

Deleuze, G., & Guattari, F. (1987). A thousand plateaus: Capitalism and schizophrenia. (B. Massumi, Trans.). Minneapolis, Minn: University of Minnesota Press.

Hickman, L. (2013, July 1). How algorithms rule the world. The Guardian. Retrieved from: <http://www.theguardian.com/sncience/2013/jul/01/how-algorithms-rule-world-nsa> [Accessed August 20, 2015].

Konczal, M. (2011, June 23). Demos on credit reporting and employment: Surveillance, inequalities and the labor market. Retrieved from: <https://rortybomb.wordpress.com/2011/06/23/demos-on-credit-reporting-and-employment-surveillance-inequalities-and-the-labor-market/> [Accessed August 20, 2015].

Nieva, R. (2015, July 1). Google apologizes for algorithm mistakenly calling black people ‘gorillas’. CNET. Retrieved from: <http://www.cnet.com/news/google-apologizes-for-algorithm-mistakenly-calling-black-people-gorillas/> [Accessed August 20, 2015].

Norris, P. (2001). Digital divide: Civic engagement, information poverty, and the internet worldwide. Cambridge, Mass: Cambridge University Press.

Pasquale, F. (2015). The black box society: The secret algorithms that control money and information. Cambridge: Harvard University Press.