Ellen HelsperA new report questioning the assumption that using the Internet and related ICTs is beneficial for everyone was released today. Here one of the authors, LSE’s Ellen Helsper, explains how increased online activity doesn’t always translate into tangible outcomes.

There may be many disagreements amongst UK politicians in the run up to the election but the one thing that they do agree upon is the importance of making sure the UK becomes a digitally savvy nation. It is worrying in this context that the most recent report from the “From Digital Skills to Tangible Outcomes” project shows that many people struggle to translate their internet use into tangible offline benefits.

 

Outcomes achieved*

graph tangible outcomes* The percentage of participants that on average agreed they achieved outcomes in a certain area less or more than if they had undertaken the same activity without using the Internet

A tangible benefit might be finding a (better) job, increased knowledge or education, higher quality relationships and interactions, membership of groups or organisations, or improved personal well-being. As the chart above shows, undertaking an activity in the digital space does not automatically lead to achieving an outcome different from the outcome achieved when undertaking it offline. Depending on the outcome under investigation, around 50 to 75% of the respondents indicated that they did not achieve an outcome that they could not have achieved offline. It seems like getting the benefits of Internet use is easier for economic and individual outcomes than for cultural and social outcomes.

These findings suggest real problems for existing policies and interventions aiming to tackle digital inclusion. Existing evaluations tend to measure success in improvement of digital skills or, in more sophisticated designs, different ways of engaging with Information and Communication Technologies (ICTs). Underlying these measurements is the assumption that they are good indicators of someone actually achieving a related tangible and satisfying outcome in his or her everyday life.

That these indicators are unsatisfactory is demonstrated by our research findings, showing that use of the internet does not directly translate into tangible outcomes. For example, the pilot survey conducted in the Netherlands and the UK shows that doing online job searches has often been used as an indicator of employment benefits of digital inclusion, but this project shows that 54% indicated that they could have probably found the job they got through their online search offline as well, and while 52% indicated that the internet had influenced how they worked, 29% did not think it had influenced their job in a positive way. In the interviews conducted this was clear when one respondent explained this as follows: “So it has influenced my job completely, but it’s not to improve my job. If anything, it’s made me and a lot of other people a bit lazier at work (…) It has influenced how I do my job, but it’s made me worse”.

The report showed that there are social inequalities in the achievement of and satisfaction with the tangible outcomes from internet use. This complicates the work of those who design digital inclusion interventions using measures such as access, skills and engagement with ICTs as indicators, since these do not necessarily lead to tangible offline benefits especially for the more disadvantaged in society.

Upon closer examination there might also be positive news, as the report argues that to translate engagement with an online activity into a tangible outcome, having digital skills is essential, and this is an area where interventions can have an impact. Differences in digital skills between different socio-economic and socio-cultural groups lead to lower levels of achievement of and satisfaction with outcomes when people engage with an activity online.

Benefits of a multi-modal approach to tangible outcomes

To be able to evaluate the social return on investment of digital inclusion initiatives it is important to be quite specific about which areas of outcomes are of interest. Taking a broad stroke approach is not useful since those who were able to achieve an outcome in one area are not necessarily able to achieve an outcome in another: being able to achieve benefits in terms of personal well-being and health does not mean that the person is able to achieve benefits at the social or economic level, for example. This study was the first to think about how to measure and conceptualise outcomes in terms of four fields of resources: economic, cultural, social and individual well-being.

pic tangible outcomes

A previous report on measuring digital skills discussed how skills should be operationalised separately from use and along different categories (e.g. operational, social, and creative skills). In the Tangible Outcomes report published today these different skills were shown to aid the translation of engagement into tangible outcomes in different areas. I argue here, as we do in the report, that a model with the inclusion of this multi-modal approach to understanding and measuring outcomes allows the researcher or evaluator to understand the “unintended benefits” of engagement with ICTs . For example, an intervention aiming to help people engage with learning might have the secondary outcome of improving health or increasing self-actualisation. This will give us not only a better way of understanding and evaluating digital inclusion, but also a way of seeing how different types of engagement and skills are related to a wide variety of social benefits that we have tended to ignore previously .

The From Digital Skills to Tangible Outcomes study presented in the report was a first attempt at measuring and explaining outcomes of digital engagement. With my co-authors Alexander Van Deursen and Rebecca Eynon we will build on this in future publications and examine the pathways that lead from social exclusion to tangible outcomes of Internet use through means of digital access, literacy, motivation and engagement. There is still room for improvement in the measurement of tangible outcomes, especially as this was the first time (that we are aware of) that an evaluation was attempted in this comprehensive a fashion.

Notwithstanding these imperfections, we are calling on organisations like the Government Digital Service, Go On UK, and various funders of digital inclusion programmes who are interested in the social return on investment and the real life impact of digital inequalities, to change their approach to evaluation so that the nuances of the causes and consequences of different forms of digital exclusion can be understood, evaluated, and countered in a more efficient and more sustainable fashion.

This post gives the views of the author and does not represent the position of the LSE Media Policy Project blog, nor of the London School of Economics and Political Science.