This article was written by Atte Jääskeläinen and is part of a JournalismAI series in which we invite industry experts to share case-studies of newsrooms experimenting with AI technology.
Sign up for the JournalismAI monthly newsletter to follow the series.
At some point, all traditional newsrooms that experiment with data and AI seem to hit a wall: the newsroom is not producing the content that the algorithm says the audience would like to read.
You may define this problem as a commercial one. If the newsroom earns its living in the marketplace, then serving the audience’s needs matters, whether you like it or not. But it also matters from a social point of view. In the UK, only around one third of the audience thinks that the media pick the right subjects and only about half thinks that the media help them to understand the news. In the 2019 General Election, only a third of the population thought that the media did a good job. Luckily, across countries, a bit more than half of the audience think that media keeps them updated.1
So, we have to conclude that professional intuition in the newsroom is far from perfect. What if a machine could help? This question is still a taboo in many newsrooms. The immediate response is often that no machine can take into account all the aspects and contexts of news. That the audience wants journalists to be responsible for choosing what to write, and how, and which pieces to promote.
Because of these concerns, we thought it was worth experimenting with machine-learning-based modern predictive analytics to see how they could be used in enhancing editorial decision-making in newsrooms. Our proposal was partly funded by the Media Industry Research Foundation of Finland and our industry partner was the second-largest newspaper publisher in Finland, Alma Media.
We took a ’constructive’ approach in our research: the aim was to find a relevant solution to a practical problem. We investigated first the field of predictive analytics to find out what had already been done, in media and elsewhere. Among 12 interviewees were people like Financial Times Chief Data Officer Tom Betts and BBC News’ Chief of Data Science Magda Piatkowska.
We also interviewed 16 key people from Alma Media, to model the causalities between actions in the newsroom and “success”, asking what that word means for each interviewee. We modelled the newsroom cognition process with system-dynamic stock-and-flow charts borrowed from engineering science. The results were striking: the news professionals said it was a positive experiment and produced “a discussion that too rarely happens in newsrooms but surely is needed.”
Based on this, we found a total of 15 possible use cases to optimise decision-making with the help of modern predictive machine-learning. After careful prioritising with the media company’s project leads, the case selected for testing was this: Is it possible to predict the contribution of a single news story to the willingness of a user to become a subscriber, or stay as a subscriber – before the story is published?
The value of the prediction is obvious: If you get feedback before publishing, you still have time to do something with the story: make a better storyline, headline, or reconsider whether it’s worth publishing after all. My own experience, shared by many of my former colleagues, of monitoring the data reports of news consumption often was unpleasant: more than half of the stories produced didn’t get any relevant digital readership.
Because of limitations in time and budget, the proof of concept had to be based on some assumptions and simplifications, but the luxury in our research project was that the data we used was real, the stories were real and, of course, the audience was real.
Building A Model
After around two months of experimenting with our research group’s machine-learning experts from Fourkind Oy and Alma Media’s audience insight specialists, we built a model based on recurrent neural networks that analysed the whole text content. It was based on language analysis and comparison with the data from all historical behaviour of the audience that predicted who would read a story and how many subscriptions would be generated after reading the story.
At first, we taught the Finnish language to the model with the full content of Finnish Wikipedia, then we taught it the correct news style with the full archives of Alma Media. We also used Universal Language-Model Fine-tuning, which allows the language model to be effectively used in other contexts. In the end, three different regression models were used for predictions.
Finally, the model had to be fine-tuned and the prediction model changed to make it work in a daily newsroom. First, the predictions were presented as actual numbers, and that made users focus on the differences between predicted and actual numbers. When the model was changed to predict five categories, aligned with newsroom targets, it started to work well. The quote from Alma Media’s project lead was: “Expectations exceeded”.
The system was brought into production, and further developed. In later phases, different kinds of metadata were added to the model. It has been showcased by Alma Media in a recent article in an industry magazine as an example of using neural networks as a practical tool to augment editorial judgment.2 In the terms of the ’constructive approach’: the solution had passed a “weak market test”.
Some traditional news leaders and academics, after hearing what we had done, said that we have created an ’infernal’ machine. My answer: You already use analytics to find out how your content is consumed. You already use that information as guidance for what to cover, how and what to promote. You even sometimes let the machines do the promoting.3 Why not test in advance what the machines are going to do, and how the audience will react? Is this information so dangerous that you rather would not like to know it?
That being said, an optimal newsroom is not one in which this kind of system is always able to predict what will be successful and what will not. News should be surprising. Predictive analytics is based on historical data. When history does not repeat itself, it’s best detectable by humans.
Or is it?
Atte Jääskeläinen is a Senior Visiting Fellow at LSE and Professor of Practice at LUT University, Finland. He was Director of News for Finnish Broadcasting Company 2007-2017 and CEO of Finnish News Agency 2004-2006. He co-authored EBU News Report 2019: The Next Newsroom. Unlocking the value of AI for Public Service Journalism and was co-director of an LSE research project The Future of National News Agencies in Europe.
You can contact Atte via: email@example.com
Link to the published paper in ACM Digital Library (open access): https://dl.acm.org/doi/pdf/10.1145/3377290.3377299
1Across 38 countries studied, 62 % feel the media are good at keeping people up to date and 51 % feel media are good at helping them understand the news. Newman, N., Fletcher, R., Kalogeropoulos, A., & Nielsen, R. K. (2019, June 12). Reuters Institute Digital News Report 2019. http://www.digitalnewsreport.org/survey/2019/overview-key-findings-2019/.
During the UK General Election 2019, 30-40 % respondents say they thought that the media as a whole did a good job. Fletcher, R., Newman, N., & Schulz, A. (n.d.). A Mile Wide, an Inch Deep: Online News and Media Use in the 2019 UK General Election. Digital News Report. http://www.digitalnewsreport.org/publications/2020/mile-wide-inch-deep-online-news-media-use-2019-uk-general-election/
2Autio, N. (2020, January 29). Tekoäly ennakoi, onko juttu hitti vai huti. Suomen Lehdistö. https://suomenlehdisto.fi/tekoaly-ennakoi-onko-juttu-hitti-vai-huti/
3Cherubini, F., & Nielsen, R. K. (2016). Editorial Analytics: How News Media are Developing and Using Audience Data and Metrics. Reuters Institute for the Study of Journalism.
Blanchett Neheli, N. (2018). News by Numbers. Digital Journalism, 6(8), 1041–1051;
Nelson, J. L. (2018). The Elusive Engagement Metric. Digital Journalism, 6(4), 528–544. Nelson, J. L., & Tandoc, E. C., Jr. (2019). Doing ‘well’ or doing ‘good’: What audience analytics reveal about journalism’s competing goals. Journalism Studies, 20(13), 1960–1976.
As anyone knows a model is only as good as the data you feed into it, and is always biased by the assumptions you make.
Fascinating experiment! As a recently educated journalist I hope to have access to tools like this in my work in the future. Although I really wish the claim that “At some point, all traditional newsrooms that experiment with data and AI seem to hit a wall: the newsroom is not producing the content that the algorithm says the audience would like to read” was founded in documented empirical research. Because that’s such an interesting statement! But I feel like I can’t in good conscience cite it because of the lack of citation, especially since it says “all traditional newsrooms”.
Anyway, great article and very cool experiment! Looking forward to what you’re gonna do next