As part of the 2020 JournalismAI Collab, an international team of journalists led by Christina Elmer of Der Spiegel and Olle Zachrison of Swedish Radio explored new ideas to make it easier for users to discover and consume quality journalism, in spite of the flood of information they have to navigate. In particular, the team decided to focus on the use of automated summaries and how they can become a tool for newsrooms to leverage evergreen articles to enhance and contextualise new stories, as well as to attract and retain new audiences. This article will give you a short summary of what they did and there are links to their much detailed report for you to explore further.
The mere knowledge that AI-powered tools can automatically extract summaries, key quotes, or Q&A’s from any article thrills the creative imagination. Having this conversation with people from different disciplines, countries, and media types has been very fruitful as one small part of the industry’s quest to develop tomorrow’s journalistic formats.
The overarching goal of the team that Christina and Olle were a part of was to foster the development of algorithmic literacy in newsrooms. After exploring a variety of topics that could help them move some steps toward that goal, they agreed on exploring how automated summaries might be used in journalism. The team started by defining three research areas:
- Discoverability: Can AI-summaries help users discover and get key elements of the very best journalism in a massive flood of output?
- Effectiveness: How effective are available AI summarisation tools at capturing the essence of a journalistic evergreen?
- Conversion: Can automated summaries attract more people to premium pieces? Does integrating summaries of evergreens enrich news articles?
What are automated summaries?
At the beginning of their work, the team needed a shared definition of what an automated summary actually is, in the context of their experiment. They went for this: “AI-summarisation is the process of condensing a text into a comprehensive synopsis by using a pre-trained data model and machine learning.”
Existing summarisation models can be divided into extractive – which pull key phrases from a document and use them to create a summary – and abstractive – which instead aim to understand the meaning behind a text and generate new sentences from it.
The most common output formats include:
- Short summaries
- Speakable summaries
- Bullet points
Testing summarisation tools
The team then proceeded to test existing summarisation tools. The testing process involved five steps:
- Selection of evergreens from premium articles – chosen from the various team members’ newsrooms archives.
- Manual benchmarking: the chosen articles were manually summarised by the team.
- Testing existing summarisation tools like Agolo (software used by AP and Forbes, among others) and a prototype developed in-house by BR – Bavarian Radio, one of the members of the team.
- Rating the output of the tools based on criteria like the ability to capture the facts, grammar, journalistic quality, and usability as teasers.
- Results analysis: Comparison of scores given to the output by the different media companies involved.
The key findings of this testing phase included:
- The summaries of short traditional news pieces in English are very good.
- The first lines of the article are crucial, as the summarisation models rely often on them.
- Grammatical accuracy is often satisfying when the summarisation tools use an extractive model.
- The longer and more creative the original pieces are, the lower is the quality of the summary.
- Automated summaries are weak as teasers: They rarely stimulate curiosity.
Test #2: Integration
To explore how the summaries could then be used to enhance and add context to new stories, the team run an experiment for three weeks in Autumn 2020: hand-picked evergreen articles on the climate crisis found in Der Spiegel’s archives were summarised and integrated into new articles on the topic. The goal was to track whether the performance of those articles was boosted by the additional context and information offered by the summaries – inserted as normal summaries, bullet points, quotes, and questions.
The A/B/C/D testing showed some hopeful results as well as some challenges. Unusual formats such as quotes and questions performed quite well as teasers, but inserting snippets into new articles did not significantly help the surrounding article to be read more intensively. Summaries were also rarely used by readers as a vehicle to reach in-depth background information.
- Customising your summarisation model to your use case and data is key.
- Machine-generated summaries need human editors – especially for long and creative formats. Start with standard news!
- Combining AI-summaries with transcription and translation tools is hard but possible.
- Questions and quotes are the most interesting formats to explore further as teasers.
To find out more about the research of the team, you can explore online their full report:
- Download the team’s presentation
- Collaboration as the key to discovering new ways to connect content and users
- Explore the complete study of the Collab team