In a previous blog post, we described how transformer AI systems can generate entire student essays at the press of a button. Type in the essay title and some opening words, press ‘submit’ and the program responds with a few hundred words of grammatically correct, plausible-looking text. Add another prompt, such as ‘example’ or ‘conclusion,’ and the transformer will continue to write. The result is an original essay or report that will pass a plagiarism check, because the text is not copied from the web, but generated by an AI program trained on the language of billions of books, articles, blog posts, and web pages.
However, transformer systems have a fatal flaw: they are models of language, not of knowledge. A transformer can write in the style of a student essay, scientific paper, or news report, but it has no deep understanding of how the world works. It, therefore, makes up research studies, invents facts to back up its arguments, and spouts references to fake papers. As the developers of GPT-3, the best-known transformer, put it: ‘large pretrained language models are not grounded in other domains of experience, such as video or real-world physical interaction, and thus lack a large amount of context about the world.’
Students now have access to cheap, easy-to-use AI programs, such as transformers, that generate plausible, yet flawed, essays, reports, and exam answers. This seems like an academic’s worst nightmare. However, this powerful new technology can also benefit education. (See video below.)
If an essay question can be answered by an AI program, albeit with flaws, then maybe it is time to rethink assessment. Transformer technology can be used creatively in the classroom to teach academic writing. It can be a starting point for teaching critical digital literacy, and for collectively exploring the ethics and limits of AI.
If an essay question can be answered by an AI program, then maybe it’s time to rethink assessment.
Teachers could set reflective and contextualised written assignments that cannot be generated by AI. For example, each student could carry out an independent research project, then produce a written report on that specific project, get feedback on the report, then write a critical reflection on the feedback and the issues raised by the project.
Universities could also embrace AI-assisted writing. Students would then be allowed to employ whatever tools they wish, to develop their skills in critical reading, accuracy, argumentation and structure. Assignments where AI is not allowed (say, in an invigilated exam) would also be assessed, as currently, for style, expression, voice and personal reflection.
An imaginative way to teach academic writing could be for the teacher to generate a set of essays with the aid of a transformer AI, then ask students to critique these and write their own, better, versions. Alternatively, students could themselves set exam questions on a topic, command the AI to answer them, then evaluate the answers. Or the teacher could first set an open-ended question, then ask each student to generate AI responses to it. Students, in small groups, could discuss these responses in relation to the marking criteria. Following this, each student could write an integrative essay drawing on the AI answers to address the original question.[1] Students would have the option to experiment with a variety of prompts to elicit different perspectives or arguments.
Transformer AI is a gamechanger
Mike Sharples, co-author of Story Machines, discusses:
- What is a transformer AI program, and why is it a gamechanger?
- Could AI be used to disrupt testing for lower-level learning?
- How could AI help students develop critical digital literacy?
Here’s a brief example, with the prompt or question in bold italics and two alternative answers generated by GPT-3 (verbatim) below.
Question: Compare Marx and Nietzsche in their analysis of the underlying problems of their day in 19th century European society.
Answer 1: There are a few key ways in which Marx and Nietzsche differ in their analysis of 19th century European society. For one, Marx was a materialist who believed that economic forces were the primary drivers of history, while Nietzsche was a idealist who believed that ideas and culture were the primary drivers of history. Additionally, Marx believed that class conflict was the main source of social tension and conflict, while Nietzsche believed that the main source of social tension was the tension between the “herd mentality” and the “individual will.” Finally, Marx believed that the way to solve the problems of society was through revolution and the overthrow of the capitalist system, while Nietzsche believed that the way to solve the problems of society was through the development of a new, higher culture.
Answer 2: Both Marx and Nietzsche were concerned with the problems of 19th century European society, such as the growing divide between the rich and the poor, the increasing number of industrial accidents, and the spread of disease. However, they had different solutions to these problems. Marx believed that the only way to solve these problems was through a revolution that would overthrow the capitalist system. Nietzsche, on the other hand, believed that the only way to solve these problems was through a revaluation of all values, which would lead to a new way of living.
Transformer AI can also become a tool for creative writing. For example, the student writes a first paragraph, the AI continues with the second paragraph, and so on. The AI writing partner helps to maintain a flow of words and may take the story in unexpected directions, to which the student must respond.
Last, but by no means least, students could experiment with transformer systems to explore the ethics and limits of generative AI. How does it feel to interact with an expert wordsmith that has no intrinsic morals and no experience of the world? Is writing with AI tantamount to plagiarism? Is generative AI a threat or a boon to higher education, or perhaps both? Because of the strong impact that AI can have on education, we need to consider what critical technical skills students need to develop, particularly in the humanities and the social sciences.
How does it feel to interact with an expert wordsmith that has no intrinsic morals and no experience of the world?
Transformer AI systems have many more acts to perform. They can translate languages, summarise scientific papers, converse as chatbots, explain computer code, and generate computer programs from written descriptions. To consider them to be solely a threat to education is to misunderstand both their potential and their limitations.
[1] These suggestions are based on responses by @jennicarr8 to a Twitter discussion on rethinking assessment in an era of generative AI. https://twitter.com/sharplm/status/1534051131978047494
___________________________________________________________________
Disclaimer: This post is opinion-based and does not reflect the views of the London School of Economics and Political Science or any of its constituent departments and divisions.
___________________________________________________________________
Image: Andrea De Santis on Unsplash
Your conclusion “This seems like an academic’s worst nightmare” rather misses the point. Please allow me to argue by analogy.
One way to get fit it so ride on a stationary bicycle. If you ride one, you practice using your muscles, and train them, and your cardiovascular system, to get stronger. Unfortunately, a new invention came along. It is call “the electric motor”. If you attach one of these to your stationary bicycle, then the axel will spin with no effort from you, but won’t get any fitter. Isn’t this a fitness trainer’s worst nightmare?
Does this mean that stationary bicycles are no longer fit for purpose? No, it just means that they only work for people motivated to engage with them, but you will only get the benefits through hard work. Essays are similar. Check the meaning of ‘essay’ in your dictionary. It is a try, an attempt. The mental exercise of taking the material you have studied, and arranging it into a coherent piece of prose writing that justifies the argument being made in answer to a specific question is a fantastic mental exercise. I am currently studying an Open University music degree, and so I have written essays recently. My experience is that the point where you really consolidate what you have learned recently is while you struggle and sweat over the essay. So, I see no need to reject a tried a tested and academic exercise, just because a new technology has come along.
There is, however, a problem. The problem is, to return to the analogy, that we only care about how fast the axle spins. That is all that we measure. Ultimately, we just assess the result of the process, the final essay, not the struggle to produce it. And that is reduced to a single number in a gradebook, and ultimately leads to a degree classification, and potentially better job, which is why there are incentives to cheat. But, really, the written essay is not the point. Reading student essays is a chore. The point is, the only way to learn is to sweat over the subject matters and make it your own. The process of writing an essay is a good way to do that (but not the only way). We should not stop getting students to that. We should think think about how we incentivise students to do that, and how we award degrees as a result.
Thanks for reading.
Is Tim Hunt’s comment an AI-generated response to the article? (Probably). A classic example of taking a sentence completely out of context. The very next sentence is “this powerful new technology can also benefit education.” The authors did not advocate the abandonment of essay writing; they just provided some sensible and creative suggestions for how to innovate written assessments, including essays.