Drawing on the findings of the GENIAL project, which focused on how generative AI tools are used in practice by students in real time, Dorottya Sallai, Jon Cardoso-Silva and Marcos Barreto analyse how students use these tools differently across qualitative and quantitative subjects and offer recommendations for how educators can integrate these findings into their teaching and assessment plans.

Jon Cardoso-Silva also chaired a panel on The Role of AI in Education at the DSI Squared Symposium on Generative AI and The Knowledge Economy hosted by Imperial and LSE.

As Generative AI (GenAI) tools have become freely available for everyone, their use in teaching and learning has skyrocketed. Chatbots and other AI-powered learning tools have started to change the teaching and learning landscape in higher education, but this transformation has not been one led by the higher education industry. In fact, while the question of how AI will affect universities is still a pressing concern for academic institutions everywhere, students have embraced GenAI tools without hesitation.

Last year, universities found themselves making quickfire decisions about how to regulate GenAI tools without a significant evidence base as to how students were actually using these tools in real-time. To address this shortcoming, we set out to investigate the learning patterns of 220 students through the ‘The GENerative AI Tools as a Catalyst for Learning’ (GENIAL) project. Led by Dr Marcos Barreto and Dr Jon Cardoso-Silva, the initiative that started in September 2023 as a focus group to assess the effectiveness of code-generation tools, has grown into a multidisciplinary project during the 2023-2024 academic year to shed light on how students use GenAI tools in their learning.

universities found themselves making quickfire decisions about how to regulate GenAI tools without a significant evidence base as to how students were actually using these tools in real-time

We collected and analysed data from four undergraduate and three postgraduate courses offered by LSE Departments of Statistics, Data Science, Management, and Public Policy, using a combination of surveys, focus group sessions, observational experiments, and chat log analysis. Participating students used a dedicated chatlog for all their course-related queries and shared their chatlogs together with the corresponding outputs with the research team through weekly surveys.

The preliminary findings suggest that most participating students were using GenAI tools for task completion and to improve their productivity. Students relied on technology most frequently with the intention of ‘speeding up’ the learning journey. For instance, by using chatbots for summarising required readings or troubleshooting coding errors in programming-heavy courses. In contrast, we found that students only occasionally used GenAI to explore concepts, to develop their understanding or to deepen their learning of the subject matter. Instead, they used chatbots predominantly for output-related interactions with the intention to save time when working on their assignments. As one student stated:

“[Using generative AI], I got my assignment done [and] submitted two weeks before the deadline. And then I went through it line by line and understood what each line of the [code] solution does.”

However, our analysis also suggests that trying to save time in this way may be counter-productive in the context of learning. We discovered that students only benefited from the engagement with GenAI tools when they clearly understood the task’s purpose and had a solid grasp of the underlying concepts that were required to complete it. For example, we observed students asking AI tools to explain how a management framework applies to a business case, which would have been an effective way of exploring how theories could be used in practice. However, instead of checking the chatbot’s response against the course material, or critically evaluating it, students often integrated the AI-generated summary into their work as it was, without any substantial content-related changes. We also found students submit AI-generated code in their assessments in statistics and data science courses, mistakenly assuming the answer they generated with the help of the chatbot was correct. In these cases, even though the code was running correctly, it was often unrelated to the task’s objectives and therefore was not sufficient to show an understanding of the course materials.

students only benefited from the engagement with GenAI tools when they clearly understood the task’s purpose and had a solid grasp of the underlying concepts that were required to complete it.

Across both quantitative and qualitative courses, we found that the convincing and authoritative tone of the AI chatbots gave even the best students the impression that the output was accurate and true. In some courses, students produced lower quality work when relying on chatbots too much than they did without its ‘support’. This is not surprising since chatbots may allow students to mimic high-quality language that signals a false impression of knowledge without actual understanding. For example, during the GENIAL focus groups, one student stated: “There is like the ‘dual-purpose’, so one of them is obviously to get high grades in the assignment and the other one is to learn what’s happening”, implying that the process of learning itself is sometimes at odds with the requirement to perform well in assessments.

While students had a more mixed view of GenAI’s benefits in the Autumn Term, over 80 percent of students on all courses in the Winter Term found AI useful for learning. Interestingly, in contrast to our previous assumptions, students on management courses found AI tools slightly less useful than those on quantitative and data science courses.

This is surprising since AI tools are often perceived as a potential threat to the future of essays. Nevertheless, our study suggests that students rely more on AI tools when they are struggling with the quantity of the reading materials or the complexity of the course content and tend to lean less on it when the pace of delivery and the content is more accessible to them. As one of the management students expressed it:

“This week’s content was pretty straightforward, and I haven’t found myself using AI”

Based on these preliminary findings, we set out some practical policy recommendations for university educators in relation to course and assessment design. Given the high likelihood that students may use AI as shortcuts, educators must find strategies to ensure that students use these tools to enhance their learning rather than bypassing it.

What should educators consider when designing their curriculum?

• Assume that students are using GenAI, even if you do not.
• Integrate non-marked activities into the course and give students feedback that includes comments on their AI use.
• Ensure that GenAI is not used in ways that limit students’ ability to fully engage with the curriculum. While with GenAI students can achieve significant results, for instance in coding, with little skill, it is critical to prepare them to move beyond AI-favoured solutions.
• Teach students about the importance of locating primary sources of information, and to be critical of AI-produced outputs.
• Avoid trying to outsmart models’ problem-solving abilities by underspecifying assignment details. These techniques may only provide short-term benefits, as future models are likely to overcome these challenges.
• In coding-heavy courses guide them how to identify and fix problems and help them recognise that alternatives exist. Aim to teach high-level engineering concepts by critiquing the inconsistencies of functions output by a model or learning to re-prompt a system to produce cleaner and more consistent results. This approach can help students understand and apply engineering principles.

What should educators consider when designing their assessments?

• Map out the process you think your students should go through as they work on their assignments and create a process-path to visualise the milestones in their learning journey. These check-in points can be used in the evaluation of students’ work during the assessment.
• Separate the learning process from the assessed ‘product’ by designing assessments that are continuous during the term or if you prefer to keep the end of course evaluation design then integrate tasks into the assessment that require the documentation of the development or thinking process.
• Find ways to measure and support students’ individual learning journey. GenAI makes it easier for educators to evaluate students’ understanding and content-related knowledge throughout the course lifecycle. The integration of in-class quizzes in the beginning, mid- and at the end of term can help measure individual progress and be used at the end of the term to benchmark students’ final grade to their learning journeys.

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: Solen Feyissa on Unsplash.