Lessons Learned from Developing L2-Bench at Oxford University Press
On the 8th of October 2026, Generation AI was joined by James Edgell, Wm Matthew Kennedy and Isaac Pattis to discuss Oxford University Press’s development of the first large-scale benchmark for AIED, L2-Bench, that measures LLM capabilities in language education. Dr Anthony Bridgen, project coordinator for Generation AI, reflects on this.
Evaluation is neither new nor novel in the education sector, one which is built around evaluating young people against standardised benchmarks in the form of assessment. Nor is the idea of automating aspects of education a recent development, with the notion of the Automatic Teacher put forth by Sidney Pressey in the 1920s as a way to give teachers more time for individual students. Whilst the Automatic Teacher didn’t catch on, technology has been gaining ground in the classroom in the 100 years since, from computers to smartboards and MOOCs to apps, technology is firmly embedded in education. Typically, where EdTech tools have been evaluated, this has been focused on whether they improve young people’s performance in standardised assessment, not whether they improve learning.
Since the launch of OpenAI’s ChatGPT in 2022, the idea of the automatic teacher has arisen once more, and governments globally are racing to implement LLM technology in schools. Estonia’s AI Leap Programme represents the most extreme example of this, undertaking a deep integration of LLMs into their education system through public-private partnership with OpenAI. Simultaneously, some countries are taking this as an opportunity to completely reevaluate the role of technology in education spaces, for example, Norway is pulling back from digital tools in the classroom, particularly restricting access to AI. Whether these tools support learning remains to be seen, but they undoubtedly raise new challenges surrounding the safety, privacy and rights of young people in education.
How, then, can we ensure that AI is having a beneficial effect on education and learning? One route is through development of independently developed evaluations of systems in order to validate claims made by developers of AI EdTech, namely, benchmarks. Typically, model optimisation occurs in areas that are easier to model (e.g. maths and coding); educational performance, because it does not follow hard, universal rules, is frequently avoided. This creates challenges for educators seeking to understand which AI tools, if any, they should be using for certain tasks. By developing benchmarks in this space, there is an opportunity to drive pedagogy-led product development and support policymakers, institutions and educators in deploying the right tool, in the right context.
At Oxford University Press, a collaborative research project also involving external researchers at Oxford University and King’s College London, has been ongoing to develop the public capacity to evaluate AI models against educational tasks. Resulting in L2-Bench, a benchmark for second language education.
Making such a benchmark a reality requires three key elements:
Establishing what good looks like, with the input of teacher frameworks and pedagogical experts.
Deploying this to develop criteria against which to test LLMs, validated by pedagogists, digital learning designers, practitioners, and other key stakeholder groups.
Involving those same practitioner groups in evaluating scores given by an LLM judge to responses produced by the LLM under evaluation.
Throughout, validation by key stakeholders is critical if benchmarks are to have validity in the ecosystem in which they are deployed. This engagement process must be carefully designed in order to ensure that all relevant perspectives are represented and to screen out bad actors.
In short, any benchmark must be created through an established, repeatable process which can be iterated on and re-used in regular evaluation cycles. It is through these systems and others like them that, when a company such as OpenAI promises “stronger built-in safety protections” in their ChatGPT for Teens, we can independently and thoroughly test these claims.
If you are interested in hearing about future seminars, reach out to global.challenges@reuben.ox.ac.uk