Every three years, an early Christmas gift arrives for the global education community from the Organisation for Economic Co-operation and Development (OECD). The Programme for International Student Assessments (Pisa) is an international test in which 15 year olds are tested on their knowledge and skills.

It relegates those far below the Pisa top ten as poor performers in desperate need of improvement, which this time includes Wales.

The Pisa scores for participating education systems around the world are unquestionably significant. But since its inception in 2000, Pisa has sparked much debate, especially among experts and policymakers, with many viewing it as a flawed assessment of educational outcomes. In 2018, around 600,000 students took part in the standardised Pisa tests, which measured their performance in maths, science and reading, and also looked at wellbeing.

Predictably, the 2023 Pisa results captured the negative impact of COVID on learners and learning, with some downward trends in performance visible across the data set.

The results signalled mixed fortunes for the UK. The BBC headline, reporting the results starkly stated: “Wales slumps to worst school test results.” Such sweeping statements are by now an anticipated byproduct of Pisa that ignore how the tests are often highly contested and controversial.

Pisa in Wales

Every three years, Pisa measures the ability of 15 year olds to use their reading, mathematics and science knowledge and skills to meet real-life challenges. In Wales, 2,568 pupils from 89 schools took a two-hour computer-based exercise. To put this in context, there are approximately 33,000 pupils in Year 11 in 178 secondary schools and 27 middle schools in Wales.

The subsequent OECD report acknowledges that “the sample for Wales, and for many other countries, did not meet some of the Pisa standards”. It is important to reflect on how a test taken by a sample of 15-year-old students, every three years for two hours, can possibly be a valid and reliable measure of a system’s performance even in a relative sense.

Pisa’s statistics show that Wales’s average score for mathematics in 2022 was significantly lower than the average across OECD countries. Wales’s average scores for mathematics, reading and science have all declined significantly since 2018. This was also the case, on average, across OECD countries for mathematics and reading. Although for science, the difference between the OECD average in 2022 was not significantly different to that in 2018.

It also noted that the gap in performance between pupils from the most disadvantaged backgrounds and the least disadvantaged backgrounds was smaller in Wales than it was on average across OECD countries for all subjects.

The important thing to observe is that Pisa deals in averages. In the latest results, those averages are derived from the 81 countries that took part, which is a huge range. The report notes a relative fall in Welsh performance against an aggregated average of OECD countries.

It then highlights that this decline was also the case on average across OECD countries for mathematics and reading. In other words, this is a trend. It also suggests that Wales has been more successful in closing the achievement gap between the most disadvantaged and least disadvantaged pupils than most other OECD countries. Yet this important indicator of success has been overshadowed by the blanket headlines of abject educational failure.

If all countries participating in Pisa now recover fully after COVID and improve their educational performance across the board, it is highly likely that Wales will “underperform” on Pisa yet again, whatever it does. If all countries in Pisa continue a steady trajectory of improvement, the country differentials will remain largely the same. Some countries may move up or down, but that movement will be marginal.

A game of relatives

There were no real surprises in the latest results. Countries like Singapore, Taiwan and Japan have retained their comparative advantage and will probably continue to do so, because Pisa is a game of relatives. The complexity and dynamic of any education system cannot (and should not) be at the mercy of a single measure of assessment however compelling or lucrative.

But what does that mean for Wales? First, it should encourage us to look at Pisa as one data set only and to not be obsessed by its findings. It is important to put Pisa in perspective by looking far beyond the simple headlines and delving into the detail of the report.

Second, it should be a reminder that Pisa is a snapshot of performance at a particular moment in time. It takes no account of the possibilities and potential of ongoing reform – deep contextual detail is not on its global radar. Hence, the danger is that every three years, Pisa fuels doubt, dissent and concern, when education systems need certainty, confidence and consensus about the reforms they are putting in place.

Third, no education system is perfect. Getting great Pisa scores is certainly no guarantee that the wellbeing and mental health of children and young people is not compromised or sidestepped along the way. The potential of human collateral damage in achieving high Pisa performance over two decades, unsurprisingly, does not feature in the OECD reports.

Wales has a choice, to either let this global compass direct its educational pathway, accepting that every three years it will derail and disrupt the reform agenda, or to hold its nerve. We cannot ignore Pisa, but we can put it in perspective and continue to focus on the learning and wellbeing of all children and young people in Wales. This is what matters most.

Author Bio: Alma Harris is Professor of Leadership in Education at Cardiff Metropolitan University