Algorithm at heart of Scottish exam results fiasco not analysed

The algorithm used by the SQA in its moderation of this year’s exam results was not made available to investigators, with the Scottish Government refusing to commit to a further independent review of the system.

John Swinney and the SQA have been heavily criticised by a report into the 2020 exam results fiasco

Its omission from the Priestley report is said to have impacted the ability of the researchers to fully investigate the failings of the system, which led to a damaging U-turn from the Scottish Government.

The report also reveals the education secretary John Swinney was warned about the likelihood of the moderation system disproportionately affecting poorer pupils in July, but did nothing to act.

Instead following the publication of the results Mr Swinney’s officials were instructed to “do lots of digging” to show the system had avoided that possibility.

The report states: “Email correspondence between the SQA and the government suggests that this issue and its explosive implications for public opinion appear to have not been fully grasped by SQA, other than through its recourse to appeals, until the EQIA was finalised in July, nor by the government until after the results and EQIA were seen at the end of July.

"Even at this late stage, the focus seemed to rest on presenting a positive picture (the attainment gap had closed in general terms) rather than seeking a fuller understanding of the nuances in the data.”

In the damning report, both the SQA and the Scottish Government are shown to have acted without appropriate regard to equalities issues and the impact of its moderation policy on individual learners.

The SQA still argue they hold “no regret” in respect of the moderation approach used.

The report states its authors were told it would not receive the dataset used by the SQA, which the authors said meant many questions on how it was applied and its subsequent impact could not be answered.

A recommendation for an independent review into the moderation system has been kicked into the long grass by the Scottish Government and listed as potential future research,.

It is the only recommendation from the Priestley report to not be accepted in full.

The SQA and Scottish Government were found to have taken decisions “often undertaken” in reaction to political and media commentary.

However, it was not until June the SQA realised extensive moderation would be necessary to maintain similarity with historical results.

One SQA official told the report: “The sledgehammer was because of the estimates and how different they were from historic distributions.”

Despite the suggestion from some within the SQA that teacher grades may have been overestimated and therefore viewed as unreliable, the report states the opposite was in fact the case with schools “cautious in their allocations”.

Both Mr Swinney and the SQA were criticised for “over-focusing” on the narrowing of the attainment gap, something the report said hid the fact that poorer students were disproportionately affected.

The “arbitrary nature” of the appeals process was also criticised, including the effect a “waterfall effect” of the downgrading of A grades leading to those in the lower end of grades below dropping further and even failing.

The report adds: “Despite the early warning about potential equality impacts, there was little evidence of systematic data analysis to identify anomalies, drawing on government and local government expertise in statistics.”

A message from the Editor:

Thank you for reading this story on our website. While I have your attention, I also have an important request to make of you.

The dramatic events of 2020 are having a major impact on many of our advertisers - and consequently the revenue we receive. We are now more reliant than ever on you taking out a digital subscription to support our journalism.

Subscribe to scotsman.com and enjoy unlimited access to Scottish news and information online and on our app. Visit https://www.scotsman.com/subscriptions now to sign up.

By supporting us, we are able to support you in providing trusted, fact-checked content for this website.

Joy Yates

Editorial Director