Jess Casey: Has the Leaving Cert calculated grades system failed the integrity test?

Jess Casey: Has the Leaving Cert calculated grades system failed the integrity test?

Minister for Education Norma Foley:/PA Wire

Do the two known errors with the Leaving Cert calculated grades process call into question the integrity of a system the Department of Education promised would be accurate and fair? 

The full extent of the flaws in the system will only really be known once a full external review is completed. 

While the department maintains the error lies with just one line of code, many will ask how, and why, two seemingly obvious flaws were not spotted sooner, particularly given the reassurances results were rigorously checked. 

Technically in development since the relatively early stages of the Covid-19 shutdown, what would eventually become the calculated grades system was first initiated back in April. 

This is around the time that it started to become undeniably clear that the traditional Leaving Cert exams wouldn’t go ahead as usual in June.  

According to official documents published by the Department of Education, while the preference was to hold the written exams as planned, it was “inevitable” that large numbers of students wouldn’t be able to sit them due to Covid-19. 

Joe McHugh, the previous Minister for Education, set up a working group within his department to scope out a possible alternative way of issuing “fair and valid” results.

The membership of the 'Technical Working Group' expanded over time, and eventually, by the end of the process, it was made up of officials from the State Examinations Commission (SEC), the Educational Research Centre (ERC), the Department of Education, and Polymetrika, an external company who we now know discovered the two known flaws in the system last week. 

The original value of the contract with the Canadian company who helped to oversee the implementation of the process was €75,000. To date, the spend has been €160,000, according to department officials. 

Back in April, there were growing calls for certainty around plans for the Leaving Cert. Students, who were understandably very anxious, called for clarity. Politically, the pressure was also mounting. 

At the beginning of May, Thomas Byrne, then Fianna Fáil’s education spokesman, was the first politician to directly call for the cancellation of the Leaving Cert. 

But while Mr McHugh may have faced political heat, there was also strong, credible evidence to suggest it was not safe or feasible to hold the traditional written exams given the pandemic. The exams were officially cancelled on May 8.  

The overall calculated grades system can be broken down into four phases: The school-based phase, ie teachers' estimates; the national standardisation phase, ie the statistical modeling and estimation process used to generate the final grade students received; the results phase; and the post-results phase. 

Schools engaged in the process in good faith, despite it being new to them. Some misgivings over legal indemnity saw the Association of Secondary Teachers, Ireland (ASTI) direct its members not to engage in the process. 

However, this was resolved within a day, and the process began promptly. 

Raw data received from schools included evidence they had "engaged in the work diligently and with integrity", according to a technical report published by the Calculate Grades Office. 

However, there was also evidence to suggest "overestimation" or "underestimation", and as a result one in five grades was changed by its standardisation process. 

The Independent Steering Committee on Calculated Grades was set up in June to oversee the implementation of calculated grades "from the perspective of quality assurance and integrity". 

This committee was chaired by Aine Lawlor, former director of the Teaching Council.  Separately, Janet Brown, the former director of the Scottish Qualifications Authority was also appointed as an external reviewer. 

At the end of June, Norma Foley, the current Minister for Education was appointed as the coalition Government was formed. In July, it emerged that results would be issued in September, much later than expected.  

Ms Foley told the Dáil that it was of "vital importance that the process is given sufficient time to execute to the highest standards and to rigorous and robust quality assurance so that the results provided in 2020 enjoy the same status and value as the leaving certificates of previous [years] and indeed those to come".

Late in August, controversy in Britain over the A-Level system saw last-minute changes to the system here, with the removal of some of the most contentious elements of school-based data. 

A school's track record in the Leaving Cert was eliminated from the process. Technical reports published by the department acknowledge that the results would have been "more accurate" if this element had been included in the overall equation. 

On September 2,  Aine Lawlor wrote to Norma Foley on behalf of the Steering Committee to sign off on the quality and integrity of the system. 

The letter notes that the Educational Research Centre (ERC) provided a "data quality assurance and verification service" on both the data processing and standardisation processes.

It had not been feasible to carry out a number of "potential validity checks" due to time constraints, the National Standardisation Group had advised the committee, but it was satisfied that "the checks that were carried out addressed the priority areas identified for validation purposes". 

Separately, Janet Brown, the external reviewer, also signed off on the approach taken, noting it was "well planned" with key elements focusing on "governance and decision making; data integrity and validation; oversight and challenge; and responsiveness and flexibility." 

Speaking at the press conference this week after the initial details of the flaws were made public, Harold Hislop, chief inspector with the Department of Education, acknowledged that a full trial of the data was never run. 

Instead, it was just sampled. 

“There were several layers of validation checks put in particularly for how the data was taken in from schools, how it was transferred from different modules and parts of the process, how it was transferred to the Canadian contractor and back again," he explained. 

These checks were signed off on by the ERC. “And they did conduct sampling tests of the code as well, including the code that dealt with the junior cert cycle data." But the code itself was only sampled, he added. 

"A full trial of all of the data, of course, couldn't have been run.

“We had never done this before, it is a completely new process. Between May and August, the model was being developed and built as the National Standardisation Group worked out how the model needed to work and what should be included in the code.” 

“The code had to be altered and changed and developed. That was the whole way the process was originally intended to run. So it wouldn't have been possible to run the full code with a full test load of data. 

"We had never actually done it ever before in Ireland. It's an entirely new piece of code. It's not code that pre-existed and or that was bought off the shelf in any way like that, it was [developed] new.”

With so much focus placed on maintaining data integrity, many will question how two seemingly basic coding mistakes could slip by. 

The review of the coding flaws is not expected to have concluded by the weekend, but it's hoped more will become clearer thereafter. 

More in this section

Lunchtime
News Wrap

A lunchtime summary of content highlights on the Irish Examiner website. Delivered at 1pm each day.

Sign up
Revoiced
Newsletter

Our Covid-free newsletter brings together some of the best bits from irishexaminer.com, as chosen by our editor, direct to your inbox every Monday.

Sign up