Further errors discovered with Leaving Cert code during review

Details of these errors were first made public last Wednesday, although Norma Foley and the department had learnt of the errors the prior week.

The Department of Education is standing over its Leaving Cert calculated grades system after further errors were discovered following an external review of the code that caused thousands of students to receive incorrect grades.

A review by external contractor Educational Testing Services (ETS) published this Saturday evening discovered further issues within the code used by the department to standardise results.

Each of these issues has since been resolved, according to Norma Foley, the Minister for Education, and department officials.

Earlier this evening, the department confirmed that 6,100 students, approximately one in ten, received grades lower than they deserved due to errors in the system.

Students also received higher grades than they should have due to system errors, but the department has not released these figures.

Students who received a higher grade will not have this taken away from them, according to Ms Foley.

Ms Foley again apologised to students for the “exceptionally difficult year” they faced. “And I’m sorry this last week delivered more uncertainty to you,” she said.

“When we found errors in the code, I decided to seek independent expert oversight in the interest of certainty, particularly for students.”

“I am glad that we can now provide students whose grades were lower than they should have been with their corrected results today and that this period of uncertainty is now over for all students.” 

Following the discovery of two errors in the code used to standardise Leaving Cert students’ calculated grades, the US-based ETS was contracted to examine the corrections made by the department and Polymetrika, the Canadian company who implemented the code.

These errors related to the Junior Cycle data used, which was meant to draw on the core subjects of Irish, English, and Maths, and combine them with students’ two best non-core subjects.

The coding error instead combined them with the students’ two weakest non-core subjects.

A separate error found that CSPE was meant to be removed from these scores, but the subject had remained included.

According to the report published by ETS, the algorithm used to treat “extreme/outlier” students in each school does not exactly match a mathematical process described by the department.

Details of these errors were first made public last Wednesday, although Norma Foley and the department had learnt of the errors the prior week.

ETS was commissioned to examine if the code that carried out the standardisation process was now operating correctly after corrections had been made. It found the two errors had been corrected by the department and Polymetrika.

However, ETS also found one new error that affected students who did not sit all three core subjects at Junior Cycle. This would mainly apply to students with special educational needs.

In these cases, the system was meant to use the average national Junior Cycle score in the missing subject of the group of students who took their Leaving Certificate in 2017, 2018, 2019 and 2020.

Instead, it chose that student’s next best subject for inclusion.

All three of these errors are contained in the same section of coding and they had an impact on students’ scores.

They have since been resolved and the code is operating correctly, according to the department.

Polymetrika was asked to amend this element of the code, and after doing so ETS rechecked the code to make sure there were no further errors.

At this time, a further discrepancy was discovered, although it did not make any significant difference to students’ scores.

Following the discovery of two errors in the code used to standardise Leaving Cert students’ calculated grades, the US-based ETS was contracted to examine the corrections made by the department and Polymetrika, the Canadian company who implemented the code.

According to the report published by ETS, the algorithm used to treat “extreme/outlier” students in each school does not exactly match a mathematical process described by the department.

“While the optimization criterion is very closely related to what is described in the Report it is not exactly equivalent [sic].” 

ETS has now signed off that the revision of the code is now correct, and that the second issue did not have a “meaningful impact” on results, i.e. no student received a lower grade as a result.

ETS also concluded that other than the two issues identified, the “other parts” of the complex algorithm it reviewed match the procedures described by the Department of Education and are "producing scores in accordance with the methodology described in the Report [sic]."

The report compiled by ETS published this Saturday also notes that due to the “very short timeframe”, a full audit of the entire coding was not carried out, and instead the coding was sampled.

“In sampling the coding, the Department [of Education] and ETS agree that certain areas of coding should be prioritized within the available time frame.” 

Areas were prioritised and ETS was provided with access to the full coding and the databases used to run the standardization process, it notes.