Ofqual exam results algorithm

Last updated

In 2020, Ofqual, the regulator of qualifications, exams and tests in England, produced a grades standardisation algorithm to combat grade inflation and moderate the teacher-predicted grades for A level and GCSE qualifications in that year, after examinations were cancelled as part of the response to the COVID-19 pandemic.

Contents

History

In late March 2020, Gavin Williamson, the secretary of state for education in Boris Johnson's Conservative government, instructed the head of Ofqual, Sally Collier, to "ensure, as far as is possible, that qualification standards are maintained and the distribution of grades follows a similar profile to that in previous years". On 31 March, he issued a ministerial direction under the Children and Learning Act 2009. [1]

Then, in August, 82% of 'A level' grades were computed using an algorithm devised by Ofqual. More than 4.6 million GCSEs in England – about 97% of the total – were assigned solely by the algorithm. Teacher rankings were taken into consideration, but not the teacher-assessed grades submitted by schools and colleges. [2]

On 25 August, Collier, who oversaw the development of Williamson's algorithm calculation, resigned from the post of chief regulator of Ofqual following mounting pressure. [3] [4]

Vocational qualifications

The algorithm was not applied to vocational and technical qualifications (VTQs), such as BTECs, which are assessed on coursework or as short modules are completed, and in some cases adapted assessments were held. [5] Nevertheless, because of the high level of grade inflation resulting from Ofqual's decision not to apply the algorithm to A levels and GCSEs, Pearson Edexcel, the BTEC examiner, decided to cancel the release of BTEC results on 19 August, the day before they were due to be released, to allow them to be re-moderated in line with Ofqual's grade inflation. [6]

The algorithm

Ofqual's Direct Centre Performance model is based on the record of each centre (school or college) in the subject being assessed. Details of the algorithm were not released until after the results of its first use in August 2020, and then only in part. [7]

Synopsis
The examination centre provided a list of teacher predicted grades, called 'centre assessed grades' (CAGs)
The students were listed in rank order with no ties.
For large cohorts (over 15)
With exams with a large cohort; the previous results of the centre were consulted. For each of the three previous years, the number of students getting each grade (A* to U) is noted. A percentage average is taken.
This distribution is then applied to the current years students-irrespective of their individual CAG.
A further standardisation adjustment could be made on the basis of previous personal historic data: at A level this could be a GCSE result, at GCSE this could be a Key Stage 2 SAT. [8]
For small cohorts, and minority interest exams (under 15).
The individual CAG is used unchanged [7]
The formulas
for large schools with
for small schools with
The variables
is the number of pupils in the subject being assessed
is a specific grade
indicates the school
is the historical grade distribution of grade at the school (centre) over the last three years, 2017-19.
is the predicted grade distribution based on the class’s prior attainment at GCSEs. A class with mostly 9s (the top grade) at GCSE will get a lot of predicted A*s; a class with mostly 1s at GCSEs will get a lot of predicted Us.
is the predicted grade distribution of the previous years, based on their GCSEs. You need to know that because, if previous years were predicted to do poorly and did well, then this year might do the same.
is the fraction of pupils in the class where historical data is available. If you can perfectly track down every GCSE result, then it is 1; if you cannot track down any, it is 0.
CAG is the centre assessed grade.
is the result, which is the grade distribution for each grade at each school .

Schools were asked to make a fair and objective judgement of the grade they believed a student would have achieved, but in addition to rank the students within each grade. This was because the statistical standardisation process required more granular information than the grade alone. [9] Some examining boards issued guidance on the process of forming the judgement to be used within centres, where several teachers taught a subject. [10] This was to be submitted 29 May 2020.

For A-level students, their school had already included a predicted grade as part of the UCAS university application reference. [11] This was submitted by 15 January (15 October 2019 for Oxbridge and medicine) and had been shared with the students. This UCAS predicted grade is not the same as the Ofqual predicted grade.

The normal way to test a predictive algorithm is to run it against the previous year's data: this was not possible as the teacher rank order was not collected in previous years. Instead, tests used the rank order that had emerged from the 2019 final results. [7]

Effects of the algorithm

The A-level grades were announced in England, Wales and Northern Ireland on 13 August 2020. Nearly 36% were lower than teachers' assessments (the CAG) and 3% were down two grades. [12]

Side-effects of the algorithm

Students at small schools or taking minority subjects, such as are offered at small private schools (which are also more likely to have fewer students even in popular subjects), could see their grades being higher than their teacher predictions, especially when falling into the small class/minority interest bracket. Such students traditionally have a narrower range of marks, the weaker students having been invited to leave. Students at large state schools, sixth-form colleges and FE colleges who have open access policies and historically have educated BAME students or vulnerable students saw their results plummet, in order to fit the historic distribution curve. [7]

Students found the system unfair, and pressure was applied on Williamson to explain the results and to reverse his decision to use the algorithm that he had commissioned and Ofqual had implemented. [13] On 12 August Williamson announced 'a triple lock' that let students appeal the result using an undefined valid mock result. [14] But on 15 August, the advice was published with eight conditions set which differed from the minister's statement. [15] Hours after the announcement, Ofqual suspended the system. [16] On 17 August, Ofqual accepted that students should be awarded the CAG grade, instead of the grade predicted by the algorithm. [13]

UCAS said on 19 August that 15,000 pupils were rejected by their first-choice university on the algorithm-generated grades. After the Ofqual decision to use unmoderated teacher predictions, many affected students had grades to meet their offer, and reapplied. 90% of them said they aimed to study at top-tier universities. The effect was that top-tier universities appeared to have a capacity problem. [1] [13]

The Royal Statistical Society said they had offered to help with the construction of the algorithm, but withdrew that offer when they saw the nature of the non-disclosure agreement they would have been required to sign. Ofqual was not prepared to discuss it and delayed replying by 55 days. [17]

Lord Falconer, a former attorney general, opined that three laws had been broken, and gave an example of where Ofqual had ignored a direct instruction of the Secretary of State for Education. [18]

Falconer said the formula for standardising grades was in breach of the overarching objectives under which Ofqual was established by the Apprenticeships, Skills, Children and Learning Act 2009. The objectives require that the grading system gives a reliable indication of the knowledge, skills and understanding of the student, and that it allows for reliable comparisons to be made with students taking exams graded by other boards and to be made with students who took comparable exams in previous years. [2] [18]

The Labour Party suggested that the process was unlawful in that the students were given no appeal mechanism, [18] [2] stating: "There will be a mass of discriminatory impacts by operating the process on the basis of reflecting the previous years' results from their institutions", and "It is bound to disadvantage a whole range of groups with protected characteristics, in breach of a range of anti-discrimination legislation." [18]

See also

Related Research Articles

<span class="mw-page-title-main">General Certificate of Secondary Education</span> British public examinations, generally taken aged 15-16

The General Certificate of Secondary Education (GCSE) is an academic qualification in a range of particular subjects, taken in England, Wales, and Northern Ireland, having been introduced in September 1986, and its first exams sat in 1988. State schools in Scotland use the Scottish Qualifications Certificate instead. However, private schools in Scotland often choose to follow the GCSE system in England.

The General Certificate of Education (GCE) is a subject-specific family of academic qualifications used in awarding bodies in England, Wales, Northern Ireland, Crown dependencies and a few Commonwealth countries. For some time, the Scottish education system has been different from those in the other countries of the United Kingdom.

National Vocational Qualifications (NVQs) are practical work-based awards in England, Wales, and Northern Ireland that are achieved through assessment and training. The regulatory framework supporting NVQs was withdrawn in 2015 and replaced by the Regulated Qualifications Framework (RQF), although the term "NVQ" may be used in RQF qualifications if they "are based on recognised occupational standards, work-based and/or simulated work-based assessment, and where they confer occupational competence".

Education in the United Kingdom is a devolved matter with each of the countries of the United Kingdom having separate systems under separate governments. The UK Government is responsible for England, whilst the Scottish Government, the Welsh Government and the Northern Ireland Executive are responsible for Scotland, Wales and Northern Ireland, respectively.

<span class="mw-page-title-main">AQA</span> British examination board and registered charity

AQA Education, trading as AQA, is an awarding body in England, Wales and Northern Ireland. It compiles specifications and holds examinations in various subjects at GCSE, AS and A Level and offers vocational qualifications. AQA is a registered charity and independent of the government. However, its qualifications and exam syllabi are regulated by the Government of the United Kingdom, which is the regulator for the public examinations system in England and Wales.

<span class="mw-page-title-main">Edexcel</span> British multinational education and examination body owned by Pearson

Edexcel is a British multinational education and examination body formed in 1996 and wholly owned by Pearson plc since 2005. It is the only privately owned examination board in the United Kingdom. Its name is a portmanteau term combining the words education and excellence.

The Advanced Extension Awards are a type of school-leaving qualification in England, Wales and Northern Ireland, usually taken in the final year of schooling, and designed to allow students to "demonstrate their knowledge, understanding and skills to the full". Currently, it is only available for Mathematics and offered by the exam board Edexcel.

<span class="mw-page-title-main">Oxford, Cambridge and RSA Examinations</span> British examination board

Oxford, Cambridge and RSA Examinations (OCR) is an examination board that sets examinations and awards qualifications. It is one of England, Wales and Northern Ireland's five main examination boards.

<span class="mw-page-title-main">International General Certificate of Secondary Education</span> English language based academic qualification

The International General Certificate of Secondary Education (IGCSE) is an English language based secondary qualification similar to the GCSE and is recognised in the United Kingdom as being equivalent to the GCSE for the purposes of recognising prior attainment. It was developed by Cambridge Assessment International Education. The examination boards Edexcel, Learning Resource Network (LRN), and Oxford AQA also offer their own versions of International GCSEs. Students normally begin studying the syllabus at the beginning of Year 10 and take the test at the end of Year 11. However, in some international schools, students can begin studying the syllabus at the beginning of Year 9 and take the test at the end of Year 10.

The Welsh Baccalaureate, or Welsh Bacc, is an educational qualification delivered in secondary schools and colleges across Wales. The Welsh Government says that it gives broader experiences than traditional learning programmes, developing transferable skills useful for education and employment. The Welsh Bacc is offered at Advanced, National Foundation and National/Foundation level, and is studied alongside a range of academic and vocational qualifications.

<span class="mw-page-title-main">A-level</span> British educational certification

The A-level is a subject-based qualification conferred as part of the General Certificate of Education, as well as a school leaving qualification offered by the educational bodies in the United Kingdom and the educational authorities of British Crown dependencies to students completing secondary or pre-university education. They were introduced in England and Wales in 1951 to replace the Higher School Certificate. The A-level permits students to have potential access to a chosen university they applied to with UCAS points. They could be accepted into it should they meet the requirements of the university.

<span class="mw-page-title-main">Ofqual</span> United Kingdom government department

The Office of Qualifications and Examinations Regulation (Ofqual) is a non-ministerial government department that regulates qualifications, exams and tests in England. Colloquially and publicly, Ofqual is often referred to as the exam "watchdog".

The General Certificate of Education (GCE) Advanced Level, or A Level, is a main school leaving qualification in England, Wales, Northern Ireland, the Channel Islands and the Isle of Man. It is available as an alternative qualification in other countries.

<span class="mw-page-title-main">Gavin Williamson</span> British politician (born 1976)

Sir Gavin Alexander Williamson is a British politician who most recently served as Minister of State without Portfolio from 25 October to 8 November 2022. He has been the Member of Parliament (MP) for South Staffordshire since 2010. A member of the Conservative Party, Williamson previously served in Theresa May's Cabinet as Government Chief Whip from 2016 to 2017, Secretary of State for Defence from 2017 to 2019, and as Secretary of State for Education under Boris Johnson from 2019 to 2021.

The national qualification frameworks in the United Kingdom are qualifications frameworks that define and link the levels and credit values of different qualifications.

<span class="mw-page-title-main">GCSE Science</span> British educational qualification

In the education system in England and Wales, science at GCSE level is studied through Biology, Chemistry and Physics.

<span class="mw-page-title-main">Impact of the COVID-19 pandemic on education in the United Kingdom</span>

In March 2020, nurseries, schools, and colleges in the United Kingdom were shut down in response to the COVID-19 pandemic. By 20 March, all schools in the UK had closed for all in-person teaching, except for children of key workers and children considered vulnerable. With children at home, teaching took place online. The emergence of a new variant of COVID-19 in December 2020 led to cancellation of face-to-face teaching across England, Northern Ireland, Scotland, and Wales the following month.

<span class="mw-page-title-main">Kathleen Tattersall</span> English educationist (1942–2013)

Kathleen Tattersall was a British educationalist, specialising in examination administration. She was the leader of five examination boards in the United Kingdom before becoming the first head of exams regulator Ofqual.

Due to the COVID-19 pandemic in the United Kingdom, all secondary education examinations due to be held in 2020 were cancelled. As a result, an alternative method had to be designed and implemented at short notice to determine the qualification grades to be awarded to students for that year. A standardisation algorithm was produced in June 2020 by the regulator Ofqual in England, Qualifications Wales in Wales, Scottish Qualifications Authority in Scotland, and CCEA in Northern Ireland. The algorithm was designed to combat grade inflation, and was to be used to moderate the existing but unpublished centre-assessed grades for A-Level and GCSE students. After the A-Level grades were issued, and after criticism, Ofqual, with the support of HM Government, withdrew these grades. It issued all students the Centre Assessed Grades (CAGs), which had been produced by teachers as part of the process. The same ruling was applied to the awarding of GCSE grades, just a few days before they were issued: CAG-based grades were the ones released on results day.

<span class="mw-page-title-main">Sally Collier</span> British civil servant and former exam regulator of Ofqual

Sally Collier is a British civil servant and former head of the Office of Qualifications and Examinations Regulation (Ofqual). She also involved in negotiating changes to European Union rules governing public procurement.

References

  1. 1 2 Adams, Richard; Elgot, Jessica; Stewart, Heather; Proctor, Kate (19 August 2020). "Ofqual ignored exams warning a month ago amid ministers' pressure". the Guardian. Retrieved 20 August 2020.
  2. 1 2 3 Ferguson, Donna; Savage, Michael (15 August 2020). "Controversial exams algorithm to set 97% of GCSE results". The Guardian. Retrieved 19 August 2020.
  3. Richardson, Hannah (25 August 2020). "Ofqual chief resigns after exams chaos". BBC News . Retrieved 25 August 2020.
  4. "Ofqual head Sally Collier resigns over exams fiasco". the Guardian. 25 August 2020. Retrieved 25 August 2020.
  5. "Ofqual statement on grading of VTQs". GOV.UK. 18 August 2020. Retrieved 20 August 2020.
  6. Richardson, Hannah (19 August 2020). "Fresh exam results chaos as BTec grades pulled". BBC News. Retrieved 20 August 2020.
  7. 1 2 3 4 Harkness, Timandra (17 August 2020). "How OfQual failed the algorithm test". UnHerd. Retrieved 20 August 2020.
  8. "Requirements for the calculation of results in summer 2020" (PDF). /assets.publishing.service.gov.uk/. Retrieved 20 August 2020.
  9. "Summer_2020_grades_for_GCSE_AS_and_A_level" (PDF). gov.uk. p. 6. Retrieved 28 August 2020.
  10. "Preparing centre assessment grades and rank order" (PDF). qualifications.pearson.com. Retrieved 28 August 2020.
  11. "Predicted grades – what you need to know for 2021 entry". UCAS. 3 January 2019. Retrieved 28 August 2020.
  12. "A-levels and GCSEs: How did the exam algorithm work?". BBC News. 20 August 2020. Retrieved 23 August 2020.
  13. 1 2 3 Anonymous (21 August 2020). "'We're picking up the pieces': a university vice-chancellor's diary of A-level chaos". the Guardian. Retrieved 21 August 2020.
  14. "Triple lock for students ahead of A level and GCSE results". Press Release. GOV.UK. 12 August 2020. Retrieved 21 August 2020.
  15. Walawalkar, Aaron (15 August 2020). "Students in England must meet eight criteria to appeal exam results". The Guardian . Retrieved 21 August 2020.
  16. P. A. Media (15 August 2020). "Ofqual suspends A-level appeal criteria hours after announcing them". The Guardian. Retrieved 21 August 2020.
  17. Murray, Jessica (24 August 2020). "Royal Statistical Society hits back at Ofqual in exams algorithm row". the Guardian. Retrieved 24 August 2020.
  18. 1 2 3 4 Elgot, Jessica; Adams, Richard (19 August 2020). "Ofqual exam results algorithm was unlawful, says Labour". the Guardian. Retrieved 19 August 2020.