Regulating student outcomes: summary of OfS proposals

Introduction

February 2022

The ‘Regulating student outcomes’ consultation outlines a new approach to setting ‘minimum requirements’ for positive HE outcomes in England. The continuing narrative is the threat of ‘low-performing providers’ to the public purse and public confidence. The consultation is part of the wider regulatory risk-based approach we’ve seen, which is couched as reducing regulatory burden for some (low risk) providers.
The proposals include ‘non-determinative’ minimum outcomes thresholds. That means that falling short of these will trigger other processes prior to a limiting judgement (ie. a penalty). A host of metric estimates will identify potential underlying performance issues which, depending on OfS’s focus, would lead to a closer look at the providers’ context submission and compliance history. This could lead to an improvement notice.
While minimum requirements are proposed for a dizzying number of indicators, no subject level thresholds are identified; although it is proposed to unpick provider performance at subject level (‘engineering’ not course, or engineering discipline) to ‘enable us to identify pockets of provision where a provider is delivering outcomes below our numerical thresholds even though its overall performance is above our numerical thresholds.’
OfS is looking to introduce the changes as soon as July 2022, the publication of all indicators in September and the identification of providers for assessment by October 2022.

Implications for engineering

The EPC has, through its Engineering Education, Employablity and Skills Committee, prepared some initial thoughts on the potential implications on engineering HE. Read our summary here.

Further information

You can read the full OfS consultation and watch introductory video presentations here. You can watch the recorded OfS consultation / Q&A event here.

Complete our survey

The EPC is conducting a full consultation of our members. Please submit your views here. Note, there is a second EPC member consultation in relation to the TEF reforms (here).

 

  • Condition B3 will include minimum requirements for ‘positive student outcomes’.
  • These will be set using a series of numerical thresholds. These are sector-wide numerical minimum expectations.
  • Minimum thresholds will be based on performance in absolute terms. While there is no benchmarking, in calculating the minimum thresholds, OfS has made some downwards adjustments to take account of sector-wide historical (data) context.
  • Numerical thresholds are calculated for three student outcomes:
    1. the proportion of students continuing on a higher education course;
    2. the proportion of students completing a higher education qualification; and
    3. the proportion of students progressing to managerial or professional employment, or further study (note, it is no longer proposed to use LEO earnings data for this measure).
  • Degree classifications metrics have also been dropped from earlier B3 proposals and will continue to form part of Access & Participation Plans.
  • A set of indicators will be produced, each with its own threshold. Each indicator will be formed by student outcome + mode + level.
    1. As described above, student outcomes are: continuation; completion; progression +
    1. Mode relates to mode of study: full-time; part-time; apprenticeships +
    1. Level relates to nine levels of study: first degree; other undergraduate; undergraduate with postgraduate components; postgraduate taught masters; postgraduate research; PGCE; other postgraduate; undergraduate apprenticeships; postgraduate apprenticeships.
  • This could mean each provider having up to 48 indicators – and 64 thresholds (there are two completion thresholds) – depending on the range of modes and levels of courses it offers.  See figure D1, p95.
  • There will additionally be split indicators to disaggregate performance by indicator: Split indicators are: time series, subject, student characteristics, course type, teaching arrangements.
  • Split indicators will not have thresholds. This means there will be a single numerical threshold across all subjects and across course types not reflected in the nine levels of study above (e.g. integrated Foundation Years and HTQs).
  • This extensive dataset will be published to providers as a dashboard every year. There will also be publicly available information of provider performance. It is not clear if all provider data will be in the public domain.
  • For the purposes of the consultation, the accountable officer in all registered providers has been given access to the provider dashboard. Meanwhile, sector level data and a fictional provider dashboard are available on the OfS website.
  • Engineering performs well above threshold in OfS sector level data, except for part-time course continuation undergraduate with postgraduate elements (where approximately half is below threshold).
  • For those of you in Materials and technology (CAH10-03), completion is predominantly below threshold for undergraduate apprenticeships; part-time first degree, postgraduate taught and other postgraduate; and full-time other postgraduate. There are also pockets of continuation under performance part-time (postgraduate taught and other postgraduate).
  • The numerical thresholds will not operate as an automatic mechanism for determining compliance with condition B3. Instead, where a provider’s outcome data is not at or above the numerical thresholds, contextual information will be used by OfS to make a judgement about whether the provider has nevertheless achieved positive outcomes.
  • This will take the form of information provided by the institution plus information to which the OfS already has access, including an assessment of benchmarked values.
  • Information which may be taken into account includes factors that may explain the reasons for a provider’s historical performance and ‘actions a provider has taken, or will take, to improve its performance, and the extent to which those actions appear credible and sustainable and capable of improving the provider’s performance.’
  • The following contextual information has been deemed irrelevant and will not inform the judgement: the mission and strategy of a provider; funding; entry tariff; sustainability; and reputation.
  • Regulatory action will take the form of an Improvement Notice, which will specify indicators requiring improvement, require a provider to take actions and/ or require evidence of sustained improvement on relevant indicators. Regulatory action will not be taken in relation to every provider with an indicator below one or more numerical thresholds each year.
  • Providers would be selected by a yet-to-be determined prioritisation exercise. The following ideas are mooted in the consultation: thematic; focus on the most severe breaches by either number or proportion of provider’s students or the distance between indicator value and threshold; focus on those breaches related to particular groups of students, where there strongest statistical confidence in the data; a random approach.
  • Regulatory action may be focused on either the indictor which is below the numerical threshold or look across all the indicators which fall below the numerical threshold across the institution. This is a trade-off between more frequent shorter assessments versus in depth assessments across institutions.
  • Numerical thresholds will be reviewed every 4 years, in line with but staggered from the proposed TEF assessment cycle.
  • The OfS does not intend to ‘ratchet up’ the numerical thresholds over time as continuous improvement is incentivised through TEF – at undergraduate level, that is.
  • Eligibility to apply for and retain a TEF award will be determined by a provider’s current and previous compliance with the B conditions, including condition B3, as it may not be considered ‘appropriate for a provider to obtain a TEF award that signals ‘teaching excellence’ if there is or has been a breach of our minimum requirements for student outcomes.’
  • Separate consultations are proposed to consider:
    1. Benchmarking approaches (see the current TEF consultation).
    2. Student outcomes for courses which are delivered in partnership agreements; these are nonetheless, expressly in scope and registered providers will take responsibility for these.
    3. Transnational education (TNE) which is out of scope of these proposals.