Regulating student outcomes (changes to B3 baseline)

• Condition B3 will include minimum requirements for “positive student outcomes”.
• These will be set using a series of numerical thresholds. These are sector-wide numerical minimum expectations.
• Minimum thresholds will be based on performance in absolute terms. While there is no benchmarking, in calculating the minimum thresholds, OfS has made some downwards adjustments to take account of sector-wide historical (data) context.
• Numerical thresholds are calculated for three student outcomes:

1. the proportion of students continuing on a higher education course;
2. the proportion of students completing a higher education qualification; and
3. the proportion of students progressing to managerial or professional employment, or further study (note, it is no longer proposed to use LEO earnings data for this measure).

• Degree classifications metrics have also been dropped from earlier B3 proposals and will continue to form part of Access & Participation Plans.
• A set of indicators will be produced, each with its own threshold. Each indicator will be formed by student outcome + mode + level.

1. 1. As described above, student outcomes are: continuation; completion; progression +
   2. Mode relates to mode of study: full-time; part-time; apprenticeships
   3. Level relates to nine levels of study: first degree; other undergraduate; undergraduate with postgraduate components; postgraduate taught masters; postgraduate research; PGCE; other postgraduate; undergraduate apprenticeships; postgraduate apprenticeships.

• This could mean each provider having up to 48 indicators – and 64 thresholds (there are two completion thresholds) – depending on the range of modes and levels of courses it offers. See figure D1, p95.
• There will additionally be split indicators to disaggregate performance by indicator: Split indicators are: time series, subject, student characteristics, course type, teaching arrangements.
• Split indicators will not have thresholds. This means there will be a single numerical threshold across all subjects and across course types not reflected in the nine levels of study above (e.g. integrated Foundation Years and HTQs).
• This extensive dataset will be published to providers as a dashboard every year. There will also be publicly available information of provider performance. It is not clear if all provider data will be in the public domain.
• For the purposes of the consultation, the accountable officer in all registered providers has been given access to the provider dashboard. Meanwhile, sector level data and a fictional provider dashboard are available on the OfS website.
• Engineering performs well above threshold in OfS sector level data, except for part-time course continuation undergraduate with postgraduate elements (where approximately half is below threshold).
• For those of you in Materials and technology (CAH10-03), completion is predominantly below threshold for undergraduate apprenticeships; part-time first degree, postgraduate taught and other postgraduate; and full-time other postgraduate. There are also pockets of continuation under performance part-time (postgraduate taught and other postgraduate).
• The numerical thresholds will not operate as an automatic mechanism for determining compliance with condition B3. Instead, where a provider’s outcome data is not at or above the numerical thresholds, contextual information will be used by OfS to make a judgement about whether the provider has nevertheless achieved positive outcomes.
• This will take the form of information provided by the institution plus information to which the OfS already has access, including an assessment of benchmarked values.
• Information which may be taken into account includes factors that may explain the reasons for a provider’s historical performance and “actions a provider has taken, or will take, to improve its performance, and the extent to which those actions appear credible and sustainable and capable of improving the provider’s performance”.
• The following contextual information has been deemed irrelevant and will not inform the judgement: the mission and strategy of a provider; funding; entry tariff; sustainability; and reputation.
• Regulatory action will take the form of an Improvement Notice, which will specify indicators requiring improvement, require a provider to take actions and/ or require evidence of sustained improvement on relevant indicators. Regulatory action will not be taken in relation to every provider with an indicator below one or more numerical thresholds each year.
• Providers would be selected by a yet-to-be determined prioritisation exercise. The following ideas are mooted in the consultation: thematic; focus on the most severe breaches by either number or proportion of provider’s students or the distance between indicator value and threshold; focus on those breaches related to particular groups of students, where there strongest statistical confidence in the data; a random approach.
• Regulatory action may be focused on either the indictor which is below the numerical threshold or look across all the indicators which fall below the numerical threshold across the institution. This is a trade-off between more frequent shorter assessments versus in depth assessments across institutions.
• Numerical thresholds will be reviewed every 4 years, in line with but staggered from the proposed TEF assessment cycle.
• The OfS does not intend to “ratchet up” the numerical thresholds over time as continuous improvement is incentivised through TEF – at undergraduate level, that is.
• Eligibility to apply for and retain a TEF award will be determined by a provider’s current and previous compliance with the B conditions, including condition B3, as it may not be considered “appropriate for a provider to obtain a TEF award that signals ‘teaching excellence’ if there is or has been a breach of our minimum requirements for student outcomes”.
• Separate consultations are proposed to consider:
  • Benchmarking approaches (see the current TEF consultation).
  • Student outcomes for courses which are delivered in partnership agreements; these are nonetheless, expressly in scope and registered providers will take responsibility for these.
  • Transnational education (TNE) which is out of scope of these proposals.

Access and participation

OfS’s proposals are based on the assumption that “a provider has a considerable influence over the outcomes students achieve, and that factors beyond its control are not so extensive as to make it impossible to establish a minimum expected level of performance”. They also hold that “meaningful” improvements in access and participation must be built on a minimum level of quality and standards including “taking steps to meet the needs of students from underrepresented groups where those needs are different from other students’ needs”. There is a vast literature on the structural barriers experienced by disadvantaged learners throughout their education. By explicitly rendering universities accountable for addressing the failings of education pre-higher education, OfS will encourage HE institutions to avoid selecting students with lower prior attainment or any other circumstances that might hamper their employment outcomes (such as socioeconomic disadvantage, ethnicity, gender, disability, etc.). The effect will obviously be the opposite, undermining OfS’s legal duty to promote fair access. Ultimately, the approach as it stands is likely to encourage universities to be risk-averse at admissions, effectively raising the bar for any students that don’t look like those who have been successful in the past. This will likely close down opportunities and make higher education ever more the preserve of those whose pre-existing privilege is the most ingrained while marginalising those for whom the transformation potential of higher education is the greatest. Furthermore, although we do not want to endorse a post-pandemic deficit model, it is unrealistic to ignore pre-HE “gaps”. Widely evidenced amplification of disadvantage will distort the creaming effect further. Universities will be incentivised by these proposals not only to recruit those who are less likely to drop out and more likely to “succeed” in the graduate labour market, but they will also face potentially severe punishments for doing anything other than that. With entry tariff cited as irrelevant context – notwithstanding the messaging that KS5 results are not an equitable proxy for potential – creaming is inevitable. Meanwhile, providers who currently make the greatest contribution to the access and participation agenda risk being penalised for their efforts.

Diversity

There will be pressures against diversity of students for the same reasons as above, and this will be felt in engineering where SEND teaching approaches are commonly applied even when there is no diagnosis. Engineering’s efforts to better balance gender (women earn less than men) would be disincentivised. It also seems likely that these proposals would trigger the re-introduction of the traditional maths and physics A level prerequisite, undoing years of work to improve access and diversity, particularly gender diversity in engineering which is essential to address the well documented skills gaps. With HE planners able to pore over a host of split indicators, the volume of data required to hunt the outliers to improve progression rates – together with the complexity of internal analysis of split indicators by other split indicators (i.e. subject by sex and domicile) – could reasonably be such that the resource to understand the student factors affecting underperformance is greater than the benefit. Why bother when the known ‘problem areas’ (i.e. courses in the arts and any students that don’t look like those who have been successful in the past) can simply be cut off at source? Even without such a sweeping approach, without the ability of adequate targeting, providers will have little choice but to use a sledgehammer to crack a walnut; a disaster for diversity.

Teaching and learning

This approach threatens multidisciplinarity and innovation (if-it-ain’t-broke-don’t-you-dare-try-to-fix-it and copy-whatever-works-for-someone-else-or-always-used-to-work). Institutional autonomy allows and encourages innovative approaches, driving quality and adaptation to an ever-changing world (which is particularly important in Engineering). It facilitates innovation and diversity to serve diverse needs and a diverse body of students, serving diverse societal, economic and labour market needs and is often cited as a contributing factor to the UK’s high quality HE sector. OfS should use its position and powers to encourage innovation rather than incentivise homogenous approaches designed to deliver metric-satisfying outcomes.

Outcomes

The sole definition of a ‘positive outcome’ being defined in terms of ‘the proportion of students progressing to managerial or professional employment, or further study’ fails to address the concerns raised by EPC and others that the purpose of higher education is broader than employment. As previous attempts to define “graduateness” have proved. By recognising only the ‘exchange value of HE’ – that is the development of skills and knowledge in order to pass the required assessments and gain a ‘qualification’ that will allow the individual to access career choices that are not possible without this qualification – the OfS reject the other aims of HE including:

• use value: the development of skills and knowledge for use in employment and everyday life;
• self- actualisation: the benefits for mental and spiritual wellbeing given to an individual when they are able to reach their full potential;
• education of the next generation: by educating one generation, we are providing the generations to come with a support-network better able to nurture their educational needs. The critical role played by the family in supporting education has never been more clear as during the pandemic where parents and guardians have been expected to home-school.

This leads to a distortion of education where “teaching to the test” becomes the norm; this also leads students to view their educational experience as transactional rather than the rich transcendent experience that it should be. This generates an employee pool which does not reflect the requirements of business; life-long learners driven to deliver their very best every day.

Context

Employment outcomes are only indirectly related to factors within the universities’ control such as the quality of teaching and support. The outcomes baselines do not adequately consider that these measures are highly influenced by region, by industrial sector and by imbalances in recruitment practices (gender, socio-economic background, etc.). Not to mention wider economic conditions. Outcomes will always be the product of the health of the economy in the context of regional, national and global market forces that are far beyond the control of the sector. We note that factors outside a provider’s control inform the historical performance aspect of the context submission, but with recession, pandemic and recovery accounting for most of the past decade and a half and accepting that employment measures for the next few years may be wildly unrepresentative of actual standards or performance over time, this surely undermines the whole process anyway?

Regional development and levelling up

The value of higher education in improving social mobility, developing local economies, and “levelling up” – especially post-pandemic – is in addressing the precise inequalities that these proposals ignore. Under these proposals, institutions where graduates supply workplaces in disadvantaged parts of the country will be in the bottom third of the table for employment and be at risk of being fined. It’s been pointed out that the best thing a university like Sunderland could do is relocate to London.

The approach would benefit from a regional appraisal and nuance to prevent local brain drains and enforced geographic mobility. Local retention / employment is not only a positive outcome for graduates, but also a really positive choice for the Government’s goal of levelling up regions by creating high-skilled employment in disadvantaged areas. Given regional variations in the labour market, the goal of regional development through educating and upskilling the local workforce is inconsistent with the desirable outcomes as set out in the strategy as it stands. It should not fall to universities to resource consultants to outline regional priorities to a government regulator.

Standards

Universities will be disincentivised to let anyone fail, leave or even repeat a year. Instead, everyone will pass (grade inflation) and students who have lost enthusiasm for their course will be conveyor belted through rather than allowed to drop out.

Engineering is heavily regulated; in any accredited engineering course there is already a very clearly defined set of standards, governed by the Engineering Council and assured by Professional Engineering Institutions. Indeed, PSRBs already set standards in terms of learning outcomes in engineering (and many other professional higher education courses). Accreditation standards are high; but if meeting accreditation standards threatens performance metrics, universities may opt not to be accredited rather than risk letting students fail or drop out. This is a threat to standards.

We would suggest prioritising subjects where there are no PSRBs.

High-cost courses

While expensive, universities are likely to see that engineering has a positive effect on their indicators overall, which could be good news for engineering. However, these proposals are clearly a threat to the arts. How will universities be able to afford to run the more costly engineering courses without cross subsidisation?

Non-standard provision

These proposals may deter future partnerships, limit student choice, and disproportionately impact upon small, specialist providers. For example, data for student cohorts of fewer than 23 will be suppressed. Typically, the proposed statistical uncertainty approach in this context relates to the size of the student population, where smaller numbers can lead to greater uncertainty.