Teaching Excellence Framework (TEF)

• Condition B3 will include minimum requirements for ‘positive student outcomes’.
• These will be set using a series of numerical thresholds. These are sector-wide numerical minimum expectations.
• Minimum thresholds will be based on performance in absolute terms. While there is no benchmarking, in calculating the minimum thresholds, OfS has made some downwards adjustments to take account of sector-wide historical (data) context.
• Numerical thresholds are calculated for three student outcomes:

1. the proportion of students continuing on a higher education course;
2. the proportion of students completing a higher education qualification; and
3. the proportion of students progressing to managerial or professional employment, or further study (note, it is no longer proposed to use LEO earnings data for this measure).

• Degree classifications metrics have also been dropped from earlier B3 proposals and will continue to form part of Access & Participation Plans.
• A set of indicators will be produced, each with its own threshold. Each indicator will be formed by student outcome + mode + level.

1. As described above, student outcomes are: continuation; completion; progression +

2. Mode relates to mode of study: full-time; part-time; apprenticeships +

3. Level relates to nine levels of study:first degree; other undergraduate; undergraduate with postgraduate components; postgraduate taught masters; postgraduate research; PGCE; other postgraduate; undergraduate apprenticeships; postgraduate apprenticeships.

• This could mean each provider having up to or 48 indicators – and 64 thresholds (there are two completion thresholds) – depending on the range of modes and levels of courses it offers.  See figure D1, p95.
• There will additionally be split indicators to disaggregate performance by indicator: Split indicators are: time series, subject, student characteristics, course type, teaching arrangements.
• Split indicators will not have thresholds. This means there will be a single numerical threshold across all subjects, and across course types not reflected in the nine levels of study above (e.g. integrated Foundation Years and HTQs).
• This extensive dataset will be published to providers as a dashboard every year. There will also be publicly available information of provider performance. It is not clear if all provider data will be in the public domain.
• For the purposes of the consultation, the accountable officer in all registered providers has been given access to the provider dashboard. Meanwhile, sector level data and a fictional provider dashboard are available on the OfS website.
• Engineering performs well above threshold in OfS sector level data, except for part-time course continuation undergraduate with postgraduate elements (where approximately half is below threshold).
• For those of you in Materials and technology (CAH10-03), completion is predominantly below threshold for undergraduate apprenticeships; part-time first degree, postgraduate taught and other postgraduate; and full-time other postgraduate. There are also pockets of continuation under performance part time (postgraduate taught and other postgraduate).
• The numerical thresholds will not operate as an automatic mechanism for determining compliance with condition B3.
• Instead, where a provider’s outcome data is not at or above the numerical thresholds, contextual information will be used by OfS to make a judgement about whether the provider has nevertheless achieved positive outcomes.
• This will take the form of information provided by the institution plus information to which the OfS already has access, including an assessment of benchmarked values.
• Information which may be taken into account includes factors that may explain the reasons for a provider’s historical performance and “actions a provider has taken, or will take, to improve its performance, and the extent to which those actions appear credible and sustainable and capable of improving the provider’s performance”.
• The following contextual information has been deemed irrelevant and will not inform the judgement: the mission and strategy of a provider; funding; entry tariff; sustainability; and reputation.
• Regulatory action will take the form of an Improvement Notice, which will specify indicators requiring improvement, require a provider to take actions and/ or require evidence of sustained improvement on relevant indicators. Regulatory action will not be taken in relation to every provider with an indicator below one or more numerical thresholds each year.
• Providers would be selected by a yet-to-be determined prioritisation exercise. The following ideas are mooted in the consultation: thematic; focus on the most severe breaches by either number or proportion of provider’s students or the distance between indicator value and threshold; focus on those breaches related to particular groups of students, where there strongest statistical confidence in the data; a random approach.
• Regulatory action may be focused on either the indictor which is below the numerical threshold or look across all the indicators which fall below the numerical threshold across the institution. This is a trade-off between more frequent shorter assessments versus in depth assessments across institutions.
• Numerical thresholds will be reviewed every 4 years, in line with but staggered from the proposed TEF assessment cycle.
• The OfS does not intend to ‘ratchet up’ the numerical thresholds over time as continuous improvement is incentivised through TEF – at undergraduate level, that is.
• Eligibility to apply for and retain a TEF award will be determined by a provider’s current and previous compliance with the B conditions, including condition B3, as it may not be considered “appropriate for a provider to obtain a TEF award that signals ‘teaching excellence’ if there is or has been a breach of our minimum requirements for student outcomes”.
• Separate consultations are proposed to consider:
  1. Benchmarking approaches (see the current TEF consultation).
  2. Student outcomes for courses which are delivered in partnership agreements; these are none-the-less, expressly in scope and registered providers will take responsibility for these.
  3. Transnational education (TNE) which is out of scope of these proposals.

Indicators

NSS Students’ self-reported satisfaction is emphatically not equivalent to teaching quality, but rather a reflection of the gap between the student’s expectation and what they perceive to have been delivered. This can be seen in the long continual variance in self-reported satisfaction rates in these categories in NSS data between students studying STEM subjects, which tend to be higher than among students studying arts subjects. Most students lack an objective point of reference for what good teaching at higher education level looks like and, while student satisfaction is an important indicator to monitor, it is not appropriate to use it as a proxy for teaching quality.

There are also other problems with NSS data, including: (a) the likelihood that satisfaction is less dependent on teaching quality than on demographic patterns (gender, age, socioeconomic background, ethnicity, etc) and on students’ circumstances (commuter students, part-time study, part-time work, etc); (b) the possibility of gaming; and (c) non-comparability owing to factors such as student boycotts.

NSS is subject to a continuing review.

Educational gain

This is a seriously underdeveloped concept intended to recognise the diversity of mission, not to be confused with learning gain, HEFCE work on which all but disappeared under OfS. Under the proposals providers will report on the education gain of their students (and, importantly, how this may vary across subjects) with little guidance. See Wonkhe for an interesting blog on this important facet.

Requires improvement

The introduction of a new award category of “requires improvement” (RI). So, what is TEF if not designed to be used heuristically?  That is to say it is used over-simplistically as a shortcut to ‘good’, ‘okay’ and ‘bad.’ Heuristic information discourages students from further consideration of their choice and encourages sub-optimal choices. A greater range of outcomes compared with the previous iterations of the TEF will just shortcut some of those institutions previously at the lower end of the Bronze level to ‘really bad’. Requires improvement can only be extremely damaging unless it is applied very leniently in terms of context.

Given the availability and presentation of the proposed student outcomes data, in Engineering the dashboards will, in almost all cases, provide the confidence students need, without the need for TEF. Research shows that students chose subjects, not universities, first.

Qualitative statements

In the context of RI, these will be essential to pick out institutions adding a great deal on the basis of prior attainment, specialist (and arts) institutions, those delivering local education and regional levelling up. However, this additional weight to providers’ case for their excellence in their submission, will create inequity; smaller universities and specialist providers (those most likely to be RI) will not have the resources for specialist consultants to write these.

Accreditation

In Engineering, accreditation is the baseline, regardless of B3 proposals. There is an established system of accreditation of engineering degrees, and degrees accredited by Professional Engineering Institutions licensed by the Engineering Council are recognised internationally through a number of international accords. The accreditation process focuses on assuring that degrees will deliver to at least a threshold standard of learning outcomes specified by the engineering profession. These learning outcomes are developed and maintained in consultation with employers and other stakeholders. There may be synergies between subject-level TEF and accreditation, although these have distinct and different purposes. At the moment, the two exercises – although they are designed for fundamentally different purposes and should never be conflated – create obfuscation rather than clarity. For example, some Bronze courses may be accredited under the Engineering Council while some courses deemed Gold may not be.  Of the two, accreditation serves its purpose more effectively.

The new four-year TEF cycle means a provider will hold an award for longer and coupled with the accreditation timescales in Engineering, this is likely to mean that anyone using the rating may be looking at information that is constantly partial, out-of-date and potentially conflicting.

Timing

The plan to open the TEF submission window in September 2022 will make for a high-pressure start to the new academic year, which could put added strain on already-stretched staff who are already set to be dealing with the third academic year under the uncertainty of Covid.

The exclusion of TNE (i.e. the Aggregate Offshore Record) and HE modules or credit only courses is not commensurate with the current policy thinking around micro-credentials and the Lifelong Loan Entitlement (LLE). We fear that the overall approach adopted across OfS does not adequately accommodate courses which allow students to do anything other than join a course, stay the duration, and graduate. Given the Government’s intention to expand flexibility throughout the education sector to encourage lifelong learning, modularity and hop-on-hop-off courses, the applicability of the TEF in the light of hop-on-hop-off and emergent HTQs is remiss. This is a game changer for the sector.