Constructing student outcome and experience indicators for use in OfS regulation

• The consultation proposes a continued use of the indicators that currently inform assessments of condition B3 and TEF awards, and of the regulation of access and participation, with some changes:

1. ‘Measures of continuation, completion and progression’ – i.e. student outcomes indicators – are proposed as the continuing basis for assessments of condition B3.
2. TEF assessments will draw on both student outcomes and student experience indicators (measures of continuation, completion, progression, and student experience.
3. Regulation of access and participation will continue to use access and outcomes indicators (‘measures of access, continuation, degree outcomes and progression’)

• The consultation proposes the introduction of a completion measure alongside continuation.
• It proposes to measure progression using data from Graduate Outcomes (for which data exists for two cohorts currently – 2017-18 and 2018-19).
• International students do not feature in the Access and Participation data dashboard student populations, but they do in some of the populations used to assess B3/make TEF awards. The proposal is to exclude them from the calculation of progression measures, with a commitment to review this in future.

While the EPC welcomes consistency of approach, we are concerned that the limitations and constraints of the existing metrics and their collection methods compromise the utility of this approach beyond fitness for purpose.

Of particular concern in engineering:

Value of HE

The use of the Graduate Outcomes dataset supports out-of-date thinking around measuring success through high-achieving entrants going on to earn large sums. It does not account for students’ own views about their post-HE destinations, nor does it capture the distance travelled for students from different backgrounds entering higher education. What’s more, the below target response rate to this survey currently has, together with structural response bias, not been helped by changes to the methodology around telephoning students.

While the use of LEO data has been dropped from the formal metrics, the OfS has shared that it expects LEO data to form part of context statements.

Professional pathways into engineering

We note the limitations of the Graduate Outcomes survey’s Standard Occupational Classification (SOC 1-3) approach. Specialist engineering graduates, in particular, may not progress onto conventional graduate work (for example aid work) yet be considered world-leading.

The 10-year revision schedule is particularly limited for the fast-paced technological discipline of engineering. Its approach to interim study and employment activities also presents as a disadvantage to engineering graduates, where those going on to professional pathways or taught postgraduate programmes, may disproportionately achieve negative interim outcomes.

International

The partial exclusion of international students from metrics – in particular progression – is likely to lead to an incomplete and potentially inconsistent picture, particularly in Engineering, where the proportion of international students is higher than in almost any other discipline.

Foundation years

Foundation year students will not be separated out from first degree numbers. This will discourage institutions from offering this pathway for access in case students do not progress. This may align with the proposal recommend by the Augar review that Foundation years should be defunded, but it is at odds with the same Review’s call for greater flexibility in the delivery of courses, particularly in the form of hop-on-hop-off models and recognition (through interim qualifications) of each level reached.

Lifelong learning

This approach is not fit for purpose in a more modular system of higher education as presented in current Government thinking for the LLE. The consultation does not adequately accommodate courses which allow students to do anything other than join a course, stay the duration, and graduate. Given the Government’s intention to expand flexibility throughout the education sector to encourage lifelong learning, modularity and hop-on-hop-off courses, the applicability of the strategy in the light of hop-on-hop-off and emergent Higher Technical Qualifications (HTQs) is remiss. This is a game changer for the sector. In Engineering, there are already calls for the Institute for Apprenticeships and Technical Education to invite OFSTED to join accreditation visits; making better use of existing quality mechanisms should be more fully considered by OfS.

Student factors

OfS has not evidenced sufficient grasp of how student factors influence progression and outcomes; evidence shows that one of the most significant influences was whether they came from an affluent background with school attainment a stronger predictor of graduate earnings than subject or university choice. BTEC students, for example, have higher non-continuation rates on engineering courses than students with A level Maths and Physics. When they do graduate, they face higher hurdles in gaining employment because they may not have the social capital and may not have had the same extra-curricular opportunities. However, the earnings premium for BTEC students in engineering is greater relatively than for their high-achieving A level counterparts. This complexity is lost in the proposals.

Local Skills Priorities

The proposed progression benchmark draws on ‘geography of employment and earnings’ quintiles but the reality of how higher education institutions help to deliver local skills priorities is much more complex than this. This is really important in engineering.

Proxy measures

The use of proxies (e.g. outcomes for teaching quality, compound indicators for completion) inevitably leads to gaming. Continuation and completion measures are particularly liable to this given that it is proposed to treat the outcomes of students who have transferred to another course or provider as neutral, and also recognise that completion of a lower award than originally intended could still be interpreted as a positive outcome. For engineering, this does mean that MEng to BEng transfers would be a positive outcome.