Opportunities to influence: Subject level TEF

The Department for Education is seeking views on the proposed design of the subject-level Teaching Excellence and Student Outcomes Framework (TEF). The TEF aims to recognise and reward excellence in teaching, learning and student outcomes in higher education providers in the UK. Following an announcement in May 2016, the Government seeks comments on the best and most proportionate way to undertake TEF assessments at a disciplinary (subject) level. The consultation closes on 21 May 2018.

The EPC fully supports the key rationale behind subject-level TEF; to provide potential students with timely and relevant information to help inform their decisions. However, the research available clearly shows that this type of information tends to be used over-simplistically as an indicator of quality or a shortcut to ‘good’, ‘okay’ and ‘bad. and does not directly support meaningful choice. The EPC strongly believes that the subject-level TEF must consciously avoid the trap of heuristic choice. The less granular the subject-level TEF data is, the less meaningful it will be to the point of, in practice, misleading potential students.

Members of the EPC Executive have worked closely with the Royal Academy of Engineering, the Engineering Council, and other members of E4E to draft a joint and detailed response to this important consultation. The key points of our draft joint response are summarised below. These are accompanied by areas we have identified to which our members may wish to contribute based on your experiences on the ground. We would like the EPC to be able to additionally submit member contexts to demonstrate the impact and/or unintended consequences of the proposals upon which the consultation is based. Please link to the appropriate feedback form through the drop-down menus below. To reduce the burden on our members, you can chose to feedback on just one element, or multiple sections, of the consultation as you prefer (there is one feedback form per drop-down). Feedback by 9th May please.

The subject level TEF proposes to define subjects using thirty-five subjects which groups Engineering with Computing and Technology

The EPC will respond though E4E that:

  • Any system of higher education (HE) subject classification will involve a degree of arbitrariness and thus limitations.
  • It is important any parameters used for subjects do no more than is strictly necessary to aggregate different subjects together for administrative purposes.
  • The system used for the purpose of the subject-level TEF should be one already in use (the proposed CAH2 will suffice).
  • We believe the underpinning principle of any system must be for the greatest degree of granularity that is reasonably possible without losing comparative impact.

Based on your experiences on the ground, the EPC would like to offer member university scenarios to demonstrate the impact and/or unintended consequences of the following:

  • The ‘subject’ of engineering alone covers a wide range of diverse disciplines and subdisciplines.
  • Although this is a reasonable fit, engineering will be impacted by the other two subjects (and vice versa).
  • Potential students may choose to study a bad computer science course, say, on the basis of this being in a wider grouping that drags up its score and conceals the weaknesses of the individual course.
  • If providers are allowed to switch subjects between groups, the obvious temptation is for providers to swap these subjects strategically to hide weaker subjects and create strong-looking (according to TEF metrics) groups.

Further information can be found on P.10 of the consultation document.

Please provide your comments on subject grouping here.

A longer duration and re-application period is proposed in subject level TEF

The EPC will respond though E4E that:

  • There is a balance between the data being relevant (timeliness is a key part of this) and the practicalities and burden of data collection.
  • The proposed extension to five or six years between reassessments is simply too great for the awarded TEF rating to be meaningful as it may reflect an assessment made seven years previously (taking into account the time between the rating being awarded and publication).
  • The HE sector is moving away from annual data collection towards in-year collections as recommended by the Higher Education Data & Information Improvement Programme (HEDIIP). The data collection that will be necessary under subject-level TEF should not act contrary to this development.

Based on your experiences on the ground, the EPC would like to offer member university scenarios to demonstrate the impact and/or unintended consequences of the following:

  • Five-year Engineering accreditation cycles which renew, say the year after a five or six-year TEF cycle, could potentially equate to nearly a decade gap before a particular course could be fully updated in terms of accreditation and TEF rating. This is particularly damaging given the rapid pace of change in engineering technologies.
  • A longer duration and re-application period could mean in practical terms there would be no way for potential students to know that the data used for the rating reflected an issue since addressed.Conversely, a poor-performing subject (in a grouping with high-performing subjects and so deemed perhaps ‘silver’) could keep running ‘silver’ rated courses for five or six years.
  • Alternatively, a subject could have reached a gold standard due to the efforts and achievement of a particular academic or academic team who then leave that provider, leaving the provider with five years of a gold rating.
  • Some providers may reapply as frequently as possible in order to obtain a higher rating: either if they believe their current rating does not properly reflect the quality of particular subjects, especially if there may have been material changes to the courses in question OR because they are engaging in game-playing or manipulation where they reapply as frequently as possible in order to obtain a higher rating.

Further information can be found on Pp.11-12 of the consultation document.

Please provide your comments on award duration here

The subject level TEF could potentially retain the existing key elements of the provider level framework (including the 10 TEF criteria, the same suite of metrics, benchmarking, submissions, an independent panel assessment process and the rating system)

The EPC will respond though E4E that:

  • Subject-level TEF must aim to encourage prospective students to explore the context of performance rather than stripping away that context.
  • We strongly advise against use of the ‘gold’, ‘silver’, ‘bronze’ grading system from the provider-level TEF as it is inappropriate.In order to avoid misrepresentation and encouragement of a heuristic misuse of the data in student choice, either the terminology of ‘TEF’ needs to be changed or the intended metrics.
  • We would not wish to see the broadly acknowledged issues with many of the key elements of the provider level TEF framework amplified by carrying them across to the subject-level TEF including that:
    • Student satisfaction is not a proxy for teaching excellence.
    • Continuation is misrepresented as indicative of teaching quality.
    • DLHE is related to outcomes and is not related to teaching.
    • Grade inflation is an issue to be addressed but TEF, particularly at subject-level, is not the appropriate vehicle to do this.
    • Whilst LEO data does represent progress as an employment outcome metric over DLHE, measuring income is inherently crude, valuing industrialist engineers over academics and (early stage) entrepreneurs.
  • Qualified to teach and student engagement metrics may be better measures of teaching excellence.
  • It is critical that the panel assessments are done by engineers with an understanding of both the discipline and HE pedagogies.
  • The subject-level TEF does not purport to, and is not capable of, comparing subjects in the latter sense, but it is highly unlikely that potential students and the wider public will realise this.
  • It is common for students to move between BEng and MEng programmes depending upon their performance and personal circumstances. We are concerned that continuation metrics at subject level may misrepresent the general picture for students on engineering courses at each institution. Furthermore, we are generally unclear on how integrated masters degrees fit into this scheme (over half of integrated masters students are engineers or scientists), a particular issue for engineering where a postgraduate qualification is still often demanded for working engineers. It is also not clear to us where higher apprenticeships (that range from level 4 to level 7) and degree apprenticeships (available at levels 6 and 7, and even 8) sit in this framework as their inclusion could have a significant impact on the data and the split between full-time and part-time courses.

Based on your experiences on the ground, the EPC would like to offer member university scenarios to demonstrate the impact and/or unintended consequences of the following:

  • The learning environment continuation metric counts full-time students between their first and second year of study regardless of subject or provider. We are unclear about how this metric would capture students switching courses in Engineering, which tends towards student switching: not least because many universities run engineering programmes with commonality in the early years of study so that students can select or switch specialisms in subsequent years of study.
  • Student employment outcomes are largely related to students’ choice of course (Engineering performs well in DLHE), their level of prior attainment, and wider economic factors: providers in low employment regions are always going to score behind regions such as London regardless of teacher quality or student experience, particularly in industry.
  • Engineering currently performs well in LEO data, a reflection of the higher salaries engineering graduates can expect to command. However, given the longitudinal nature of LEO data (and the proposed five- or six-year gap between TEF gradings) possible prospective students will be looking at data that relates to a course that was taught more than a decade previously. This is inherently problematic for engineering, where the pace of change is driven by both technological and industrial revolution, meaning a decade is likely to involve a complete change in industry, labour market and subject currency.
  • LEO data, as regards engineering students, is also likely to have significant gaps. In particular, UK graduates who secure work abroad will not be included which is likely to have a pronounced impact on engineering which has a particularly mobile workforce. This is true in both industry and academia and across all skill levels. Engineering companies tend to recruit from a global talent pool; UK engineers are in high demand internationally and can readily secure employment in other countries.
  • Benchmarking does not fully take into account geographical patterns of economic deprivation and social disadvantage. A further layer of complexity is added when the engineering (and the many courses that encompasses) is placed alongside computing courses and technology courses in its CAH2 aggregation and the different intakes on those courses.
  • Within the current benchmarking methodology, the male-dominated makeup of HE engineering courses effectively means those providers that do succeed in attracting more women in to study engineering will not be made sufficiently visible and will be disproportionately disadvantaged under LEO data (owing to differences in average male and female earnings).
  • Providers have very different missions and approaches (for example, they may choose to focus and excel on access issues, PhDs, regional or industrial engagement) and that they should be measured against that in a way that requires continuous improvement and genuinely stretching targets rather than allow coasting. These dimensions could form part of a more inclusive framework that would explicitly acknowledge the very different focuses and strengths of higher education providers rather than trying to force them all into one fairly rigid model (the TEF).

Further information can be found on Pp.13-16 and 17-24 of the consultation document.

Please provide your comments on the key elements of subject level TEF here.

PSRB accreditation or recognition could be made a mandatory declaration

The EPC will respond though E4E that:

  • We cannot think why an Engineering provider would not wish to declare their course was PSRB accredited, but we would object to this being used as a metric.
  • Course accreditation is completed on a five-yearly cycle and so there is a time lag factor to be considered, as well as that many good engineering degrees will not have current accreditation.

Based on your experiences on the ground, the EPC would like to offer member university scenarios to demonstrate the impact and/or unintended consequences of the following:

  • There is a risk that universities give incorrect information about the accreditation status of their programmes, an issue that the Engineering Council is aware of in relation to Key Information Sets (partly due to the long lead-time for submitting KIS data, the window for which may close before the accreditation process is started or decisions are confirmed).

Further information can be found on P.25 of the consultation document.

Please provide your comments on reporting of accreditation in subject level TEF here

New measures are proposed

The EPC will respond though E4E that:

  • We agree all of forms of contact and learning should be captured.
  • We are opposed to a measure of teaching intensity.
  • The subject-level TEF should encourage diversity and innovation to ensure continued progress in effective teaching practice.
  • There are many models of student learning not captured in the current framework and further sophistications that attempted to capture all these different ways of learning would serve only to complicate the metric and would still exclude other learning approaches.

Based on your experiences on the ground, the EPC would like to offer member university scenarios to demonstrate the impact and/or unintended consequences of the following:

  • Engineering, in particular, is a practical subject, heavily reliant on practical experience and project work where ‘teachers’ may not be lecturers but industry supervisors, laboratory technicians, researchers, student peer groups working as teams or as individual students, making mistakes and learning from them how to improve.

Further information can be found on Pp.28-31 of the consultation document.

Please provide your comments on proposed new subject level TEF measures here.