Research Excellence Framework 2028 consultation response

The EPC Research, Innovation and Knowledge Transfer Committee responded to a targeted REF consultation following its earlier input into the key decisions on the high-level design of the next research assessment exercise.

What practical challenges may institutions face in implementing these changes?

The proposal to determine volume through Unit of Assessment (UoA) values for staff in the Dept/Faculty/School HESA staff return will require a consistent mapping between the two, and that this does not change over time. Presumably institutions would still need to select UoA for each individual submitted to HESA? There is the potential for disproportionate impact to disadvantage smaller units through, for example, a large group from a small unit being bought out outside of the census period. There appears to be an inconsistency with the recommendations allowing non-academics/PhD students to be included. Would this be HESA as a minimum plus others?

How might the funding bodies mitigate against these challenges?

Clear definition is essential of how the assignment of expected UoA to groups/depts/individuals within HESA will work in practice. We recommend consideration to allow a mechanism to revise the volume calculation determined in the census period if it is not an accurate picture over the full REF period (due to significant departure or arrival of groups, for example).

What would be the impact of these changes on individual researchers and particularly those with protected characteristics?

None expected, though consistency between UoA and HESA is important, many staff may not know which they are returned in except when it comes to REF.

Submission of outputs:

What impact would these changes have on individual researchers and particularly those with protected characteristics?

Whilst the inclusion of all staff in the submission is a positive development, we have concerns over removing the minimum requirement for 1 output for research staff. There can be adverse career consequences for staff who have no outputs returned to the REF, it risks papers being concentrated from a small number of high-performing individuals, possibly at the expense of those who for good reason may not be as productive in the period (e.g., transitioning to, or at the beginning of, their independent research careers, with caring responsibilities and such like). A good compromise would be to keep the minimum requirement for staff with significant responsibilities for research and not expecting the same for other categories of staff.

The intention of removing the link to individuals is good and could be considered helpful in encouraging ‘practitioners’ as well as researchers on the staff. A genuinely blind assessment process would be difficult to implement but at least the link between the contributing author and the output should not be displayed for the assessors to see. To drive submissions that are representative of the staffing base, we recommend consideration be given to a submission from each UoA, that is not disclosed to panels, describing outputs and attribution across the staff cohort (e.g., a table of staff members and number of papers with their name on).

What impact would these changes have on institutions in preparing output submissions?

There is already very rigorous institutional checking and selection of outputs and it takes considerable time and effort in choosing between similarly excellent papers in sometimes widely different fields. This will be exacerbated by the removal of a fixed number per individual, checking of staff eligibility (contracts, dates), and the need to monitor leavers’ outputs – there will be considerably more work required in selecting papers under the proposed changes.

Inclusion of disciplinary-level evidence statements adds additional burden to institutions. There is potential for this to have a negative impact on submissions with a small staff cohort.

What would be appropriate indicators of a demonstrable and substantive link to the submitting institution?

A 0.2 FTE for six months contract is a very low bar and does not support the development of a progressive research environment/culture over a sustained period, with clear potential for “buy in” game playing leading to submissions being non-representative of the submitting unit/institution. We recognise it goes against the intention of breaking the link to individuals but we recommend consideration be given to implementing a limit to the number of outputs submitted for obviously ‘seconded’ staff members – perhaps proportional to time/FTE for individuals below (say) 0.4 FTE and 1year?

What may be the unintended consequences of allowing the submission of outputs produced by those on non-academic or teaching-only contracts?

This introduces a whole new level of game playing. It completely negates the idea of using HESA data to include all ‘research staff’. Staff employed on teaching-only contracts could be encouraged/pressured to do research despite heavy teaching loads.

It will likely impact or see an impact from those institutes who have created new job families that are not Teaching & Research or Research only. Those staff in the Research Engineer job family that exists in some institutes are excluded from outputs but do contribute to impact in a major manner. Currently there is an advantage to having such a job family focused on impact. However, including them will change the nature of their roles to include publications, reducing their time focussing on impact, whilst simultaneously increasing the number of impact case studies required. On the Teaching side, where much of the changes might be seen, we may see more pedagogic research evident in engineering – virtually none in 2021 (if any). Some staff do change job family as they near retirement, we might see inclusion of more of these types of staff, and also very early career graduate teaching assistants, who are not yet at lecturer level, but developing as academics.

Should outputs sole-authored by postgraduate research students be eligible for submission? If so, should this

include PhD theses?

We do not support the inclusion of sole-authored outputs, including PhD theses, by postgraduate research students. PhDs are already included in REF separately and carry significant weight. PhD theses are in most cases not peer-reviewed. Institutional and REF panel output assessors cannot be expected to review theses, which are often 150+ pages. In engineering, most PhD projects are collaborations and PhD theses arising from the fostering of good future researchers generally result in peer-reviewed papers which include supervisors as co-authors. These peer-reviewed papers can already be included as outputs. The additional complication of submitting both a thesis and research papers based on the same work risks double counting.

Do the proposed arrangements for co-authored outputs strike the right balance between supporting

collaboration and ensuring that assessment focuses on the work of the unit?

In reality we do not think this will be a significant issue. We would be surprised if a paper were able to score highly in more than two, perhaps three disciplines. Some consideration should perhaps be given to limiting outputs to one discipline per institution only, to avoid double counting at institutional level. We are strongly supportive of maintaining the 100 words statement on impact of each paper after publication – this has worked well previously.

Are there any further considerations around co-authored outputs that need be taken into account?

We would like to see a requirement for outputs to be derived from substantive research undertaken in the UK. Too many papers in REF2021 had all the work done overseas and little or no apparent work by the UK author, or the ‘UK author’ was originally at that overseas institute and moved to UK. We should be measuring the vitality of the UK research environment.

Engagement and impact:

What will be the impact of reducing the minimum number to one?

This might see different decisions at institutes on the number of UoAs they submit to and increase the number to better reflect the granularity of their staff base – so small units do not get absorbed into larger submissions.

What will be the impact of revising the thresholds between case study requirements?

The proportional reduction in impact case study requirements for larger volume submissions has potential for further exaggeration of existing advantage for larger research-intensive institutions. There is a case to look at this very carefully, given a lot of ‘place based’ investments that have changed the playing field around impact type resources (and also environment), e.g. city deals channelling funding through local institutes, devolved authorities placing strategic investment (Scotland, Wales and N Ireland have very different opportunities to England) and we have catapults or parts of catapults that vary in terms of their ownership/legal status. A more levelling up approach that takes account of imbalances in resource better would be welcome.

We do not understand why removal of the requirement of 2* or better underpinning research leading to impact is being proposed. Eligible impact arising from 2* or better research has worked well in the previous two REFs and seemed well justified.

To what extent do you support weighting the impact statement on a sliding scale in proportion to the number of case studies submitted?

We are very concerned about this as it risks setting up rankings of submissions before any assessment occurs – artificially giving more value to some impacts. It also introduces more thresholds for universities to have to try to be the optimum side of. We are not convinced a change is needed, it worked quite well in engineering during REF2021. The ICS should stand on its own and have equal merit/value irrespective of size of submission.

More broadly, there is some concern regarding the weightings change and the increase on environment, which was the most challenging part of REF to assess. Making it very prescriptive and questionnaire-like is not helpful and the validity of a staff survey approach is questionable. There are resource challenges affecting many institutes who do not enjoy the legacy of resource over many decades. Additional weighting to impact would demonstrate more clearly value for tax payer’s money, and more meaningfully tell the story of the research and its purpose.

Unit of Assessment structure and panel appointments:

If the UOA structure is relevant to you/your organisation, please indicate clearly any changes that you propose to the UOA structure and provide your rationale and any evidence to support your proposal.

We are seeing much larger societal challenges being addressed by funding bodies and some of the major journal channels. So, attention to this growing multidisciplinary research is worthwhile. For example, we will see, and have already seen, changes to various institutes to create cross-academy entities that comprise staff from many different HESA/REF UoAs – how do we best organise the UoA structure to assess the significance of such cross-UoA outputs, environment and impact of these on a level playing field, i.e. where different approaches have been taken by different institutes?

Impact of Covid 19:

What is your view on the proposed measures to take into account the impact of the Covid pandemic?

The impacts are quite diverse across the academic disciplines and, even within engineering, this will be a challenge to get right.

What other measures should the funding bodies consider to take into account the impact of the Covid pandemic?

We are not sure there are any warranted so long after pandemic, much of this period will have fallen into a new REF period. Brexit is a more important impact as it has affected volume of research and talent pool coming to the UK.

Subscribe
Notify of
0 Comments
Inline Feedbacks
View all comments
Related articles

IfATE Lifelong Learning Entitlement Consultation

Lifelong Learning Entitlement Consultation The Institute of Apprenticeships and Technical Education is consulting on how the employer stamp of approval...

Consultations

Lifelong Learning Entitlement

By Joshua Mitchell, EPC Data Fellow What is the Lifelong Learning Entitlement The Lifelong Learning Entitlement (LLE) is a scheme...

News

DfE call for evidence on Generative AI in education consultation response

The EPC’s Education, Employability and Skills Committee responded to The Department for Education (DfE) position on generative AI in and...

News

DfE Advanced British Standard consultation

The Department for Education (DfE) is seeking to introduce  a new Baccalaureate-style qualification framework for 16 to 19 year-olds, the...

Consultations
Let us know what you think of our website