Measuring Research Excellence: where does the UK REF go next ?

April Fool’s Day 2021 marks the start of a new cycle for UK universities. Yesterday saw the deadline for submissions to the national census of research effectiveness, now branded as the Research Excellence Framework or REF. This is the sixth such census dating back to 1992. This time around the sector was granted a short, Covid-related extension and now the long wait for feedback begins. That feedback will impact individual careers and institutional bragging rights as well as being a key mechanism by which billions of pounds of public funding flows into the sector. If you aren’t familiar with the process, every few years, each academic discipline is asked to report on its best research in the form of papers, research grant income, doctoral completions and impact. The last exercise took place in 2014 meaning that this cycle which closed yesterday covered 7 years for each of these categories.

Alongside some simple facts and figures (how many people, how much grant income, how many PhD graduates) there is a lengthy narrative about research strategy, the organisational support and environment which accounts for 15% of the overall grade. There are then two other more contentious and subjective elements.

First, are research outputs. REF 2021 is the first time that universities have been compelled to return everyone that they employ to do research. Up until 2104, universities employed careful selection processes to find the sweet spot between the highest rated individual researcher and the financial reward which was made available on a per person basis. Gaming the system in this way was frowned upon but everyone did it. The regulatory response was to insist that everyone must be returned and each individual returned must put forward their best work. Those items (journal articles, books, reports, etc.) are reviewed by judging panels and rated. Such judgements are made by leading scholars but remain open to concerns about subjectivity. After all, how easy is it to classify an individual paper as a primary or essential point of reference (4 star), an important point of reference (3 star) or a recognised point of reference (2 star)? In my own discipline of management research there are stark differences within and across a diverse range of sub-disciplines; these are further exacerbated by methodological differences meaning that defining “good” is not unproblematic. Your “essential” reference point might only be “recognised” to me. This is challenging given that outputs account for 60% of the overall grade. Whilst a REF panel, consisting of recognised experts will allocate the eventual score, most universities have spent the last several years evaluating and revaluating outputs in an attempt to second guess the nuances around a subtle scoring system. This represents a significant investment of time fretting about research which has already been published and, in the vast majority of cases, peer-reviewed. By extension, that time is not being spent on the doing of new research.

The second subjective element of REF 2021 is that of impact. Using case studies, each discipline in each university is asked to present evidence that their research has made a difference in the world. Perhaps a theoretical model has informed policy or practice such that it has been adopted widely. This accounts for the remaining 25% of the overall outcome but again is subject to interpretation and opinion. Referring to my own discipline of management again, it is reasonably clear to see how ideas, models and concepts could influence the ways in which people actually manage in practice. Arguably a less straightforward translation process might face those in some other disciplines. Even in management impact is a complex, temporal phenomenon which, with colleagues, I have written about at length (see here).

Over the last three decades, REF and its predecessor exercises (the Research Assessment Exercise or RAE) has profoundly changed the behaviour of academics and universities. Peer-reviewed publication had long been a key activity, testing as it does, the quality of ideas and contributions. The top rating in REF terms is referred to as four star and producing a four star paper is arduous. It typically involves multiple rounds of double or triple blind review and high rejection rates. Success in management journals rests on demonstrating a novel theoretical contribution often through recourse to theories which are difficult to follow. Using management as an example, the kind of four star work which appears in top-rated journals isn’t easy to read unless you are another well-read and informed academic researcher. Elliot Green’s excellent blog piece sets out the most heavily cited works in social science. Most professional management researchers would find this list familiar and could probably guess a fair number of entries on the list were it a game show format. Most practicing managers would probably not be familiar with the entries on this list. Why does that disjuncture matter?

Take the global pandemic as an example. It has been tragic for each and every family that has lost a loved one but it has also wrought rapid and radical changes to the world of work. Organisations in a variety of sectors are rethinking whether they need all that office space. Employees are rethinking whether they can face all the time and cost associated with commuting. Given this real world problem, it would seem sensible to do some research on who might do what over the next few years since such findings could inform both policy decisions and the choices for individual firms. A survey, by sector, geography, organisation size, etc. would be very helpful and indeed many such surveys have been undertaken by me and others. The kind of insight required to help decide whether to renew your office lease is not however, the kind of insight required to get published in a top-tier journal. Few policy makers or business leaders would thank you for extensively theorising the social processes underpinning the decision they need to make but this is what will help secure a spot in a prestigious journal.

 Taking this issue of mismatched incentives a little further, let’s consider the similarities between academic and sporting excellence. Tokyo has finally set the Olympic torch alight. Packed stadia may not fill our screens this summer but we can expect coverage of crowning achievements, gold medals and world records. Amidst the sweat, tears and elation, elite athletes will hope that their victory inspires more participants to take up the sport. The public health benefits of mass participation in sport is a laudable aim but only loosely related to the specialist training, infrastructure and skills required to top the medal table. Similarly in disciplines such as management, education, nursing or medicine elite, peer-reviewed research may push back the boundaries of knowledge but those on the ground delivering services are just as important when it comes to making a positive difference in the world. Educating managers, teachers, doctors and nurses is closer to mass participation in sport than to Olympic gold medals. Knowledge doesn’t always have to be world-leading to be world-changing. Reaching a wide cross-section of practicing professionals depends on the knowledge concerned being relevant and digestible. Since these are not the primary criteria by which elite peer-reviewed publication operates, these are not always the criteria that incentivise academics.

The basic premise of measuring research performance, especially in connection to the dispersal of public monies, is a good one. After all, some means of dispersing £14bn is needed and there seems little appetite for a flat, per capita arrangement though this would be radically simpler. The problem that REF faces is that it is labour intensive, focuses too heavily on a narrow, specialist form of research activity and has become institutionalised in ways which are not particularly helpful. As the panel members for REF2021 gear up to read thousands of pages of research it would seem timely to pause. Rather than turning the handle of what will probably be REF 2027, we should really consider a radical rethink of the purpose, process and unintended consequences of the current system. To not do so would seem (April) foolish.