April Fool’s Day 2021 marks the
start of a new cycle for UK universities. Yesterday saw the deadline for submissions
to the national census of research effectiveness, now branded as the Research
Excellence Framework or REF. This is the sixth such census dating back to 1992.
This time around the sector was granted a short, Covid-related extension and now
the long wait for feedback begins. That feedback will impact individual careers
and institutional bragging rights as well as being a key mechanism by which billions of pounds of public funding flows into the sector. If you aren’t familiar with the process,
every few years, each academic discipline is asked to report on its best
research in the form of papers, research grant income, doctoral completions and
impact. The last exercise took place in 2014 meaning that this cycle which
closed yesterday covered 7 years for each of these categories.
Alongside some simple facts and
figures (how many people, how much grant income, how many PhD graduates) there
is a lengthy narrative about research strategy, the organisational support and
environment which accounts for 15% of the overall grade. There are then two
other more contentious and subjective elements.
First, are research outputs. REF
2021 is the first time that universities have been compelled to return everyone
that they employ to do research. Up until 2104, universities employed careful
selection processes to find the sweet spot between the highest rated individual
researcher and the financial reward which was made available on a per person
basis. Gaming the system in this way was frowned upon but everyone did it. The
regulatory response was to insist that everyone must be returned and each
individual returned must put forward their best work. Those items (journal
articles, books, reports, etc.) are reviewed by judging panels and rated. Such
judgements are made by leading scholars but remain open to concerns about subjectivity.
After all, how easy is it to classify an individual paper as a primary or
essential point of reference (4 star), an important point of reference (3 star)
or a recognised point of reference (2 star)? In my own discipline of management
research there are stark differences within and across a diverse range of
sub-disciplines; these are further exacerbated by methodological differences
meaning that defining “good” is not unproblematic. Your “essential” reference
point might only be “recognised” to me. This is challenging given that outputs
account for 60% of the overall grade. Whilst a REF panel, consisting of recognised
experts will allocate the eventual score, most universities have spent the last
several years evaluating and revaluating outputs in an attempt to second guess the
nuances around a subtle scoring system. This represents a significant investment
of time fretting about research which has already been published and, in the
vast majority of cases, peer-reviewed. By extension, that time is not being
spent on the doing of new research.
The second subjective element of
REF 2021 is that of impact. Using case studies, each discipline in each university
is asked to present evidence that their research has made a difference in the
world. Perhaps a theoretical model has informed policy or practice such that it
has been adopted widely. This accounts for the remaining 25% of the overall
outcome but again is subject to interpretation and opinion. Referring to my own
discipline of management again, it is reasonably clear to see how ideas, models
and concepts could influence the ways in which people actually manage in
practice. Arguably a less straightforward translation process might face those
in some other disciplines. Even in management impact is a complex, temporal
phenomenon which, with colleagues, I have written about at length (see here).
Over the last three decades, REF
and its predecessor exercises (the Research Assessment Exercise or RAE) has
profoundly changed the behaviour of academics and universities. Peer-reviewed
publication had long been a key activity, testing as it does, the quality of
ideas and contributions. The top rating in REF terms is referred to as four
star and producing a four star paper is arduous. It typically involves multiple
rounds of double or triple blind review and high rejection rates. Success in management
journals rests on demonstrating a novel theoretical contribution often through
recourse to theories which are difficult to follow. Using management as an
example, the kind of four star work which appears in top-rated journals isn’t easy
to read unless you are another well-read and informed academic researcher. Elliot
Green’s excellent blog piece sets out the most heavily cited works in
social science. Most professional management researchers would find this list
familiar and could probably guess a fair number of entries on the list were it
a game show format. Most practicing managers would probably not be familiar
with the entries on this list. Why does that disjuncture matter?
Take the global pandemic as an
example. It has been tragic for each and every family that has lost a loved one
but it has also wrought rapid and radical changes to the world of work. Organisations
in a variety of sectors are rethinking whether they need all that office space.
Employees are rethinking whether they can face all the time and cost associated
with commuting. Given this real world problem, it would seem sensible to do
some research on who might do what over the next few years since such findings
could inform both policy decisions and the choices for individual firms. A
survey, by sector, geography, organisation size, etc. would be very helpful and
indeed many such surveys have been undertaken by me and others. The kind of insight
required to help decide whether to renew your office lease is not however, the
kind of insight required to get published in a top-tier journal. Few policy
makers or business leaders would thank you for extensively theorising the
social processes underpinning the decision they need to make but this is what will
help secure a spot in a prestigious journal.
Taking this issue of mismatched incentives a
little further, let’s consider the similarities between academic and sporting
excellence. Tokyo has finally set the Olympic torch alight. Packed stadia may
not fill our screens this summer but we can expect coverage of crowning
achievements, gold medals and world records. Amidst the sweat, tears and
elation, elite athletes will hope that their victory inspires more participants
to take up the sport. The public health benefits of mass participation in sport
is a laudable aim but only loosely related to the specialist training,
infrastructure and skills required to top the medal table. Similarly in disciplines
such as management, education, nursing or medicine elite, peer-reviewed
research may push back the boundaries of knowledge but those on the ground
delivering services are just as important when it comes to making a positive
difference in the world. Educating managers, teachers, doctors and nurses is
closer to mass participation in sport than to Olympic gold medals. Knowledge doesn’t
always have to be world-leading to be world-changing. Reaching a wide
cross-section of practicing professionals depends on the knowledge concerned
being relevant and digestible. Since these are not the primary criteria by
which elite peer-reviewed publication operates, these are not always the criteria
that incentivise academics.
The basic premise of measuring
research performance, especially in connection to the dispersal of public
monies, is a good one. After all, some means of dispersing £14bn is needed and there seems little appetite for a flat, per capita arrangement though this would be radically simpler. The problem that REF faces is that it is labour
intensive, focuses too heavily on a narrow, specialist form of research
activity and has become institutionalised in ways which are not particularly
helpful. As the panel members for REF2021 gear up to read thousands of pages of
research it would seem timely to pause. Rather than turning the handle of what
will probably be REF 2027, we should really consider a radical rethink of the
purpose, process and unintended consequences of the current system. To not do
so would seem (April) foolish.
