The Stern Review – a review of the Research Excellence Framework (REF) by Lord Nicholas Stern at the behest of Jo Johnson, Minister of State for Universities and Science – cannot easily be caricatured. Yes, the Review is fairly unambiguous – ‘to deliver quality-related research funding we need a REF’ (p. 7) – but one needs to look closely. As pointed out by SOAS Senior Lecturer Brenna Bhandar, ‘though broadly in favour of it, [the Review] includes some important criticisms’ of the REF. Indeed, it can hardly be cast as yet another piece of government-sponsored policyspeak: according to Bhander it stands in ‘distinct contrast to the government’s recent White Paper, Success as a Knowledge Economy: Teaching Excellence, Social Mobility and Student Choice’. Stern speaks of, for example, academic ‘career choices, progression and morale’ (p. 19). – the academic as potentially adversely affected worker makes a welcome return to HE discourse.
Reflecting this nuance, UCU members in Bristol (and no doubt beyond) have mixed feelings about the Sterne Review. Many welcome much of the Review; more critical comment has largely focused on specific aspects of the Review, notably the question of portability.
In the Review, Stern suggests stopping portability; that is, stopping academics taking their research output (their book or articles) with them if they move to a different university. ‘Outputs should be submitted only by the institution where the output was demonstrably generated’ (p. 21). Stern’s non-portability stance is designed to address REF ‘gaming’ –employment of academics with attractive outputs on short-term contracts, hardly indicative of research-as-process.
Diversity of views amongst Bristol UCU members reflects broader differences about the REF itself. In principle, many retain deep misgivings about the REF. This is, though, tempered by a concern that a reformed REF is the only realistically available option in the current HE marketplace.
This speaks to a wider strategic and tactical dilemma for the REF-sceptical, the UCU’s default position for many years. Should one support another improved REF-type exercise in 2020 – the main conclusion of the Review; that is, reform the flawed nature of the exercise (coincidence standing in for causation; dubious data such as citation data). Or is something more fundamental required from a UCU perspective? Should REF- scepticism reflect more UCU’s long-standing principled objection to the REF itself?
Many Bristol UCU members are sympathetic to Stern’s proposed reduction in the average number of outputs per fulltime equivalent academic post, and the suggested flexibility over the number of outputs submitted by a given individual (p. 19-20). This could improve the exercise’s sensitivity to the differing circumstances of individuals, and potentially provide a less-stigmatising means of addressing issues around fractional working, caring responsibilities, and disability.
At Bristol there is widespread support for Stern’s proposal of a genuinely inclusive REF exercise that reduces games playing by universities as regards the selection of REF-able staff, along with the concomitant stresses upon individuals, and the conflicts that have resulted in some institutions over this.
The very process of ‘selecting’ individuals for submission has clearly led within the sector as a whole to instances of discrimination. As ever, the goal of eliminating such games-playing and selectivity is challenging, given the tendency of universities to seek the ‘best’ REF submission possible not necessarily the most ‘accurate’ or ‘holistic’. Concerns around selection centre on whether certain staff will be moved on to teaching-only contracts as the census date approaches, moving them out out the REF’s beady eye, given the Review’s suggestion that all research-active, research-employed staff would be entered into the REF (p. 19).
Stern recognises difficulties in assessing truly inter-disciplinary work: disciplinary ‘silos’ are replicated by the REF’s Unit of Assessment (UoA) panel structure [webpage] for instance (p. 15). Repeated iterations of the REF, its research activity predecessors such as the Research Assessment Exercise and Research Selectivity Exercise have said similar things, and this problem clearly remains.
At this point, it would make more sense to see this as a fundamental shortfall of a UoA-based REF-style exercise, rather than as something that will – finally! – be solved.
Concentration of Funding
UCU as a union has long noted and criticised the ever increasing concentration of funding in a small number of institutions. In the case of Bristol UCU members, it needs acknowledging that the University of Bristol is an institution that benefits from this, and members here are not unaware of that.
As a matter of public policy, however, many Bristol UCU members would continue to support a more pluralist and equitable approach to allocating funding. Strikingly, perhaps because it does not obviously fall within his terms of references, Stern says little about how funding should be related to scoring, except in the very interesting Appendix c, point 13 on international comparisons (p. 49). Point 13 makes clear just how unusual the UK is in the degree of concentration (…one of the few countries…), and shows how this flows from an approach whereby funding rises exponentially not linearly as scoring goes upward.
Many UCU members in Bristol remain opposed to the use of impact in a REF-style exercise, with some exceptions. The sections of the Review on impact contain some of the weakest arguments of what is generally – whether one agrees with its various proposals or not – a thoughtful and serious piece of work (pp. 16-17; 21-23). The ‘benefits’ of the impact agenda are asserted rather than established, and the very considerable cost [£55 million!] is glossed over. The claim – no doubt made by some universities – that the costs of impact will diminish in its second iteration is dubious. Many universities – including Bristol – that see themselves as having under-performed in impact in the REF are likely to invest heavily in buyouts, consultancy, and support for impact case studies authors. Competition will be fierce; all universities will seek to provide further evidence – including expensively generated evidence – of the ‘impact’ of research. It seems more likely that costs will rise from their already high levels than that they will fall.
That said, some of the proposed changes on impact have the potential to improve aspects of this component. It makes sense to consider a body of work, rather than to insist on tight connections between particular outputs and impact. Stern is ‘clear that impact case studies should be based on research of demonstrable quality – over a period of time (which could be quite long)’ (p. 49). The scope to consider impact on teaching is welcome (p. 17), though more might have been done here. UCU has made this point previously, but the exclusion of teaching from notions of impact – not to mention ‘public engagement’ – has always been problematic. Given that work is already well in train on potential impact case studies, these changes will of course have costs in terms of refocusing existing projects. Previously, there has been much discussion in universities of the official distinction between ‘impact’ and ‘public engagement’ – the Stern Review renders those (often lengthy) discussions obsolete!
The reduction in the minimum number of impact case studies required is worthwhile (p. 22). Stern recognises a problem UCU has previously identified, namely the tendency of the impact component to lead to increased selectivity in terms of the number of individuals submitted, and attempts to respond to this. It is, though, worth noting that impact was always presented as inclusive: in practice, it had the opposite effect.
As UCU has noted, there has been some discussion of the proposed loss of portability of outputs. Some early career academics in particular have expressed concerns about this. At Bristol there is a variety of views on this question. Concern is partly a principled objection to the idea that an academic’s work somehow ‘belongs’ to the institution, rather than to the individual academic. There is also significant concern amongst ECRs about the potential impact on job prospects, and the uncertainty created by the change in system.
My personal view is that the demise of portability would be a net good. A strong track record in research is bound to recommend a candidate, whether previously published outputs are portable or not. The focus on existing outputs in early career appointments has not made it easy for colleagues in teaching fellowships to compete against those with research fellowships. It is at senior, not early career, level that portability has driven the market, and the yields of this have been captured by the few not the many, and have distorted the REF.
Again, UCU members are not united here. Most members I have spoken to at Bristol – certainly in arts, humanities and social sciences – are sceptical about metrics, but some in science differ.
My own view would be relief that the Review has not gone further down the direction of increased reliance on metrics. Stern only notes ‘[d]ata and metrics are increasingly used by HEIs and funders to manage and assess research’ (p. 29). Certainly, in my own discipline of History, the very citation data itself is unreliable, leaving aside the question of what, if anything, it is supposed to tell us. Most obviously, researchers working in less populated fields can do superb work that generates fewer citations than that of similarly able researchers working in more densely populated parts of a subject. UCU should support the position of many learned societies and other professional bodies of seeking to minimise use of metrics.
Strengths of the Review
Stern recognises a number of the problems with the operation of the REF. It does seem more connected to the lived experience of academic life than some of its predecessors. There is some good analysis of gaming (pp. 28-32), and selectivity (pp. 32-35). The section on encouraging focus on ‘big problems’ (p.14) is suggestive, though that kind of language fits better with the way research proceeds in some areas than in others. It is good that there is some more explicit recognition of the need to treat part-time workers equitably (p.16), and at least some attention to the idea that (certainly in some subject areas) teaching and research are jointly-produced (p. 17).
Methodology and Production of the Report
It is noteworthy that the Stern Review panel –including Vice-Chancellor of Aston University, Professor Julia King, Past President of the Academy of Medical Sciences, Professor Sir John Tooke, Professor Sir Leszek Borysiewicz, immunologist and Vice-Chancellor, Cambridge University, Professor Anton Muscatelli, economist, Principal and Vice-Chancellor, Glasgow University, and Professor Linda Colley – met only 4 times over 6 months. Given its composition, no doubt scheduling difficulties were acute. Even allowing for non face-to-face communication, this does raise questions about the degree of ownership of the Report on the part of the panel as a whole. The composition of the panel is also striking: Oxford and Cambridge are highly atypical UK universities yet a fairly small panel included a representative from each.
It is regrettable that the panel included no early, or mid-career, members: UCU should surely argue that future panels should better represent those who usually bear the greatest burden in combining teaching and research, and are closer to the day to day experience of the majority of UCU members working in higher education.
Lastly, the money spent on external consultants does not appear good value: I see nothing here that civil servants couldn’t have done, as well, or better. I suspect that it is now thought to be the case that the input of highly paid external consultants is essential for ministers to take a review seriously.
While views amongst UCU members at Bristol on the Stern Review differ, a common view – shared by the author – is that – leaving aside fundamental misgivings about the very existence of the REF – the Stern Review is better than some of its predecessors, and that parts of it have the potential to make REF less of a burden on the HE community.
Bristol UCU Vice President