REF 2021: Seven lessons learned about impact case studies

Lesson 1: Consider research impact from the very outset of your research and plan for it

Returning to our line of thinking about ratings on impact case studies, how does this information about significance and reach affect our goal? We could focus on the accountability targets of the REF and consider research impact from this point of view, but this essentially treats impact like an add-on and does the concept a disservice. Since research impact is the good our research does in the world, surely this is something that should be considered as we develop and conduct out research. Interestingly, the easiest way to have significance and reach in relation to the REF is to embed impact into your research from the outset, through planning for impact, cultivating long-term, mutually beneficial relationships beyond academia, and evaluating progress as research develops. In other words, and perhaps counter intuitively, focusing less on a REF rating for impact and more on the good your research can do for society can improve your REF rating. It’s not that aiming high in the REF is a poor strategy, but if it’s the only strategy, this becomes more difficult.

Lesson 3: Determine potential research users and beneficiaries early, and continually review

Now that we have an idea of what impact is and how to identify it, we can turn to why embedding research impact from the outset is advantageous. Here, specific difficulties that arose during the process of writing impact case studies can help.

By engaging with potential users and beneficiaries, and doing so early, it can develop their understanding of your research and how it might positively affect them and/or their organisation, which can increase the chances of your research being used more meaningfully. One difficulty was knowing who all the research users and beneficiaries of the research were. Some were obvious, other less so. Sometimes, they were not recognised until discussions between a researcher and specific research users indicated beneficial results for certain groups, often as a result of indirect impact. For example, a new methodology for evaluating the effectiveness of youth work was developed by researchers. Its aim was to improve reporting of youth work and its impact, in order that agencies and organisations both within and beyond the sector could understand the importance of the work and ensure accountability. It was obvious that changing practice to employ this new methodology would have an impact on youth workers, who would implement it, and on stakeholders, who would hopefully better understand youth work and its outcomes.

However, as discussions occurred with research users, it became clear that the young people whom the youth workers engaged with were also positively impacted. The conversations central to the evaluation methodology also provided a space for young people to talk about youth work practices in a way that didn’t arise organically or from previous methods of evaluation. This, in turn, improved both youth workers’ practice and their relationships with young people. Thus, while the methodology had one intention, unexpected indirect consequences included improvements in the service provided by youth workers and a benefit for an unexpected group. But, since impact on this group was not anticipated, nothing had been done to hear the voices of young people and understand how the research underpinning changes in practice benefited them. It was therefore much harder to document this impact for the REF Impact Case Study.

Not all research users or beneficiaries will be recognised at the outset of a research project. However, taking time to consider who might use or benefit from the research and reviewing this as implementation develops will facilitate connections that may only otherwise be seen in retrospect. And the earlier they are seen, the easier it is to plan for and measure impact.

Lesson 5: Plan tangible ways research can be mobilised to action

Another lesson linked to this planning for impact relates to how impact may occur. It doesn’t just happen; people have to make it happen. Consideration not only of who beneficiaries might be but of which aspects of the research they might use and what resources and skills will be needed to put it into practice will enhance impact. Things to consider are: how findings will be made available for use (accessibility) and the most suitable language for each group targeted (2); who will be involved in making findings available; how to promote the use of research in practice and what resources are needed for this; and what steps can be undertaken if unexpected problems or new opportunities arise.

There are many ways of creating impact. Here, pathways to impact are important. For this planning, you have to find tangible, realistic ways to disseminate research and help users structure practical ways to implement the findings:

• You may want to create a stakeholder advisory group.
• Briefings, seminars, workshops and conference presentations often provide information about how findings might be applied; information leaflets could accompany these.
• A project website can bring all the relevant information – including academic research – together in one easily accessible place, and it could be used to facilitate feedback and engagement with research users.
• Twitter, Instagram or press releases can be used to update on new developments or events.

These pathways could also be designed in collaboration with stakeholders or project partners. 

Lesson 7: Prioritise gathering and recording evidence of impact throughout the project

This brings us to our final lesson. Now that research users and beneficiaries are known, research has been disseminated and used, and impacts are recognised and evaluated, we must communicate what those impacts are and why they’re important. That is, we must ensure they are demonstrable. And to do this, we must gather evidence of research impact.

But it’s not only because the REF tells us we must. Funding applications are increasingly requiring such evidence of how our research benefits society, with a better chance of securing funding if research impact is embedded from the outset. But arguably even more importantly, planning, tracking, assessing and communicating your research’s impact enables your research to do even more good in the world. It has the potential to enhance practice, improve engagement outside academia, refine future research, and help us meet the needs of stakeholders and the public better. And in order to best communicate our research’s impact, we must record that impact throughout the project.

It’s best when the evidence we collect is robust, transparent and preferably gathered independently. It can be numerical, and sometimes that’s appropriate. But it’s not about the more, the merrier. Numbers can indicate the extent of a change, but not what the change itself is nor why it’s significant. When gathering evidence, consider what kind of evidence best suits your research and its use, as well as what is available or feasible to collect as evidence of change. Also, determine how you’ll gather than information, including the tools or techniques you’ll use, and how and when you’ll record it.

How many of us put off a task until the deadline? Who prioritises their work based on when each task has to be completed? We try not to, but we’re human and it happens. But when you put off recording impact – and if you’re only recording impact for a short time before the REF – the quality of what’s recorded suffers. Quite apart from missed opportunities and ineffective processes that we fail to notice if we don’t record and assess impact throughout, we are reliant on fallible memories and fate. We, and those who use our research, forget: how the research was put into practice, how it impacted them or their organisation, omitting many crucial details and leaving our account patchy and partial. Research users change jobs, and sometimes no one remains at an organisation who can attest to the impact we need to evidence shortly before the REF. Furthermore, researchers and users / beneficiaries who engage in ongoing dialogue about impact from early on will undoubtedly highlight impacts that would otherwise go unrecognised or the implications of which would fail to be fully understood. And this process of embedding research impact early will improve our ability to communicate it, presenting it to those who need to hear it, for the REF and funding applications, yes, but vitally, for the improvement of our world (3).