Skip to main content Accessibility information

REF 2021: Seven lessons learned about impact case studies

A jigsaw where all the pieces are question marks and one is lifted to reveal a lightbulb

Dr Philly Iglehart, Research & Knowledge Exchange Impact Officer

Our seven lessons learned from preparing impact case studies for REF 2021 are:

  1. Consider research impact from the very outset of your research and plan for it
  2. Recognise the difference between ‘pathways to impact’ and ‘impact'
  3. Determine potential research users and beneficiaries early, and continually review
  4. Develop mutually beneficial, long-term relationships with research users
  5. Plan tangible ways research can be mobilised to action
  6. Track progress of implementation of research to evaluate impact
  7. Prioritise gathering and recording evidence of impact throughout the project

 

REF Impact Case Study submissions were made earlier this year, and we have some time before having to think about research impact again. Or do we?

Well, that depends on how you view research impact, which can be stated simply as the demonstrable good that researchers can do in the world, consisting of the non-academic benefits that arise, directly or indirectly, from research (1).

In terms of REF accountability, it will be some time before the next submission. And with the overwhelming responsibilities academics have, it would be easy to put this on the back burner. But for the sake of argument, let’s begin by following a line of thinking that emphasises research impact as a means of gaining the highest possible rating in REF impact case studies.

According to the REF guidelines, impact has to have significance and reach. These two terms have often been misunderstood. Significance deals with whether or not the impact of research is meaningful, valuable and/or beneficial. Reach relates to whom the impact affects. It’s generally been assumed that impact has to be big, showy or extensive for it to be rated highly. This is not necessarily the case. But it does have to be valuable or meaningful, and it does have to impact a significant extent and diversity of relevant individuals, systems or organisations within the groups where it’s focused.

For example, let’s say your research involves a new medical intervention to improve outcomes for patients under the age of 5. Based on your research, this intervention begins to be used by 90% of hospitals across England. How should this be judged? First, we have to ask whether the impact of the research, that is, the use of the intervention itself, is valuable: has it produced any beneficial effects for under 5s? Is their health or wellbeing improved by this intervention? If it’s no better than conventional treatment methods, this impact is not particularly valuable or beneficial. The fact that its reach is very good – occurring in 90% of hospitals – then becomes irrelevant. But let’s say there is a significant improvement in the health of under 5s. In this case, the research impact would have both significance and reach.

But what if the intervention helps under 5s, and only under 5s? That is, use of the intervention in older children and adults is not shown to significantly change their health outcomes. Let’s assume the treatment in hospitals covers all patients over the age of 5, and under 5s are treated exclusively in GP surgeries. This new intervention would be used extensively in English hospitals, but it would do little for the group it actually helps because they are not treated in hospitals. The impact in this case would not be significant (since it’s being used with patients it hasn’t been shown to help) and poor in reach (since those treating under 5s are not using it).

However, if this intervention were to be used in these circumstances in just one GP surgery instead of in many hospitals, but it is used consistently with all patients under the age of 5, this research impact would have better significance and better reach. Getting the significance right with small, achievable reach first makes it easier to expand reach as research and implementation develop. Once the value of the research put into practice is demonstrated well in one GP surgery, it makes it easier to extend impact by scaling up to other surgeries, primary care networks, NHS trusts, or perhaps even other countries.

Lesson 1: Consider research impact from the very outset of your research and plan for it

Returning to our line of thinking about ratings on impact case studies, how does this information about significance and reach affect our goal? We could focus on the accountability targets of the REF and consider research impact from this point of view, but this essentially treats impact like an add-on and does the concept a disservice. Since research impact is the good our research does in the world, surely this is something that should be considered as we develop and conduct out research. Interestingly, the easiest way to have significance and reach in relation to the REF is to embed impact into your research from the outset, through planning for impact, cultivating long-term, mutually beneficial relationships beyond academia, and evaluating progress as research develops. In other words, and perhaps counter intuitively, focusing less on a REF rating for impact and more on the good your research can do for society can improve your REF rating. It’s not that aiming high in the REF is a poor strategy, but if it’s the only strategy, this becomes more difficult.

Lesson 2: Recognise the difference between ‘pathways to impact’ and ‘impact’

Having established that research impact is ‘the good research does in the world’, next we need to determine how we can identify what this is. One of the most important points is to keep asking ‘who benefits, and how?’. If this question is repeated, especially as research is developed and we focus on how the research can be implemented, it helps us distinguish between ‘pathways to impact’ and the impact itself. The former – pathways to impact – are the ways in which the research becomes available and usable to those beyond academia, including through various avenues of knowledge exchange. The latter – the impact – is the change that happens.

Some examples of pathways to impact include podcasts, press releases, seminars or workshops, exhibitions, media coverage. Examples of the impact itself include changes that flow from new understanding or knowledge disseminated via these pathways, such as switching to a restorative justice classroom behaviour policy which you’ve learned about in a seminar, or changing your running technique based on one described and praised in a podcast. In these instances, asking ‘who benefits, and how’ provides the answers (assuming the changes are effective). Who benefits from the podcast or seminar? The teacher or runner has new knowledge, but there is no tangible benefit. Who benefits from the new policy or running technique: the students in the class where disruption is caused, the teacher, and the student causing the disruption; the runner. How do they benefit: less classroom disruption, more time to learn, poor behaviour reduced, those behaving poorly feel heard, root causes of poor behaviour recognised and dealt with; more effective technique, faster running times, longer distances covered. Asking these questions shows that no one benefits from the seminar or podcast unless the acquisition of knowledge gained from them is put into practice. That is, the teacher or runner has to do something with their new knowledge for there to be demonstrable impact.

Lesson 3: Determine potential research users and beneficiaries early, and continually review

Now that we have an idea of what impact is and how to identify it, we can turn to why embedding research impact from the outset is advantageous. Here, specific difficulties that arose during the process of writing impact case studies can help.

By engaging with potential users and beneficiaries, and doing so early, it can develop their understanding of your research and how it might positively affect them and/or their organisation, which can increase the chances of your research being used more meaningfully. One difficulty was knowing who all the research users and beneficiaries of the research were. Some were obvious, other less so. Sometimes, they were not recognised until discussions between a researcher and specific research users indicated beneficial results for certain groups, often as a result of indirect impact. For example, a new methodology for evaluating the effectiveness of youth work was developed by researchers. Its aim was to improve reporting of youth work and its impact, in order that agencies and organisations both within and beyond the sector could understand the importance of the work and ensure accountability. It was obvious that changing practice to employ this new methodology would have an impact on youth workers, who would implement it, and on stakeholders, who would hopefully better understand youth work and its outcomes.

However, as discussions occurred with research users, it became clear that the young people whom the youth workers engaged with were also positively impacted. The conversations central to the evaluation methodology also provided a space for young people to talk about youth work practices in a way that didn’t arise organically or from previous methods of evaluation. This, in turn, improved both youth workers’ practice and their relationships with young people. Thus, while the methodology had one intention, unexpected indirect consequences included improvements in the service provided by youth workers and a benefit for an unexpected group. But, since impact on this group was not anticipated, nothing had been done to hear the voices of young people and understand how the research underpinning changes in practice benefited them. It was therefore much harder to document this impact for the REF Impact Case Study.

Not all research users or beneficiaries will be recognised at the outset of a research project. However, taking time to consider who might use or benefit from the research and reviewing this as implementation develops will facilitate connections that may only otherwise be seen in retrospect. And the earlier they are seen, the easier it is to plan for and measure impact.

Lesson 4: Develop mutually beneficial, long-term relationships with research users

For our case studies, research users and beneficiaries had been considered early on to some degree. Relationships with research users were forged, and researchers recognised the significant and substantial changes underpinned by the research made outside academia – by individuals, orgranisations, and within public policy arenas. However, in many cases these relationships could have been developed further for mutual benefit, and they weren’t always relied upon to collect evidence of impact.

Ways to deepen these relationships and make it easier to understand, measure, demonstrate and enhance impact include:

  • Keeping-in-touch messages
  • Reporting to research users on new developments in the research
  • Asking how research has been implemented
  • Obtaining feedback on what happened after implementation
  • Acknowledging the expertise and active roles research users play in making impact happen
  • Having a regular, two-way dialogue on what works well and less well, and how feedback from use of the research might inform the future direction of the research.

When possible, it’s even better to involve users in all stages of the research and facilitate collaborative reflection, from design of project through to implementation. Considering the impact of the research at the outset of a project and engaging with those who are likely to use the research later can help focus research on areas most useful to stakeholders and the public and thereby increase the good the research does in society. It may also take your research in a new but meaningful or more relevant direction.

Lesson 6: Track progress of implementation of research to evaluate impact

Engaging with earlier lessons leads to a greater understanding of the impact you’d like to see your research have in society. As noted earlier, beneficiaries aren’t always easily identified, nor are the impacts. Those using your research have their own goals, tasks and expected outcomes. Further, they may be too close to the practice, or too far removed from the research itself, to be thinking about impact. Or they may recognise impact but not connect it with the research. You’ll need to gather this information. In addition, (co-)generation of new knowledge and knowledge exchange activities should also be tracked to identify how this occurs and by what methods. Evaluating the progress of research as it’s disseminated and used in practice, for consequences both intended and unintended, positive and negative, can help us know how best to engage effectively with different groups, anticipate challenges and avoid unproductive methods of engagement, as well as help us assess the impact of the research. And remember, negative consequences are impact, too, which are significant because they can be used to help prevent mistakes or harm in the future.

The different means used to accomplish this should vary depending on the stakeholders, beneficiaries and context. For example, feedback from teachers on how many implemented a classroom behaviour policy after attending a workshop is appropriate early on, while a questionnaire for teachers about what works and what doesn’t can only be completed effectively once a strategy has been in place for some time. A different approach, providing different evidence of impact, might be to ask students about the effect of the current behaviour policy on their learning or wellbeing, or to examine their grades on assessments, immediately before a new policy is implemented and again six months later.

Such evaluation will hopefully provide evidence of impact, although the research may not be the only factor underlying that impact. Moreover, impact may be indirect, as attested by the aforementioned methodology’s positive effect on young people (lesson 3). It can thus be helpful to evaluate the degree to which research makes a significant contribution towards the impact. This information can then be used to evidence the significance and reach of impacts and the extent to which the research caused them.

Lesson 5: Plan tangible ways research can be mobilised to action

Another lesson linked to this planning for impact relates to how impact may occur. It doesn’t just happen; people have to make it happen. Consideration not only of who beneficiaries might be but of which aspects of the research they might use and what resources and skills will be needed to put it into practice will enhance impact. Things to consider are: how findings will be made available for use (accessibility) and the most suitable language for each group targeted (2); who will be involved in making findings available; how to promote the use of research in practice and what resources are needed for this; and what steps can be undertaken if unexpected problems or new opportunities arise.

There are many ways of creating impact. Here, pathways to impact are important. For this planning, you have to find tangible, realistic ways to disseminate research and help users structure practical ways to implement the findings:

  • You may want to create a stakeholder advisory group.
  • Briefings, seminars, workshops and conference presentations often provide information about how findings might be applied; information leaflets could accompany these.
  • A project website can bring all the relevant information – including academic research – together in one easily accessible place, and it could be used to facilitate feedback and engagement with research users.
  • Twitter, Instagram or press releases can be used to update on new developments or events.

These pathways could also be designed in collaboration with stakeholders or project partners. 

Lesson 7: Prioritise gathering and recording evidence of impact throughout the project

This brings us to our final lesson. Now that research users and beneficiaries are known, research has been disseminated and used, and impacts are recognised and evaluated, we must communicate what those impacts are and why they’re important. That is, we must ensure they are demonstrable. And to do this, we must gather evidence of research impact.

But it’s not only because the REF tells us we must. Funding applications are increasingly requiring such evidence of how our research benefits society, with a better chance of securing funding if research impact is embedded from the outset. But arguably even more importantly, planning, tracking, assessing and communicating your research’s impact enables your research to do even more good in the world. It has the potential to enhance practice, improve engagement outside academia, refine future research, and help us meet the needs of stakeholders and the public better. And in order to best communicate our research’s impact, we must record that impact throughout the project.

It’s best when the evidence we collect is robust, transparent and preferably gathered independently. It can be numerical, and sometimes that’s appropriate. But it’s not about the more, the merrier. Numbers can indicate the extent of a change, but not what the change itself is nor why it’s significant. When gathering evidence, consider what kind of evidence best suits your research and its use, as well as what is available or feasible to collect as evidence of change. Also, determine how you’ll gather than information, including the tools or techniques you’ll use, and how and when you’ll record it.

How many of us put off a task until the deadline? Who prioritises their work based on when each task has to be completed? We try not to, but we’re human and it happens. But when you put off recording impact – and if you’re only recording impact for a short time before the REF – the quality of what’s recorded suffers. Quite apart from missed opportunities and ineffective processes that we fail to notice if we don’t record and assess impact throughout, we are reliant on fallible memories and fate. We, and those who use our research, forget: how the research was put into practice, how it impacted them or their organisation, omitting many crucial details and leaving our account patchy and partial. Research users change jobs, and sometimes no one remains at an organisation who can attest to the impact we need to evidence shortly before the REF. Furthermore, researchers and users / beneficiaries who engage in ongoing dialogue about impact from early on will undoubtedly highlight impacts that would otherwise go unrecognised or the implications of which would fail to be fully understood. And this process of embedding research impact early will improve our ability to communicate it, presenting it to those who need to hear it, for the REF and funding applications, yes, but vitally, for the improvement of our world (3).

Sources:

[1] Mark S. Reed, The Research Impact Handbook, 2nd edn (Huntly: Fast track Impact, 2018), p. 15; UKRI, ‘What Is Impact?’, Economic and Social Research Council, Impact Toolkit, https://esrc.ukri.org/research/impact-toolkit/what-is-impact/.

The handbook is an excellent, detailed work on how to ensure your research has the potential for the best possible impact. I would also recommend Mark Reed’s website Fast Track Impact (https://www.fasttrackimpact.com/), which has a wealth of free information, resources and videos / podcasts, as well as fee-based training available.

The UKRI Impact Toolkit is briefer but also has some very useful information.

[2] A good resource for choice of language for writing a REF Impact Case Study is Bella Reichard et al., ‘Writing Impact Case Studies: A Comparative Study of High-Scoring and Low-Scoring Case Studies from REF2014’, Palgrave Communications, https://doi.org/10.1057/s41599-020-0394-7.

[3] Some further blog resources that may be helpful:

For a somewhat different viewpoint on impact (in HE): https://juliebayley.blog/2021/03/21/next-steps-for-ref-we-need-to-repair-the-sectors-health-first/