PERU is working with TASO to develop guidance on impact evaluation with small cohorts. We have undertake an extensive review of so-called ‘small n’ impact evaluation methodologies and then developed guidance on using these methodologies. The guidance is published on the TASO website and we are now working with TASO to support a number of institutions who are undertaking impact evaluations using the methodologies. Learning from these evaluations will be used to further refine the guidance.

What is the need?

At the heart of impact evaluation is the requirement to link cause and effect, to explain how and why activities lead to changes in outcomes of interest. While experimental and quasi-experimental evaluation methods can provide the sector with the confidence required to be able to state causal inference, it’s not always possible to use these methods. Small – or specialist – providers face issues when attempting to generate causal impact evidence, and even in larger institutions, existing methods can be difficult to use with smaller cohorts of students, for example, when evaluating highly targeted support. The authors of TASO’s own report The Impact of Interventions for Widening Participation, for example, concluded that although there has been increased focus on robust evaluations across the HE widening participation sector, ‘there are still some gaps in the research base, and the evidence often does not demonstrate causality’.

What are we doing?

We reviewed a number of different impact evaluation methodologies that can be used with small cohorts. Our focus was on methods that support causal inference. We developed a methodology taxonomy and identified common elements of the methodologies including their use of mid-level theory (theories of change) and different understandings of causality. We drafted a detailed guidance document as well as material for the TASO website.

We are now working with TASO to support Higher Education Providers and colleges to use the guidance we have developed to deliver interventions with small cohorts and pilot ‘small n’ evaluation methodologies.

What will be the outcomes?

Our review, including a taxonomy of small n impact evaluation methodologies and guidance on their use will be part of a suite of tools to support HE providers. The insights we gather from the pilots will be used to further refine the guidance over time.

What are the timescales?

The project started in 2021 and will run until the end of 2022.