This guest blog is by James Noble from NPC—an organisation that supports and promotes the voluntary sector. He recently spoke at the All-Party Parliamentary Group on Penal Affairs on how voluntary organisations in criminal justice can show their impact. This blog is an adaption of that talk.

Voluntary organisations use resources. They use funding from sources such as foundations and commissioners, and they use volunteer time and energy. People in prison themselves give their time, and we rely on the efforts of prison and probation staff to enable the work to happen. So, asking whether the voluntary sector’s work is worthwhile, and makes a difference, is a legitimate question.

The problem is that it’s difficult to answer.

The obvious measure of success is to prevent offending or reoffending. But this is not something that happens right away. In a prison, you might work with someone who’ll not be released for years. While in the community, keeping track of what happens to people is a huge practical problem.

Irrespective of this, voluntary organisations will often say that preventing reoffending is not their primary goal. That given the complexity of peoples’ needs, it’s better to focus on more immediate, but less measurable, things like attitudes and behaviours that could contribute to long-term change - we call these intermediate outcomes. Also, desistance theory is quite clear that it’s very rare for a single intervention to be the thing that turns someone’s life around - rather it is the cumulative effect of lots of things.

All of which makes it difficult to be definitive about what any one organisation is achieving.

I have begun by outlining these challenges so we can be realistic. But we should not be despondent. There are ways for voluntary sector organisations to test the value of their work.

Voluntary organisations need to be clear about their model

Firstly, I think voluntary organisations need to be clear about their model; who they are trying to work with, what they are trying to achieve and how they will go about it. This might sound obvious, but voluntary organisations don’t always describe the reasoning behind their work clearly; which makes it hard to evaluate. ”The ‘theory of change’ approach is one way to go about this. We would also like to see academia and the voluntary sector working together more so that insights from research contribute to the design of programmes.

Once organisations are clear about their model, they should consider which questions they really need to answer. Broadly I think there are three main questions, which should be tackled in sequence.

1. Does the organisation deliver well? Does it reach and engage the right people? Is it a high quality service?
2. Are there early signs of it working? (The intermediate outcomes)
3. Is there impact? Do people commit less crime? Are there other positive results like employment and better health?

Unfortunately, there’s often not enough clarity about which of these questions is important. For Government, the third impact question is critical, but local commissioners often focus on the first question – about delivery. Prisons also want organisations that can work efficiently and reduce pressure on themselves. Hence, when NPC surveyed voluntary organisations about what kind of evaluation support they most wanted, the top answer was ‘clarity about funders' expectations’.

To actually collect data, organisations need to do a range of qualitative and quantitative research. To make this easier we should:

• Work towards standardised approaches
• Help people make better use of digital technology
• Encourage organisations to think about sampling, i.e. not collecting data from everyone.

And we should remember the ultimate aim is as much about learning what is effective as it is about demonstrating impact.

A free analysis of impact

It is also worth mentioning something which is unique in the world, something which NPC lobbied for, and is a tremendous step forward—the Ministry of Justice’s Datalab.

Since 2014, organisations that have worked with more than 60 people in total over the last 10 years can get a report on the reoffending rates of the people they’ve worked with drawn from the Police National Computer alongside the reoffending rate for a statistically matched control group. For example, the Prisoners’ Education Trust (PET), which supports education and provides educational materials in prisons, found that 19% of their service users reoffended, compared to 26% in the control group.

It is important to be clear that this does not prove that their service will always make a difference—other factors may have contributed to the success, and like any research it’s a one-off. But it does give us a valuable piece of information that would otherwise be nearly impossible to get hold of. And which—alongside other research—offers a good assessment of PET’s impact.

The potential of the Datalab will be seen over time—as more analysis is done. NPC and Middlesex University have reviewed the first 100 reports and found that the median effect size is two percentage points; so a typical project might reduce reoffending from 34% to 32%. This might not sound like a lot, but it does show how difficult rehabilitation work is, and when grossed-up, still represents a lot less crime. As the Datalab continues we can start looking at which types of programmes are more effective for different types of people. But sadly, even though it’s free, not enough voluntary organisations have used the Datalab yet and I would encourage you to look into it if you haven’t already.

What next?

In conclusion, demonstrating impact is a thorny issue. But at NPC we think it’s possible for voluntary organisations to take a realistic and effective approach. And through the Data Lab, we have an opportunity to learn a lot more.

There are lots of evaluation resources available on the dedicated Clinks pages. And this introductory document is probably the best place to start. We have also written a more detailed review of the use of evidence by the voluntary sector in criminal justice which is available here.