Why we like theory of change

by James Noble, NPC [Researcher]

Over the last year, NPC and Clinks have run the Improving Your Evidence project. The aim was to help Clinks members and other organisations improve their use of existing evidence, as well as produce better evidence themselves about the work that they do. We have provided training and one-to-one support for over 200 organisations, as well as  guidance documents that you can find here.

One of the main points we hoped to convey was that you cannot measure the impact of a project unless you can first describe who the beneficiaries are, what you want to achieve, and how you will go about it. In short, you need a theory of change.

But when we began the project there wasn’t enough information on how to do this. And so we also set about producing guidance and a library of examples. We also wanted to demonstrate the full potential of the approach, and we therefore teamed up with TheHorseCourse, an innovative charity that engages young offenders through a horsemanship skills programme.

• First of all, we thought about the needs of the client group for the service—young people in custody not engaging with traditional interventions.
• After this, we agreed the long-term goals —reduced social exclusion and criminal behaviour—followed by the intermediate outcomes (the changes we want to encourage in young people to help them towards these goals).
• The final part of theory of change involved describing exactly how TheHorseCourse achieves this, based on evidence from other research that suggests why the approach might be effective.

But this is only half the picture. To really test whether TheHorseCourse actually  works we needed to collect a range of evidence from the project itself and use a process called ‘contribution analysis’ to assess it.

The final report includes a detailed description of this process and results. We think other organisations could find the approach useful to demonstrate their own impact. Indeed, not all projects are as novel as TheHorsecourse, and you may find it easier to articulate what you do and to find evidence to support it. At the end of the process you should be able to make a really clear argument about your impact.

How was it for me? A practitioner's view

by Harriet Laurie, TheHorseCourse [Practitioner]

The process of building the theory of change with NPC was a bit like one of those TV home makeover shows where it all starts out as a hideous mess, and then the experts come in and before you know it the place looks clean, attractive and rational.

From the start of TheHorseCourse project four years ago, when I led two horses into HMP Portland to try out a new approach, I have been an obsessive data-gatherer, wanting to test and challenge my ideas (frankly I found it hard to believe that they could be good enough!)

As the project developed it became apparent that not only did the horse-stuff work, it seemed to make really big important changes in people who were pretty much written off—the so-called “difficult to reach” high-risk offenders.

I have been very lucky in that several top class academics have taken an interest in the project and written TheHorseCourse up in various papers. However, each study has focused on a specific set of data; there was no one document to draw it all together.

NPC started the theory of change work by asking me to draw a diagram of how the course works, step by step. That was fun and interesting, making me think about which elements are important and which might not matter. It helped that by this time I had trained a number of people to replicate the intervention with their own horses, so I had seen various versions of approximation versus replication, and could learn from that.

Once we had finalised this part, James wanted to test each element against existing theories drawn from criminology, psychology, learning theory and neuroscience… AND double check our own data AND look back at the specific studies on TheHorseCourse. Yikes, it seemed like a daunting task to bring it all into one document! Pretty soon though, it was all set out in sensible tables. Finally, he went through and assessed it all, step by step and wrote up the Contribution Analysis, even checking the results against specific standards like NOMS and NESTA.

Just as the home-owners on those TV shows come in all shiny eyed, I feel very much the same way— surprised and proud and pleased by the results.

For the organisation and for the horsemanship professionals we have trained (currently 10 fully qualified and seven in progress), this is more about sustainability. This document should provide the evidence required to command statutory commissioning and therefore set them up to earn a reasonable living in their own regions doing a difficult and important job.