Have you measured the impact of your impact measurement?

I was pleasantly surprised by the Third Sector impact conference.

That’s not something you’re really supposed to admit about a conference organised by your own organisation, but there we go: impact measurement encourages honesty.

I went along wondering if it would prove to be one for the wonks – full of technical jargon and impact-speak, incomprehensible to a humble journalist. But instead it was full of good, sensible, productive theories about how to use measurement to help your organisation.

A lot of what I learned can be encapsulated pretty well in one simple solution, put forward by Alison Braybrooks, a consultant who is working with bankers UBS to develop their CSR: if you can’t put into a single sentence what you want to do, you probably don’t really know.

So here’s how to do it, Braybrooks-style.

First, decide what problem you want to solve. Write that in a single sentence. Then, work out what you plan to do to solve it. Write that in a single sentence. Then work out what change you want to have achieved. Write that – you guessed it – in a single sentence.

If you can’t articulate what you want to measure, and measure it, relatively simply, she suggested, you’re doing it wrong.

This feeds into another concept much in evidence – the “theory of change” principle – which says impact measurement should primarily be used not to measure whether you’ve succeeded – although that’s certainly important to know – but to identify what changes you need to make to succeed next time.

Failure is a useful outcome, said philanthropist Nick Jenkins, so long as you know why you failed, and tell everyone else, so they don’t fail in your footsteps.

This leads to yet another idea much in evidence at the conference: don’t work out how to measure impact after you’ve finished, because it will be pretty hard to tell whether you’ve achieved something if you don’t know what you were trying to do.

And if you’re a funder, don’t insist on an impact measurement tool that tells you whether your money has been well spent, but is no use to the organisation you’re funding. Also, don’t insist on spending so much money on impact measurement that the people you’re funding don’t have much cash left to get anything done.

Finally there was another message for funders and specialists, put forward by Caroline Forster of the Social Investment Business, who said that too often these organisations demand a lot of measurement from others, but don’t do enough themselves.

I’m pleased she mentioned it, because when phrased another way, this is actually my favourite question to ask impact measurements specialists: have you measured the impact of your impact measurement?

P.S. Please respond and let me know whether you found this useful. I would like to measure the impact of my impact measurement blog.

  • Graham Hodgkin

    Thank you for this. Excellent article. I have some experience in impact measurement when I set up Deutsche Bank’s Impact Investment Fund but now as CEO of London’s Air Ambulance I want us to start thinking about our impact, not just the number of our patients and their direct outcomes…

    Consider yours an impactful article, for me at least!

  • David Floyd

    For me, despite spending several years as trustee of a charity that had officially signed off on one, ‘Theory of change’ is a concept that I find more confusing every time I hear someone talking about it or see it being written about.

    Is it a description of how you’ll seek to bring about change, a description of how you’ll understand whether change has happened, neither or both?

    • David Ainsworth

      When I came to write about it, I realised I thought I understood it, but had the same dilemma – what is the change? Is it the social change you want to bring about, or the change you make in your organisation when you realise you aren’t having much luck.

      In impact measurement, when people mention “theory of change” what they mean is “you should measure stuff for the benefit of the beneficiary and the service provider, not the funder”. This is obviously sensible, but it doesn’t throw much light on the actual meaning.

      I have decided that to me, at least, following a theory of change means doing this:

      1. Identify a social problem
      2. Decide the intervention you think will solve the problem
      3. Decide on a measure to see whether your intervention works.
      4. Try solving the problem
      5. Use your measurement structure to see whether you’ve succeeded.
      6. Change anything that didn’t work.
      7. Tell everybody what worked and what didn’t.
      8. Go back and have another stab.

      I rather hope that’s what it means to everyone else. If not, perhaps someone could let me know.

      • Peter Desmond

        David. A very helpful summary. Seems the conference had quite an impact on you. Did you use a theory of change to determine how you might apply the learning so the next conference you attend might have an even greater impact on your writing – and therefore your readers?

  • David Smith

    Excellent article and excellent comment from David Ainsworth. Im sure Im not the only one who has been stuck in an impact measurement loop before and will be printing this out and saving it for the future!