Introduction To Impact Measurement
If you’re a social entrepreneur, you’ll need think about impact measurement.
Everyone else is going to have an opinion on your idea, on what you can realistically achieve, and how you should be defining success.
They might be right, or they might be totally wrong, and it’s hard to tell at first.
Impact is tricky, because there aren’t always natural numbers attached to your achievements:
· How do you gauge a reduction in social stigma?
· How will you measure the number of plastic bags that have been re-used or saved?
· How should you quantify the meaningfulness of people’s jobs?
There’s also no sense of comparison between different ideas – is it better to plant 1,000 trees or to create jobs for five people with disabilities?
It’s also tough to define success and failure – if you aim to lift 1,000 people out of poverty, but only end up helping 850, have you really failed?
What will your investors/funders say?
Fortunately, there are a few simple ways of setting benchmarks and reviewing your progress.
Logic Models
A logic model (or program logic) is a way of thinking about an idea or challenge from start to end.
For example, you might set a desired outcome, then work backwards to identify what will be required to make it a reality.
e.g. I want to win the respect of my brother in law.
In this case, you determine what outputs you need (winning an arm wrestle), then the activities that are required to achieve those outputs (going to the gym, eating well, learning proper arm wrestling techniques), and finally sourcing the right inputs (gym membership, protein powder, YouTube tutorials).
This turns a vague aspiration into an actionable plan.
It also allows someone else to understand your thought process, and offer advice that will be more effective than what you’ve got in mind
e.g. perhaps being more thoughtful in conversation will get you further than winning a petty competition?
That’s a deliberately silly example, but you’d be surprised at some of the abstract ways entrepreneurs hope to solve tough problems – sometimes they’re great, and sometimes they’re misguided.
Fortunately, this framework helps to clear up bad ideas.
A logic model should be straightforward, like writing out a formula.
It can be used in presentations and proposals, or simply for your own mental clarity.
Logic models describe four things:
Inputs – The raw ingredients that go into your organisation.
It might be people, buildings, your beneficiaries, materials, whatever you start with.
Put another way, these are the things you can’t do without.
Activities – What you then do with your inputs.
It might be running a campaign, mentoring, selling a product, publishing articles, manufacturing, public speaking, etc.
These are the ways in which inputs are turned into outputs.
Outputs – The results that emerge from your activities.
These are tangible, visible and measurable.
E.g. “We sold 500 widgets, spoke to 2,000 people, created 45 jobs, ran 300 hours of mentoring, published four reports that were read by 60,000 people, etc.”
Outcomes – The difference or impact made by your outcomes, like a “So what?”.
E.g. “We spoke to 2,000 people, who then donated…”
or “We ran 300 hours of mentoring, and those mentees have since…”
I use this process in a lot of my work.
It’s great for reverse-engineering training sessions (In order to arrive at…we need to end the day with…so by lunch time we need to have…which means I need to build handouts for…)
It’s also useful for evaluating various business models, by understanding how our investment into a new venture creates jobs, which lead to higher incomes, which then translate into improved health and education.
If it’s a good idea, it will be apparent in the logic model.
Outcomes vs Outputs
Outputs are easy to identify, but outcomes are what really create impact.
If you reach a lot of people but don’t change their behaviour, you haven’t achieved much at all.
Vice versa, if you only reach a few people but can change their lives for the better, then you have done something remarkable.
It may well be that the outcomes of your work aren’t visible for several months or years.
If that’s the case, it’s worth articulating from the start how you’ll measure success.
Storytelling
The program logic framework is really useful for telling your social story.
Starting with your inputs, you set the scene for your audience and explain the problem. Next you describe what work gets done, what comes out of that work, then finally talk about the long term impact and results and why they matter.
Your audience will appreciate it.
The temptation is to talk just about impact, without explaining how it’s achieved.
By describing your inputs and activities, the audience understands what business you’re in, and how your results are possible.
Cards On The Table
The additional benefit is that you’ve exposed your (previously unspoken) assumptions.
Like with the Business Model Canvas, this is a good way of addressing weaknesses within an idea.
If an assumption is accurate and valid, it will withstand basic scrutiny.
If an assumption is faulty, then we want to highlight and correct it straight away.
Your social enterprise idea most likely has a flawed assumption buried within it.
Would you rather have it come to light now, or once you’ve committed a lot of time and money?
Deriving Meaning
In order to meaningfully evaluate your impact, it’s worth identifying three things;
Intent – what you set out to do
Measurement – how you’ll quantify what you’ve done
Comparison – how it stacks up with what else it out there
“We’re trying to end homelessness, this year we’ve housed 89 people at a government cost of $X, while others achieved the same impact at a cost of $2X”
“Our aim is to give everyone access to clean water in the developing world, our programs gave 100,000 people access this year, whereas traditional aid would have only helped 35,000”
Again, this takes courage – it may mean admitting that your program is not as effective as you’d hoped.
Fortunately, this honesty allows you to address weaknesses in the model, and rebuild a stronger version to trial in the future.
Context Is Everything
Imagine a fund manager achieves a 6% return on their investments for the year.
Is that good or bad?
It depends on the context.
What did they set out to do?
How much risk did they take?
How did the rest of the market perform?
If the fund manager put their money into 6% p.a. term deposits right before the market crashed, they’d be praised for their intelligence.
If the market then goes up 11%, they’re condemned for their ignorance.
If they set out to buy safe stocks and returned 6%, that’s a good outcome.
If they took big risks and then only returned 6%, customers will take their money elsewhere.
That’s why for our social enterprise, we need to state our intent upfront, rather than shooting the arrow and painting the target around it.
At The Difference Incubator, we call this quality “the willingness to be held accountable for social and financial performance”.
You’ve got to put your name on your work, and declare what success looks like at the start of the project.
Measurement Tools
There are a tonne of different metrics out there, the best collection I’ve seen are from a set called IRIS – 400+ aspects that a company can measure.
The wrong metrics tell the wrong story, so it’s important to only focus on the factors that tell us something meaningful about our performance.
A good metric needs a baseline – like the “before” photo on an infomercial.
Without a proper baseline, it’s hard to take credit for any future improvements.
That’s why comparison is crucial.
Our brains can’t make much sense of an isolated number, but we’re good at making comparisons.
For example, if you told me that Victoria’s youth unemployment rate was at 8.5%, I would have no reaction.
I’d then ask about the rates in other states, other countries, and what it has been in the past 20 years.
Those figures would then enable me to feel outraged or pleased, depending on which way the trend is going.
It’s worth also measuring the circumstances of similar people who you didn’t impact.
Academics and researchers know that for an experiment to be valid, we need to have a control group (e.g. some people are given a new treatment, some are given a placebo, we compare the results), so that we can see what outcomes would have been achieved naturally.
What should I do now?
The most important thing is that we’re honest with ourselves.
By being clear about what we’re aiming for, we have a “guiding star” to follow when things get messy.
This should probably be one metric, maybe two.
When you’re talking to investors/donors/partners, ask them about how they measure success – it might change how you describe your results in the future.
Then grab your team and map out your logic model, spelling out how you’re going to turn mundane ingredients into something meaningful.
For more on impact measurement, there’s a good collection of wisdom on The Guardian and a huge collection of measurement tools at IRIS and B Analytics
For incorporating impact into a business model, check out Where Does Impact Fit?
You also might enjoy learning more about the Centre For Social Impact or the Social Impact Measurement Network
