Categories
Impact measurement

Measuring impact of social innovation

Impact. You know the feeling don’t you – you’ve been working on a brilliant initiative, and then someone turns up and asks you “So – what impact are you making”?

It’s a fair question – indeed, it’s a question we should be asking ourselves. If we are not making a difference, we are wasting our time, aren’t we? In this blog post we look at what we mean by impact, and how we can measure it.

Impact – what is it?

First, let’s be clear what we mean by ‘impact’  –  it’s the powerful and long-lasting effect that something we’re doing has on a situation or on people. So, for example, if we run a programme encouraging women to become entrepreneurs, hopefully some of them will set up successful businesses –  that’ll be our impact.

The question now is:  how do we know if we are achieving what we want?  Well, we are going to have to measure the impact… And this is not always as difficult as it first sounds. Most things can be measured . Check out my brief blog post on We can measure (nearly) anything.

Temple in Bhutan with three monks in the background

The 5 steps to measure impact of social innovation

To measure impact, we are going to need data –  in other words, factsSo, to go back to our example, we could look at what percentage of trainees who set up a business, how the business grows, and how much income they generate.

Importantly, we need TWO measures  – we need the data before our intervention and the data following our intervention –  hopefully, when we compare the two, we will see that there’s been an improvement.  If not, we have wasted out time!

So how can we get our data? This is where the five steps to measure impact of social innovation come in.

Decision tree that describes five steps how to measure impact of social innovation.

Step 1: Dig deep into what we already know

You would be surprised; we often have more data available than we realise. In step 1, we carefully think through what data we are already collecting. For example, we are likely to have administrative or financial data. If we do training, we will also have data on trainees.

Example

We are training young people on digital innovation

Think about it!  We already know how many attend our training and for how long;  we just have to look at the attendance sheet the participants sign every day during our training. This is valuable data

We also know who the people are that attend our training, including their sex and age, just by looking at the trainee profiles filled out by applicants.

We can find out the extent to which participants have acquired new skills by comparing their skills before and after the training

And we may even have an idea of how many of our participants have managed to set up businesses or find employment –  just check the call log and notes from meetings with previous participants who have come back to us to ask for further support.

Step 2: Do some research about others

If we don’t have data, it’s possible that somebody else has useful data. So, before thinking about getting new data ourselves, let’s see if it is already being measured in some way by someone else. That is our step 2.

This will require some research – at least a careful search on the internet and government and non-governmental websites.

This is our chance to be a ‘clever detective’: Consider using big data, national statistical databases and reports, international data repositories, national or international surveys and indices.

Example

A youth organisation is sending a caravan across Morocco to promote the Sustainable Development Goals. We want to measure if people become more aware of the SDGs.  OK, so here’s an easy way: use Google Trends to track how many people search for the term “SDG” over time.

Example

A Ministry runs an awareness campaign to stop sexual harassment. After some research, we find out that HarassMap, a volunteer-based initiative in Egypt, already records reported incidences of sexual harassment. This data can be analysed and used to track high-level impact of the awareness campaign over time.

Step 3: Measure impact it yourself

If we do not have data ourselves – and nobody else has it either – it’s time to put our thinking hats on : We need to measure it ourselves.

Just about every imaginable phenomenon leaves some evidence that it occurred. Let us look for any trails it leaves, consider tagging it or carry out experiments:

a. Can we observe it directly?

For example, we have done some training for unemployed people in Somalia, and this requires us to measure to what extent trainees are successful in producing mobile apps. To do that, we regularly count the number of published apps with at least four stars on Google Play with the keyword “Somalia”.

b. If we can’t observe it directly, can we tag it to start tracking?

For example: 500 young people in Iraq are trained in entrepreneurship and design thinking. Six months after finishing the training, we offer 50 randomly selected trainees an additional day of tutoring with a group of established businesswomen and men. During this tutoring, we ask them to fill out a one-page questionnaire that helps us measure their success and ability to obtain additional loans.

c. If all else fails, can we create an experiment to create the conditions to observe it?

For example: A network of youth organisations support young people in political participation. To measure success, we compare how many young people under 21 are elected to councils in three supported cities compared to three similar councils in the same region that were not supported

To collect data ourselves, we have a full toolbox from Social Sciences available to us. I wrote about this toolbox in another blog post.

Step 4: Use sampling to measure impact

This is my favourite part: Step 4 is about sample surveys to collect data.

Sampling is like magic: We observe just some of the things we are interested in, and from this we can learn something about all things.

Sample surveys can be used for people, things and documents.

Sampling can be done for people (through interviews), things (through observations) and documents (through desk reviews)

And sample surveys can be small, simple and cheap, including only a single observationor one or two questions.

Example

An organisation in Somalia provides 2,000 young people with new skills in digital innovation. We want to know the impact.

Rather than interview all of them, we randomly select 100 young people at the training graduation and ask them to leave an email address. Six months later, we ask them if they have found employment, in what area and how much they earn now.

Then, we ‘extrapolate’.  That is to say: if we find that, for example, 60 of our 100 people we track have found work in the ICT sector and are earning an average of, say, $400 a month, then we can assume the same pattern will be found in all 2,000 trainees  – i.e. that 60% of the 2,000 trainees are working, and that our training has created a total additional monthly income of $480,000.  Multiply that over twelve months, and that’s well over $5 million in a year!  That’s a BIG impact!

Step 5: Estimations for measuring impact

Ok. If nothing has worked so far, we have one last option up our sleeves: estimations. No, I didn’t say ‘make things up’ (that wouldn’t be right) –  but we can get indications of impact by estimating data based on what we know already. Not convinced? Let’s look at an example:

Example

We want to know how many people our Sustainable Development Goals campaign reaches. We want to know how many young people we have reached in a year through our the campaign.

Counting every single participant at every of our 200 events per year would be a nightmare. However, we can take a photo of 15 randomly selected events. We roughly count the number of people on the picture and take an average. Let’s say 50 people on average show up.

Nothing works? Rethink what you do!

Ok. If nothing has worked so far, we may have a problem.

If we cannot measure it at all, we may need to think again about what we are trying to achieve!

Categories
Planning Results Based Management

A Theory of Change

Probably as an inevitable result of the buzz in the past few years, many Theories of Change are often useless.

One reason for that is – is my feeling – that they are typically not developed in a logical sequence: without a thorough problem analysis, with a possible intervention already in mind, just as the opposite side of a result chain, etc.

Here is my suggestion in what sequence we should go about it:

Categories
Planning for M&E Results Based Management

The result chain: a beginner’s guide

Monitoring and Evaluation is about measuring and tracking results. That is why it is important to understand what results are, and how to distinguish between different levels of the results chain.

In general, a “result” is something that happens or exists because of something else that has happened:

  • the results of a football game
  • the final value of a mathematical calculation, or
  • the outcomes of an election.

In development and governance, we use a more nuanced understanding of different types of ‘result’: the so-called result chain.

The result chain distinguished between five logically connected elements:

  • inputs
  • activities
  • outputs
  • outcomes, and
  • impact

A REAL-LIFE EXAMPLE

It’s not complicated – and we use this logic all the time. Let’s take a simple example:

I want to do something about living a healthier life. This is the desired impact. To do that, I want to reduce my weight. This is my planned outcome. To reduce my overall weight, I plan on eating ore vegetables and exercise regularly. These are my planned outputs. Eating healthier requires more conscious shopping habits. More exercise requires me to go running or join a gym. These are some of my planned activities. These activities require some extra time and money. These are the inputs.

ELEMENTS OF A RESULTS CHAIN

Let us look more in detail at a typical results chain of a development or government programme, policy or service. This time we start at the lower end of the result chain and work our way up:

INPUTS

Any programme, policy or service requires resources of some kind. We call these resources inputs.

For example: to put together this course, it took me time to record this video; you need an internet connection and a computer to watch it.

Typically, inputs refer to money, staff time, materials and equipment, transport costs, infrastructure, etc.

ACTIVITY

These inputs are required to carry out a number of activities.

For example: you watch videos of this lesson; you do a quiz; you do some additional reading; you watch the next videos; etc.

So: Activities are actions taken that use inputs to produce higher level results – ‘outputs.’

Typical activities in governance and development are the drafting of a policy document for a ministry, the organization of a media outreach campaign, the training of midwives in a new approach, etc.

OUTPUT

The next level, outputs are typically the result of several completed activities.

For example: after reading this blog post, you have the knowledge to critically review and design your own results chain.

In development and governance, an output is delivered if a group of people or an organization has improved capacitiesabilitiesskillssystemspolicies or if something is created, built or repaired.

Outputs are the direct result of a set of activities and delivered during the implementation of a programme, a policy or a service. Outputs are different from the next level of results – outcomes – because you largely have control over delivering outputs.

That means that if we – and our partners – have the resources and the time to deliver a certain output, we can largely guarantee that the output will be delivered.

That also means, that in turn, we are fully responsible for delivering an output.

Typical outputs are a draft policy document for a ministry, a media outreach campaign, improved skills for, etc.

OUTCOME

Now, this is very different from the next level of results: An outcome is something we hope to achieve as a result of what we do.

In development and governance, an outcome implies that institutions or people do

  1. a) something differently (behavioural change) or
  2. b) something better (change in performance)

The difference of an outcome is that – unlike outputs which we largely control – we can only influence the achievement of an outcome, but it ultimately goes beyond our control.

Typical outcomes are a parliament passing a new law, people changing their behaviour because of a media campaign, midwifes that apply new skills in their daily routine, etc.

Outcomes are typically achieved at the end or even after a programme, policy or service has been implemented.

IMPACT

Finally, outcomes should contribute to a broader impact.

An impact is the long-term effect of programmes, policies or services. It implies a detectable improvement in people’s lives.

Impact typically relates to positive economic, social, cultural, institutional, environmental, technologicalchanges in the lives of a targeted population. An impact is often related to broad national goals or international aspirations like the Sustainable Development Goals. Impact is typically much broader than a programme, policy or service. And: an impact is typically detectable only after a few months or even years.

What about ‘results’?

So: Which of these elements of a result chain do we considered ‘results’?

We usually define ‘results’ – in the context of governance and development – as the top three elements of the result chain: outputs, outcomes and impact. And most importantly, results are not inputs or activities.

Example: The Land of Smokistan

Let us look at a few simplified but typical examples from the world of development and governance:

The country of Smokistan wants to reduce smoking.

The desired impact is that less people die of smoke-related illnesses within five years.

The planned outcome is that fewer people smoke fewer cigarettes.

Smokistan plans to achieve that by increasing taxes on cigarettes and making smoking more difficult in public spaces – these are the outputs.

This requires several activities: for example drafting and passing a new anti-smoking law or funding and implementing smoke-free public zones, etc.

These activities require additional time and money – the inputs.

Example: Domestic violence

Let’s take another example:

The same country aims at fewer people experiencing violence in a domestic setting within three years – the desired impact.

The planned outcome is that women and men openly discuss domestic violence.

The country aims at achieving that by drastically increasing public discussions on domestic violence in social media and traditional media outlets – the outputs.

This requires setting up – for example – a social media unit in a ministry and training influential bloggers and journalists in properly reporting on domestic violence – the activities.

Again, these activities require time and money – the inputs.


Did you like this post? If you are interested, check out my mini-course on Monitoring and Evaluation on Udemy, a learning platform: