Categories
Impact measurement

Measuring impact of social innovation

Impact. You know the feeling don’t you – you’ve been working on a brilliant initiative, and then someone turns up and asks you “So – what impact are you making”?

It’s a fair question – indeed, it’s a question we should be asking ourselves. If we are not making a difference, we are wasting our time, aren’t we? In this blog post we look at what we mean by impact, and how we can measure it.

Impact – what is it?

First, let’s be clear what we mean by ‘impact’  –  it’s the powerful and long-lasting effect that something we’re doing has on a situation or on people. So, for example, if we run a programme encouraging women to become entrepreneurs, hopefully some of them will set up successful businesses –  that’ll be our impact.

The question now is:  how do we know if we are achieving what we want?  Well, we are going to have to measure the impact… And this is not always as difficult as it first sounds. Most things can be measured . Check out my brief blog post on We can measure (nearly) anything.

Temple in Bhutan with three monks in the background

The 5 steps to measure impact of social innovation

To measure impact, we are going to need data –  in other words, factsSo, to go back to our example, we could look at what percentage of trainees who set up a business, how the business grows, and how much income they generate.

Importantly, we need TWO measures  – we need the data before our intervention and the data following our intervention –  hopefully, when we compare the two, we will see that there’s been an improvement.  If not, we have wasted out time!

So how can we get our data? This is where the five steps to measure impact of social innovation come in.

Decision tree that describes five steps how to measure impact of social innovation.

Step 1: Dig deep into what we already know

You would be surprised; we often have more data available than we realise. In step 1, we carefully think through what data we are already collecting. For example, we are likely to have administrative or financial data. If we do training, we will also have data on trainees.

Example

We are training young people on digital innovation

Think about it!  We already know how many attend our training and for how long;  we just have to look at the attendance sheet the participants sign every day during our training. This is valuable data

We also know who the people are that attend our training, including their sex and age, just by looking at the trainee profiles filled out by applicants.

We can find out the extent to which participants have acquired new skills by comparing their skills before and after the training

And we may even have an idea of how many of our participants have managed to set up businesses or find employment –  just check the call log and notes from meetings with previous participants who have come back to us to ask for further support.

Step 2: Do some research about others

If we don’t have data, it’s possible that somebody else has useful data. So, before thinking about getting new data ourselves, let’s see if it is already being measured in some way by someone else. That is our step 2.

This will require some research – at least a careful search on the internet and government and non-governmental websites.

This is our chance to be a ‘clever detective’: Consider using big data, national statistical databases and reports, international data repositories, national or international surveys and indices.

Example

A youth organisation is sending a caravan across Morocco to promote the Sustainable Development Goals. We want to measure if people become more aware of the SDGs.  OK, so here’s an easy way: use Google Trends to track how many people search for the term “SDG” over time.

Example

A Ministry runs an awareness campaign to stop sexual harassment. After some research, we find out that HarassMap, a volunteer-based initiative in Egypt, already records reported incidences of sexual harassment. This data can be analysed and used to track high-level impact of the awareness campaign over time.

Step 3: Measure impact it yourself

If we do not have data ourselves – and nobody else has it either – it’s time to put our thinking hats on : We need to measure it ourselves.

Just about every imaginable phenomenon leaves some evidence that it occurred. Let us look for any trails it leaves, consider tagging it or carry out experiments:

a. Can we observe it directly?

For example, we have done some training for unemployed people in Somalia, and this requires us to measure to what extent trainees are successful in producing mobile apps. To do that, we regularly count the number of published apps with at least four stars on Google Play with the keyword “Somalia”.

b. If we can’t observe it directly, can we tag it to start tracking?

For example: 500 young people in Iraq are trained in entrepreneurship and design thinking. Six months after finishing the training, we offer 50 randomly selected trainees an additional day of tutoring with a group of established businesswomen and men. During this tutoring, we ask them to fill out a one-page questionnaire that helps us measure their success and ability to obtain additional loans.

c. If all else fails, can we create an experiment to create the conditions to observe it?

For example: A network of youth organisations support young people in political participation. To measure success, we compare how many young people under 21 are elected to councils in three supported cities compared to three similar councils in the same region that were not supported

To collect data ourselves, we have a full toolbox from Social Sciences available to us. I wrote about this toolbox in another blog post.

Step 4: Use sampling to measure impact

This is my favourite part: Step 4 is about sample surveys to collect data.

Sampling is like magic: We observe just some of the things we are interested in, and from this we can learn something about all things.

Sample surveys can be used for people, things and documents.

Sampling can be done for people (through interviews), things (through observations) and documents (through desk reviews)

And sample surveys can be small, simple and cheap, including only a single observationor one or two questions.

Example

An organisation in Somalia provides 2,000 young people with new skills in digital innovation. We want to know the impact.

Rather than interview all of them, we randomly select 100 young people at the training graduation and ask them to leave an email address. Six months later, we ask them if they have found employment, in what area and how much they earn now.

Then, we ‘extrapolate’.  That is to say: if we find that, for example, 60 of our 100 people we track have found work in the ICT sector and are earning an average of, say, $400 a month, then we can assume the same pattern will be found in all 2,000 trainees  – i.e. that 60% of the 2,000 trainees are working, and that our training has created a total additional monthly income of $480,000.  Multiply that over twelve months, and that’s well over $5 million in a year!  That’s a BIG impact!

Step 5: Estimations for measuring impact

Ok. If nothing has worked so far, we have one last option up our sleeves: estimations. No, I didn’t say ‘make things up’ (that wouldn’t be right) –  but we can get indications of impact by estimating data based on what we know already. Not convinced? Let’s look at an example:

Example

We want to know how many people our Sustainable Development Goals campaign reaches. We want to know how many young people we have reached in a year through our the campaign.

Counting every single participant at every of our 200 events per year would be a nightmare. However, we can take a photo of 15 randomly selected events. We roughly count the number of people on the picture and take an average. Let’s say 50 people on average show up.

Nothing works? Rethink what you do!

Ok. If nothing has worked so far, we may have a problem.

If we cannot measure it at all, we may need to think again about what we are trying to achieve!

Categories
Data collection Measurement

The Age of Data

Data, information and knowledge

Professional Monitoring and Evaluation is based on hard facts: data, information, knowledge and understanding. Let us take a closer look at these concepts:

Hierarchy of data (know what) that can lead to information (know what), knowledge (know how) and understanding (know why)

WHAT IS DATA?

As you will know, data is a collection of objective facts, such as numbers, words, images, measurements, observations or even just descriptions of things. In other words: Data is chunks of raw facts about the state of the world.

For example: crime rates, unemployment statistics, but also handwritten notes of interviews, or a recorded description of an observation.

Data is raw, unorganized and lacks context. To be useful, it needs to be turned into information.

WHAT IS INFORMATION?

Information is data that has meaning and purpose. It can help us understand what is happening.

For example: the noises that I hear are data. The meaning of these noises – for example a running car engine – is information.

WHAT IS KNOWLEDGE?

Information can serve to create knowledge. Knowledge can instruct how to do something.

And finally, knowledge can be turned into understanding that explains why it is happening.

Categories
Measurement

Measuring = reducing uncertainty

A lot of what we do in Monitoring and Evaluation is related to measuring. And that’s the problem: Many of us think of measuring as something precise. But that’s not necessarily the case in M&E. What we need to be is roughly right. Those of us in Monitoring and Evaluation should carefully considering to this advice from John Maynard Keynes, a British economist.

“It is better to be roughly right than precisely wrong.”

Measurements are not always exact – but can be more or less precise. But after any measurement, we know more after measuring than we did before. Even an imprecise measurement will reduce our uncertainty to some degree. And depending on what we need, such an imprecise measure may just be good enough for a specific purpose.

For example: If you want to get yourselves now shoes, it is sufficient to know your shoe size. You don’t need to precisely measure your feet.

In Monitoring and Evaluation, this is how we need to look at measuring: as a set of observations that reduce uncertainty where the result is expressed as a number. In most cases in Monitoring and Evaluation, we are not interested in a scientifically precise measurement. Instead, we aim for information that are sufficiently accurate to tell us what is going on – and to give us an indication how it changes over time.

Bertram Russel, the British mathematician and philosopher, has formulated that well (apart from the gender insensitive ‘man’):

Categories
Measurement

Seven steps to measure results

Infograph with seven steps how to measure results
Source: Thomas Winderl

STEP 1: WHAT DO WE WANT TO KNOW?

What is the information we would like to know in the future? How often do we want to know it?

For example: Our goal is to help 10.000 women increase their income. What we would like to know is a) if this is the case, b) by how much their income has increased, c) how sustainable this increase in income really is after our support ends.

STEP 2: SHOULD IT BE MEASURED? WHY?

Ensure that the measurement will be used; if yes, clarify for what it will be used; estimate how much the information is worth (e.g. by thinking about how much you would pay to get that information).

For example: Data in changes in household income and sustainability will help us to adjust the project and tell us if we are overall successful or not. And: Since we absolutely need to know if what we do works, we would be ready to pay up to 10% of the project funds for this information.

PREPARATION FOR MEASURING A RESULT

STEP 3: DE/REFINE WHAT RESULT YOU WANT TO MEASURE

Check if the result is really formulated on right level (e.g. that it is not an activity or an output that can be delivered with nearly 100% certainty); check if it’s clear who is supposed to change behaviour/performance; check if the result statement is time-bound;

For example: An increase in income is clearly not an output, since it is beyond our control. And: we expect that 10,000 women from low-income households in 2 provinces increase their income. We expect that to happen within 3 years.

STEP 4: DECOMPOSE WHAT SHOULD BE MEASURED

‘Decompose’ any uncertain variable into constituent parts to identify directly observable things that are easier to measure.

For example: Income from non-formal economic activities of 10,000 women before taxes.

STEP 5: CHECK SECONDARY SOURCES

Have others already measured it (or parts of it)?

For example: The Bureau of Statistics collects this data, but only every 5 years – not frequent enough for our purpose.

OK, LET’S MEASURE IT

STEP 6: WHAT LEVEL OF ACCURACY DO WE WANT? WHAT SHOULD THE ERROR BE?

Measure “just enough” (“optimal ignorance”); keep the information value in mind (see step 2).

For example: We would like to know the changes in income with a degree of uncertainty of ca. 15% (meaning + or – 15%.

STEP 7: DETERMINE MEASUREMENT INSTRUMENTS

Can the change be observed? If observing it in total is not feasibly, can you observe a sample of it?

For example: Yes, increased income can be observed by frequently visiting the 10,000 women. However, that might not be feasible. That leaves us with the option of measuring a small, randomly selected sample of these women.

Does it leave a trail? Think like a forensic investigator; if there is no direct trail, does it lead to consequences which leave a trail (think of a proxy measurement)?

For example: Increased income will leave a trail. Expenditure might raise with increasing income; local tax payments might increase; living conditions might raise; saving rates might increase. However, we conclude that none of these measurements provide us with the sufficiently accurate measurement.

If not, can it be ‘tagged’? Tagging involves adding some sort of ‘tracker’ so that it does leave a trail and/or can be observed.

For example: We can provide 5% randomly selected women with an inexpensive mobile phone, a SIM card and some credit after they complete our training. We can then call them or send them an SMS with 3 simple questions on a regular basis to collect the information we want.

If not, can it be forced to occur (e.g. through an experiment)?

If not, the result is probably not sufficiently well defined. Repeat steps 1 to 7.

Categories
Measurement

We can (nearly) measure anything

“We had some great results -but you can’t measure them!”

“Not everything that counts can be counted!”

“You can’t measure everything!”

These are some of the critical arguments we often hear in Monitoring and Evaluation.

The reason for this confusion is the common misconception that we in Monitoring and Evaluation always aim for precise, scientific measurements.

However, our work is about measuring to reduce uncertainty – and not necessary about precise numbers.

Let’s take an example: Can we measure happiness? Yes, we can. It is done all the time. Not precisely, but we can measure approximate levels of happiness and how they change over time. For example, we can ask people regularly how happy they feel on a scale from 1 to 10. Or we can define a set of criteria that we know from research make people happy – a warm, dry place to sleep, food on the table, a sense of self-control over their lives, and so on. Or we can use face recognition software to track over time how often people smile per day.

In fact, the so-called World Happiness Report regularly ranks countries according to their level of happiness. And the Himalaya kingdom of Bhutan sets policies based on a Gross National Happiness index.

In a nutshell: If we can observe a thing in any way at all, we can also measure it.

And we know: What is getting measured, gets done.