STEP 1: WHAT DO WE WANT TO KNOW?
What is the information we would like to know in the future? How often do we want to know it?
For example: Our goal is to help 10.000 women increase their income. What we would like to know is a) if this is the case, b) by how much their income has increased, c) how sustainable this increase in income really is after our support ends.
STEP 2: SHOULD IT BE MEASURED? WHY?
Ensure that the measurement will be used; if yes, clarify for what it will be used; estimate how much the information is worth (e.g. by thinking about how much you would pay to get that information).
For example: Data in changes in household income and sustainability will help us to adjust the project and tell us if we are overall successful or not. And: Since we absolutely need to know if what we do works, we would be ready to pay up to 10% of the project funds for this information.
PREPARATION FOR MEASURING A RESULT
STEP 3: DE/REFINE WHAT RESULT YOU WANT TO MEASURE
Check if the result is really formulated on right level (e.g. that it is not an activity or an output that can be delivered with nearly 100% certainty); check if it’s clear who is supposed to change behaviour/performance; check if the result statement is time-bound;
For example: An increase in income is clearly not an output, since it is beyond our control. And: we expect that 10,000 women from low-income households in 2 provinces increase their income. We expect that to happen within 3 years.
STEP 4: DECOMPOSE WHAT SHOULD BE MEASURED
‘Decompose’ any uncertain variable into constituent parts to identify directly observable things that are easier to measure.
For example: Income from non-formal economic activities of 10,000 women before taxes.
STEP 5: CHECK SECONDARY SOURCES
Have others already measured it (or parts of it)?
For example: The Bureau of Statistics collects this data, but only every 5 years – not frequent enough for our purpose.
OK, LET’S MEASURE IT
STEP 6: WHAT LEVEL OF ACCURACY DO WE WANT? WHAT SHOULD THE ERROR BE?
Measure “just enough” (“optimal ignorance”); keep the information value in mind (see step 2).
For example: We would like to know the changes in income with a degree of uncertainty of ca. 15% (meaning + or – 15%.
STEP 7: DETERMINE MEASUREMENT INSTRUMENTS
Can the change be observed? If observing it in total is not feasibly, can you observe a sample of it?
For example: Yes, increased income can be observed by frequently visiting the 10,000 women. However, that might not be feasible. That leaves us with the option of measuring a small, randomly selected sample of these women.
Does it leave a trail? Think like a forensic investigator; if there is no direct trail, does it lead to consequences which leave a trail (think of a proxy measurement)?
For example: Increased income will leave a trail. Expenditure might raise with increasing income; local tax payments might increase; living conditions might raise; saving rates might increase. However, we conclude that none of these measurements provide us with the sufficiently accurate measurement.
If not, can it be ‘tagged’? Tagging involves adding some sort of ‘tracker’ so that it does leave a trail and/or can be observed.
For example: We can provide 5% randomly selected women with an inexpensive mobile phone, a SIM card and some credit after they complete our training. We can then call them or send them an SMS with 3 simple questions on a regular basis to collect the information we want.
If not, can it be forced to occur (e.g. through an experiment)?
If not, the result is probably not sufficiently well defined. Repeat steps 1 to 7.