The myth of clean data
If you work on impact, you rarely get a neat dataset. Usually it is a pile of emails with attachments. How we work with that, and why we don’t make it look better than it is.
For charities and non-profits, foundations and 1 moreImprovementAccountabilityBelongs to
At the start of a project we ask what data is already there. The answer is hardly ever a database. Sometimes it is a folder of spreadsheets and old annual reports, if it is a folder at all. Often it is a collection of emails with all kinds of attachments. And someone who knows there was a survey once. Nobody quite knows where it is now.
Often the figures are old too. Or they are about the people who filled in a questionnaire, not about those who didn’t. Sometimes the same number has been in every report for years, and nobody remembers where it came from.
We don’t say this to blame anyone. Time is short, and measuring comes after the real work. But it also stays this way because it suits many parties. A report that repeats last year is done quickly. Nobody likes to check a figure that looks good. A grantee would rather not write in a report to a funder that the grant achieved less. And a policy evaluation sometimes relies on figures from years ago, because there is nothing newer.
An example from our own work. How many disposable bottles does it take to make one sustainable water bottle? That figure stayed in use for years, from presentation to website and report. Nobody worked it out again. That only happened when we made a life cycle assessment together with a partner. It calculates the whole chain, from raw material to waste.
So we don’t start with a new system, but with what is there. We look at what is usable. For every figure we write down where it comes from and how certain it is. What is missing, we show. Where we can, we ask the people it is about what changed for them. That often says more than one more figure.
At The Green Village of TU Delft it was twelve and a half years of research data. It was spread out, in all kinds of forms. We brought it together in one place, with the source next to every figure. The gaps are in there too.
An overview like that looks less neat than a dashboard where everything seems to add up. A board, a funder or a minister also sees what is still uncertain. That takes getting used to. But you then account for what is really there, and you see where there is still something to learn.
