Nothing strikes fear in the hearts of nonprofit organizations more than when a funder asks about impact.

Foundations demand that nonprofits measure and understand the impact of their work, but rarely give their grantees the resources to do that kind of evaluation. On top of that, many foundations have a low tolerance for risk or failure (i.e. lack of impact) as well as a reluctance to provide stable, long-term support. This is an especially difficult combination for start-ups, which need the time and patience from funders to show results.

On the flipside, nonprofits argue that evidence of impact is not so black and white – that they can’t know for sure if their work is directly resulting in change because there are too many factors outside their control. Which is often true. We all (nonprofits and funders) have to be comfortable with some level of uncertainty. But shying away from assessment because we can’t prove things is also a cop out for trying to understand whether the work is making a difference.

Which Metrics Matter?

In Media, there is so much being written about metrics these days—from page views and unique visitors to “attention minutes,” conversion rates, and renewal rates. The common thread through all of these articles and blog posts is that there is no consensus on what metrics should and do matter most. Among funders too, there are many unanswered questions about how to evaluate media grants and not enough open dialogue on the subject, which is why Media Impact Funders just released “Funder Perspectives: Assessing Media Investments.” This report, which is well worth reading, shares the results of survey responses from 30 large and small foundations as well as follow-up interviews with a handful of foundation staff (including me) to shed light on how foundations of differing sizes and media strategies are thinking about impact.

In the report, I point to the work of ProPublica and its president Dick Tofel’s leadership on the subject of impact:

“Some funders prefer to allow grantees to develop methods from the bottom up. ‘We’d like to see more of our grantees approach measurement like ProPublica – being clear about their mission and goals, and then assessing their work against what they say they intend to accomplish. Tracking it carefully and honestly, in order to learn and adapt along the way,’ noted Molly de Aguiar, Program Director of Media & Communications at the Geraldine R. Dodge Foundation.”

Allow me to elaborate on that. And full disclosure: ProPublica is a grantee of the Dodge Foundation.

Impact Starts with a Clear Mission

Dick Tofel wrote a paper for the Gates Foundation called “Nonprofit Journalism: Issues Around Impact” that I would recommend to anyone looking for a very straightforward approach to this topic. Measuring impact, he emphasizes, starts with being clear about your mission. If your news organization’s mission is to inform the public, you should first understand where the gaps in information are. Measuring your impact means trying to assess whether your community believes it is more knowledgeable as a result of your reporting. Did your coverage directly help fill their gaps in information and understanding? Was it an important factor? Did it play some role?

Likewise, if you’re an investigative news organization, like ProPublica, and your mission is to expose abuses of power and spur reform, you should be asking, “What does it take to fix this, and who can fix it?” and then track whether or not anything changed as a result of your reporting. The same questions apply: Did your work directly lead to the change you wanted to see? Was it an important factor? Did it play some role?

Why Honest Assessment Matters

Assessment requires a level of honesty that can be difficult to admit to, but is necessary for gaining the loyalty and trust of your community first, and second, the loyalty and trust of your funders. If you’re not purposefully asking your community whether and how your work is making a difference, you’re just guessing about your impact. You’re not learning what you’re doing right and what you’re doing wrong, and therefore you do not have a clear understanding of how to effectively allocate your resources. Tofel’s advice to news organizations is to operate as transparently as possible. “Tell the world what kind of change you think you are having. The world will let you know whether they agree.”

There are plenty of free tools (e.g. polls and surveys and interviews) that news organizations can incorporate into the reporting process – preferably at all stages of the reporting process – and use regularly to engage audiences on what they do and don’t know, what issues they’d like to see covered, and what knowledge and experience the community can contribute to the understanding of a topic. It’s an iterative process that reveals how your news organization’s value is perceived by the community, as well as tremendous insight about where you should focus your energies based on the kinds of stories and issues your readers say they care about most.

Funders, You’re on the Hook Too

What’s good for the goose is also good for the gander, and so honest assessment is a requirement for funders too.

Foundations approach their support for media in different ways. Some support journalism for journalism’s sake – toward a more informed and engaged public. Others support media to advance understanding of a particular issue, like health or education. The aforementioned Media Impact Funders report does a good job outlining the diversity of approaches. Regardless of the strategy though, I think one of the key questions for funders to consider is whether we’re funding at the right level to achieve the kind of impact we want to see.

Imagine the issue you’re trying to address as an iceberg, with the easier, perhaps less permanent (but still valuable) solutions at the tip – what you see above the water – and the deep, systems-level solutions at the massive bottom of the iceberg underneath the water. The solutions at the top of the iceberg require far fewer resources than the solutions for enduring change required at the bottom of the iceberg.

So, at what level are you trying to have impact on a particular issue – where are you on the iceberg? – and do your grants match the resources required for that level of change?

If we’re not honestly assessing these questions and attempting to understand our impact, then we don’t know what we’re doing right and what we’re doing wrong, and therefore we do not have a clear understanding of how to effectively allocate our resources.

Funny how that works both ways.

Impact is not a four letter word. The more we ask questions and challenge our assumptions, the more we are transparent about what we’re trying to accomplish and share what we’re learning, the closer we get to making the difference that we want to see in this world.

For additional reading on impact, I also highly recommend Chalkbeat founder Elizabeth Green’s excellent paper “What We Talk About When We Talk About Impact.”

Ruler image by Humusak and iceberg image by Martin Fuchs, both via Creative Commons