What is Think Tank impact? (And what is it not?)

Why no one should care about your 18 reports

January is a time when my email inbox is full of “impact reports” from think tanks. Reflecting on accomplishments of the past year is a super useful exercise. And think tanks all need to share with donors what they got for their money. The pressure to show a beehive of activity is strong.

Yet I’m struck by how many organizations seem to stop their impact reporting at the number of events hosted or reports published – often without even explaining the purpose of such things toward any specific goal. Meetings and pubs are outputs, not impact. And when disconnected from a particular aim, they aren’t even a clear step toward progress.

Getting your name in the New York Times or speaking on a panel at Davos is a pretty good sign of influence (in certain circles) but that’s not impact either. Maybe some funders want their grantees in the press or on a stage, but most that I know don’t care about that. They want their donations to lead to some meaningful change in the world. They’ll support a meeting or report or their grantee in the press if it leads to that change. But those are all means, not ends.

I’ve written recently about lessons I’ve learned from Nancy Birdsall on how to build an effective think tank, including a relentless focus on outcomes over egos. I also wrote up how a few orgs worked together over many years to remove the World Bank’s nuclear ban as a window into how think tanks can get sh*t done in Washington DC. That policy change aimed to expand the range of clean energy technologies available to poor countries so they can grow faster. It was accomplished with zero reports, zero public events, and made the NY Times only after it was all over. Now, we’re laser focused on helping the policy change lead to actual projects and, eventually, to more energy creating jobs and lifting livelihoods.

This can all be confusing. People conflate outputs, impacts, and results all the time. A whole industry of nonprofit consulting exists to help organizations and foundations figure it out. It can seem like a pedantic debate over terms. But clarity of purpose and what counts as impact goes to the heart of why philanthropy and their grantees exist.

Think tanks have an especially steep challenge since they are not doing direct service delivery but trying to dance a two-step toward impact. A nonprofit vaccinating kids or delivering solar home systems can report on the exact number of people they helped and measure how much their individual lives changed. But altering a government rule is not like vaccinating kids or installing solar panels. Think tanks are trying to influence policies that, if successful, might lead to more vaccinations or lower electricity tariffs for an entire country. Success reaches many more people (scale!) but attribution (or a precise ROI) is far more tenuous. That raises the bar on think tanks to explain what they do, why, and then tell a plausible story when it works.

A Simple 6-Step Model for Think Tank Impact

After years of trying different things at the Center for Global Development and the Energy for Growth Hub, here’s how I now think about the impact value chain and how think tanks can explain it. None of this is unique to me. Others may use different words. I’m just sharing what I’ve learned (so far) from working with at least seven think tanks over 30+ years.

One quick caveat: this schematic is linear for simplicity, but think tank impact never happens linearly. Every individual effort at policy change is going to follow its own crazy path of twists, stop-starts, and pivots. The military maxim that no war plans survive first contact also applies to think tank impact. Thinking through steps toward your goal is useful. But a rigid plan is almost immediately useless. I’m just sharing a crude mental model I use to think about how different steps may lead eventually to impact – and the kinds of things we try to track along the way to see how we’re really doing. This approach also helps us tell a credible impact story after a big win.

To think through the path to success, I like to have, in this order:

1. Ultimate goal. Start with a shared final objective such as people living healthier lives or having decent jobs.

2. A near-term result. Posit a credible idea of how to do that by solving some specific problem standing in the way, like reducing drug prices or attracting more capital to build power plants.

3. Impact. Identify a specific mechanism to solve that problem, like changing a rule or creating a new initiative. This is the think tank sweet spot.

4. Influence. Have multiple ideas for how we might spark the rule change or initiative, such as figuring out exactly who needs to act, what they really care about, and how to convince them. This is think tank strategy.

5. Outputs. Conduct activities to share, shape, and sell the idea. This is think tank tactics.

6. Inputs. Find people and money to get started. This is think tank operations.

So what exactly to track and report?

We can then track progress in the exact reverse order. Here is my crude schematic along with examples of what to document or measure.

1. Inputs are people and money you need to operate. Reporting how we’re doing on recruitment and fundraising is a good signal, but of course just the first step.

2. Outputs are activities that could eventually lead to later steps. These should be tracked and can sometimes be quantified. But be careful: quantity does not tell us anything meaningful and is usually misleading about effectiveness. That’s because “100 meetings” may be far less valuable to reaching the end goal than “one important phone call with the right person.” I’ve also found that one killer graphic or a good 1-pager can change minds more than a 125-page report.

3. Influence is tracking proxies for credibility with target audiences. This might mean New York Times quotes if that really helps us move the ball, but more often it’s things like use of our work by policymakers or requests for briefings from the Interagency or Hill staff. These are signs that we are reaching our audience and responding to a pressing demand, not just throwing out clever analysis. Again, quantifying this can be misleading and create weird incentives. Testifying to a Congressional subcommittee ten times might be far less influential than one White House request for a 5-minute call.

4. Impact is the target action. This is the think tank sweet spot. We know we’re having an impact when something concrete changes in the real world and we can trace our role in making it happen. At the Energy for Growth Hub, we explain this through our Impact Stories (ten, soon to be eleven!). I wrote more about why we landed on this approach with Sarah Jane Staats. It’s still a work in progress but far better than counting publications or press quotes.

5. Results are near-term steps that help, but are not quite all the way. These are the intended direct effects of the think tank’s impact toward the ultimate goal. Clear attribution is sometimes possible but often not. Did the rule change we sparked result in lower prices or was something else going on in the market? Did the new initiative we helped launch actually lead to more investment? We need to do our best to try to follow (or even shape) the effects, but we also accept this ambiguity. Our donors understand the world is messy.

6. Ultimate goals are the North Star. This is the world we want to create. Ideally, the intermediate results played some role, catalyzed by our impact step. But attribution here is truly difficult. Life expectancy goes up, but did lower drug prices from our rule change play a role? Incomes rise, but how much of that is due to more energy investment from our new initiative? Rough orders of magnitude and direction of trends may be the best to hope for. But always having an eye on an ultimate goal keeps us grounded and focused on the mission.

None of this is science. Trying to pretend think tank impact is a quantifiable linear process is a mistake. Yet merely listing your reports or media mentions is not even close to good enough.

Not good enough

Think tanks have a tremendous role to play in making public policy more effective and helping build a better world. That’s why the onus is on us to explain what we do, when it’s working… and what counts as impact.