More evidence won’t save development

We don’t need more studies. We need a new coalition rebuilt on political priorities

Washington DC is awash with groups trying to strategize what’s next for development. After USAID’s shock dismantling and the budget bonfire, aggressive brainstorming is absolutely needed. Yet many seem headed in odd, unhelpful directions.

I’m bemused that so many of these conversations seem to be falling into myopic, defensive huddles that do not grapple honestly with why the constituency for development turned out to be vaporware. Congressional support – supposedly buttressed by a bedrock civil society coalition that included business and churches – never rallied and, maybe, was so shallow that it barely existed.

One strategy to rebuild that seems to be gaining traction in some circles is a push for “more rigorous evidence.” In particular, I’m hearing calls to fund far more experimental evaluation of interventions, especially randomized controlled trials or RCTs. I suppose the notion is “if we show skeptics that aid actually works, then they’ll spend political capital defending it.” This strikes me as politically tone-deaf and likely damaging.

If you’re not a dev nerd, RCTs randomly divide people into groups, give one the treatment and not the other, and measure the difference. Because the split is random, whatever gap shows up later must be the treatment’s effect. The method works to test & prove pharmaceuticals. That’s because a drug trial is a clean setup: one well-defined treatment, a short causal chain, and an effect that’s easy to isolate. It’s enticing to hope we can use the same approach to test & prove development ideas. The thinking goes: Let’s identify roughly similar groups, randomly assign a treatment [give cash, a laptop, cheap fertilizer, a solar lantern, etc] to one and not the other, and then compare the two groups later to measure the isolated effect. Voilà, we can prove impact. And, advocates hope, policymakers and politicians can use all this evidence to justify development aid – because they’ll be spending money on what works and not on stuff that doesn’t.

RCTs for development are all the rage. The researchers who pioneered the idea earned a Nobel Prize in 2019, penned a best selling book, and launched multiple well-funded research organizations to deploy armies of grad students to run experiments. This approach has pretty much overtaken the entire field of development economics. It’s nearly a cult.

The RCT wave did help focus the aid industry on impact and cost-effectiveness. It’s been useful for some social sector programs, especially health and safety nets, and for designing more efficient humanitarian relief. I used RCT evidence on cash transfers to make the case for citizen dividends in new oil economies in my book, Oil to Cash. Some of my favorite people are diehard Randomistas.

But RCTs’ dominance has also created some very weird – and I’d say ultimately detrimental – effects on researchers’ ability to influence government policies that might, you know, promote economic development.

RCTs are useful for discovering some things…

Here’s an exhaustive list of what RCTs are good for:

1. Evaluating the specific effect of a designed experiment, provided that treatment can be randomized among willing participants and cleanly isolated from external factors

2. Academic publications

3. Tenure

…but RCTs are not actually useful for much of anything that most politicians care about

Here’s my non-exhaustive list of what RCTs cannot help us understand because they are not randomizable or easily isolated:

1. How to make poor countries rich

2. How to spur economic growth

3. How to foster industrialization and economic transformation

4. How to add value to critical mineral supply chains

5. How to use AI and demand for data center infrastructure to promote economic opportunities

6. How to build political coalitions for policies that have a fighting chance to overcome low-equilibrium rent seeking that is strangling many low-income economies

In other words, the dominant tool in development economics is not useful for macroeconomic questions, political economy puzzles, or any of today’s economic policy priorities in most developing countries. A minister of health or education might want to know how to squeeze 10% efficiency gains out of their budget by weighing different options. That’s useful, on the margins.

But the finance minister, the infrastructure minister, the industrial policy cabinet lead, and indeed the president all hold much higher ambitions. They are, in nearly every emerging market country I know, hoping to turn their economies into wealthy, high-productivity engines that create full employment and compete in the global economy. RCTs can’t help them do that.

The multiple damaging effects of the randomization revolution

If this was just one academic subdiscipline driving itself into irrelevance, it wouldn’t matter much. But the total dominance of RCTs is creating damage to the range of policy questions research can help us answer. Others far smarter than me have slammed RCTs and their overhyped misuse, most famously Angus Deaton and Lant Pritchett. My criticisms could be summarized in three ways:

• Confuses what’s measurable with what’s important. The Dismal Science was once about big ideas facing humanity: How do we make the world richer? How do societies become productive? How do we free people to make choices in their own lives? RCTs have skewed problem identification from what matters to what’s easily experimental. Development economics has gone from dreaming about humanity’s future to getting lost down rabbit holes about measurement and methodology. The arguments are now more about statistics than ideas. The Worm Wars, for example, is a debate over one seminal study from 2003 that helped initiate the whole RCT push by finding a cost effective way to get poor kids in western Kenya to attend school by giving them deworming pills. After 23+ years, the war is still raging. This is a very long way from the big questions that created development economics – or what’s top of mind for today’s policymakers.

• Presupposes what counts as evidence. While most economists agree that other kinds of non-RCT evidence are perfectly valid, it’s also very common to hear RCTs described as “the gold standard.” In other words, other approaches are less valuable. I hear this often, like when people try to dismiss our Empty Quadrant scatterplot as irrelevant since it’s not directly causal. We know. But it’s still useful evidence about a complex relationship.

• Creates powerful incentives for thinking small. Both problems above combine to push the field into ever more micro interventions that are easily measurable rather than the hard-to-isolate macro stuff that determines which countries become prosperous. Are we trying to make poor people a little less miserable? Or finding ways that human progress can benefit everyone, everywhere? RCTs have literally nothing to say about long-run growth.

RCTs are also undermining the push for universal energy abundance

I see these negative dynamics play out in my own work on electricity. I even hear people complain that we lack evidence to show electricity is even needed for development. The article often cited lately is Does Household Electrification Supercharge Economic Development? by my friend Catherine Wolfram along with Ken Lee and Ted Miguel. They’re all great economists and I have nothing but respect for their work. Yet this study, with its bold headline question, answers clearly: no. That’s because they found, based on their own randomized experiments, that very poor households in western Kenya who connected to the grid didn’t use much electricity or show much benefit after 2 to 3 years.

This is no surprise. Connecting extremely poor people who can’t afford to pay for power (much less buy the machines that make electricity useful) doesn’t have a transformational short term impact. Poor people are poor precisely because they are stuck in low-productivity jobs and in markets that are not functioning. So, no, connecting their homes to a wire (that often does not deliver power) will not magically change their lives. I know Catherine et al are thoughtful and more nuanced than this. But I’ve personally experienced people interpreting their work as “RCTs show little effect of an electricity connection, so it’s not worth bothering.”

My interpretation is very different: their paper shows the limitations of RCTs to answer the most pertinent energy question. Why did they opt to randomize electricity for people at home rather than where they actually earn a living, which would be at work on a farm or in a workshop? It’s the bias of the tool: they’re looking for impact where an RCT can spot it, not where we’d expect it. Another academic friend compared RCTs to the old streetlight joke, where a man who lost his keys in the dark only searches under the lamp post because that’s where the light is better.

This is why, for me, Wolfram et al makes a good case that if you want energy to lead to better economic outcomes, don’t prioritize connecting the poorest households. Instead, focus on delivering cheaper and more reliable power for businesses in towns and cities. And let’s just accept that it’s impossible to randomize power for industry. No business would ever knowingly volunteer to suffer blackouts to help an academic study. That means we need a different way to gather evidence. If we stick with randomized experiments in energy it will inevitably push studies down to the micro level of initial access for the extreme poor – and thus back into irrelevance for policymakers focused on growth.

An alternative interpretation, of course, is that the Government of Kenya is pursuing rural electrification for a political or social reason, rather than as a strictly economic one. Most countries do this. (The United States is still subsidizing rural electricity today.) Governments rationally choose to connect everyone, not because they expect an immediate economic payoff, but because they are trying to build more inclusive societies – and politicians are trying to win elections.

We need durable coalitions built on an honest reckoning, not more academic studies

The methodological problem and the political one are really two sides of the same coin: RCTs are not built for the questions that actually move ministers -- or members of the US Congress. Which brings me back to the misguided rationale in some circles that more rigorous randomized studies can save smart development policy in Washington DC and help us rebuild something akin to USAID again. I don’t buy it.

I have seen nothing to suggest that what allowed USAID to be burned to the ground is a lack of peer-reviewed evidence. Does anyone believe that Republicans on Capitol Hill would have defended USAID from DOGE if they had 100 (or even 1000) more academic studies showing the positive impact of aid spending? Of course not. So the way to rebuild a durable political constituency for US development policy is not to double down on randomized experiments.

We need evidence, but not that kind. I would start with old school qualitative research that is way out of fashion. We should be asking all of the old congressional champions what happened, what went wrong, and how did the political incentives shift. Only by understanding the motivations and actual priorities of legislators, senior executive branch officials, and the multitude of external forces that influence them, will we ever come to a new, better understanding of how to rebuild. One good oral history would be far more valuable than a mountain of new impact studies.

If you want US development policy to come roaring back, don’t waste money on another RCT. Spend time interviewing Lindsay Graham.