Hi Todd! I agree with a lot of this — especially the critique of RCTs crowding out the big questions and fixating on what's answerable (maybe...) through a specific methodology. But I also want to make sure we don't swing too far away from recognizing the utility of micro data/evidence.
Of course you can't randomize a country's growth trajectory. But understanding what allows people to get healthier, learn more, earn more, and transition into higher-quality work is relevant to unlocking broad-based growth (a recent piece by David Oks here -- https://davidoks.blog/p/why-china-got-rich-and-india-didnt).
The problem isn't micro evidence per se, but that microeconomists have largely failed to draw the connective tissue between their findings and macro outcomes. The more useful intervention, I'd argue, isn't to abandon the micro toolkit but to reorient it around a different definition of success. Right now, a study succeeds if it establishes a clean causal effect on a proximate outcome. What if success instead meant: does this finding help us understand/address a binding constraint on structural transformation/growth? (to your point about electricity for businesses versus households)
Definitely curious about how to shift incentives in that direction, recognizing how many obstacles (tenure included...) stand in the way.
The slashing of PEPFAR is the proof that evidence wasn’t the problem. PEPFAR had superb, easily documented outcomes— people who didn’t die, people who didn’t give their kids HIV and so on and it didn’t matter. It didn’t matter because the administration, by empowering known white-supremacist Elon Musk, made cutting off medicine to Black people a top priority. And the Rs in Congress are, almost to a man, afraid to push back on the administration. They’d push back on tax hikes on the rich, gun control, abortion, but that’s about it.
How many evangelicals who genuinely care about PEPFAR—and there were a lot—were willing to penalize Trump and his enablers for this? Not many. Even for the Rs for whom it’s a priority, it usually a second or third tier priority
To summarize my arguments (feel free to copy/paste the next time this debate gets a sequel release):
1. Micro and macroeconomics both care about development but specialize in answering different kinds of questions. We need both.
2. No single method is the gold standard for everything. Choose good questions and then best viable tool for that question.
3. I’ve found you can study, and sometimes even randomize, more things than people first assume, including parameters to models that speak to “big” questions in macro.
4. If we have a choice between guiding critical decisions with more information or less information, why not choose more?
Bottom line: Todd’s right that more evidence might not have been a political solution in these times, but it can help us spend scarce dollars where they’ll do the most good. Hopefully that’s a goal on which we can agree.
The conclusion is spot on ... So many of these discussions presume that what doomed USAID was development (in)effectiveness. Thus the solution is "do development better." As much as I would like to see it, I tend to agree that any proposal for a more (developmentally) effective set-up is not going to do much to rebuild a domestic constituency for foreign assistance. What I think may be needed instead is a cogent argument that can appeal to the American people and their representatives for why US support for development - understood as structural transformation and progressively more inclusive governance in low-income countries - remains in the interests of the United States. As a member of the development community, the argument is self-evident to me, but what seemed like a popular consensus - particularly during the Cold War but even as recently as 15 years ago - feels like it has eroded considerably. Why that is, I don't quite know ... So I second your qualitative research proposal.
Dear Todd, Thank you for this article. I am a fan of your work, but I respectfully disagree with your conclusions here. Much of what you argue about the limitations of RCTs are widely acknowledged by those who use them: they cannot answer macroeconomic or political economy questions; evidence is not the only thing or even the main thing that policymakers use to make decisions; and qualitative research is valuable and provides another vantage point. But for all these critiques, your article fails to acknowledge that rigorous evaluative methods, like RCTs, have fundamentally been a net positive for the aid sector.
Despite much talk of systems-level or policy change, a lot of aid, particularly in the humanitarian sector, still funds discrete interventions or services. In this context there are real choices like what interventions to invest in, what dosage and through which modalities. These are the types of questions that RCTs or other impact studies can answer and answering them can lead to better decisions by some decision makers who control aid budgets, in ways that can improve outcomes for people.
As Marco Di Natale points out in his excellent SSIR article, “Measurng More and Learning Less”, we often focus on this debate for and against RCTs, whereas the underlying issue in the aid sector--which still largely holds itself accountable for the number of people reached rather than how those people's lives have meaningfully improved--is rigor and standards in how we measure results. Impact studies push us towards greater accountability.
Finally, I have worked on both qualitative, political-economy research and impact evaluations in the aid sector. My experience tells me both are needed for different reasons, and neither can sway political support for policies where preferences are determined a priori, regardless of the evidence (i.e. the decision to dismantle USAID)
"We need evidence, but not that kind. I would start with old-school qualitative research, which is way out of fashion."
That one line stood out for me. It feels irritating to see the development sector's obsession with RCTs. Hopefully, the sector can go back to building meaningful research frameworks.
The populist critique of aid is pretty simple. Americans have always overestimated the amount we spend on aid, and it is easy for populists to claim that we could make the average American’s life better if we just stopped spending money on people overseas.
RCTs are vital for demonstrating how money spend causes less death, less disease, better educational outcomes, and better quality of life abroad. If you want more populist support, the focus needs to be on how these changes make Americans lives better in conservative and populist terms. Does it reduce migration to the US? Does it reduce likelihood of terrorist cell formation? Does it prevent spread of disease to the US?
All good points (show how a specific aid program reduces terrorist cell formation, reduces asylum-seeking immigration to the US, prevents spread of disease to the US, etc.) But we also need to constantly correct that misinformation about how much America spends on foreign aid, or we'll never get anywhere with that demographic. They keep thinking (and reiterating) there's already "all that money pouring into foreign bank accounts..."
Strongly agree! And a bit disappointing that the responses here seem to slip back into "is evidence / RCTs good or bad"). A few thoughts come to mind:
1) what can we learn from the growing parts of aid, like high net worth philanthropy? When I look to the TED Audacious Project and donor collectives, I see evidence as table-stakes, but that willingness to give also requires storytelling and clear leadership.
2) what about what we may have lost along the way in terms of professionalization of aid? Reductions in volunteer flows, global solidarity relationships, missionaries; there is surely no efficiency gain from sponsorship relationships (and many problematic aspects), but it's certainly easy to see how evidence may leave something to be desired in terms of inspiration / relevance to developed country donor. (My mother, in spite of my work with GiveDirectly, continues to give through her church as she trusts them to manage the money.)
3) what skills are required for this? The kind of expertise you're calling for isn't really seen as development - it's more Marshall Ganz / organizing (or lobbying) than economics.
Cute. It is naive to think "economists" do RCTs becasue they believe in human progress - as you said - tenure and academic careers are bulit on getting causal inference results that end saying little about the human experience, much less helping to improving it. .
"Interview Lindsay Graham" is an interesting one... but you didn't finish your idea. why?
That's one solution, another is qualitative research.
BUT!
WHAT IF YOU/WE ACTUALLY DO THE THING??
Like you know industrial energy is important for growth so lets just make a company and start installing geothermal power in Rift Valley. What are we waiting for? Sure there are blockers to geothermal. So let's start a company that removes those blockers.
We don't need qualitative OR quantitative evidence.
Ministers in emerging economies everywhere are looking for ways to turn their countries to economic powerhouses without interfering with their personal low-entropy rent seeking.
Hi Todd! I agree with a lot of this — especially the critique of RCTs crowding out the big questions and fixating on what's answerable (maybe...) through a specific methodology. But I also want to make sure we don't swing too far away from recognizing the utility of micro data/evidence.
Of course you can't randomize a country's growth trajectory. But understanding what allows people to get healthier, learn more, earn more, and transition into higher-quality work is relevant to unlocking broad-based growth (a recent piece by David Oks here -- https://davidoks.blog/p/why-china-got-rich-and-india-didnt).
The problem isn't micro evidence per se, but that microeconomists have largely failed to draw the connective tissue between their findings and macro outcomes. The more useful intervention, I'd argue, isn't to abandon the micro toolkit but to reorient it around a different definition of success. Right now, a study succeeds if it establishes a clean causal effect on a proximate outcome. What if success instead meant: does this finding help us understand/address a binding constraint on structural transformation/growth? (to your point about electricity for businesses versus households)
Definitely curious about how to shift incentives in that direction, recognizing how many obstacles (tenure included...) stand in the way.
The slashing of PEPFAR is the proof that evidence wasn’t the problem. PEPFAR had superb, easily documented outcomes— people who didn’t die, people who didn’t give their kids HIV and so on and it didn’t matter. It didn’t matter because the administration, by empowering known white-supremacist Elon Musk, made cutting off medicine to Black people a top priority. And the Rs in Congress are, almost to a man, afraid to push back on the administration. They’d push back on tax hikes on the rich, gun control, abortion, but that’s about it.
How many evangelicals who genuinely care about PEPFAR—and there were a lot—were willing to penalize Trump and his enablers for this? Not many. Even for the Rs for whom it’s a priority, it usually a second or third tier priority
I've posted a longer response on my new substack: https://deankarlan.substack.com/p/more-evidence-wont-save-development.
To summarize my arguments (feel free to copy/paste the next time this debate gets a sequel release):
1. Micro and macroeconomics both care about development but specialize in answering different kinds of questions. We need both.
2. No single method is the gold standard for everything. Choose good questions and then best viable tool for that question.
3. I’ve found you can study, and sometimes even randomize, more things than people first assume, including parameters to models that speak to “big” questions in macro.
4. If we have a choice between guiding critical decisions with more information or less information, why not choose more?
Bottom line: Todd’s right that more evidence might not have been a political solution in these times, but it can help us spend scarce dollars where they’ll do the most good. Hopefully that’s a goal on which we can agree.
The conclusion is spot on ... So many of these discussions presume that what doomed USAID was development (in)effectiveness. Thus the solution is "do development better." As much as I would like to see it, I tend to agree that any proposal for a more (developmentally) effective set-up is not going to do much to rebuild a domestic constituency for foreign assistance. What I think may be needed instead is a cogent argument that can appeal to the American people and their representatives for why US support for development - understood as structural transformation and progressively more inclusive governance in low-income countries - remains in the interests of the United States. As a member of the development community, the argument is self-evident to me, but what seemed like a popular consensus - particularly during the Cold War but even as recently as 15 years ago - feels like it has eroded considerably. Why that is, I don't quite know ... So I second your qualitative research proposal.
Dear Todd, Thank you for this article. I am a fan of your work, but I respectfully disagree with your conclusions here. Much of what you argue about the limitations of RCTs are widely acknowledged by those who use them: they cannot answer macroeconomic or political economy questions; evidence is not the only thing or even the main thing that policymakers use to make decisions; and qualitative research is valuable and provides another vantage point. But for all these critiques, your article fails to acknowledge that rigorous evaluative methods, like RCTs, have fundamentally been a net positive for the aid sector.
Despite much talk of systems-level or policy change, a lot of aid, particularly in the humanitarian sector, still funds discrete interventions or services. In this context there are real choices like what interventions to invest in, what dosage and through which modalities. These are the types of questions that RCTs or other impact studies can answer and answering them can lead to better decisions by some decision makers who control aid budgets, in ways that can improve outcomes for people.
As Marco Di Natale points out in his excellent SSIR article, “Measurng More and Learning Less”, we often focus on this debate for and against RCTs, whereas the underlying issue in the aid sector--which still largely holds itself accountable for the number of people reached rather than how those people's lives have meaningfully improved--is rigor and standards in how we measure results. Impact studies push us towards greater accountability.
Finally, I have worked on both qualitative, political-economy research and impact evaluations in the aid sector. My experience tells me both are needed for different reasons, and neither can sway political support for policies where preferences are determined a priori, regardless of the evidence (i.e. the decision to dismantle USAID)
Thanks. We mostly agree. Not sure on net +/- tho. I’d just like to see far less self-serving advocacy from academics who are overselling one approach.
"We need evidence, but not that kind. I would start with old-school qualitative research, which is way out of fashion."
That one line stood out for me. It feels irritating to see the development sector's obsession with RCTs. Hopefully, the sector can go back to building meaningful research frameworks.
The populist critique of aid is pretty simple. Americans have always overestimated the amount we spend on aid, and it is easy for populists to claim that we could make the average American’s life better if we just stopped spending money on people overseas.
RCTs are vital for demonstrating how money spend causes less death, less disease, better educational outcomes, and better quality of life abroad. If you want more populist support, the focus needs to be on how these changes make Americans lives better in conservative and populist terms. Does it reduce migration to the US? Does it reduce likelihood of terrorist cell formation? Does it prevent spread of disease to the US?
All good points (show how a specific aid program reduces terrorist cell formation, reduces asylum-seeking immigration to the US, prevents spread of disease to the US, etc.) But we also need to constantly correct that misinformation about how much America spends on foreign aid, or we'll never get anywhere with that demographic. They keep thinking (and reiterating) there's already "all that money pouring into foreign bank accounts..."
Strongly agree! And a bit disappointing that the responses here seem to slip back into "is evidence / RCTs good or bad"). A few thoughts come to mind:
1) what can we learn from the growing parts of aid, like high net worth philanthropy? When I look to the TED Audacious Project and donor collectives, I see evidence as table-stakes, but that willingness to give also requires storytelling and clear leadership.
2) what about what we may have lost along the way in terms of professionalization of aid? Reductions in volunteer flows, global solidarity relationships, missionaries; there is surely no efficiency gain from sponsorship relationships (and many problematic aspects), but it's certainly easy to see how evidence may leave something to be desired in terms of inspiration / relevance to developed country donor. (My mother, in spite of my work with GiveDirectly, continues to give through her church as she trusts them to manage the money.)
3) what skills are required for this? The kind of expertise you're calling for isn't really seen as development - it's more Marshall Ganz / organizing (or lobbying) than economics.
Anyway, great piece!
Cute. It is naive to think "economists" do RCTs becasue they believe in human progress - as you said - tenure and academic careers are bulit on getting causal inference results that end saying little about the human experience, much less helping to improving it. .
"Interview Lindsay Graham" is an interesting one... but you didn't finish your idea. why?
That's one solution, another is qualitative research.
BUT!
WHAT IF YOU/WE ACTUALLY DO THE THING??
Like you know industrial energy is important for growth so lets just make a company and start installing geothermal power in Rift Valley. What are we waiting for? Sure there are blockers to geothermal. So let's start a company that removes those blockers.
We don't need qualitative OR quantitative evidence.
Does anyone here have any recommendations of research/studies on US political incentives?
Ministers in emerging economies everywhere are looking for ways to turn their countries to economic powerhouses without interfering with their personal low-entropy rent seeking.
Yes! Yes!! YES!!!
I agree - we do not need one more white paper.
There is a lot of fuzzy thinking around what caused the USAID shutdown, and what could bring it back. https://usaidmedia.org/posts/what-happened-to-usaid
Good science is empirical. Where have poor countries made significant progress?
What were Adam Smith’s findings? How do you avoid corruption. We are not doing well in California with state corruption at high levels.