Absolutely, knowing your role, your audience, and bringing new information/views is absolutely key. And you hardly ever have an audience craving 300 page reports.
Bravo, Todd. Now you just need to write a long report explaining why </snark>
I dare say - academia is not much better (journal publishing).
Anything counter to CW (conventional wisdom) and the gatekeepers is rejected. So authors create a typical template of "deep analysis" to "prove" their findings. And the longer length makes it look legit, and few people bother understanding the details.
I have cut down my reviewing because I will go back to read citations to check what *assumptions* are key. That takes time. But no one has time thanks to information overload, instead of smaller, incremental work.
I love the two page brief story. I agree that long reports often don’t have the impact we hope for them and that other approaches can be way more effective. I do think that we still need the bulky evidence recorded so that the 2 pager is based on solid data. When we do write a long report we should definitely suppliment it with other communication approaches (and be less long winded).
I think there is also a space for reader reform. Sometimes the complexity of energy and development can’t be captured in 2 pages (especially for the less informed). We have to be prepared to engage with the depth if we are going to break from our polarised world, where, often, neither side is correct.
Yep, you need analysis to back up the memos. But what I see from most TT & IO reports is repeated pablum. A crisp 1-2 pager paired with a good technical paper is an ideal combo.
And, of course, the memo is just the conversation starter.
I admit, I'm worried that we're moving towards a world of less analysis and more vibes, shorter papers that are read by decisionmakers directly rather than being processed by staffers who don't share the writer's agenda, and a degradation of decision-making quality.
I agree about death to the long, published policy report. I do think, however, there are benefits to writing up the long version of the research and analysis that leads to a short report, especially if the short report is just a two-pager or a blog or video. That write-up shouldn't cost millions of dollars and doesn't need to be shiny with pictures and ten layers of edits and approvals, but it should be publicly available online and provide a careful documentation of the sources and analysis that led to the policy brief. I've read far too many policy briefs with bold claims, one or two footnotes at best, and no supporting document. That doesn't establish credibility, even if the brief includes the logo of a well-known institution.
We were deliberately crisp and provocative as a way to constructively stir the pot.
A call to action - classic marketing and sales approach.
Too often the aid and development industry is wrapping itself in pseudo academia rather than being what it is - an industry now definitely in its mature stage with a few organisations requiring euthanasia?
It feels like policy reports are being judged against a linear model of impact, when in practice their effects are much more distributed. Even when they don’t drive immediate decisions, they still shape interpretive frames and what becomes possible later on.
So maybe the issue isn’t that they fail—but that we haven’t designed them for how systems actually absorb meaning.
Todd, having spend some 13 yrs on USG (USAID and DOS) CAs, it's not clear that on the ground efforts accomplish much either........... If people don't want change, it doesn't happen. So, a great deal more money and time could be saved (avoid the cost of trying to execute on the recommendations in reports that people don't read in the first place, because there is a lack of sincerity all around). Which means the loads of college degrees obtained with intentions of effecting public policy all need to be deployed toward something else, hopefully more constructive.
yeah, I feel like the key question here isn’t whether reports are read or not, but whether their writers do in fact have something better to do or the whole thing is a “jobs program”
The key phrase here: "I certainly learned a lot (mostly about how the World Bank works) and met some incredibly smart, dedicated people."
While it'd be great that such works had more meaningful impact, sometimes the real goal is for an individual or team that will go on to do other work to show they've done the work.
That said, I think some honesty about that could save a lot of time, because there's a lot that could be dropped when that is the goal, and 18+ months is a long time.
This post really resonated with me and the hands-on examples are excellent evidence of the importance of knowing how to share knowledge and insight for impact. The "full report" can also be shared through facilitated dynamic briefings and exchanges.
However, and to my own surprise I caught myself wanting to add a point to the "What are policy reports good for?" - list. Writing extensively and coherently can help refine your thinking, and if its done in collaboration it can align a team around core ideas. Obviously that requires time and explicit intent, and will be lost as soon as the pasting cutting starts.
Agree with much of this! That said, as an economic historian, I plough through a lot of old IMF and World Bank reports for data and context - given the lack of systematic administrative data collection in many low-income countries. So at least those meaty, policy reports that hoover up everything systematic there in a country on a given topic, and include long appendices, serve a certain long-term record-keeping purpose.
Love this and unlike so many policy reports I both read it to the end & started at the top to savour it again. I often think the reason why they are so long is very point: the hope is no one reads them. Plus all of tbe explanatory reasons for why things actually are the way they are tend to be fly specked out , as I wrote in thi criminally unread blog 😀
Do they have a pedagogical value nonetheless? my background is having written thesis for my bachelors/masters in engineering - even though entire corpus of documentation is not needed to be shown, producing each of those plots and analysis had immense pedagogical value for me
"We use social media, videos, and all kinds of communications tools. But we also write loads of long reports."
Conducted an experiment over the last 7 months placing comments, commentaries, analysis building on posts, outright/direct questions and seen very little, less than 3 in over 50plus additions to posts, engagement to develop a conversation, dialogue, discussion.
It is attitude? Or a check and this person is not going to get me business? Or a discounting as useless input?
When people have a salary and secured on inputs and activity rather than outcomes and causing impact, then we will witness the plateauing of knowledge (since where is the insight to come from? Let alone the challenges for Kuhn-esque shifts). AI in all its forms will replace the churn when we are not being insightful and presenting 'stuff' drawing people into reading and interacting on the basis of reading to think and produce further
"About 13 percent of policy reports were downloaded at least 250 times while more than 31 percent of policy reports are never downloaded. Almost 87 percent of policy reports were never cited."
This echoes findings from (at least) the 1980s looking at business book buying and what are even opened, read and used. It led to changes in formating and yet the pseudo academia of numerous aid and development organisations continued to produce quantity rather than quality.
Quality with defined recipients and calls to action - invariably we have witnessed justifying positions and what was rather than developing specific elements for defined people and organisations. Perhaps, with the beauty of hindsight, it is why we reached a number of decision points these last couple of years regarding aid effectiveness?
If someone hasn’t been involved in writing one of these massive reports before then this article may not land fully. I have however spent up to 18 months on a single report that involved a gang of other people too. A huge effort that ultimately didn’t have much impact! So I thought what you wrote very accurate and, given the current media and content landscape, equally timely. Thank you. An opinion driven Substack article sometimes now has more impact than a 120 page report. Plus the ability to write freely.
This is SO good. Thank you for writing it!
Thanks Amy!
Absolutely, knowing your role, your audience, and bringing new information/views is absolutely key. And you hardly ever have an audience craving 300 page reports.
Bravo, Todd. Now you just need to write a long report explaining why </snark>
I dare say - academia is not much better (journal publishing).
Anything counter to CW (conventional wisdom) and the gatekeepers is rejected. So authors create a typical template of "deep analysis" to "prove" their findings. And the longer length makes it look legit, and few people bother understanding the details.
I have cut down my reviewing because I will go back to read citations to check what *assumptions* are key. That takes time. But no one has time thanks to information overload, instead of smaller, incremental work.
I love the two page brief story. I agree that long reports often don’t have the impact we hope for them and that other approaches can be way more effective. I do think that we still need the bulky evidence recorded so that the 2 pager is based on solid data. When we do write a long report we should definitely suppliment it with other communication approaches (and be less long winded).
I think there is also a space for reader reform. Sometimes the complexity of energy and development can’t be captured in 2 pages (especially for the less informed). We have to be prepared to engage with the depth if we are going to break from our polarised world, where, often, neither side is correct.
Yep, you need analysis to back up the memos. But what I see from most TT & IO reports is repeated pablum. A crisp 1-2 pager paired with a good technical paper is an ideal combo.
And, of course, the memo is just the conversation starter.
I admit, I'm worried that we're moving towards a world of less analysis and more vibes, shorter papers that are read by decisionmakers directly rather than being processed by staffers who don't share the writer's agenda, and a degradation of decision-making quality.
I agree about death to the long, published policy report. I do think, however, there are benefits to writing up the long version of the research and analysis that leads to a short report, especially if the short report is just a two-pager or a blog or video. That write-up shouldn't cost millions of dollars and doesn't need to be shiny with pictures and ten layers of edits and approvals, but it should be publicly available online and provide a careful documentation of the sources and analysis that led to the policy brief. I've read far too many policy briefs with bold claims, one or two footnotes at best, and no supporting document. That doesn't establish credibility, even if the brief includes the logo of a well-known institution.
We were deliberately crisp and provocative as a way to constructively stir the pot.
A call to action - classic marketing and sales approach.
Too often the aid and development industry is wrapping itself in pseudo academia rather than being what it is - an industry now definitely in its mature stage with a few organisations requiring euthanasia?
This really clicked.
It feels like policy reports are being judged against a linear model of impact, when in practice their effects are much more distributed. Even when they don’t drive immediate decisions, they still shape interpretive frames and what becomes possible later on.
So maybe the issue isn’t that they fail—but that we haven’t designed them for how systems actually absorb meaning.
Absolutely, and possibly how feedback from that meaning is reincorporated into the system.
Todd, having spend some 13 yrs on USG (USAID and DOS) CAs, it's not clear that on the ground efforts accomplish much either........... If people don't want change, it doesn't happen. So, a great deal more money and time could be saved (avoid the cost of trying to execute on the recommendations in reports that people don't read in the first place, because there is a lack of sincerity all around). Which means the loads of college degrees obtained with intentions of effecting public policy all need to be deployed toward something else, hopefully more constructive.
yeah, I feel like the key question here isn’t whether reports are read or not, but whether their writers do in fact have something better to do or the whole thing is a “jobs program”
I think part of the answer here is that it's not about the report, it's about the process: https://substack.norabble.com/p/the-mirage-of-deep-research
The key phrase here: "I certainly learned a lot (mostly about how the World Bank works) and met some incredibly smart, dedicated people."
While it'd be great that such works had more meaningful impact, sometimes the real goal is for an individual or team that will go on to do other work to show they've done the work.
That said, I think some honesty about that could save a lot of time, because there's a lot that could be dropped when that is the goal, and 18+ months is a long time.
This post really resonated with me and the hands-on examples are excellent evidence of the importance of knowing how to share knowledge and insight for impact. The "full report" can also be shared through facilitated dynamic briefings and exchanges.
However, and to my own surprise I caught myself wanting to add a point to the "What are policy reports good for?" - list. Writing extensively and coherently can help refine your thinking, and if its done in collaboration it can align a team around core ideas. Obviously that requires time and explicit intent, and will be lost as soon as the pasting cutting starts.
Either way, many thanks for an insightful post!
Agree with much of this! That said, as an economic historian, I plough through a lot of old IMF and World Bank reports for data and context - given the lack of systematic administrative data collection in many low-income countries. So at least those meaty, policy reports that hoover up everything systematic there in a country on a given topic, and include long appendices, serve a certain long-term record-keeping purpose.
Love this and unlike so many policy reports I both read it to the end & started at the top to savour it again. I often think the reason why they are so long is very point: the hope is no one reads them. Plus all of tbe explanatory reasons for why things actually are the way they are tend to be fly specked out , as I wrote in thi criminally unread blog 😀
https://devpolicy.org/why-is-it-that-so-few-of-us-appear-to-read-reports-because-life-is-squelched-out-of-them-20140523/
Do they have a pedagogical value nonetheless? my background is having written thesis for my bachelors/masters in engineering - even though entire corpus of documentation is not needed to be shown, producing each of those plots and analysis had immense pedagogical value for me
"We use social media, videos, and all kinds of communications tools. But we also write loads of long reports."
Conducted an experiment over the last 7 months placing comments, commentaries, analysis building on posts, outright/direct questions and seen very little, less than 3 in over 50plus additions to posts, engagement to develop a conversation, dialogue, discussion.
It is attitude? Or a check and this person is not going to get me business? Or a discounting as useless input?
When people have a salary and secured on inputs and activity rather than outcomes and causing impact, then we will witness the plateauing of knowledge (since where is the insight to come from? Let alone the challenges for Kuhn-esque shifts). AI in all its forms will replace the churn when we are not being insightful and presenting 'stuff' drawing people into reading and interacting on the basis of reading to think and produce further
"About 13 percent of policy reports were downloaded at least 250 times while more than 31 percent of policy reports are never downloaded. Almost 87 percent of policy reports were never cited."
This echoes findings from (at least) the 1980s looking at business book buying and what are even opened, read and used. It led to changes in formating and yet the pseudo academia of numerous aid and development organisations continued to produce quantity rather than quality.
Quality with defined recipients and calls to action - invariably we have witnessed justifying positions and what was rather than developing specific elements for defined people and organisations. Perhaps, with the beauty of hindsight, it is why we reached a number of decision points these last couple of years regarding aid effectiveness?
If someone hasn’t been involved in writing one of these massive reports before then this article may not land fully. I have however spent up to 18 months on a single report that involved a gang of other people too. A huge effort that ultimately didn’t have much impact! So I thought what you wrote very accurate and, given the current media and content landscape, equally timely. Thank you. An opinion driven Substack article sometimes now has more impact than a 120 page report. Plus the ability to write freely.