Governance Assessments: Worth investing in or not?

[Editor’s Note: Nathaniel Heller (@Integrilicious) is a Managing Director at the Results for Development focused on harnessing citizen-centric transparency and accountability efforts as drivers of global development outcomes. This blog is cross-posted on the Governance Data Alliance blog. The Governance Data Alliance aims to bring together a wide range of world-class organizations committed to the effective production and use of high-quality governance data.]

On the heels of our wonderfully successful launch of the Governance Data Alliance’s data dashboards and newly commissioned report on understanding governance data users, a number of Alliance members and extended family have been writing about the ways in which governance assessments might need to evolve in the future. Global Integrity’s Johannes Tonn has called for “Governance Assessments 2.0” that focus primarily on serving as learning tools for domestic change agents interested in advancing reforms. Over at Development Gateway, Dustin Homer has put forward compelling arguments that caution against over-fetishizing data gathering efforts at the expense of their practical return on investment.

Based on research Development Gateway has been advancing in the past few years, there’s indeed reason to worry that many “results”-oriented data gathering efforts in international development come with significant negative opportunity costs that call into question their ultimate utility. The same concern could be applied to governance assessments. I wanted to build from that and offer another way of thinking through whether and why to consider investing in governance assessments (or, frankly, any assessment of public sector performance): the likely cost-benefit ratio.

We know from early costing work within the Governance Data Alliance that the rough median cost of a governance assessment on a country basis is, historically, about US$25,000. This is indeed a “rough” figure with some non-trivial outliers, but it tends to approximate the usual lift required to hire a quality in-country team, manage their research and data gathering, and publish the final results. It also takes, on average, nearly a year to go from start to finish with most governance data gathering efforts.

Imagine that you are a decision-maker considering whether to invest in a new 50-country scan of, say, civic space issues. The back-of-the-envelope math suggests that generating new governance data will cost more than $1.2 million for a single round and consume a small project team for the better part of a year. Is it worth the effort, and what opportunity costs are associated with saying “yes?”

One question to ask yourself is, “Can I articulate a scenario in which the positive changes that might emerge from the production of these data are valued at more than $1.2 million?” This is admittedly a difficult answer to pin down in most cases, but not necessarily impossible.

Consider, for example, a data-gathering effort around procurement transparency. Reforms around transparent procurement and “open contracting” can lead to measureable, large cost savings for public coffers. It’s not completely beyond the realm of the possible to make a case for why generating other types of governance data couldn’t lead to similar downstream savings, whether through improved government efficiency or reduced waste, fraud or corruption.

A second question to ask is about the time spent to generate governance data. It could potentially be deployed to other activities with equal or greater chance of affecting positive social change. Would spending the year that’s required to generate your new civic space data set be better put to use by lobbying political leaders in key countries to adopt more progressive approaches to protecting civil liberties?

Finally, there are always tradeoffs associated with the political capital that is required to leverage governance data towards concrete downstream reforms. Policymakers and other domestic change agents have only so much time and attention to focus on and digest any data set or index (something we know in more concrete terms thanks to AidData’s terrific new research around who governance data users really are). Are we convinced that asking them to spend time mastering our new data to launch new reforms is better than shoring up existing efforts with which they are already familiar?

The answers to these (highly stylized and imperfect) questions will never be clean. But trying to articulate the tradeoffs and costs is important if we’re serious about investing intelligently in future governance data production efforts.

My hope is that future efforts will always able to offer a compelling rationale for their expected return on investment…and why it’s at least marginally better than no new data at all.