Is it possible to calculate the return of investment on a research policy project? Well, the Indonesian think tank SurveyMETER and the Knowledge Sector Initiative have given it a go for a policy research project on services and infrastructure for the aging population in the Indonesian municipality of Balikpapan. Here is what we did and what we learned along the way.
[Editor’s Note: This post was written by Tanty Nurhayati Djafar, Program Officer, Knowledge Sector Initiative; Wayan Suriastini, Director, SurveyMETER; and Arnaldo Pellini, Senior Research Fellow, Overseas Development Institute, and Learning Lead, Knowledge Sector Initiative.]
First, a few disclaimers
This is the first time we have used this methodology and so it is an experiment and we’re learning as we go (Infographic).
It is important to remember that calculating the return on investment is just one piece of the puzzle when it comes to understanding research impact and contribution. The evidence produced should complement other monitoring, evaluation and learning work. In the case of SurveyMETER, we also undertook three episode studies to describe how the research and engagement activities contributed to the policy decisions in three Indonesian municipalities.
It is not possible to calculate the return on investment for all policy research projects. In this case study, the policy change resulted in a budget commitment by the government which provided a monetary sum to calculate against the research investment.
The policy research project and impact
Indonesia’s aging population (over 65 years of age) is increasing rapidly. It is expected to increase from 18 million (or 7.5 percent of the population) in 2010 to 41 million (or 14 percent of the population) by 2030. If we consider Indonesia’s rapid rate of urbanisation (2.6 percent per year), the number of elderly citizens living in urban areas will grow considerably. Some municipalities and local governments have started to think about how to prepare for this change. In 2013, Balikpapan, a municipality in south-west Kalimantan, began a collaboration with SurveyMETER to produce and discuss research evidence to better understand how well the municipality was addressing senior citizens’ access to services and infrastructure and decide which investments would be required to provide senior citizens with a high quality of life.
During 2013 SurveyMETER collaborated with the Centre for Ageing Studies at the University of Indonesia. They conducted a study in 14 cities across Indonesia to assess their preparedness for this increase in the senior citizen population. As a result, the municipality committed some US$ 8.5 million over 2015-2020 to fund better services and infrastructure for elderly citizens.
Estimating the return on investment
To test the return on investment with the policy engagement by SurveyMETER we used a five-step methodology which has been developed by the Redstone Strategy Group. Redstone used this methodology in 2013 with 3 policy research organisations supported by the Think Tank Initiative: Research on Poverty Alleviation (REPOA) in Tanzania, Fundación ARU in Bolivia, and the Institute of Economic Affairs (IEA) in Ghana (see examples from this exercise for further information).
Steps 1 – 4 are relatively straightforward:
- Define the policy problem: in this case, it was an aging population and the need for policies and programs to prepare appropriate services and infrastructure within the city.
- Define the research contribution to addressing the problem: the research provided a comparative assessment of preparedness across 14 municipalities.
- Define the ‘benefit’ of the policy change: based on the available data, we defined the benefit as the budget commitment by the municipality to fund services and infrastructure.
- Estimate the cost of research and policy engagement activities: this was calculated by the research finance team.
Step 5 is where it gets tricky. We had to estimate the portion of benefits for which we could argue a contribution.
We took the six conditions for policy change suggested by the Redstone Strategy Group and discussed the conditions before and after the research took place. We then estimated the contribution of the research to any changes. Ideally this process would involve external as well as project stakeholders, but as this was an experimental first go, we kept it within the project team.
Of course, without external stakeholders, internal biases are inevitable. We think we may have underestimated the contribution in favour of other external factors.
Initially we struggled to relate the six conditions to the context of SurveyMETER’s policy engagement in Balikpapan. A paper that really helped us understand the definitions of the conditions and facilitate the discussion was Assessing Advocacy by Barkhorn et al. (2013).
A ratio is a comparison of like quantities; estimating the benefits of policy research and engagement requires good access to data and the availability of financial or monetary values for both the benefits and the costs. It took some time for us to gather the data about the budget commitment of the municipality of Balikpapan. The lesson we learned is that if the decision is made to use the return on investment for a policy research and engagement project, it is better to start planning data collection at an early stage of the project.
As is often the case, commitment of the leadership of the institute has been a key enabling factor for putting in place a team and providing it with the space and resources to test (and learn) this and other tools for assessing the policy influence of policy engagement and research. This enabled the SurveyMETER team to gain some additional experience with qualitative research methods and storytelling that are relatively new to it.