Integrated development is an approach that employs the design and delivery of programs across sectors to produce an amplified, lasting impact on people’s lives. Integrated programs are based on the premise that the interaction between interventions from multiple sectors will generate benefits beyond a stand-alone intervention. As human development interventions take this more holistic approach, funders and program implementers alike recognize the importance of understanding the impact of multi-sector interventions. While we can continue to use sector specific measures of impact – for instance, Disability-Adjusted Life Years (DALYS) – this creates an apples and oranges problem if one wishes to compare across interventions. It begs the question: can we move towards a single performance metric to assess effectiveness and cost-effectiveness of integrated programs? Here at FHI 360, we are attempting to answer this question by developing a new measurement tool – MIDAS (Measuring the Impact of Development Across Sectors).
In this post, we discuss the need for better measures, describe our conceptual framework, and then present some of the key components of the tool. We conclude by demonstrating how the tool worked when piloted in a current FHI 360 project.
Currently available performance measures
Traditionally, multi-sectoral development indices report at the population level and combine measures of health (life expectancy), education (years of schooling) and socio-economic status (per capita income) into a single metric, e.g., the UN Human Development Index. These types of indices are unitless and allow us to rank order countries. While rankings are useful for cross-national comparisons, a 10% change in such an index is not interpretable and it fails to provide a measure specific to sub-populations or persons reached by a specific program or set of interventions.
The Poverty Probability Index (PPI®), formerly called the Progress out of Poverty Index, by Innovations for Poverty Action (IPA) is a new single dimension index. It provides a quick assessment of the likelihood that a household is living below or near the poverty line in 60 countries (see an overview here). This measure is well-suited to assess how well a program or intervention is reaching its intended target group (people in need) but is less well-suited to assess changes in an individual’s risk of poverty due to participation in a program or intervention. [1]
In the health sector, the most widely accepted performance metric is DALYs. This measure captures the impact of both morbidity (decrease in functional capacities) as well as mortality (reduced life-expectancy). Therefore, it is applicable to any health intervention that produces changes in morbidity and/or mortality among the recipients of the intervention.
Education performance metrics often focus on literacy and/or numeracy and are generally adjusted for age (for a more complete list of possible indicators see here). Standardized tests measure these skills, which limits the ability to capture this information outside of a formal education setting. Functional literacy/numeracy assessments exist for use outside of formal education settings, but those data are not widely available in public data sets.
The challenge therefore is to try and build from the strengths of the single dimension indices in such a way that summarizes multiple sector impacts into an easily interpretable measure of impact. Ideally such a measure will not require the application of a lengthy assessment tool but instead leverage easily obtained information from intervention recipients.
A new measure for integrated development
MIDAS is a new Excel-based measurement tool to model the impact of multi-sector interventions. It employs a combination of DALYs and a newly developed measure, Poverty-Adjusted Life Years (PALYs), to estimate the impact of interventions in one, cross-sector comparable metric. MIDAS is being developed with internal funding from FHI 360 and is currently a prototype model in the beta testing phase.
Some of the most common spheres of human development interventions are health, education, and economic development. Many interventions attempt to protect individuals from the negative effects of poor health, limited education, and/or poverty or attempt to assist individuals to improve their potential with respect to health, education, and economic security. We developed MIDAS based on the conceptual framework shown below, which depicts the six factors that we have begun to operationalize via modelling.
Figure 1: Conceptual framework
Impacts 1–3 show the direct effects of better health, better education, and increased economic opportunity on wellbeing, and correspond to the direct change in outcome due to each component addressed by the program or intervention. Relationships 4–6 depict the “spillovers” between sectors, and represent the potential for increased impact due to an integrated intervention.
Our goal is to build a tool that allows a user to enter readily available project and other data common to most international development programs. By avoiding the need for an additional formal questionnaire, the tool should be easy to use by project implementers. Examples of required data include: location of the program, number of persons reached by interventions, recipient population characteristics (age, gender, household size), and relevant outcome data from two time periods (ideally baseline and endline) such as household income or prevalence of health risk factors.
To date, we developed the beta version of MIDAS to be used by program monitoring and evaluation (M&E) staff to estimate impacts 1 and 3, and relationship 6 from the figure above for sub-Saharan Africa. The goal of the beta version was to develop and test the user-interface and get feedback on the display of impact results. For expediency, we have used publicly available country- and household-level data sets to generate initial parameter estimates used in the beta version of the tool.
Methods – better health
For the direct impact of health interventions (impact 1), we use DALYs averted. As described above, a DALY is a measure of the burden of morbidity and premature mortality in terms of years of life. Our DALYs averted model uses changes in the prevalence of selected health risk-factors among program recipients to estimate the impact of the program.
The DALYs averted model in MIDAS is based on incremental impact, or the difference between the burden of disease before the intervention and after the intervention, which is a similar approach DALYs averted models use in other impact estimation tools (Winfrey et al., Yang et al.). In general, DALYs averted models are calculated based on reduction of risk factors among the population of interest, namely the risk factors already modeled as part of the GBD study. Based on risk-factor specific health data and population-level estimates of population and prevalence, the GBD estimates the reduced risk of morbidity and mortality due to reduction in each risk factor by gender. These estimates can be used in the development of models tailored to different risk factor reduction. For more about the Global Burden of Disease study see this recent blog post by Tim Mastro.
The key parameters needed to model the gains from reducing a given risk factor are burden of disease attributable to the risk factor, prevalence of risk factor, and demographic data related to population size of the population of interest. Here is an example of how our DALYs averted model estimates the impact of reducing the prevalence of tobacco use among a sample of men benefiting from an intervention. Table 1 shows the parameters and data sources in the tobacco use model for males.
Table 1: Parameters and data sources in the smoking model for men in sub-Saharan Africa (SSA)
The model starts by using DALYs due to smoking, population size, and prevalence of smoking to estimate the disability weight, which is the reduction in disease burden per man who quits smoking. The model then estimates the number of people who quit smoking due to the intervention using the change in prevalence and the number of participants. Finally, the model calculates DALYs averted by applying the disability weight to the number of people who quit and then multiplying by the life expectancy minus the average age. In the example in table 1, there were 500 participants with an average age of 23, and at the beginning of the intervention they were all smokers. By the end of the intervention, 60% had stopped smoking. Applying the current model to this example data, the DALYs averted due to this intervention are 1,306.
We will further refine this component of the tool by introducing discrete time discounting as described by Bruce Larson for each DALY averted model. Discrete time discounting accounts for diminishing intervention affect over time and adjusts for age-specific life expectancy of an individual.
Methods – increased economic opportunity
For the direct impact of socio-economic development interventions (impact 3), we built the tool to estimate the number of PALYs averted. PALYs reflect the cumulative annual risk of living below the international poverty line of $1.90 per day summed across the individual’s remaining years of life. In MIDAS, PALYs averted equals the estimated probability of poverty over the life-course based upon the baseline measure of household income minus the estimated years probability of poverty over the life-course based upon the endline measure of household income. By aggregating the differences over the remaining life-course, we generate the PALYs averted associated with the change in household income.
Our current parameter estimates in the tool are based on a series of cross-sectional estimates of poverty risk (probability of living on less than $1.90 per day) for sub-Saharan Africa from UNStats. We used these data to create a synthetic modal person in year 2002 and then treated the subsequent cross-sectional estimates as reflecting future poverty risk for this modal person as he/she aged from 35 to 45 years old. Based on these poverty risk estimates, we then estimated values for the risk of poverty for the intervening years assuming a steady growth/decay rate between subsequent time points. Finally, we regressed the risk of poverty as a function of age and age-squared over the 10-year interval to estimate poverty risk over the life course from age 15–75. These estimates are then used to generate a risk of poverty profile for individuals as a function of income with and without the benefit of the socio-economic development intervention. The difference between the two poverty risk distributions (area between the curves) yields our estimate of PALYs averted.
Figure 2. Example of a PALYs averted estimation
Figure 2 shows the example of how the model estimates the PALYs averted for a 35-year-old living at the poverty line at baseline and experiencing an increase of income to 110% of poverty at age 36 due to the intervention. The area between the curves (PALYs averted) = 3.60 or a gain of just over 3.5 years of life above the poverty line.
In subsequent versions of the tool we plan to improve this component by incorporating individual-level data into our parameter estimates and, if possible, generate estimates for groups of countries stratified by economic development rather than generating a single sub-Saharan Africa estimate.
Methods – interaction between better health and increased economic opportunity
To include the interaction between health and wealth in the tool (impact 6), we use data on both DALYs and household income to estimate the relationship between the two. We created a simple analysis data set using World Bank Open Data comprised of country-level data on per capita GNP and per capita DALYs. As suggested by the figure above we assume that health can impact wealth and that wealth can impact health. This introduces a problem of simultaneous causality (or endogeneity). See Bloom and Canning for a discussion of the interactions between demographic transitions and economic growth. Because of this, a simple regression of one impact on the other yields biased estimates of the coefficients. To solve this problem, the standard approach is to use instrumental variable techniques to estimate the independent variable of interest and use these predicted values in a two-stage least squares regression to generate an unbiased estimate of its relationship to the dependent variable of interest. Unfortunately, our simple analysis data set is insufficient to support this analysis.
Given these data limitations, we used simple regressions to model the relationships between health and income and estimate the expected impact of changes in household income or health risk factors on the risk of poverty over the life course (PALYs averted) and the risk of morbidity and mortality over the life course (DALYs averted) in our beta version of the tool. At this stage of development, the goal was to test having target users enter readily available data from their program and let the tool apply the embedded models to produce sector-specific estimates of program impact. We are developing a richer data set containing individual-level data on DALYs, wealth, and education to support the proper instrumental variable and two-stage least squares estimation of interactions in the model for future versions.
Results of pilot application
This beta version of MIDAS has been tested by M&E staff from FHI 360’s Community Connector Project in Uganda using existing M&E data. Community Connector (CC) was a USAID-funded project operating in 15 districts in Uganda. The project sought to improve the nutritional status of women and children through an integrated suite of interventions focused on nutrition and health; agriculture and food security; water, sanitation, and hygiene (WASH); gender; and economic livelihoods. The results shown below are calculated using pre-intervention and post-intervention data on household income from 76,140 households and maternal and early childhood health risk prevention interventions delivered to 997 individuals. These calculations do not use information captured directly about other components of the program (e.g., agriculture, food security, WASH and gender).
Prior to the use of the tool, the project could report separately on a series of 32 performance indicators (outcome and process measures) in their Activity Monitoring, Evaluation and Learning Plan (AMELP). Many of these measures were cumulative counts of persons reached or referred to other programs. However, a few of these indicators seemed well suited to the application of our estimation tool: change in average household income, change in prevalence of anemia among women of reproductive age, change in prevalence of exclusive breastfeeding among children under 6 months measured at baseline, mid-point and at endline.
We asked the M&E team in Uganda to populate the tool using their database. The only additional analysis required was to stratify the household income data by age cohorts and compute breastfeeding prevalence by infant gender. With these data manipulations, the team populated the tool with their data and generated the results dashboard shown in figure 3.
Figure 3: Results dashboard
The top panel reports the two impact measures by intervention component and shows the direct and indirect effects of the two intervention components. The middle panel shows these same results graphically but now instead of grouping by intervention, groups the results by impact measure. One challenge in interpreting these charts is the different numbers of persons reached by the health intervention (n = 997) vs. the economic intervention (n = 376,052 in 76,140 households). Therefore, the last panel highlights the impacts at the person level. Here we can easily see that the direct effects of the interventions are larger than the indirect effects, that total DALYs and PALYs averted per person are roughly equivalent, and that the indirect effect of health interventions are much larger than the indirect effect of the economic interventions. Part of this is due to the health interventions targeting women of reproductive age and infants who have a longer life-expectancy over which to reap the benefit of the health risk reduction interventions.
The feedback from the program staff indicated they found the tool easy to populate with required data and that they would have liked to have a more comprehensive list of health-related risk behaviors to select from to reflect more of their program activities in the health sector.
Future refinements
Based upon our experience with the beta version, we have identified further refinements to improve the tool.
Future work will include adding models for components 2, 4 and 5 to the tool to further explore the role of education in wellbeing. Furthermore, we will seek to combine the measures used in the tool into a positive metric we are calling Wellbeing-Adjusted Life Years gained (WALYs).
Once we have addressed any measurement challenges and revised the tool, a final step will be to develop a web-based user interface where virtually any program (FHI 360 or otherwise) can access to input program specific data on size, scope, sectors, as well as changes in key program outputs to generate estimates of program outcomes and program impact. This is a critical component designed to increase the visibility and utility of this approach to the larger development community. We expect that once we engage with the larger development community, there will be further refinements to the approach.
Conclusion
We asked the question – can we move towards a single performance metric to assess effectiveness and cost-effectiveness of integrated programs? We feel that MIDAS can do just that. Moving towards a life-course focused, unified, multi-sector impact measure allows us to start examining multi-sectoral impacts on wellbeing. A method to measure wellbeing is directly related to the desire to promote person-focused, evidence-based development interventions. We are currently looking for additional resources to accelerate development of this tool and introduce it to the larger development community.
What do you think?
Comment below to let us know what you think of MIDAS, and how you would use it in your work.
Photo credit: 2014 Alexaya Learner/ GlobeMed at UCLA, Courtesy of Photoshare
###
[1] The authors updated this sentence on January 5, 2018.