My personal interest in evaluation ethics goes back to my days at MDRC, where I was responsible for developing survey questions and accompanying protocols to capture domestic violence among mothers who participated in a DHHS-funded welfare-to-work program called JOBS (Job Opportunity and Basic Skills). MDRC was about three and a half years into a five-year evaluation of JOBS when our program officer asked us to include questions specifically about domestic violence in the next wave of our survey. As I recall, no one wanted to touch this – too sensitive, too volatile, too many ethical loopholes to jump through – so, as the lowest rung in the food chain at the time, I was given the task. With that, I entered the world of evaluation ethics where I learned quickly the challenges in getting it right, and the consequences of getting it wrong.
Reflecting on that experience, I felt an immediate sense of recognition – almost déjà vu, if you will – as I came upon three excellent articles on ethics in evaluation featured in a recent issue of the Journal of Development Effectiveness (Volume 8, 2016 issue 4). Although tailored for a global development audience, and presented within an international development political and organizational context, the articles contain lessons for all evaluators and everyone involved in the evaluation process. The authors describe many scenarios that can lead to lapses in ethics, but here I’ll focus on three: lack of overall consensus in defining ethics; conflicts between research goals and organizational (or funder) priorities; and a results-based organizational mindset that turns an environment of learning and improvement into an “audit culture.”
Leslie Groves Williams describes the consequences of failing to articulate what constitutes ethical behavior in evaluation. It starts with cognitive dissonance: we think we know what ethics is, we’re sure that everyone else knows what it is, we think that we all agree on what it is, so no one needs to explain what it is. We think that knowing right from wrong, or knowing what is legal or illegal, is plenty. In addition, ethics is often relegated to the evaluators: “That’s not my problem: that’s your job.” These unspoken assumptions make it difficult to build a common language around ethics that can be understood and appreciated by everyone involved in an evaluation effort. Here, “everyone” includes the evaluators but also program developers, ministries/government agencies, funders, stakeholders and, most importantly, study participants. The solution, according to Groves Williams, is to build a multi-sector consensus and shared language about ethics. This effort will create buy-in across and within sectors, which will result in a culture that embraces ethics itself as part of evaluation; as well as a collective, or body of sectors, in which every party is accountable for ensuring that ethics is front and center at all stages of any given study. Put another way, everyone knows that ethics is their job, too.
Peter O’Flynn and colleagues discuss how the dual visions of many evaluations – to inform the field as well as to serve the goals of the organization (which I will use as a catch-all for program managers, developers, and funders) that commissioned the study – can impact ethics in evaluation. In the former, the evaluators seek to increase knowledge; accordingly, they treat evaluations as rigorous research projects that require IRB oversight. With the latter, organizations use evaluation as a project management tool, designed for monitoring program activities and guiding program improvements. These visions can and often do co-exist: the due diligence of the IRB protects study participants from harm, and the organizational perspective provides important context to the study. But, when they compete for primacy, what results are inconsistencies in how evaluation ethics is viewed and applied. For example, organizations might feel hamstrung by the IRB regulations, which they see as obstructing program activities. Or, an organization’s need for program data can compromise human subjects protections, as in requiring an evaluator to release personally identifiable information. Thus, the lines between evaluation and project management become blurred: Who controls the study – the evaluator or the organization? How much autonomy should organizations give their evaluators? In the end, whose interests are being served – the people that the program is supposed to help, or those who developed and/or funded the program?
Caitlin Scott takes the theme of competing priorities a step further by describing what happens when organizations adopt a “results-based” approach to evaluation: mainly, that a culture of learning and improvement is replaced with what she calls an “audit culture.” It typically starts when an organization hires an evaluator to collect some hard data to confirm its assumptions regarding the positive impacts of its programs: “I know it works! We’ve been running this program for years. Our clients love it. Can you get me some numbers that I can show to my board?” Predictably, they get nervous when evaluators find that a program lacks implementation fidelity and/or positive impacts. (Show of hands: how many of you have ever been asked to re-analyze the data, collect more data, or tweak the “findings” section in the report?) In a best-case scenario, negative findings are treated as learning tools to improve the program. But in a worst-case scenario, organizations that rely heavily on metrics and results treat such findings as threats because so much is on the line: program sustainability, funding, professional reputation and even employment. For evaluators, it might mean the loss of a client or even the breaking of a contract for cause, or the loss of an opportunity to use the lessons to inform the field. With so much at risk, it’s not difficult to imagine how ethics might take a back seat to self-preservation or simply “keeping the customer satisfied.”
I encourage everyone who is interested in ethics to read the articles because they contain far more than I can summarize here. If you’re an evaluator, you will find little that is new. Indeed, you might get the same sense of recognition that I did and finding yourself saying, “Yeah, that happened to me,” “I know someone who did that,” or “Ha! Now I know why I felt uneasy about [fill in the blank].” I’ll bet that we all have at least one ethics-related horror story. Here’s mine: I was ordered, by an organization with a strong audit culture, to report “alternative facts” (sorry, but the phrase fits here) on their program outcomes. I quit shortly thereafter. So, for me, these articles confirmed what I’ve long felt and experienced but not quite articulated as well as the three authors: every party involved in an evaluation should be held accountable for ethical practices, and to be mindful of competing priorities and develop strategies for reconciling them.
And the domestic violence questions and protocols? MDRC gave me free rein to engage academics, survey methodologists, lawyers, criminologists, case workers, domestic violence advocates and survivors, etc., to help me design the most psychometrically and ethically sound tools possible. In the end, the clients were happy with the product, as were several other welfare-to-work program evaluators who adopted the questions and protocols for their own studies. My job was done, and we were able to get it right.
Photo credit: Mike Goad/Public Domain Mark 1.0
