We are part of a team that recently conducted three virtual HIV Prevention Ambassador Trainers’ Workshops with more than 75 trainers from four countries, and a fourth workshop is planned later this month. From the outset, we knew how important monitoring and evaluation (M&E) of the training-of-trainer workshops would be, especially in a virtual setting. User feedback would be crucial to understanding whether our training methods were resulting in real impact. Through a comprehensive M&E approach, we hoped to learn what was working to prepare HIV Prevention Ambassador trainers to conduct their own trainings with adolescent girls and young women in their communities. We also wanted to know what was not working, and how we might create a better, more sustainable workshop.
What we measured
We chose the Kirkpatrick Model, developed more than 50 years ago, as a framework for our M&E plan. Ultimately, we wanted to learn what was working, what was not and how we might deliver better trainings with more sustainable impact. So, we settled on this model because of its succinct steps for analyzing that impact. Kirkpatrick focuses on four specific levels of evaluation: reaction, learning, behavior and results. The reaction level is measured by direct feedback from our participants. We also measured learning through pre- and post-test assessments. With the behavior level, we were interested in discovering how participants applied what they learned during our workshop when they began training HIV Prevention Ambassadors in their communities. Finally, the results level asks: what impact has our training had on the program? With each level, new information emerged.
In addition to our pre- and post-tests, we also conducted facilitator debriefs, participant feedback surveys and interviews to learn how we needed to adjust and improve our training program. Six months after each workshop, a post-training follow-up was done via virtual interviews with trainers, their supervisors and program managers. We gathered information about post-training activities (such as the number of training events held and number of ambassadors trained), the skills trainees are using or wish to develop, and ongoing support activities (such as office hours or mentoring groups) they have accessed or need.
What we learned
During our facilitator debriefs, we identified that participant performance was negatively affected – especially during the teach-back activities – by the short time participants were given to prepare. Participants expressed a need for greater support, guidance and orientation to the HIV Prevention Ambassador Toolkit package. To address these issues, we added a robust mentorship component, a package overview section and more detail about expectations for the teach-backs.
Based on feedback from participants and pre- and post-test results, we moved away from content focused on HIV prevention and pre-exposure prophylaxis (PrEP) since participants already had a high level of understanding in those areas. We shifted the curriculum toward training delivery and facilitation skills, managing gender-based violence, increasing self-care, and handling challenging participants and situations.
The post-training follow-up revealed that all trainers carried out ambassador trainings after the workshop. Trainers made further recommendations including:
- Post-training follow-ups to trainers needed to be the most robust in the first quarter after the workshop because trainers were not yet confident in their skills.
- Trainers expressed a need for support from master trainers during the ambassador trainings. They wanted assurance and guidance during the training to confirm that, as one trainer said, “I am doing the right job.” Some trainers noted they needed continuous interaction with master trainers until they were confident in their ability to execute the work.
- Some trainers expressed they struggled with simplifying concepts in the local language and felt their greater knowledge made it difficult to connect to ambassadors.
- Some suggested increasing sustainability by including district trainers in the training of trainers (cascading the role of training facilitator) because projects are time bound.
- Trainers mentioned that simple visual job aids could help ambassadors better understand the program and their roles.
Next steps
Nonetheless, we plan to continue capturing activities and outputs, such as number of ambassadors trained, ambassador training events that have occurred post-training, program impact and output, and outcomes such as increased skills and confidence of ambassador trainers in leading ambassador training, on a quarterly basis. Our M&E plan will help to identify inevitable challenges so that quick and effective solutions can be implemented.
Photo credit:Pangaea Zimbabwe AIDS Trust
Photo caption: Workshop facilitator conducting a session from her home office
A version of this post originally appeared on the EpiC blog. It is the eighth post in a series on the design, development, implementation and iterative processes of the interactive and virtual HIV Prevention Ambassador Trainers’ Workshop, led by the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR)- and the United States Agency for International Development (USAID)-supported Collaborative for HIV Prevention Options to Control the Epidemic (CHOICE) through the terms of cooperative agreements of the EpiC and RISE projects. In the series, we share our experience and reflect honestly on the ups and downs of designing and delivering a comprehensive training virtually.
