In this post, we describe how after just 3 months of intervention, Grade 1 students participating in the Mahay Mamaky Teny (MMT) or “I know how to read” pilot program in Madagascar demonstrated greater gains than control students in reading outcomes. For this evaluation, not only do we analyze Early Grade Reading Assessment (EGRA) data but we also examine the effects of intervention dosage by integrating measures of fidelity of implementation of the MMT program.
Fidelity of implementation (FOI) refers to the extent to which an intervention is implemented as designed. In this case, we measured FOI with a tablet-based classroom observation tool to assess how well teachers implemented MMT’s teaching and learning materials. With this analysis, we show that the net causal effect of receiving the MMT program and implementing at a high level of fidelity is positive in boosting Grade 1 student reading outcomes after only 3 months of intervention.
Background
The goal of the MMT program was to provide the Ministère de l’Education Nationale (MEN) with key knowledge, tools and expertise to be capable of sustainably developing and improving a set of core inputs into an effective early grade reading instructional approach. It was implemented from January to July 2018 with school-based activities starting in April 2018.
The project accompanied the MEN to (1) develop and pilot teaching and learning materials (teachers’ guides and pupil books) designed to teach Grade 1 children how to read in the Malagasy language, (2) train Grade 1 teachers and headteachers in the use of these teaching and learning materials, (3) train headteachers to coach and supervise teachers, (4) train MEN staff in the design of impact evaluations and data collection instruments, and (5) train MEN staff in the use and administration of data collection instruments. In sum, the project used a “learning-by-doing” approach whereby project staff worked hand-in-hand with MEN staff throughout all phases of the project including both development and implementation of the early grade reading interventions but also monitoring and evaluation activities. This blog post presents the result of the 3-month evaluation (the pre-test was conducted in April 2018 and the post-test in July 2018).
Evaluation design
Working hand-in-hand with MEN staff, we designed an experimental evaluation of MMT, using a multi-stage cluster sampling strategy involving 60 treatment schools and 59 control schools across two Directions Régionales de l’Education Nationale (DREN) of Madagascar – Analamanga and Boeny. We sought to answer two main research questions:
- What is the impact of MMT on students’ performance in reading and are there differences by sub-groups (gender, socio-economic status)?
- What are the links between student learning outcomes and teacher behavior especially as it relates to implementation of practices promoted by MMT?
To do this, we employ a Difference-in-Differences methodology using EGRA scores and data collected through our formative evaluation tools used for our Collaborating, Learning and Adapting approach. In particular, we link student EGRA data with teacher classroom observation data to understand whether student reading outcomes are better in classrooms where teachers implement MMT with higher fidelity.
As we have discussed in this blog post, we believe that understanding the extent to which an intervention was implemented as designed (i.e., with fidelity) is key to understanding intervention effectiveness. More specifically, of the four elements of implementation fidelity discussed in this past blog post, our classroom observation tool measures adherence to the reading lesson plan.
The main measures of interest therefore include:
| Tool/Subtask | Measure |
|---|---|
| Early Grade Reading Assessment | |
| Letter sound identification | Number of letter sounds correctly identified out of 100 in 60 seconds |
| Syllable identification | Number of syllables correctly read out of 50 in 60 seconds |
| Familiar word reading | Number of familiar words correctly read out of 50 in 60 seconds |
| Oral reading fluency | Number of words in a reading passage read with accuracy in 60 seconds |
| Reading comprehension | Number of questions (out of 5) about a reading passage read by the student answered correctly |
| Classroom Observation Tool | |
| Adherence to lesson plan | Binary variable indicating whether teacher implemented most/all steps of the lesson plan or not |
Analytical framework
As mentioned we employ a Difference-in-Differences methodology and also disaggregate the overall treatment effect by levels of program fidelity.
The identification assumption of this approach is that, in the absence of treatment, students in pilot and control schools would experience the same changes in the outcomes of interest and, therefore, any differential change between the two groups can be attributed to the program. The inclusion of student fixed effects enables us to account for individual level heterogeneity in observable and unobservable factors that are time-invariant.
To answer research question #2, among the treatment schools only, we formally test whether teachers who complete most or all steps of the lesson plans exhibit larger reading gains than their counterparts who do not.
This approach allows to test for statistical significance whether following the lesson steps is associated with higher reading scores.
Findings
Key finding 1: Initial descriptive analyses show that across all EGRA subtasks, we observe a consistent pattern of EGRA performance gaps between the two groups in favor of the treatment group at baseline that gets larger by endline, the latter being the intended effect of the intervention. Figure 1 shows the distribution of the scores on the letter sound identification subtask at pre-test and post-test as an example.
Figure 1. Letter Sound Identification Subtask: Percentage of students for each score
Key finding 2: The average effect sizes for all subtasks except reading comprehension are large in magnitude for a 3-month intervention and range between .20 and .37 standard deviation for the full sample, with the largest estimated effect being in syllable identification.
Figure 2. Effect Sizes by Subtask and Student Subgroup
Key finding 3: 83% of teachers in treatment schools implemented the lesson plans with high fidelity and these teachers consistently had better student reading outcomes than teachers who did not implement with high fidelity.
For instance, in the table below, we see that average letter sounds per minute (lspm) scores grew by about 2.73 lspm more for the treatment students whose teachers did not implement with high fidelity relative to the control group. We also find that students whose teachers did implement with high fidelity score an additional 3.36 lspm. This means that relative to the control group, students whose teachers implemented with high fidelity scored 6.09 lspm higher (2.73+3.36 lspm).
Table 1. Regression results – MMT treatment effects on EGRA subtasks
Discussion and conclusions
This study, conducted between April and July 2018, served to evaluate the impact of approximately one-third of the entire reading and writing program (10-12 weeks of a 33-week program). Overall, the results presented are encouraging in the context of the short pilot period.
It is encouraging to see the large effect size of the Grade 1 MMT program on all subtasks, except reading comprehension, ranging between .20 and .37 standard deviation, with the largest estimated effect being in syllable identification. It is unsurprising that the program had no significant effects on student’s reading comprehension, given that most teachers had not yet reached the program lessons focusing on building reading accuracy, fluency and comprehension at the time of the post-test EGRA administration.
Further, our analysis shows that teachers who implemented most or all steps of the lesson plan obtained better student learning outcomes than teachers who did not. Standard lesson plan activity steps reflect the “I do, we do, you do” approach and allocate at least 60% of the whole lesson to the “you do” step or for individual student practice reading letters, syllables, words and/or text or writing while the teacher circulates to monitor and assist.
In Madagascar’s daily timetable for Grade 1, there are two 20-minute reading lessons per day and one 30-minute writing lesson, this means students should have approximately 24 minutes per day to practice independent reading and 18 minutes to practice their writing. Students were also allowed to take their books home to practice reading and writing. This focus on increased time allocation aligns with the research showing that extended and supportive reading practice is critical to the development of reading proficiency (Cipielewski and Stanovich, 1992).
Overall, most students in treatment schools were eventually afforded more time to practice their skills individually with support from the teacher. It is therefore no surprise that treatment school students showed significant improvements in the mechanics of reading (decoding).
Read the final endline evaluation report here.
Photo credit: AG Klei/USAID Madagascar
