Considerable effort has gone into perfecting the art of tablet data collection, which is the method typically used to collect data for evaluating education programs. The move away from paper has been a welcome shift, as for many research and evaluation professionals, paper conjures images of junior staff buried under boxes of returned questionnaires manually entering data into computers. Indeed, when our team recently began experimenting with paper-based data collection in our education projects, one colleague with decades of experience remarked warily, “It just seems like we’re moving backwards here!”
Improvements in the software, however, allow us to merge new technology with “old school” methods. Digital scanners can now replace manual data entry, powered by software that is able to read completed questionnaires, and quickly format responses into a data set for subsequent analysis. Our team has been experimenting with a new digital scanning software called Gravic to easily and quickly enter data from paper-based surveys. The Gravic digital scanning tool introduces flexibility and opens a new option for data collection across our projects, but not without some drawbacks. In this post, we make the case for paper surveys combined with the Gravic software and then review the drawbacks.
Why did we opt to use paper surveys and the digital scanning software?
While our typical work involves large-scale data collection to support sophisticated program evaluations, smaller projects or sub-components of larger projects may only require one-time, small scale, or targeted data collection. For example, a small project recently requested our help in evaluating outcomes from school and community-level trainings on gender sensitivity and sexual/reproductive health. The program was implemented in only four secondary schools in southern Nigeria, which were in close geographic proximity (although not in the same communities).
Because the program only involved a few schools, it was important to be able to administer questionnaires to all students and teachers in the school, in order to ensure a sufficiently large sample to meaningfully assess program impact with statistical validity. Sometimes when working in a small number of schools, we use group-level data collection, which entails an enumerator gathering responses from a group of students as a “quick and dirty” alternative to full-scale assessments. In this case, however, that wasn’t the right approach given our need to directly survey a large number of respondents.
In addition to the project in Nigeria, we have relied on paper-based data collection and Gravic in three other projects, while several more are in the planning stages. To our surprise, despite the buzz around ICT and mobile data in development, paper-based data collections have been popular among our project implementers and field-based M&E staff. The logistics, procurement, training and hiring involved in using tablets with a large number of enumerators introduces additional work that staff perceive as unnecessary or counterproductive for some projects. Mobile data collection tools are supposed to make data collection more efficient, but conversations with field staff revealed that sometimes, paper-based tools are actually more practical and efficient.
In addition, paper-based surveys can reduce certain types of biases. For example, programs evaluating sensitive outcomes such as gender-based violence or adolescent health outcomes are best served by data collection that doesn’t require interaction with an enumerator to record a response. Respondents may be more open to providing truthful answers on paper.
Drawbacks to using paper surveys and the Gravic software
From our experience, there are some drawbacks to the use of paper surveys and the Gravic software, which means that caution should be used in relying on paper for data collection.
- Limitations to question types: While sometimes practical and efficient, paper surveys limit the types of questions that can be asked. Interactive assessments, such as learning assessments that incorporate visual or auditory stimuli (metrics we typically collect with the mobile assessment software Tangerine), are not possible. These types of questions require an enumerator conduct an interview.
- Missing data: Using paper surveys usually means more missing data. While enumerators or respondents filling out a survey on a tablet are often required to enter a response (i.e., “Don’t know/refused”) before the next question is shown, paper allows the respondent to skip a question and still proceed with the survey. Surveys programmed into tablets are also often able to detect and prevent other errors in responses, which paper surveys cannot do. With paper-based data collection, our experience tells us to expect roughly 5-10% missing values on key demographic variables earlier in the survey and perhaps 10-15% on items later in the questionnaire.
- Digitization of surveys: Gravic, the scanning software used with paper surveys, requires careful scanning to properly digitize surveys. The software is highly sensitive to paper shape, size, and formatting, and minor changes in how the surveys are scanned can prevent the software from reading the documents properly. Careful training for field staff is required to ensure that the documents are digitized correctly.
Overall, while it’s unlikely that paper surveys will supplant tablets anytime soon, for certain types of evaluations they represent a reliable, cost-effective supplement or alternative. Coupled with Gravic’s digitization software, paper surveys offer a way to survey a large number of respondents very quickly at a low cost. As our team has learned, the more flexibility and options available for data collection, the more thoughtfully we can customize the support we provide to a project’s monitoring or evaluation needs.
