Given the growing focus on PYD, researchers have emphasized the development of assessment tools to measure PYD effectiveness. A recent study by Michael Sieng, Scott Cloutier, and Katherine Irimata published in the Journal of Social Change discusses the development of a self-reporting survey tool known as the Positive Youth Development Sustainability Scale (PYDSS). The PYDSS expands on an earlier assessment tool known as the Positive Youth Development Inventory (PYDI) by focusing on sustainability of PYD approaches and universal application across various settings.
In this post, I summarize the article and offer my own thoughts on how the PYDSS can continue to improve the effectiveness of PYD programs.
The Five-Cs and Contribution
The structure of the 4-H PYDI assessment tool was influenced by Richard Lerner’s Five-Cs model identifying five core characteristics of youth development: competence, confidence, connection, character, and caring. The PYDI is based on data collection (in survey form as a 55-question Likert-scale questionnaire) from youth participating in PYD programs and measures youth perceptions of change in their Five-Cs through participation in the programs. Additionally, improvement in all Five-Cs leads to a sixth C known as contribution, which considers how youth contribute to their communities.
Happiness and Resilience as Measurements of Sustainability
In expanding the PYDI into the PYDSS, Sieng, Cloutier, and Irimata added a construct to measure happiness as studies have indicated that happiness may contribute to greater sustainability, and vice versa. Consequently, the authors argue that happiness is an important metric for assessing the effectiveness and sustainability of PYD programs.
The researchers also incorporated resilience metrics into the PYDSS, addressing the presence of support networks (such as teachers and parents) as a measure of the sustainability of youth development. The resilience metrics in the PYDSS were influenced by Richard Catalano and J. David Hawkins’ social development model that argues positive relationship networks suppress risk behaviors among youth. The PYDSS surveys both youth perceptions of their support networks as well as the perceptions of support networks themselves.
As the researchers noted, these happiness and resilience metrics enabled the PYDSS to provide a more in-depth approach to evaluating sustainability of PYD programs. In particular, I believe the PYDSS’s incorporation of resilience concepts underscores an important evolution in the evaluation of youth programs by incorporating input from social networks in the survey process.
Development and Testing of the PYDSS
Refining the Survey Questions: Literature Review
The literature was analyzed to develop the types of survey questions and statements that measure each category of the PYDSS: the Five-Cs, contribution, happiness, and resilience.
Ultimately, the PYDSS survey expands the PYDI’s 4-point scale of 1 (“strongly disagree”), 2 (“disagree”), 3 (“agree”), and 4 (“strongly agree”) to a 5-point scale including a “neutral/unsure” score. The total number of survey questions was also reduced to 32 instead of 55 under PYDI to increase clarity (Page 85).
Refining the PYDSS in Practice: Pilot Studies
Importantly, the researchers intentionally chose two programs with both similarities—such as their foundation of fostering PYD—as well as differences—such as their techniques and youth demographics—to test PYDSS’s applicability across different contexts.
Youth participants completed a PYDSS survey both before and after the intervention programs, as did each member of the youths’ support networks. Researchers collected 580 surveys in Thailand and 407 surveys in Phoenix (Page 84).
PYDSS researchers then used Exploratory Factor Analysis (EFA) to determine how well the pilot study survey items reflected the underlying constructs of the PYDSS variables. Principal components analysis and Kaiser’s eigenvalue rule were used to identify the number of factors to retain. Researchers then employed several fit indices based on the factor solution set obtained from the EFA using Confirmatory Factor Analysis (CFA) to evaluate the overall fit of the PYDSS. Model fit tests were conducted using both the Thai and Phoenix datasets to assess the PYDSS’s potential for application across different PYD programming and cultural settings.
Changes Made After the Pilot Studies
Analysis of the pilot studies indicated several questions should be removed to ensure greater model fit in both datasets to support global application.
For example, based on CFA results, the researchers removed survey items such as “I can manage my emotions” and “I have close friendships” due to low factor loadings and repetition in the given category, respectively. With the removal of these items, the PYDSS is now within statistical significance of most of the model fit tests used in the CFA for both datasets, demonstrating its ability to be used in multiple settings.
Strengths and Limitations
However, as noted by the researchers, one challenge with this—and any—evaluation tool is the time it takes away from actual programming. This survey on average took between 20 and 40 minutes to complete, while the ideal time would be between 10 and 15 minutes (Page 93). However, the authors highlighted that, given high Cronbach’s alpha values for both datasets, it would be possible to remove additional questions in future versions of the PYDSS to reduce the time required to complete the survey without adversely impacting result quality.
Areas for Further Research
The authors noted that future studies should further experiment with how the survey is administered in different project settings—such as verbally, in written form, or as a guide for interviews and focus groups. From my experience, the inclusion of qualitative (discussive) methods always strengthens evaluations of human-centered programs as personal narratives are vital to understanding a project’s impact on lived experiences.
The authors also stressed the importance of future studies testing the PYDSS across a range of grade levels, cultures, and countries beyond Brighter Thailand Foundation and Future for Kids to reaffirm the tool’s application in multiple settings.