Introduction

International donors and community groups are increasingly investing in programs centered around Positive Youth Development (PYD)—a project approach that seeks to connect young people to their communities with a focus on skills advancement and leadership opportunities. For example, during my time as a Youth Policy Development Intern with the U.S. Agency for International Development (USAID), I took part in USAID’s ten-year update of its Youth Policy which heavily emphasized the concept of PYD. Now, as an intern at FHI 360, I have seen how this approach has been put into practice on the ground with various projects and activities implemented by FHI 360.

Given the growing focus on PYD, researchers have emphasized the development of assessment tools to measure PYD effectiveness. A recent study by Michael Sieng, Scott Cloutier, and Katherine Irimata published in the Journal of Social Change discusses the development of a self-reporting survey tool known as the Positive Youth Development Sustainability Scale (PYDSS). The PYDSS expands on an earlier assessment tool known as the Positive Youth Development Inventory (PYDI) by focusing on sustainability of PYD approaches and universal application across various settings.

In this post, I summarize the article and offer my own thoughts on how the PYDSS can continue to improve the effectiveness of PYD programs.

Background

The Five-Cs and Contribution

PYDSS is adapted from the PYDI assessment tool developed by Mary Arnold et al. for the 4-H program, the largest youth mentoring program in the U.S.

The structure of the 4-H PYDI assessment tool was influenced by Richard Lerner’s Five-Cs model identifying five core characteristics of youth development: competence, confidence, connection, character, and caring. The PYDI is based on data collection (in survey form as a 55-question Likert-scale questionnaire) from youth participating in PYD programs and measures youth perceptions of change in their Five-Cs through participation in the programs. Additionally, improvement in all Five-Cs leads to a sixth C known as contribution, which considers how youth contribute to their communities.

Happiness and Resilience as Measurements of Sustainability

In expanding the PYDI into the PYDSS, Sieng, Cloutier, and Irimata added a construct to measure happiness as studies have indicated that happiness may contribute to greater sustainability, and vice versa. Consequently, the authors argue that happiness is an important metric for assessing the effectiveness and sustainability of PYD programs.

The researchers also incorporated resilience metrics into the PYDSS, addressing the presence of support networks (such as teachers and parents) as a measure of the sustainability of youth development. The resilience metrics in the PYDSS were influenced by Richard Catalano and J. David Hawkins’ social development model that argues positive relationship networks suppress risk behaviors among youth. The PYDSS surveys both youth perceptions of their support networks as well as the perceptions of support networks themselves.

As the researchers noted, these happiness and resilience metrics enabled the PYDSS to provide a more in-depth approach to evaluating sustainability of PYD programs. In particular, I believe the PYDSS’s incorporation of resilience concepts underscores an important evolution in the evaluation of youth programs by incorporating input from social networks in the survey process.

Development and Testing of the PYDSS

Refining the Survey Questions: Literature Review

The authors conducted a literature review of PYD theory and subjective well-being (i.e., happiness) concepts using six electronic databases. Common themes and findings from each database were used to develop the PYDSS survey.

The literature was analyzed to develop the types of survey questions and statements that measure each category of the PYDSS: the Five-Cs, contribution, happiness, and resilience.

Ultimately, the PYDSS survey expands the PYDI’s 4-point scale of 1 (“strongly disagree”), 2 (“disagree”), 3 (“agree”), and 4 (“strongly agree”) to a 5-point scale including a “neutral/unsure” score. The total number of survey questions was also reduced to 32 instead of 55 under PYDI to increase clarity (Page 85).

Refining the PYDSS in Practice: Pilot Studies

The researchers tested and further refined the PYDSS at two programs: Brighter Thailand Foundation and Future for Kids. Brighter Thailand Foundation hosts a global leadership camp for high school and primary school students throughout rural areas of Thailand. Future for Kids is an out-of-school-time program in Phoenix, Arizona focused on academics, athletics, and ethics activities.

Importantly, the researchers intentionally chose two programs with both similarities—such as their foundation of fostering PYD—as well as differences—such as their techniques and youth demographics—to test PYDSS’s applicability across different contexts.

Youth participants completed a PYDSS survey both before and after the intervention programs, as did each member of the youths’ support networks. Researchers collected 580 surveys in Thailand and 407 surveys in Phoenix (Page 84).

PYDSS researchers then used Exploratory Factor Analysis (EFA) to determine how well the pilot study survey items reflected the underlying constructs of the PYDSS variables. Principal components analysis and Kaiser’s eigenvalue rule were used to identify the number of factors to retain. Researchers then employed several fit indices based on the factor solution set obtained from the EFA using Confirmatory Factor Analysis (CFA) to evaluate the overall fit of the PYDSS. Model fit tests were conducted using both the Thai and Phoenix datasets to assess the PYDSS’s potential for application across different PYD programming and cultural settings.

Changes Made After the Pilot Studies

Analysis of the pilot studies indicated several questions should be removed to ensure greater model fit in both datasets to support global application.

For example, based on CFA results, the researchers removed survey items such as “I can manage my emotions” and “I have close friendships” due to low factor loadings and repetition in the given category, respectively. With the removal of these items, the PYDSS is now within statistical significance of most of the model fit tests used in the CFA for both datasets, demonstrating its ability to be used in multiple settings.

Analysis

Strengths and Limitations

The authors noted various strengths of the PYDSS. First, analysis indicated the PYDSS has Cronbach’s alpha reliability statistics of .951 (Thailand) and .952 (Phoenix), which is far greater than the generally-accepted reliability minimum of .7 (Page 86). The authors also highlighted that the PYDSS allows for modification, such as the inclusion of program-specific questions, so that this evaluation tool can be used as a baseline assessment of all PYD programs.

However, as noted by the researchers, one challenge with this—and any—evaluation tool is the time it takes away from actual programming. This survey on average took between 20 and 40 minutes to complete, while the ideal time would be between 10 and 15 minutes (Page 93). However, the authors highlighted that, given high Cronbach’s alpha values for both datasets, it would be possible to remove additional questions in future versions of the PYDSS to reduce the time required to complete the survey without adversely impacting result quality.

Areas for Further Research

The PYDSS was developed to be universally applicable. As such, translation and cultural adaptation are important considerations when applying this assessment tool to PYD programs. Therefore, the researchers emphasized that future studies should consider how to best develop a qualitative or mixed-methods approach to PYDSS data collection for specific PYD projects. For example, when responses to the PYDSS questions during the pilot process in Thailand were gathered via focus groups and interviews, participants were often more open to sharing personal experiences.

The authors noted that future studies should further experiment with how the survey is administered in different project settings—such as verbally, in written form, or as a guide for interviews and focus groups. From my experience, the inclusion of qualitative (discussive) methods always strengthens evaluations of human-centered programs as personal narratives are vital to understanding a project’s impact on lived experiences.

The authors also stressed the importance of future studies testing the PYDSS across a range of grade levels, cultures, and countries beyond Brighter Thailand Foundation and Future for Kids to reaffirm the tool’s application in multiple settings.

Sharing is caring!