Northern Illinois University
STEM Interest and Engagement (STEM IE)

Experts have developed frameworks that define what quality entails in both formal and informal learning settings. In the Science, Technology, Engineering, and Mathematics Interest and Engagement (STEM IE) study, we collected data about quality using one of these frameworks. We then examined how quality contributed to youth experiences in the programs.

The term program quality in the field of afterschool is commonly attached to specific assessment tools that provide a definition of quality, allow program staff to self-assess how well they are meeting these criteria, and craft action plans to intentionally improve in specific areas. Common assessment tools include the Youth Program Quality Assessment (YPQA) and the Assessment of Afterschool Program Practices Tool (APT-O). Generally, these tools describe how to create developmentally appropriate settings for youth, where participants feel safe and supported and are afforded opportunities to form meaningful relationships, experience belonging, and be an active participant in their own learning and development. In some cases, these assessment tools include additional quality criteria for a particular content area like STEM (for example, outlining ways to improve scientific or mathematical reasoning, develop STEM-oriented inquiry skills, etc.). Examples of tools like this include the STEM version of the YPQA and the Dimensions of Success (DoS) tool. Both the Providence After School Alliance and Boston After School and Beyond used these tools to help program staff understand what constitutes quality in the design and delivery of summer programming. 

For the STEM IE study, our task was to assess the quality of programming that was being offered in the 15 minutes before a given experience sampling method (ESM) signal was issued. To support this task, we chose to use specific dimensions from the Upper Elementary Classroom Assessment Scoring System (CLASS). CLASS is typically used to support teacher growth and development in school-day classrooms. We opted to use it for the STEM IE study because it could be used to feasibly assess programming in the 15-minute segments we were working with before the ESM signal was issued. 

The portions of CLASS that we used assessed the extent to which activity leaders (ALs) provided opportunities and supports that pushed youth thinking and cultivated a deeper understanding of the content being explored. Quality was coded as higher when ALs

• provided clear learning targets, encouraged youth participation, and conveyed genuine interest and passion about activity content;
• focused on promoting youth understanding of key concepts and how they apply to the real world, while taking into consideration youth prior knowledge;
• provided opportunities for youth to examine and analyze information, modeled processes for solving problems, and provided opportunities for youth to plan and reflect on activities;
• provided scaffolding and feedback on youth work, encouraged persistence, and recognized youth accomplishments;
• promoted and supported discussions that build on one another and allowed youth to play a substantive role in moving the discussion forward; and
• maintained a productive learning environment characterized by few disruptions and efficient movement through planned activities. 

These practices are largely described in the Instructional Support domain of the CLASS tool.