Northern Illinois University
STEM Interest and Engagement (STEM IE)

The research study was unique because of the innovative methods used to collect data. We collected the reactions of youth as they were participating in activities while simultaneously videotaping and observing the program in real time. Much of the previous research on the attitudes and experience of youth participants in informal STEM programs has asked youth to report on their general attitudes and experiences after they participated in a program. The methods used in the STEM IE study permit reliable detection of systematic patterns in the experience of youth and more importantly, the identification of practices and approaches that are related to the presence of high levels of interest and engagement on the part of participating youth.

During the summer of 2015, data were collected in nine programs (five from one and four from another city-based afterschool intermediary). A total of 203 youth and 32 of the 33 staff members who had primary responsibility for them participated in the STEM IE study. The majority of the participants were rising sixth to eighth graders but a few rising fifth, ninth, and 10^th graders enrolled in the programs also participated in the study.

Data were collected using multiple methods

Pre & post surveys of youth characteristics and dispositions

An initial (pre) survey provided information about youth characteristics and their interest in STEM prior to participation in the summer program. At the end of the program, youth took a (post) survey on which they provided general feedback about the program, reported their future goals and aspirations, and their interest in STEM after participating in the summer program.

The Experience Sampling Method (ESM)

The Experience Sampling Method (ESM) was used to gather the perspectives of the participating youth during the classroom-based and/or the field-based STEM sessions youth participated in each week during the second, third, and fourth weeks of each program. We only time scheduled as STEM activity time. Participants were signaled via a mobile hand-held device to respond to a series of in the moment questions at four randomly selected time points during each STEM session. In response to the signal, students briefly recorded their perceptions about what they were experiencing just before they were signaled. Each participant was asked to rate how challenging, relevant, educational (how much are you learning), and autonomous (how much control do you have) their experience was at that moment and how engaged (composite of their reports of interest, enjoyment, concentration, and hard work) they were during that time.

Videotaping and Observations 

Videotaping and Observations captured the instructional practices being used by the primary activity leader during the same lessons in which ESM data were collected. We videotaped and observed STEM activity time, which included traveling to outdoor sites or field trips. Breaks, lunch, and scheduled time for recreation were not recorded and analyzed. A videographer focused the camera on the activity leader. After each lesson, trained observers completed a brief global rating scale characterizing the nature of the class session they observed in terms of behavioral indicators of engagement and entered notes about the session. The videotape was coded by trained staff and yielded measures of program quality, activity leader support for youth agency, and the extent to which activity leaders promoted the relevance of the STEM content.

Surveys of Observed STEM Activity Leaders

Surveys of Observed STEM Activity Leaders included questions about their backgrounds, perceptions of effective practices, and sense of competence in designing and delivering STEM activities.

Interviews of Observed STEM Activity Leaders

Interviews of Observed STEM Activity Leaders included their descriptions of the programs and participants and included review and discussion of selected video clips of lessons to determine their perspectives.