Abstract

This study evaluates the effectiveness of the think-aloud method used during cognitive interviewing for survey-item testing with children. This study examines the application of a think-aloud technique during testing of a computer-based assessment, across two different grades and age-groups: grade 4 (age 9-10) and grade 8 (age 14-15). We compared the effectiveness of the think-aloud method between the two grades by evaluating the verbal reports length, time of response, and the quality and the completeness of the student responses. We found that the think-aloud method does not vary in the quality of the verbal output obtained from children in the two age groups. Although research suggests that young children typically find the think-aloud method to be difficult (Someren et al, 1994), this research shows that there is no difference in the effectiveness of the think-aloud method between students in Grade 4 and in Grade 8.

Introduction

High-quality survey data often requires laboratory pretesting of the survey questionnaires or educational assessments. During the last two decades, cognitive interviewing became one of the most widely used pretesting methods. Cognitive interviewing is intended to uncover problems with the questions that may compromise the quality of responses before the survey or assessment fielding. One of the procedures used during the cognitive interviews is the think-aloud that requires respondents to verbalize their thoughts as they answer survey or test questions.

There is little empirical research on the effectiveness of the think-aloud method utilization during the pretesting of surveys or assessments with children. Previous research has shown mixed results. Some reported that young children and teenagers lacked the ability or motivation to articulate their thoughts, while others showed that the think-aloud works well with teenagers but requires age-relevant instructions and more follow-up probing with younger children (Leeuw, et al. 2004; Someren et al, 1994).

Objectives

1. To evaluate effectiveness of the think-aloud procedure and quality of the verbal protocols produced by children in Grade 4 (ages 9-10) and Grade 8 (ages 14-15).
2. To compare the effectiveness of the think-aloud procedure and quality of the verbal protocols between children in Grade 4 (ages 9-10) and Grade 8 (ages 14-15).
3. To compare the ability to recognize and categorize the information from verbal reports into four processes (comprehension, retrieval, judgment, and response) identified the Question Response Model (Tourangeau, 2000) across grades.
4. To evaluate the ability to evaluate question processing at each of four response stages (comprehension, retrieval, judgment, and response) based on the information provided in the student think-aloud.

Methodology

• Ten 4th grade students and ten 8th grade students participated in cognitive interviews.
• Each student group completed 4 grade-specific science questions.
• Concurrent think-aloud procedure utilized during the cognitive interview.
• Students were provided with detailed think-aloud instructions, a demonstration, and a think-aloud practicing session before attempting to answer each question while thinking-aloud.
• Consistent think-aloud procedure and instructions across students ensured with a detailed script.
• The following four (4) factors were controlled for between 4th and 8th grade in terms of question type assignment:

o Subject: Science NCES Questions administered in 2009
o Content Classifications: Physical Science, Earth and Space Science, Life Science
o Question Difficulty: Easy, Medium, Hard
o Question Order: Presentation

Demographics

• A total of 20 students completed the cognitive interview. The sample consisted of a mix of SES, race/ethnicity (Table 1), and gender (Table 2).
• Controlled for experience and frequency of use of technology (tablet, phone, and computer) across grade 4 and grade 8 students.

Results

• There was no significant difference in the verbal report’s length (number of words) provided during the student think-aloud across questions between grade 4 and grade 8 students.
• We found a significant difference in the amount of time the student spent doing a think-aloud across student grades. Overall, grade 8 students took more time on their think-aloud than did grade 4 students.

Relevance of the Verbal Reports

• There was no significant difference in the relevance of the student think-aloud to the topic of the question between the two grades.
• Both 4th-grade and 8th-grade student’s think-aloud remained equally focused on the topic of the question.

Response Process Evaluation

• We examined whether there are differences in the ability to evaluate question processing at each of four response stages of the question response process (3-point Likert Scale: 1–Unable, 2–Somewhat, 3–Able).
• Overall, we found no significant differences between grades in the ability to evaluate information at each of the response stages.
• Results showed differences in the ability to identify and evaluate the judgment process between the two grades, where it was more difficult to assess judgment in 4th graders than in 8th grades.

Verbal Reports Completeness

• We examined whether there are differences in the completeness of verbal reports regarding the inclusion of information illustrating four stages of the question response process (3-point Likert Scale: 1–No, 2–Somewhat, 3–Yes).

• Results showed no differences between grade 4 and grade 8 students in their completeness of the verbal reports in terms of information relevant to comprehension, retrieval, judgment, and response processes.

Analysis

Student verbal reports were coded by three coders. The following measures were included to evaluate verbal reports quality and completeness:

• Verbal report length: number of words
• Duration of the think-aloud (seconds)
• Verbal report quality score based on the following criteria:
  1. Relevance of verbal reports content to the question tested on a 3-point Likert Scale (1–No, 2–Somewhat, 3–Yes).
  2. Completeness of the verbal report content in terms of including information illustrating four stages of the question response on a 3-point Likert Scale (1–No, 2–Somewhat, 3–Yes).
  3. Ability to evaluate question processing at each of four response stages: (1–Unable, 2–Somewhat, 3–Able).