It was going to be the next best thing to a Consumer Reports-like review of four of the most popular reading intervention programs: Spell Read Phonological Auditory Training (P.A.T), Corrective Reading, Wilson Reading and Failure Free Reading.
The $9 million Power4Kids study funded by the U.S. Department of Education, Haan Foundation and other funders promised to be a landmark randomized trial that met the “highest possible standards of quality” for research design and statistical analysis that would help educators make a more informed choice about when and how to use these programs.
Instead, after reading Closing the Reading Gap: Findings from a Randomized Trial of Four Reading Interventions for Striving Readers many administrators might wonder if they should invest in any reading intervention program at all given the disappointing results of this prestigious study.
“I have nightmares that people will interpret it that way,” says Joseph Torgesen, principal investigator of the study also known as Power4Kids and director of the Florida Center for Reading Research at Florida State University.
In the study of almost 800 3rd- and 5th-grade struggling readers (below the 30th percentile in reading) in the Pittsburgh area, intervention students scored below controls in the state test, the Pennsylvania System of School Assessment (PSSA), a year after the interventions. After an average of 90 hours in intervention programs with well-trained teachers, there was no improvement in comprehension for 5th graders, although there were improvements in decoding and other skills for the 3rd-grade cohort. In general, the study showed that interventions helped students in the 3rd-grade cohort more than students in the 5th -grade cohort.
Although it was an “extremely well-designed and well-implemented study, “Torgesen says, “with the results properly analyzed, it ended up not being exactly the study we wanted to conduct…” How should the study be interpreted?
Torgesen says it “underlines the idea that intervening with older students is a much steeper challenge” than many even imagine.
No idea of level of intervention needed
“Most educators and policymakers have no real idea about the level that will be required to do this (close the gap) for kids who have already been struggling with reading for three-five years,” Torgesen told Educational Research Newsletter. Many older students struggling with reading may need a second year in a reading intervention program because after 80-90 hours “they are just getting going,” he says.
Students in the four intervention programs spent an average of 90 hours in the programs, five days a week in groups of three students in sessions that were approximately 55 minutes long, according to the report. Torgesen says struggling readers in 3rd-grade and 5th-grade may need 90-minute sessions. Teachers received an average of 70 hours of training and professional development support in using the programs during the intervention year. Students were pulled out of the classroom for instruction.
Why did the study produce such lackluster and, in some cases, negative results for students in the intervention programs?
One reason, Torgesen says, is that the study was planned for the inner-city school district of Pittsburgh, but was carried out in a nearby suburb after the urban district decided not to dedicate time and resources to the study. The use of word-level interventions in the new venue likely did not match the students’ needs, he says.
“Unfortunately, the interventions we had already selected focused their instruction primarily on word level skills (decoding accuracy and fluency), so the instruction did not match the students most critical instructional needs, which lay more in comprehension skills in vocabulary,” he told ERN. “It caused me, if I’m going to be honest about this, to develop a more cautious attitude about doing word-level interventions, ” he says, “and I’ve done a lot of them in my career.”
Children with dyslexia have more problems with word-level reading, he says, but in the classroom, more struggling readers have problems with comprehension and vocabulary and thinking skills. If he were to do the study again, instead of focusing on word-level interventions, there would be one intervention for word level reading and comprehension and another for comprehension and vocabulary. (Three of the four interventions in the study were focused on word level and one, Failure Free, was considered a combination of comprehension and word-level skills). Only parts of Wilson Reading and Corrective Reading were used in the intervention because of the decision to focus on word level. Torgesen noted that motivation of children in such clinical studies is also always an issue.
A scientifically rigorous study
For the scientifically rigorous study, 50 schools from 27 school districts were randomly assigned to one of the four interventions; and within each school, eligible children in grades three and five were randomly assigned to a treatment group or to a control group. Eligible children scored at or below the 30th percentile on a word-level reading test and at or above the 5th percentile on a vocabulary test and were identified by their teachers as struggling readers. From an original pool of 1,576 3rd- and 5th-grade students identified as struggling readers, 1,042 met the test-score criteria. A total of 772 children who had a parent’s permission to participate were randomly assigned–558 to the treatment group and 214 to the control group.
Wilson Language Training which publishes The Wilson Reading System (WRS) made the following comments about the study: “The WRS instruction in the study focused on word-level skills only. WRS students in the study showed a statistically significant impact in this domain. This is noteworthy given that the control group generally had more positive gains than is normally expected according to the researchers.
“Regretfully, however, we agreed (at researcher’s request) to exclude the comprehension and vocabulary components of WRS in order to test a specific hypothesis about the impact of word-level instruction.”
The company also said that three of nine teachers who instructed Wilson groups were replaced in mid-year and that many were not part of the initial training and month-long practice that all treatment teachers completed.
“It is very important to properly group students for instruction,” the company wrote. “It is not recommended that students with adequate decoding skill are grouped with students who have more significant decoding issues for targeted instruction. Both the pacing and lesson focus should differ for these students.”
One of the dilemmas in education research, Torgesen says, is that there is pressure to use evidence-based instruction, yet the government and other funders are reluctant to set up “horse race” studies that compare performances of reading intervention programs, even though that is what many administrators want.
Closing the Reading Gap: Findings from a Randomized Trial of Four Reading Interventions for Striving Readers, “National Assessment of Title I Final Report. Volume II, October 2007.
Published in ERN March 2008 Volume 21 Number 3