- Intervention: A school-wide reform program, primarily for high-poverty elementary schools, with a strong focus on reading instruction.
- Evaluation Methods: A large, multi-site randomized controlled trial.
- Key Findings: The program increased second-grade reading achievement in Success for All schools by 25-30% of a grade-level, three years after random assignment.
- Other: Strong evidence of effectiveness applies to the program as implemented in grades K-2 (as opposed to later elementary school). Per-student program cost is low. Longer-term study follow-up would be desirable to see if effects continue beyond second grade.
Download a printable version of this evidence summary (pdf, 4 pages)
Finding of the Top Tier Evidence Initiative’s Expert Advisory Panel
Success for All meets the Top Tier Evidence Standard, defined by recent Congressional legislation to include: Interventions shown in well-designed and implemented randomized controlled trials, preferably conducted in typical community settings, to produce sizeable, sustained benefits to participants and/or society.
Description of the Intervention
Success for All is a comprehensive school-wide reform program, primarily for high-poverty elementary schools, with a strong emphasis on early detection and prevention of reading problems before they become serious. Key program elements include: (i) daily 90-minute reading classes, each of which is formed by grouping together students of various ages who read at the same performance level; (ii) a K-1 reading curriculum that focuses on language development (e.g., reading stories to students and having them re-tell), teaching students the distinct sounds that make up words (i.e. phonemic awareness), blending sounds to form words, and developing reading fluency; (iii) daily one-on-one tutoring (in addition to the classes) for students needing extra help with reading; and (iv) cooperative learning activities (in which students work together in teams or pairs) starting in the grade 2 reading classes.
The program costs approximately $120,000 per elementary school (for implementation in grades K-5) in the first year, $55,000 in the second year, and $45,000 in the third year, in 2008 dollars. These costs include materials and training; schools may incur additional costs of reallocating staff from other functions (e.g., to provide a higher ratio of tutors).
EVIDENCE OF EFFECTIVENESS
This summary of the evidence is based on a systematic search of the literature, and correspondence with leading researchers, to identify all well-designed and implemented randomized controlled trials of the Success for All school-wide reform program. Our search identified one such trial, summarized below. Importantly, this trial evaluated the program as implemented in grades K-2 but not grades 3-5; thus, its findings apply only to the K-2 elements of the program.
Overview of the Study Design: Large, multi-site randomized controlled trial evaluating Success For All in a sample of 41 high-poverty elementary schools across 11 states, during 2001-2006.
Prior to random assignment, at least 80% of the schools’ teachers had voted in favor of adopting Success for All (a step that the program provider typically requires before working with a school), and the schools had agreed to allow data collection over the course of the study. The 41 schools were randomly assigned either to a group that implemented Success for All in grades K-2 or a control group that did not (most implemented the program in grades 3-5 instead). The schools contained a total of 2,694 entering kindergarten students administered a pretest at the start of the study. The student population in these 41 schools was 56% African-American and 10% Hispanic, and 72% of students were low-income (i.e., eligible for federally subsidized lunches).
Approximately three years after random assignment, the study assessed reading outcomes for all second-grade students in the sample schools. Sixty-nine percent of these students had been exposed to Success for All, or the control condition, for all three years of the study (i.e., in grades K-2); the other 31% had enrolled in the Success for All or control schools during the study, and so had received partial exposure.
Effects of Success for All on school-wide second-grade reading outcomes, three years after random assignment (versus the control schools):
These are the effects on all outcomes that the study measured at the three-year follow-up. All effects shown are statistically significant at the 0.05 level unless stated otherwise.
On average, second graders at Success for All schools –
- Scored higher in passage comprehension than approximately 58% of their counterparts at control group schools (this equates to a standardized effect size of 0.21).
- Scored higher in word identification skills than approximately 60% of their counterparts at control group schools (this equates to a standardized effect size of 0.24); and
- Scored higher in word attack skills than approximately 64% of their counterparts at control group schools (this equates to a standardized effect size of 0.36).
- To express these effects as grade level equivalents: On average, second graders at Success for All schools score approximately 25-30% of a grade level higher in reading ability than their counterparts at the control schools.1
- The program’s effects generally grew in size from the first to the third year of the study (the earlier, smaller effects were mostly not statistically significant) –
- Passage comprehension: From an effect size of -0.10 in year 1, to 0.12 in year 2, to 0.21 in year 3.
- Word identification skills: From 0.09 to 0.19 to 0.24.
- Word attack skills: From 0.32 to 0.29 to 0.36.
(The earlier, smaller effects were mostly not statistically significant.)
Discussion of Study Quality
- This was a large, multi-site study evaluating Success for All as it is typically implemented in high-poverty elementary schools, thus providing evidence about the program’s effectiveness in real-world public school settings.
- The study had a reasonably long-term follow-up, and low-to-moderate attrition: Three years after random assignment, reading test scores were obtained for students in 85% of the sample schools – i.e., 35 of the original 41. (Of the 6 schools lost at follow-up, 5 closed due to insufficient enrollment and 1 dropped the Success for All model due to local political problems and refused to participate in data collection.) The number of schools lost in the Success for All versus control group were the same (3 each).
- The 35 Success for All and control schools remaining in the sample at the three-year follow-up were highly similar in their observable pre-program characteristics (e.g., average receptive vocabulary score, school enrollment, demographics).
- The study measured outcomes for all students in the second grade at the sample schools, regardless of the amount of exposure they had had to the program (i.e., the study used an “intention-to-treat” analysis).
- The study measured reading outcomes using tests whose reliability and validity are well-established (namely, the Woodcock Reading Mastery Test-Revised, subtests on word attack, word identification, and passage comprehension).
- These tests were administered by trained testers who were blind as to whether students attended Success for All or control schools.
- The study’s statistical analysis accounted for the fact that schools, rather than individual students, were randomly assigned to the Success for All versus control group.
- The results for the subsample of individual students who remained in the study for all three years were approximately the same as the school-wide results summarized above, providing a degree of corroboration for the school-wide results.
Other Issues Considered by the Expert Advisory Panel
Click here for a brief summary of the Panel’s reasoning on whether the study findings (i) meet the Top Tier Evidence initiative’s guidelines on “how many randomized controlled trials are needed to establish strong evidence of effectiveness”; and (ii) constitute a showing of “sustained” effects, as required by the Top Tier evidence standard.
SUMMARY OF THE INTERVENTION’S BENEFITS AND COSTS
If taxpayers fund implementation of this program, what benefits to society can they expect to result, and what would be their net cost? The following table provides a summary. This is intended to be a general overview of social benefits in relation to taxpayer cost, rather than a comprehensive benefit-cost analysis. It assigns monetary value to particular benefits and costs only when doing so requires minimal assumptions. All monetary amounts shown are in 2008 dollars.
Benefits To Society
Cost To Taxpayers
*This is the cost to implement the program in grades K-5, and includes one-time start-up costs of approximately $70,000 per school. If the program were implemented only in grades K-2, the total cost per school would be lower, but the cost per student would be higher. The cost shown in the table includes materials and training; schools may incur additional costs of reallocating staff from other functions (e.g., to provide a higher ratio of tutors).
(Click on linked authors’ names for their contact information)
- Borman, Geoffrey D., Robert E. Slavin, Alan Cheung, Anne Chamberlain, Nancy Madden, and Bette Chambers, “Success for All: First-Year Results from the National Randomized Field Trial,” Educational Evaluation and Policy Analysis, 2005, vol. 27, no. 1, pp. 1-22. Click here for a link to this study.
- Borman, Geoffrey D., Robert E. Slavin, Alan Cheung, Anne Chamberlain, Nancy Madden, and Bette Chambers, “The National Randomized Field Trial of Success for All: Second-Year Outcomes,” American Educational Research Journal, Winter 2005, vol. 42, no. 4, pp. 673-696.Click here for a link to this study.
- Borman, Geoffrey D., Robert E. Slavin, Alan Cheung, Anne Chamberlain, Nancy Madden, and Bette Chambers, “Final Reading Outcomes of the National Randomized Field Trial of Success for All,” American Educational Research Journal, September 2007, vol. 44, no. 3, pp. 701-731.
- Bloom, Howard S., Carolyn Hill, Alison Rebeck Black, and Mark Lipsey, “Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions,” MDRC Working Paper on Research Methodology, October 2008.
Note: Panel members Jonathan Crane, Dan Levy, and Steve Raudenbush did not participate in the Advisory Panel’s review of this intervention.
1Specifically, the average annual gain in reading for U.S. students between the end of first grade and the end of second grade on seven nationally normed tests is 0.97, expressed as a standardized effect size (see Bloom, Hill, and Lipsey 2008, referenced at the end of this summary). The reading improvement in Success for All schools compared to the control group, summarized above under the first main bullet, represents a gain of about 25-30% of this amount.