Targeted school-based interventions for improving reading and mathematics for students with, or at risk of, academic difficulties in Grades 7–12: A systematic review

BACKGROUND
Low levels of numeracy and literacy skills are associated with a range of negative outcomes later in life, such as reduced earnings and health. Obtaining information about effective interventions for educationally disadvantaged youth is therefore important.

OBJECTIVES
The main objective was to assess the effectiveness of interventions targeting students with or at risk of academic difficulties in Grades 7–12.

SEARCH METHODS
We searched electronic databases from 1980 to July 2018. We searched multiple international electronic databases (in total 14), seven national repositories, and performed a search of the grey literature using governmental sites, academic clearinghouses, and repositories for reports and working papers, and trial registries (10 sources). We hand searched recent volumes of six journals and contacted international experts. Lastly, we used included studies and 23 previously published reviews for citation tracking.

SELECTION CRITERIA
Studies had to meet the following criteria to be included:
- Population: The population eligible for the review included students attending regular schools in Grades 7–12, who were having academic difficulties, or were at risk of such difficulties.
- Intervention: We included interventions that sought to improve academic skills, were performed in schools during the regular school year, and were targeted (selected/indicated).
- Comparison: Included studies used a treatment‐control group design or a comparison group design. We included RCTs, quasirandomised controlled trials (QRCTs) and QESs.
- Outcomes: Included studies used standardised tests in reading or mathematics.
- Setting: Studies carried out in regular schools in an OECD country were included.

DATA COLLECTION AND ANALYSIS
Descriptive and numerical characteristics of included studies were coded by members of the review team. A review author independently checked coding. We used an extended version of the Cochrane Risk of Bias tool to assess risk of bias. We used random‐effects meta‐analysis and robust‐variance estimation procedures to synthesise effect sizes. We conducted separate meta‐analyses for tests performed within three months of the end of interventions (short‐run effects) and longer follow‐up periods. For short‐run effects, we performed subgroup and moderator analyses focused on instructional methods and content domains. Sensitivity of the results to effect size measurement, outliers, clustered assignment of treatment, missing values, risk of bias and publication bias was assessed.

RESULTS
We found 24,411 potentially relevant records and screened 4,244 in full text. In total 247 studies met our inclusion criteria and we included 71 studies in meta‐analyses. The reasons for not including studies in the meta‐analyses were that they had too high risk of bias (118), compared two alternative interventions (38 studies), lacked necessary information (13 studies), or used overlapping samples (7 studies). Of the 71 studies, 99 interventions, and 214 effect sizes included in the meta‐analysis, 76% were RCTs, and the rest QESs. The total number of student observations in the analysed studies was around 105,700. The target group consisted of, on average, 47% girls, 73% minority students, and 62% low income students. The mean grade was 8.3. Most studies included in the meta‐analysis had a moderate to high risk of bias.

The average effect size for short‐run outcomes was positive and statistically significant (weighted average effect size [ES] = 0.22, 95% confidence interval [CI] = [0.148, 0.284]). The effect size corresponds to a 56% chance that a randomly selected score of a student who received the intervention is greater than the score of a randomly selected student who did not. All measures indicated substantial heterogeneity across effect sizes. Seven studies included follow‐up outcomes. The average effect size was small and not statistically significant (ES = 0.05, 95% CI = [−0.096, 0.192]), but there was substantial variation.

We focused the analysis of comparative effectiveness on the short‐run outcomes and two types of intervention components: instructional methods and content domains. Interventions that included small group instruction (ES = 0.38, 95% CI = [0.211, 0.547]), peer‐assisted instruction (ES = 0.19, 95% CI = [0.061, 0.319]), progress monitoring (ES = 0.19, 95% CI = [0.086, 0.290]), CAI (ES = 0.17, 95% CI = [0.043, 0.309]) and coaching of personnel (ES = 0.10, 95% CI = [0.038, 0.166]) had positive and significant average effect sizes. Interventions that provided incentives for students did not have a significant average effect size (ES = 0.05, 95% CI = [−0.103, 0.194]). The average effect size of interventions that included none of the above components, but for example provided extra instructional time, instruction in groups smaller than whole class but larger than 5 students, or just changed the content had a relatively large, but statistically insignificant effect size (ES = 0.20, 95% CI = [−0.002, 0.394]).

The differences between effect sizes from interventions targeting different content domains were mostly small. Interventions targeting fluency, vocabulary, multiple reading areas, meta‐cognitive, social‐emotional, or general academic skills, comprehension, spelling and writing, and decoding had average effect sizes ranging from 0.14 to 0.22, all of them statistically significant. Effect sizes based on mathematics tests had a relatively large effect size (ES = 0.34, CI = [0.169, 0.502]).

Including all instructional methods and moderators without missing observations in meta‐regressions revealed that effect sizes based on mathematics tests were significantly larger than effect sizes based on reading tests, and QES showed significantly larger effect sizes than RCTs. Small group instruction was associated with significantly larger effect sizes than CAI and incentive components. The unexplained heterogeneity remained substantial throughout the comparative effectiveness analysis.

AUTHORS’ CONCLUSIONS
We found evidence of positive and statistically significant average effects of educationally meaningful magnitudes (and no significant adverse effects). The most effective interventions in our sample have the potential of making a considerable dent in the achievement gap between at‐risk and not‐at‐risk students. The results thus provide support for implementing school‐based interventions for students with or at risk of academic difficulties in Grades 7–12.

We want to stress that our results do not provide a strong basis for prioritising between earlier and later interventions. For that, we would need estimates of the long‐run cost‐effectiveness of interventions and evidence is lacking in this regard. Furthermore, there was substantial heterogeneity throughout the analyses that we were unable to explain by observable intervention characteristics.

We want to stress though that our results do not provide a strong basis for prioritising between earlier and later interventions. For that, we would need estimates of the long-run cost-effectiveness of interventions, and evidence is lacking on whether the short-run effects are lasting. Furthermore, we found little robust evidence of differences between intervention types and there was substantial heterogeneity throughout the analyses that we were unable to fully explain by observable intervention characteristics.