Invalid estimates and biased means. A replication of a recent meta-analysis investigating the effect of teacher professional development on pupil outcomes

Summary

Meta-analyses can play an important role in educational research. Aggregating results from different, but comparable, studies to demonstrate the range of plausible effect sizes can be highly informative. However, the validity of a meta-analysis mean depends on the quality of the studies included. If a meta-analysis includes studies that are not valid tests of the meta-analysis hypothesis this can introduce bias into the estimate. In this review we replicate a recent meta-analysis investigating the effect of teacher professional development on pupil outcomes to illustrate the importance of maintaining data quality. We demonstrate that when we exclude the studies that are invalid tests of the meta-analysis hypothesis, adjust the authors biased selection criteria and include a valid study that the authors inappropriately exclude the effect size falls from 0.09 to −0.008.