Understanding Effect Size Through A Study On Homework

December 8th, 2015 § 2 comments

Part 2
A continuation from Part 1. As I study John Hattie’s comparison of the effect size of direct instruction vs inquiry learning and problem-based learning I am first trying to understand how to interpret is effect size calculations. Today we look at a quick study of what his effect size of homework looks like and means.

To further clarify how to interpret effect size, Hattie describes his examination of meta-analyses for how homework effects achievement. Hattie studied 5 meta-analyses from 1984, 1989, 1994, 1994, and 2006. These covered 161 studies and more than 100,000 students to analyze the effect on achievement of giving homework. After studying them all and calculating, Hattie came up with homework having an effect size of 0.29.

This means that Homework has a positive effect on student achievement because it has a value greater than 0. The question is how much positive effect? Hattie attempts to explain it the following ways:

1. Compared to classes without homework, the use of homework was associated with advancing children’s achievement by about one year.
2. Homework improved a child’s learning rate by 15%.
3. 65% of the effects were positive, and 35% of the effects were zero or negative.
4. The average achievement levels of students in classes that were given homework exceeded 62% of the achievement levels of the students in classes where homework was not given.

Once again, these do not appear to all be the same interpretation of the data, but apparently they are. Overall, this sounds positive to me. According to this study of meta-analyses it seems that giving homework is a good thing that raises achievement. However, Hattie advises that this is actually a very small improvement and barely noticeable.

Hattie quotes a statistician who helped to originally craft the idea of effect size for the social sciences: Jacob Cohen. Cohen describes the effect size of 1.0 to be like the height difference of between a person 5’3″ and someone else who is 6’0″. He is obviously illustrating that the difference is drastic and easy to see. The effect size of 0.29 however would be akin to a comparable height of 5’11” and 6’0″. Thus, he is attempting to illustrate that although there is a difference in students who experience a 0.29 effect size, it is barely noticeable.

This takes us to the ambiguous nature of effect size. How can 0.29 be both such a minor change that “would not be perceptible” (like the difference between 5’11” and 6’0″) and also reflect advancing a child’s achievement by 1 year, improving a child’s learning rate by 15%, and having achievement levels exceed 62% of their peers without homework?

So what are we to make of this? The effect size of homework is positive so you should use homework? The effect size is negligible so using homework is not worth it? Does a positive effect size mean use a strategy?

In the final installment of understanding effect size we will look at what Hattie deems is optimum value to utilize and why. Hopefully we can gain an understanding of what Hattie thinks we should utilize and why. However, if the interpretation of what a 0.29 effect size means is this ambiguous then I do not hold much hope for moving forward.

§ 2 Responses to Understanding Effect Size Through A Study On Homework"

  • PJ says:

    I see you never made it to part 3. Hattie’s choice of the 0.4 cutoff is based on the average of all his measured effect sizes, regardless of meaning. As the fundamental error of comparing different interventions (and non-interventions, such as, socio-economic status or, the way he coded it, not being sick, rather than having a negative effect of being sick) remains, it is a meaningless average thus an arbitrary cutoff. Just on the effect(s) of gender, Hattie chose to put those in favor of boys (assuming school-aged children) to be positive and those where girls do better to be negative. A more sensible choice on that front would have been taking the absolute value of the gender effects, but that just serves to show the flawed reasoning, as forcing effects to be non-negative again illustrates the arbitrary nature of the rankings and comparisons.

    • Matt says:

      Hah, Part 3 should be coming in the next 2 days (holidays got in the way), but I greatly appreciate this knowledge drop on his .4 cutoff. I will take this all into account when writing my next piece!

Leave a Reply

Your email address will not be published. Required fields are marked *