Record: Sandra Martin-Chang, Odette N. Gould, and Reanne E. Meuse, “The Impact of Schooling on Academic Achievement: Evidence from Homeschooled and Traditionally Schooled Children.” Canadian Journal of Behavioural Science 43, no. 3 (July 2011): 195-202.
Summary: The authors of this study of 74 children, half homeschooled, half institutionally schooled, conclude that structured homeschooling is best, public schooling next, and unstructured homeschooling worst at producing high levels of academic achievement. They begin with a nice summary of the limits of previous studies of academic achievement, especially of the infamous Rudner study of 1999. They wanted to overcome the sampling biases of this and many other HSLDA-funded studies of academic achievement.
To do this they sent out ads in every medium possible to recruit public and homeschooled students between ages 5 and 10 in Canada. They chose 37 public schooled and 37 homeschooled children from the recruits. These children were then matched up so as to create pairs of demographically similar children. Later, the homeschoolers were further subdivided into “structured” and “unstructured” once researchers figured out there was a real distinction here in how homeschoolers went about their daily education. There were 25 “structured” homeschoolers and 12 “unstructured.”
Parents filled out a demographic questionnaire and children took the Woodcock-Johnson Test of Achievement. Both homeschoolers and conventionally schooled children were given the test in the same way by the same people, thus avoiding one of the most common problems with most academic achievement studies.
The results were that the structured homeschoolers scored higher than the public schoolers on every part of the test, even though their families reported slightly lower incomes and their mothers had a bit less formal education.
Unstructured homeschoolers, on the other hand, consistently scored lower than institutionally schooled students, and in four of the seven categories they performed below grade level. The authors are quick to note that their sample of 12 is inadequate to make compelling generalizations, but they find this “exploratory analysis” to be intriguing.
The authors end with a discussion of some of the limitations of their study. They note that the study’s reliance on volunteers likely contributed to an inflation of homeschooler scores, but in their case (unlike all other studies) the institutionally-schooled students were ALSO volunteers. They too scored above grade level, just not as much as the structured homeschoolers. The researchers also note that the small sample size did not allow for more nuance about level of structure homeschoolers employ, and they wonder if over time maybe the unstructured kids would catch up.
Appraisal: I have three points to make about this article:
1. Part of me is skeptical about the whole thing. The authors say that at the outset of their study they were only planning on comparing the results of homeschoolers to those of traditionally schooled children. But that’s not what they ended up doing. They say that they later came to figure out that there was a distinction between structured and unstructured homeschooling, but it is very convenient for their findings that this division has allowed them to pull out the low scoring homeschoolers from their sample. The authors never end up giving us the actual comparison between their 37 public schoolers and their 37 homeschoolers. Instead, we get a comparison between the 37 public and the 25 best scoring homeschoolers, and then a separate comparison between the 37 public and the 12 worst scoring homeschoolers.
It is impossible to tell from the data provided in the article itself what the results of a straight-ahead comparison between the 37 and 37 would have given, but it looks to me like if you combined the two groups of homeschoolers you’d end up with scores that were pretty close to the average obtained by the public schooled students.
Martin-Chang, Gould, and Meuse do not seem to have any political agenda driving their disaggregation of the homeschoolers into a high-performing group and a low-performing group, so I don’t want to accuse them of intentionally cooking the books, but it does seem very strange to me that they don’t even report at all their findings on the original question that inspired their study.
2. I found the distinction between “structured” and “unstructured” homeschoolers a bit unsatisfying, especially given how significant it is in this analysis. As any veteran homeschooler knows, there’s no hard-and-fast distinction here. There’s a continuum. Furthermore, researchers have consistently found that over time families tend to move along that continuum from structure to unstructure. This paper’s facile distinction misses both of those points. The paper also fails to consider length of time homeschooling among its subjects. Had the “structured homeschoolers” been homeschooling their entire lives? For just a few months? We don’t know, though the researchers themselves pointed out the significance of this question in their opening discussion of Rudner.
3. Finally, despite these significant misgivings, I want to applaud the study for trying to right some of the flagrant wrongs of nearly all previous studies of academic achievement. They’re not comparing a select group of homeschoolers to national averages. They’re not comparing public school kids who take their tests under stressful conditions in school, proctored by outsiders, to homeschoolers who take them in their living rooms with mom looking over their shoulder. They control for demographics. All of this is wonderful. If they had only given us the generic data comparing all homeschoolers to all public schoolers in their sample and given us data comparing the results of longtime homeschoolers with those who have only just begun, the results would be much more valuable. Of course larger sample sizes would be nice too, as would a longitudinal follow-up in 10 years to see if the results from these young children remained consistent over time. But despite its failings, this study provides a great model future studies could build upon to garner real data on homeschool/public school comparisons.
Milton Gaither, Messiah College, author of Homeschool: An American History
Disclaimer: The views expressed in reviews are not the official views of ICHER or of its members. For more information about ICHER’s Reviews, please see the « About these Reviews » Section.