Record: Chelsea McCracken, “How to Mislead with Data: A Critical Review of Ray’s ‘Academic Achievement and Demographic Traits of Homeschool Students: A Nationwide Study’ (2010).” Coalition for Responsible Home Education (15 January, 2014). [Available Here]
Summary: McCracken, who serves as the senior research analyst for the Coalition for Responsible Home Education, an organization advocating for increased regulatory protection of homeschooled children in the United States, here scrutinizes Brian Ray’s most recent study of homeschooler academic achievement. For my own summary and critique of Ray’s study click here.
McCracken begins by noting that Ray’s sample is much whiter, richer, more married, more Christian, and more educated than the general population. They are not representative Americans nor even representative homeschoolers. His study thus, “does not prove that homeschoolers have higher academic achievement than other children. It merely gives the description of the demographics of a particularly privileged subset of homeschoolers.” (p. 2)
McCracken next discusses Ray’s method of data collection. He basically tapped into several networks of conservative Christian (mostly Protestant but some Catholic) homeschoolers by way of testing organizations these homeschoolers use. Many of these tests, McCracken notes, are administered by parents in the home, which itself is enough to make comparison with students taking the same tests in more controlled environments invalid.
After summarizing the study’s findings (which are basically that the subjects performed remarkably well when compared to national averages), McCracken lays out several criticisms of Ray’s approach:
1. Lack of Crucial Details–Ray does not provide the actual instrument he used to obtain his data. Noting that another questionnaire that Ray has compiled is full of spurious and misleading questions, she is skeptical of this instrument as well.
2. Non-Representative Sample, as we have already seen.
3. Self-Selected Sample–Only a very small percentage of homeschooled students take these tests. McCracken speculates that the parents who want their children to take such tests and furthermore who volunteer to have their children’s scores submitted to a major study like this self-select for success. She notes that the three largest testing groups used by Ray in this study, plus all of the other tests homeschoolers use, capture about 44,000 homeschoolers total. But there were about 1.9 million homeschooled children in the country the year of Ray’s study, which means that Ray’s findings are really only relevant for the 2 to 3% of the homeschooling population that signs up for tests like this.
4. Low Response Rate–Depending on the testing service, only 11%-25% of the surveys Ray sent out were returned. While low response rates are a serious problem for a lot of social science research, Ray’s rates here are notably abysmal and probably bias his results even further in the direction of success in that the most motivated parents with the most successful children are more likely to take the trouble to fill out the survey.
McCracken thus concludes that Ray’s study really finds that “the top-scoring 2-3% of homeschooled elementary- and middle-school-age children of wealthy white Christian married couples who volunteer them for standardized tests score approximately 36 percentage points higher than the national average of all children in public schools. This is not a very meaningful result.” (p. 17)
Appraisal: As my initial review of Ray’s study makes clear, I agree completely with McCracken’s central claim about the unrepresentative nature of Ray’s sample. A few of her criticisms I find to be a bit more harsh than necessary. While it is true that Ray does not here reveal the specifics of his questionnaire, I am not troubled by that like she is, nor am I as scandalized by his low response rate. Her threshold of 60% as a minimally acceptable rate would disqualify most social science surveys these days. Over the years I’ve reviewed several survey-based studies of various aspects of homeschooling, and I don’t recall ever reviewing an article that enjoyed a 60% or higher response rate.
But those are trifles. Her central claim that Ray’s sample is whiter, richer, better educated, more Christian, and more married than the general population and even than the overall population of homeschoolers is spot on. I’d only add, as I noted back in 2012, that an equally profound problem with Ray’s study is his baseless claim that homeschooling itself is somehow responsible for the high scores he reports. That is obviously not the case, as his own data makes clear. Ray’s own data found no difference in scores between children who had homeschooled all their lives vs. those who had done so for only a short time, nor was there any difference between his scores and those of demographically comparable children attending private schools taking these same tests. What Ray has found in this and all of his earlier studies following the same format is simply that rich white kids from stable Christian homes do well on tests regardless of what kind of school they attend or don’t attend.
Milton Gaither, Messiah College, author of Homeschool: An American History
Disclaimer: The views expressed in reviews are not the official views of ICHER or of its members. For more information about ICHER’s Reviews, please see the « About these Reviews » Section.