Record: Bruce Stafford, “Bad Evidence: the Curious Case of the Government-Commissioned Review of Elective Home Education in England and How Parents Exposed its Weaknesses” in Evidence and Policy 8, no. 3 (August 2012): 361-381. [Abstract Here]
Stafford, a Professor of Public Policy at the University of Nottingham in the United Kingdom, here relates the remarkable story of how a group of home educators succeeded in undermining the status of a high-profile report on home education commissioned by England’s Department for Children, Schools, and Families (DCSF). Stafford is interested in this story not because of home education but because of what it reveals about the flaws in the government’s tendency to farm out its research needs to private entities absent any sort of rigorous peer-review system. Stafford himself, so far as I can tell, is not a homeschooling insider. He’s an expert on various elements of government social policy, especially Disability Services. He’s also done a good bit of contract work for the government himself, which might explain his interest in this particular case. Here’s the story:
In January 2009 the DCSF, responding to allegations that home education in the UK was being used as a cover for child abuse, announced that it was commissioning a review of Home Education regulations, to be conducted by Graham Badman, former Director of Children’s Services at Kent County Council. Badman was to work in conjunction with a panel of 10 experts, none of whom in fact had any expertise in home education. It seems from the evidence that this panel exercised very little if any oversight over Badman’s study.
Badman’s review was published in June of 2009. He found no direct evidence that home education in Great Britain was associated with higher than average rates of child abuse, but he nevertheless recommended increased regulation of home education–specifically that all home educators needed to register every year with local education authorities (LEAs). LEAs would also have the authority to make home visits whenever they wanted. Badman’s proposals were greeted warmly by the DCFS, who planned on implementing them in the 2009 Children, Schools and Families Bill.
Home Educators in Great Britain, unsurprisingly, were taken aback by these proposals. What evidence, they wondered, had Badman found that would merit this increased scrutiny of their practices? The British homeschooling community is much smaller and less organized than that in the United States, but this proposal galvanized them. Blogs, Yahoo groups, and other channels of communication that were typically local and small scale affairs suddenly sprung into action. Learning that the 2000 Freedom of Information Act empowered citizens to gain access to government funded research, home educators began requesting documents. They also presented a record number of petitions against the 2009 bill that would have made the Badman recommendations law. Read more about the political side of this conflict here.
What the homeschoolers found out as they gradually gained access to more and more of Badman’s data was that the evidence basis for his proposal was very thin. First of all, Badman’s literature review was very poor. You wouldn’t know this from the published report, but documents obtained under the Freedom of Information Act reveals that Badman based his review on only 16 sources. There is no explanation given for why these 16 sources were chosen, though a memorandum presented later after Badman’s report was called into question admits that the actual research base on home education is “extensive” and that this particular review was not meant to be “comprehensive.” (p.369)
Badman’s lit review was also inconsistent in its approach to qualitative research. It generally discounted the many convenience sample studies that have found high homeschooler achievement. But it did not exercise the same critical scrutiny of convenience samples of school officials’ views.
The only evidence Badman’s original report gave that there was a problem that needed to be addressed by increasing regulation of home education was a claim that “local authority evidence” and some case studies suggested that there were a disproportionately high number of home educated children known to social service organizations. Press reports quoted Badman as saying that home educated kids were about twice as likely as the general population to be “known to social care” groups (though no precise figure is given in the actual report). It was this claim that the skeptical home educating parents wanted to know more about. Badman’s study made the claim, but it didn’t cite or explain the evidence base underlying it. What was it?
First of all, the phrase “known to social care” is slippery. It could be easily interpreted as meaning “reported because of child abuse.” But in fact, lots of home educators register their children with social care agencies because of these children’s special needs. These are not at-risk kids. In addition, it is actually impossible to compare populations here, for nobody knows the precise number of home educators in the UK. If, for example, as many think, there are twice as many home educators in the UK as the government estimates, then a “known to social care” figure that is twice that of the general population turns out to be exactly the same as that of the general population. Finally, lots of “reports” about home educated kids come in the form of concerned third parties, such as nosy neighbors, who might not understand that home education is legal and who contact local authorities worried that neighborhood children are playing truant. These cases abound and are frequently closed almost as soon as they’re opened. Yet they still count in the overall tally. The point here is simply that none of this is explained in the report itself.
But where did Badman get this evidence of greater numbers in the first place? After numerous Freedom of Information Requests were filed, the parents finally found out. Badman had conducted two surveys of local education authorities. The first one got 90 of 150 LEAs to respond. The second got 25 out of 90. Nowhere in any of the documents about Badman’s study is there any discussion of how representative of the entire population of LEAs these samples are. Were the LEAs who responded to Badman’s request more likely to have an animus against home education? Less? Remarkably, AFTER the report had been issued and roundly criticized by home educators, Badman conducted a third survey of LEAs to try to get a larger sample size. Badman claimed that this third sample was representative, but Stafford argues that it in fact was biased toward urban areas. Stafford here pulls out all the stops, for his concern is not so much about what all this means for home education but that government policy is being made on such flimsy grounds.
By this point it should be clear why Badman didn’t present any sort of statistical breakdown of the evidence he compiled from LEAs. Instead he gives selective quotes that give the reader a general sense that LEAs think homeschooling needs to be regulated more. But here again the evidence is poor. Stafford shares a couple of examples where (to use one of them) Badman’s report quotes a juicy morsel out of context to make it sound like an LEA is concerned about abuse. But when homeschoolers obtained the full data thanks to the Freedom of Information Act, we see that in fact the LEA was himself quoting somebody else, and that the LEA himself thinks homeschooling is great.
Despite all of these concerns being raised, the record number of petitions submitted, and all the rest, the government did not withdraw the report. The 2009 Children, Schools, and Families Bill did not pass, but only because of a lack of support across parties and the calling of a general election by Prime Minister Gordon Brown. Stafford concludes that this entire debacle shows that if government is going to continue contracting out research to private entities, it needs to create rigorous review safeguards to make sure the research it is paying for is credible.
Milton Gaither, Messiah College, author of Homeschool: An American History
Disclaimer: The views expressed in reviews are not the official views of ICHER or of its members. For more information about ICHER’s Reviews, please see the « About these Reviews » Section.