{"id":537,"date":"2013-01-21T16:31:52","date_gmt":"2013-01-21T16:31:52","guid":{"rendered":"http:\/\/icher.org\/blog\/?p=537"},"modified":"2015-05-21T17:13:13","modified_gmt":"2015-05-21T17:13:13","slug":"bad-evidence-faulty-research-uncovered-by-home-educating-parents","status":"publish","type":"post","link":"http:\/\/icher.org\/blog\/?p=537","title":{"rendered":"BAD EVIDENCE: Faulty Research Uncovered by Home Educating Parents"},"content":{"rendered":"

Record:<\/strong> Bruce Stafford, \u201cBad Evidence: the Curious Case of the Government-Commissioned Review of Elective Home Education in England and How Parents Exposed its Weaknesses\u201d in Evidence and Policy<\/em> 8, no. 3 (August 2012): 361-381. [Abstract Here<\/a>]<\/p>\n

\u00a0Stafford<\/a>, a Professor of Public Policy at the University of Nottingham in the United Kingdom, here relates the remarkable story of how a group of home educators succeeded in undermining the status of a high-profile report on home education commissioned by England\u2019s Department for Children, Schools, and Families (DCSF).\u00a0 Stafford is interested in this story not because of home education but because of what it reveals about the flaws in the government\u2019s tendency to farm out its research needs to private entities absent any sort of rigorous peer-review system.\u00a0 Stafford himself, so far as I can tell, is not a homeschooling insider.\u00a0 He\u2019s an expert on various elements of government social policy, especially Disability Services.\u00a0 He\u2019s also done a good bit of contract work for the government himself, which might explain his interest in this particular case.\u00a0 Here\u2019s the story:<\/p>\n

In January 2009 the DCSF, responding to allegations that home education in the UK was being used as a cover for child abuse, announced that it was commissioning a review of Home Education regulations, to be conducted by Graham Badman<\/a>, former Director of Children\u2019s Services at Kent County Council.\u00a0 Badman was to work in conjunction with a panel of 10 experts, none of whom in fact had any expertise in home education.\u00a0 It seems from the evidence that this panel exercised very little if any oversight over Badman\u2019s study.<\/p>\n

Badman\u2019s review was published in June of 2009.\u00a0 He found no direct evidence that home education in Great Britain was associated with higher than average rates of child abuse, but he nevertheless recommended increased regulation of home education\u2013specifically that all home educators needed to register every year with local education authorities (LEAs).\u00a0 LEAs would also have the authority to make home visits whenever they wanted.\u00a0 Badman\u2019s proposals were greeted warmly by the DCFS, who planned on implementing them in the 2009 Children, Schools and Families Bill.<\/p>\n

Home Educators in Great Britain, unsurprisingly, were taken aback by these proposals.\u00a0 What evidence, they wondered, had Badman found that would merit this increased scrutiny of their practices?\u00a0 The British homeschooling community is much smaller and less organized than that in the United States, but this proposal galvanized them.\u00a0 Blogs, Yahoo groups, and other channels of communication that were typically local and small scale affairs suddenly sprung into action.\u00a0 Learning that the 2000 Freedom of Information Act empowered citizens to gain access to government funded research, home educators began requesting documents.\u00a0 They also presented a record number of petitions against the 2009 bill that would have made the Badman recommendations law.\u00a0 Read more about the political side of this conflict here<\/a>.<\/p>\n

What the homeschoolers found out as they gradually gained access to more and more of Badman\u2019s data was that the evidence basis for his proposal was very thin.\u00a0 First of all, Badman\u2019s literature review was very poor.\u00a0 You wouldn\u2019t know this from the published report, but documents obtained under the Freedom of Information Act reveals that Badman based his review on only 16 sources.\u00a0 There is no explanation given for why these 16 sources were chosen, though a memorandum presented later after Badman\u2019s report was called into question admits that the actual research base on home education is \u201cextensive\u201d and that this particular review was not meant to be \u201ccomprehensive.\u201d (p.369)<\/p>\n

Badman\u2019s lit review was also inconsistent in its approach to qualitative research.\u00a0 It generally discounted the many convenience sample studies that have found high homeschooler achievement.\u00a0 But it did not exercise the same critical scrutiny of convenience samples of school officials\u2019 views.<\/p>\n

The only evidence Badman\u2019s original report gave that there was a problem that needed to be addressed by increasing regulation of home education was a claim that \u201clocal authority evidence\u201d and some case studies suggested that there were a disproportionately high number of home educated children known to social service organizations.\u00a0 Press reports quoted Badman as saying that home educated kids were about twice as likely as the general population to be \u201cknown to social care\u201d groups (though no precise figure is given in the actual report).\u00a0 It was this claim that the skeptical home educating parents wanted to know more about.\u00a0 Badman\u2019s study made the claim, but it didn\u2019t cite or explain the evidence base underlying it.\u00a0 What was it?<\/p>\n

First of all, the phrase \u201cknown to social care\u201d is slippery.\u00a0 It could be easily interpreted as meaning \u201creported because of child abuse.\u201d\u00a0 But in fact, lots of home educators register their children with social care agencies because of these children\u2019s special needs.\u00a0 These are not at-risk kids.\u00a0 In addition, it is actually impossible to compare populations here, for nobody knows the precise number of home educators in the UK.\u00a0 If, for example, as many think, there are twice as many home educators in the UK as the government estimates, then a \u201cknown to social care\u201d figure that is twice that of the general population turns out to be exactly the same as that of the general population.\u00a0 Finally, lots of \u201creports\u201d about home educated kids come in the form of concerned third parties, such as nosy neighbors, who might not understand that home education is legal and who contact local authorities worried that neighborhood children are playing truant.\u00a0 These cases abound and are frequently closed almost as soon as they\u2019re opened.\u00a0 Yet they still count in the overall tally.\u00a0 The point here is simply that none of this is explained in the report itself.<\/p>\n

But where did Badman get this evidence of greater numbers in the first place?\u00a0 After numerous Freedom of Information Requests were filed, the parents finally found out.\u00a0 Badman had conducted two surveys of local education authorities.\u00a0 The first one got 90 of 150 LEAs to respond.\u00a0 The second got 25 out of 90.\u00a0 Nowhere in any of the documents about Badman\u2019s study is there any discussion of how representative of the entire population of LEAs these samples are.\u00a0 Were the LEAs who responded to Badman\u2019s request more likely to have an animus against home education?\u00a0 Less?\u00a0 Remarkably, AFTER the report had been issued and roundly criticized by home educators, Badman conducted a third survey of LEAs to try to get a larger sample size.\u00a0 Badman claimed that this third sample was representative, but Stafford argues that it in fact was biased toward urban areas.\u00a0 Stafford here pulls out all the stops, for his concern is not so much about what all this means for home education but that government policy is being made on such flimsy grounds.<\/p>\n

By this point it should be clear why Badman didn\u2019t present any sort of statistical breakdown of the evidence he compiled from LEAs.\u00a0 Instead he gives selective quotes that give the reader a general sense that LEAs think homeschooling needs to be regulated more.\u00a0 But here again the evidence is poor.\u00a0 Stafford shares a couple of examples where (to use one of them) Badman\u2019s report quotes a juicy morsel out of context to make it sound like an LEA is concerned about abuse.\u00a0 But when homeschoolers obtained the full data thanks to the Freedom of Information Act, we see that in fact the LEA was himself quoting somebody else, and that the LEA himself thinks homeschooling is great.<\/p>\n

Despite all of these concerns being raised, the record number of petitions submitted, and all the rest, the government did not withdraw the report.\u00a0 The 2009 Children, Schools, and Families Bill did not pass, but only because of a lack of support across parties and the calling of a general election by Prime Minister Gordon Brown.\u00a0 Stafford concludes that this entire debacle shows that if government is going to continue contracting out research to private entities, it needs to create rigorous review safeguards to make sure the research it is paying for is credible.<\/p>\n

Milton Gaither, Messiah College, author of Homeschool: An American History<\/a>\"\"<\/em>.<\/p>\n

Disclaimer<\/strong>:\u00a0 The views expressed in reviews are not the official views of ICHER or of its members.\u00a0 For more information about ICHER\u2019s Reviews, please see the \u00ab\u00a0About these Reviews \u00bb Section.<\/p>\n","protected":false},"excerpt":{"rendered":"

Record: Bruce Stafford, \u201cBad Evidence: the Curious Case of the Government-Commissioned Review of Elective Home Education in England and How Parents Exposed its Weaknesses\u201d in Evidence and Policy 8, no. 3 (August 2012): 361-381. [Abstract Here] \u00a0Stafford, a Professor of … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[10,12,891],"tags":[196,491,481,490,484,483,485,492,493,486,487,489,488,195,482],"class_list":["post-537","post","type-post","status-publish","format-standard","hentry","category-international","category-policyregulation","category-research-methodology","tag-and-families","tag-and-families-bill","tag-bruce-stafford","tag-children","tag-dcsf","tag-department-fo-children","tag-disability-services","tag-freedom-of-information-act","tag-gordon-brown","tag-graham-badman","tag-kent-county-council","tag-lea","tag-local-education-authorities","tag-schools","tag-university-of-nottingham"],"_links":{"self":[{"href":"http:\/\/icher.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/537","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/icher.org\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/icher.org\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/icher.org\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/icher.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=537"}],"version-history":[{"count":7,"href":"http:\/\/icher.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/537\/revisions"}],"predecessor-version":[{"id":2068,"href":"http:\/\/icher.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/537\/revisions\/2068"}],"wp:attachment":[{"href":"http:\/\/icher.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=537"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/icher.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=537"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/icher.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=537"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}