Lies, Damned Lies and Statistics

A few weeks ago I ran across a little article on the New York Times website entitled, In Gaps in School, Weighing Family Life. The implied thesis of the piece is that it takes both a good school and a supportive family to insure academic success in our public school system. You see teachers’ and school administrators’ unions saying this sort of thing a lot lately when they are complaining that the No Child Left Behind school accountability legislation unfairly punishes some schools in areas of high minority and/or impoverished populations. The thesis is sound, as far as it goes.

Being an educational consultant, I expect that I am a bit less sympathetic when I hear this sort of thing than I ought to be. The reason lies with an interesting little conversation that I had a few years ago with a former dot com entrepreneur who’d got into the educational publishing business. Here was the essence of what she said. Consider. The US public school system takes about $120,000 and between 12-15,000 hours over a period of 12-13 years to bring the average American child up to a 9^th grade level skills level. Overall, the Public Schools industry in the US spends over half a trillion dollars annually, over 80% of which goes to salaries to do that What other industry in the US would we tolerate operating at such a level of inefficiency?

Well, actually we tolerate more than a few. Our medical and legal delivery systems come to mind immediately, but that’s quite another story. The point is, though, that in a highly competitive global economy we really can’t afford to put up with these sorts of inefficiencies.

As I worked my way through the Times article, I discovered that it was little more than a retelling of a much longer policy paper entitled The Family: America’s Smallest School, put together by the Policy Information Center of the Educational Testing Service (ETS) way back in September. For those of you not versed in the arcanae of educational consulting ETS is the organization that has developed and administered the Scholastic Aptitude Test (SAT) to graduating seniors seeking to enter universities here in the United States since the 1920’s. In general, I’ve had great respect for the quality of their research. With this particular report, however, I’ve had cause to re-evaluate my admiration for their publications. Here’s why.

The ETS report, which you can recover from their website, is 48 pages long. The first 36 of those pages are a rather long, tedious exegesis on the various kinds and flavours of dysfunctional families, viz, poverty, single mother homes, and etc, into which so many of this generation of American children are born.

From all of that they eventually cherry pick four variables that more or less relate to those dysfunctionalities...

• percent single parent (read that as mother-controlled) household
• percent children read to daily
• percent children watching more than 5 hours TV/day on weekdays
• percent children missing 3 or more days of school/month

With those variables they try to predict NAEP (National Assessment of Educational Progress) scores on a state-by-state basis. Again, for those of you not into public education arcanae, NAEP scores are as close a thing as we have to a nationally normed measure of educational attainment in the US. The idea was that in states with higher values for these four dependent variables, you’d see poorer NAEP scores, statewide.

They got about a 0.8 correlation, which explains about two-thirds of the variation in the scores, in trying to predict NAEP scores with those variables. At that point both the Times article and the ETS policy paper invited readers to suspend disbelief and accept that since they could predict NAEP scores with their variables you ought to support a whole buffet of programmes aimed at providing taxpayer funded support for kids who find themselves in such family settings. You were supposed to do it, as usual, “for the children”, never mind that the people who will be actually getting the money are their parent(s). All it is is a justification for new welfare payments and services using a different rationale than you usually see.

I frankly couldn’t manage to suspend quite that much disbelief. Thumbing through the ETS policy paper I discovered in the appendices that they had published the data that they’d used to produce their model. ETS reports tend to include enough of the material used to generate their conclusions in appendices. This has been one of the reasons that I’ve heretofore respected their work. In this case ETS published the data set they’d used to develop their model.

While I use basic statistical techniques from time to time, I ordinarily employ predictive modeling techniques more usually seen in engineering disciplines than the liberal arts. In recent years, I’ve been far more likely to be developing C&RT trees and neural nets to predict things than classical linear regression models. Indeed, last year I transferred the license to my Statistica software package to a colleague, leaving me without most standard statistical analysis tools.

One of my neural network modeling packages, however, will give you a linear regression model if you force the nodes trained to zero. Initially, I thought it would be fun to see if I could replicate ETS’ results. I popped the data into that neural network analysis pack I have and tried to predict 8th grade NEAP scores.

Here’s what I got.

Indeed, I was able to duplicate ETS’ results, and then some.

First thing you see is that you get a 0.81 correlation which means that their little model predicts about 66% of the variation in the data. That’s more or less what ETS got. That’s okay, but it ain’t wonderful. What ETS and the Times didn’t quite show you comes next.

The height of the histogram bars on the above figure indicates the relative importance of the four dependent variables in the model. Single parenthood, as a factor, might as well have not been in there. Reading to the kids and missing 3+ days of school monthly, the second and fourth columns, respectively) are sorta/kinda important but the big deal is the third column. That is 5+ hours of TV daily on weekdays. Well, duh... If the kids are glued to the tube, you can just about bet that they aren’t doing their homework. Game over...

Just to be mean and to show you how much hot air most of these little "studies" are, I threw in another dependent variable that I just grabbed out of the air. It is the number of the letter of the first letter in the state’s name in the alphabet. That value shouldn’t have anything to do with the NAEP performance by state. If it does, it is a very strange world we live in.

Here’s what you get when you do that...

Note that the number of the first letter of the state’s name in the alphabet, the fifth column, is considerably more predictive than the single parent variable and easily within shouting distance insofar of predicting the NEAP reading scores for 8th graders as is percentage of kids read to and kids missing 3 or more days of school per month.

Basically, what you can take away from this is that from the ETS study the percentage of kids watching 5+ hours of TV on weekdays might, just might, be predictive of the state’s NAEP scores. No promises on that, just a rather crude indication that there might be a relationship. TV watching habits might be causing lower NAEP scores.

The lesson of all this is that correlation and causality are two very different things. Just about anybody out there with an agenda to get you to do what they want can cobble together a study like this and then throw it in your face demanding more public money to fix the variables they’ve chosen.

It ain’t ethical, in my opinion, and it for sure ain’t professional.