Bloggo Science ~ A Hypothetical Post
One of the, well-deserved, knocks on Ancel Keys is his cherry-picking of evidence. A little while back Chris Masterjohn posted a bit of an admonishment to those who might fall prey to Keys' error. In that post Chris presented the "evidence" of Keys' cherry picking from Uffe Ravnskov's The Cholesterol Myths.
In the interest of full disclosure, I've never really bought into cholesterol scares and I can't really tell you why. My folks had a number of nutrition books in our home growing up and my maternal grandfather was a bit of a "health nut" so I picked it up from here or there. Thus, the dietary fat type issue per se or LDL hysteria was never much of an interest to me and I've not done a lot of recent reading/research in that regard. I'm not well versed in Ancel Keys' work though the cherry picking is legendary. However, I was a bit surprised to read the following in Chris' post:
"Keys had presented data from six countries, purporting to show a clear linear relationship between the amount of fat consumed in a country and its incidence of heart disease. This graph is shown on the left below. The one problem was that data was available for 22 countries at the time, and including that data demolished the relationship."
I would hardly say that adding the scatter to that curve "demolishes" the relationship. Does anyone know if a regression on all 22 countries has been done? Pardon me if I'm asking the obvious from some well known blog or treatise on this. As I've said, it's not something I'm all that intimately familiar with. Still, I see a clear positive relationship, it's just not nearly as definitively "perfect" for the line Keys drew.
Imagine, dear readers, that yours truly discussed a small hair growth study involving a dozen participants. To further complicate matters, the amount of hair growth reported was for similar durations but duration did vary. Subject 1's growth was assessed at 20 weeks while subject 2's at 22 weeks and 5 days ... that sort of thing.
What would you think of my analysis of said study were I to say something like:
I played around with the numbers and noticed that if I looked at just results of the half of the participants with dark hair, a pretty convincing positive correlation between initial hair length and total growth could be shown. If I include the light haired participants, however, this correlation is falls apart. But I'm going to go ahead anyway, and use half the data make the case that a commonly held notion that hair growth is greater for shorter hair than longer hair is debunked by this well designed study.
This is a rhetorical question, of course. I would hope you would think I'd gone over to the science bloggo darkside with such an analysis, although I'd love to hear how someone would defend this! Ahh but you get curious and are armed with the raw data from the study ... it's not that many numbers since it was such a small study, so you put them into a spreadsheet in between checking FB while nobody's watching at work. It's a simple spreadsheet, for a dozen participants you have initial length, final length and time elapsed between measurements.
The first thing you find is that I told the truth. The dark haired group presents a rather convincing graphic. Add in the light haired participants and your scatter plot of total growth vs. initial hair length turns to a wash. But it is also curious that there's almost faint support for what I've debunked looking just at the light haired group!
Next, you think, heck, with different time periods, total growth is misleading, let's look at the rate, say growth/week, instead. You find that even for the dark haired group, where my graphic was so convincing in support my debunking, the plot of growth rate v. length already becomes a wash.
Ultimately you say, jehosephat!, let's look at the whole lot and see what we have for length vs. growth rate. Hmmmm.... perhaps not statistically significant, this we would have to test, but a trend in favor of the hypothesis I "debunked" with my cherry-picking.
Does CarbSane survive as a credible voice?? If you blog on my analysis and expose the swiss cheese science, will you be embraced?
For Part II, click HERE