What can we learn from studies with "conflicting results"?

After training for and running my first marathon this spring, I've naturally taken an interest in distance running and a lot of the literature surrounding it. I wrote about it once before, after I read Christopher McDougall's fantastic book, Born to Run.

So I was naturally drawn to this item in the New York Times last week. Entitled "Do Marathons Hurt Your Knees?", Gretchen Reynolds' column attempted to make sense of the sea of studies surrounding marathon running and its effects on the body.

The idea that distance running inexorably leads to arthritis is deeply entrenched, despite the publication of a number of recent studies that have found otherwise. In one representative experiment, the knees of experienced marathoners, with multiple races behind them, were scanned with magnetic resonance imaging technology, and then scanned again 10 years later. The runners’ knees were and remained robust throughout that time, with few significant cartilage abnormalities. The only truly unhealthy knee in the study belonged to a former marathoner, who had quit the sport. In the years since he stopped running, his joint had deteriorated badly...

But then came the latest study on the issue, this one from researchers at the University of California at San Francisco, using a more sensitive type of M.R.I. technology than had been available in the past... The results, published earlier this year in The American Journal of Sports Medicine, are eye-opening. On these more-sensitive M.R.I. scans, the researchers found evidence of significant biochemical changes in the runners’ knee cartilage, particularly in the days immediately after the race. According to their postrace scans, the racers had elevated values for two technical measures of the health of their cartilage matrix...

In other words, the issue of whether distance running does or does not harm your knees would appear still to be open (to the considerable satisfaction of some of my nonrunning friends).

The column, unfortunately, didn't do much to try to reconcile these two studies with each other. This has become typical of much research (and especially media reporting on such research). When faced with studies with conflicting results, a staid "more research is needed" is uttered as a conclusion, and we all move on knowing little more than we did before, inevitably confused.

In my opinion, when we see conflicting results like these, it's usually an indication that we may be studying the wrong thing. It's not a matter of whether running itself is inherently good or bad, but the way we choose to run--how we train for marathons, what type of shoes we wear, the surfaces we run on, what type of shape we are in to begin with, et cetera, ad nauseum.

It's difficult bordering on impossible to execute a study on "running a marathon" while holding all other variables constant. There are nearly infinite factors that could separate one marathoner from the next--not to mention marathoners from non-marathoners. So we need to be very careful not to assume that any study has properly isolated the variable that it is attempting to examine. That is, after all, the essence of scientific research.


[New York Times]