When statistics lie (NFL edition)

One of the primary reasons I started this blog back in 2010 was to shed light on some of the statistical tricks that politicians, corporate executives, and talking heads use (and misuse) so as to confuse, mislead, and otherwise divide the general population. There's any number of tricks that these individuals will use, but almost all of them rely on the same basic dynamic--highlighting one piece of data while ignoring or obfuscating another that would allow for a more complete and realistic picture. It's basic hand-waving misdirection that would make any amateur magician proud.

To see how widespread these statistical games have become, note that I've previously highlighted this dynamic with respect to cancer, budget deficits, oil prices, student loan debt, tennis, Rick Perry, apple juice, education, airlines, and investing. I've also poked fun at the issue here and here. Simply put, given the nature of the public discourse these days, I don't think you can be an informed person (or make good decisions) unless you fully understand statistics and the way that they can be manipulated. The most recent example of this is the GAO's recent takedown of the Obama administration's claim that TARP made money--in a nutshell, the claim is technically true, but only if you ignore a lot of other things that are also true. Pretty standard statistic-manipulating stuff.

With the NFL Playoffs continuing this weekend with a big game between the Patriots and Broncos, a couple of sports journalists have taken a few liberties with a similar statistic, one that allegedly speaks volumes about the Patriots. Here's the statistic, courtesy of SI's Kerry Byrne:

If there's a legitimate statistical and historical reason to doubt the validity of New England's No. 1 seed and 13-3 record, it's the fact that they faced one cream puff after another -- and then lost each time they faced something close to the iron of the NFL. New England did not beat a single team with a winning record in the 2011 season.

We track something over at Cold, Hard Football Facts.com called Quality Standings -- how well you perform against Quality Teams, or teams with winning records. It's an effective way to separate the contending wheat from the pretending chaff each NFL season. Super Bowl champs typically prove along the way that they can consistently beat Quality Opponents. And that historic fact is not good news for the Patriots.

Not only did they face fewer Quality Opponents than any team in football this year (two), but also they lost to both of them (Steelers, Giants). Would the Patriots have gone 13-3 had they faced eight Quality Opponents like the lowly 2-14 Rams? What if they faced the league-high 10 Quality Opponents who made the Peyton Manning-less season in Indianapolis such a daunting challenge?

Cool. Great statistic, right? The Patriots, as it turns out, didn't beat a "quality opponent" all year. The problem is, the meaningfulness of this statistic depends 100% on a completely arbitrary definition of what comprises a "quality opponent".

A deeper look at the Patriots' 2011 schedule reveals that Brady & Co. ended the season with a staggering 7 wins (and no losses) against teams that finished the season 8-8--including their next opponent, the Broncos. Not a single one of these six teams (the Pats played and won two games against the 8-8 Jets) counts as a "quality opponent", so the Patriots get no credit for the victories, despite the fact that every one of them had a winning record in games not played against the Patriots.

Of course, had the Patriots instead lost each of those 7 games against teams that finished 8-8, those teams would have finished at least 9-7, and therefore the Patriots would have finished 0-9 against "quality opponents". If a team is only a "quality opponent" if you lost the game, but not if you won the game, then clearly there's a problem with your definition of "quality opponent".

A closer look at the Gang of Six reveals that three of them (Chargers, Jets, Eagles) were preseason favorites to reach the Super Bowl, and that nearly all of them could have reached the playoffs had they in fact beaten the Patriots--one of them, the Broncos, made it (and won their first-round game--ironically defeating the "quality opponent" Steelers) despite their loss. It seems far-fetched and dishonest to refer to none of these teams as "quality opponents", because the statistic itself is self-referential and relies on circular references.

Ultimately, this is just another example of the way that people misuse statistics to show a slightly skewed version of the world. In this case, the authors are hoping that you won't notice the weird statistical anomaly that defined the Patriots' 2011 schedule. In the case of TARP, the Obama administration is hoping that you won't notice that the "profitability" of TARP depends entirely on the massive expansion of the Federal Reserve's balance sheet (the so-called "money printing" you've been hearing so much about).

It is exceedingly rare that the truth of the world can be easily distilled into one catch-all statistic, but that doesn't keep our favorite talking heads from trying. It's our job to know when they're telling the truth, and when they're using smoke and mirrors. More often than not, it's the latter. Go Pats.

[SI]