When statistics lie (NFL edition)

One of the primary reasons I started this blog back in 2010 was to shed light on some of the statistical tricks that politicians, corporate executives, and talking heads use (and misuse) so as to confuse, mislead, and otherwise divide the general population. There's any number of tricks that these individuals will use, but almost all of them rely on the same basic dynamic--highlighting one piece of data while ignoring or obfuscating another that would allow for a more complete and realistic picture. It's basic hand-waving misdirection that would make any amateur magician proud.

To see how widespread these statistical games have become, note that I've previously highlighted this dynamic with respect to cancer, budget deficits, oil prices, student loan debt, tennis, Rick Perry, apple juice, education, airlines, and investing. I've also poked fun at the issue here and here. Simply put, given the nature of the public discourse these days, I don't think you can be an informed person (or make good decisions) unless you fully understand statistics and the way that they can be manipulated. The most recent example of this is the GAO's recent takedown of the Obama administration's claim that TARP made money--in a nutshell, the claim is technically true, but only if you ignore a lot of other things that are also true. Pretty standard statistic-manipulating stuff.

With the NFL Playoffs continuing this weekend with a big game between the Patriots and Broncos, a couple of sports journalists have taken a few liberties with a similar statistic, one that allegedly speaks volumes about the Patriots. Here's the statistic, courtesy of SI's Kerry Byrne:

If there's a legitimate statistical and historical reason to doubt the validity of New England's No. 1 seed and 13-3 record, it's the fact that they faced one cream puff after another -- and then lost each time they faced something close to the iron of the NFL. New England did not beat a single team with a winning record in the 2011 season.

We track something over at Cold, Hard Football Facts.com called Quality Standings -- how well you perform against Quality Teams, or teams with winning records. It's an effective way to separate the contending wheat from the pretending chaff each NFL season. Super Bowl champs typically prove along the way that they can consistently beat Quality Opponents. And that historic fact is not good news for the Patriots.

Not only did they face fewer Quality Opponents than any team in football this year (two), but also they lost to both of them (Steelers, Giants). Would the Patriots have gone 13-3 had they faced eight Quality Opponents like the lowly 2-14 Rams? What if they faced the league-high 10 Quality Opponents who made the Peyton Manning-less season in Indianapolis such a daunting challenge?

Cool. Great statistic, right? The Patriots, as it turns out, didn't beat a "quality opponent" all year. The problem is, the meaningfulness of this statistic depends 100% on a completely arbitrary definition of what comprises a "quality opponent".

A deeper look at the Patriots' 2011 schedule reveals that Brady & Co. ended the season with a staggering 7 wins (and no losses) against teams that finished the season 8-8--including their next opponent, the Broncos. Not a single one of these six teams (the Pats played and won two games against the 8-8 Jets) counts as a "quality opponent", so the Patriots get no credit for the victories, despite the fact that every one of them had a winning record in games not played against the Patriots.

Of course, had the Patriots instead lost each of those 7 games against teams that finished 8-8, those teams would have finished at least 9-7, and therefore the Patriots would have finished 0-9 against "quality opponents". If a team is only a "quality opponent" if you lost the game, but not if you won the game, then clearly there's a problem with your definition of "quality opponent".

A closer look at the Gang of Six reveals that three of them (Chargers, Jets, Eagles) were preseason favorites to reach the Super Bowl, and that nearly all of them could have reached the playoffs had they in fact beaten the Patriots--one of them, the Broncos, made it (and won their first-round game--ironically defeating the "quality opponent" Steelers) despite their loss. It seems far-fetched and dishonest to refer to none of these teams as "quality opponents", because the statistic itself is self-referential and relies on circular references.

Ultimately, this is just another example of the way that people misuse statistics to show a slightly skewed version of the world. In this case, the authors are hoping that you won't notice the weird statistical anomaly that defined the Patriots' 2011 schedule. In the case of TARP, the Obama administration is hoping that you won't notice that the "profitability" of TARP depends entirely on the massive expansion of the Federal Reserve's balance sheet (the so-called "money printing" you've been hearing so much about).

It is exceedingly rare that the truth of the world can be easily distilled into one catch-all statistic, but that doesn't keep our favorite talking heads from trying. It's our job to know when they're telling the truth, and when they're using smoke and mirrors. More often than not, it's the latter. Go Pats.

[SI]

On multiple moving variables

In complex systems with multiple changing variables, it's often difficult to determine which variable is producing which outcome--or which interactions are working together in which ways. It's in large part what makes solving our economic problems so difficult, and it's a dynamic that I first explored in this post. As I wrote then,

I'm often writing here about the perils of bad science, and bad statistics, and how people try to draw conclusions from data that is inherently biased (or at least not properly controlled). In complex systems (and almost everything in human interaction is a complex system), it is almost impossible to reliably isolate and determine the impact of just one variable--but that never stops people from trying...

[In] any system with that many moving parts, it's absolutely impossible to know which one of them is having a positive impact, negative impact, or no impact on the overall outcomes. Remember that next time you see a politician trying to take credit for the supposed successes of "his" policy (or, in the case of Ben Bernanke, simultaneously taking credit and blame-shifting)--there's simply no way of knowing what's helping and what's hurting when we can't isolate just one variable.

So what brings me to revisit this dynamic today? Not the economy (for once), but the NFL. As we prepare for the first weekend of the football playoffs (Wild Card weekend, which hopefully won't be as big a letdown as Week 17), we're all still basking in the glow of a record-setting season. Both Drew Brees and Tom Brady broke Dan Marino's long-standing single-season passing record (and Matt Stafford and Eli Manning weren't far behind), leading to countless columns trying to discuss the record's significance and the reasons that led to its sudden shattering.

Many of the explanations correctly focused on a series of league rule changes meant both to protect players and to open up the passing game--the illegal contact penalty, the "Brady rule", and most recently the renewed emphasis on "helmet to helmet" hits on defenseless wide receivers. All of these--in addition to even wider usage of artificial turf and domed stadiums, and even a mild winter so far--helped to push passing numbers higher across the board.

But throughout all of this, one of the simplest variables has largely been ignored, and it's a variable that could easily help explain why this year, of all years, saw multiple quarterbacks take down the record. Let's turn things over to ESPN for a minute.

Before the NFL season, one of the rule changes that received the most discussion was how moving kickoffs up five yards to the 35-yard line would affect the return game...

Did the rule change deprive fans of excitement? From a numbers standpoint, the answer is yes. Nine kickoffs were returned for a touchdown in 2011, compared to 23 in 2010...

The change becomes glaringly obvious when looking at average starting field position. The average drive after a kickoff started just past the 22-yard line, down almost five yards from the previous year.

With each team averaging 189 possessions this season, that adds up to an additional 888 possible yards per team, which may account for part of the offensive boom in 2011.

Now, just because there are 888 yards "available" to be gained certainly doesn't mean that teams are going to gain them. But when you're talking about some of the more powerful offenses in the game, backed up closer to their own endzones, it's a pretty fair bet they're going to come out throwing. Even if we only give these QBs credit for 300 or 400 of that 888 yard figure, that's enough to bring both Brady and Brees back to even with (or slightly below) Marino's record figure. And yet, this ESPN article is the first time I've heard that statistic mentioned, with most people electing to focus more on the "helmet to helmet" rule.

Realistically, neither rule change can be given 100% credit for having spurred this offensive explosion, but similarly neither can be ignored. In a complex system with several moving variables, it's the interaction among all of them that leads to the outcome we observe. Are Brees and Brady's feats any less laudable just because there were a couple of rule changes? Not really. But when we're dealing with small margins between "record" and "not a record", every little bit counts.

This year, the kick returners' loss was the quarterbacks' gain--but that's not the story you'll end up hearing most often. The lesson, as always, is beware of the popular story--it's almost always too simple to actually be correct.

[ESPN]

Clip of the Week

Well, last week we didn't have a whole lot of options for Clip of the Week, so I recycled an old NFL clip. This week was a different story.

First of all, we had a bunch of entertaining clips from random sources, like this one (good if you like cats), this one (good if you like monkeys), and this one (good if you like surfing and/or Kelly Slater). But it was in the world of sports that this week really delivered the goods.

If you hate soccer (or at least hate lame European soccer players who take dives all day long), you'll appreciate this clip. Hell, I like soccer and I even get a kick out of that one. There's also this clip of a guy making an insane catch of a foul ball at a Rangers game (yeah, I know, old guys shouldn't bring their gloves to the park, but whatever--great snag).

And while we're on the topic of great catches and baseball, there's no way I can overlook this clip of the Indians' Shelley Duncan making three consecutive amazing leaping catches--it's impressive to make three catches like that in one week, let alone three batters. The look on his face after the third one is priceless.

But last week I promised NFL highlights from Week 1, and I'm a man of my word. Besides, how can I overlook my hometown team pulling off the kind of play that only happens once or twice in a decade? That's right, I'm talking about Tom Brady, Wes Welker, and 99.5 yards of awesomeness. Go Pats.