Cutler or McCown? The NFL’s Sample Size Problem


Photo Credit: Chicago Bears Official Site

There are 16 games in an NFL season. That is not breaking news for you, I’m sure. But it creates a problem when evaluating player performances; as anyone with a passing interest in sabermetrics can tell you, 16 games for a baseball player is not nearly enough to tell you anything statistically significant. But for football, that’s what we’re stuck with. Obviously the two sports are not strictly analogous; a quarterback has an active role in many more plays in any given game than any single baseball player would. So in that sense, we can mine more data on a per-game basis than we can for a baseball player. Which has some merit, for sure; I’m not saying that there are zero valuable statistics in football. I think there are multiple metrics that can be used to discern how well a player performed in any given game, or over the course of the season.

But I think the problem comes into play when people attempt to use stats derived from an inherently small sample to predict player performance. This phenomenon is currently playing a major role in the ongoing media debate surrounding the potential return of Jay Cutler. By any reasonable metric, Josh McCown has been very good, and that includes the “eye test.” By all the same measures, Jay Cutler was very good as well, when healthy. But due to a variety of factors (the obvious coolness of McCown’s story and the general dislike of Cutler (itself a tired and by now obsolete narrative) chief among them) there has been a growing chorus of voices who think McCown should remain the starter regardless of Cutler’s health situation.

I can understand that urge. No one ever wants to rock the boat, and the Bears have waited so long for a competent offense that it’s easy to want to handle it gingerly, like a baby bird, afraid to break something so seemingly fragile. But when you consider that the offense didn’t miss a beat with the transition to McCown, I think it’s fair to speculate that they wouldn’t miss a beat with the transition back to Cutler, and if any bumps were to arise I doubt they’d be too jarring. Which leaves the “McCown is just better” crowd, and those people have to be basing their case strictly on McCown’s performance this season; before this year, he had done little that would lead you to believe he was capable of playing as well as he’s played this season. And make no mistake about it, he’s playing well. His performance Monday night tied for the second-highest rated quarterback performance of the season according to ESPN’s QBR metric. (Which isn’t a perfect stat, but it’s better than passer rating, in my opinion. I’ll also note that if a Cowboy catches either of the two balls McCown threw right to defenders, or if the refs don’t throw a flag for defensive holding on a ball that was actually intercepted, things might look a little different.) But here we run into the problem with the sample size, as McCown has started just five games. Jay Cutler has started and finished six games, and he only played poorly in one of them, the Week 4 loss to Detroit. McCown’s body of work is more recent, and his performance Monday was the best performance by either quarterback this season, but it also came against the league’s worst passing defense. Context matters.

McCown’s performance has also been buoyed by the very impressive play of his skill position players, notably the suddenly crazy exploits of Alshon Jeffery. Marc Trestman’s game-planning, play-calling, and game-management have all been very, very good. Those things would be there for Jay Cutler, as well; McCown is not doing things that Cutler is incapable of doing. (Well, maybe the spinning, Elway-like dive into the end zone; I’m not sure Cutler’s ankle would be up for that one.) But because he’s played nearly a half-season’s worth of games, it becomes easy for fans, media, and players alike to view that as a large enough sample to justify an ongoing entrenchment as the starter.

But that’s not enough. Data collected from 5.5 games just isn’t enough to be used to gauge future performance. On Twitter, I equated this “QB controversy” to an above-average MLB player missing two weeks, while his below-average replacement filled in and played well. That drew questions as to the validity of that analogy, and I’d like to try to clear that up a bit. As was noted, 5 games in the NFL roughly correlates to 50 MLB games in terms of share of the season. But it does not correlate to 50 games in terms of statistical sampling. (I’ll also note that 50 games is still a small sample by baseball standards.) It’s not a difficult difference to illustrate; football teams within the past decade have finished the regular season with every possible finishing record, from 16-0 to 0-16. Teams in the NFL routinely go on lengthy winning streaks; the Chiefs began this season at 9-0. But no MLB team has ever gone 162-0, or 0-162. No team wins or loses 90 games in a row. The NFL is often praised for its parity, the idea that any team could beat any team at any time, or that any team could make the playoffs in any given year.

Those things are true, but the first point is true in baseball as well. The second part isn’t, because the sample size is much greater. If the MLB season were 16 games, any team could make the playoffs. (I’m using team results to compare the leagues, but the sample size issues work for analyzing individuals as well.) The sports are different, and it’s not an apples-to-apples comparison (the nature of baseball success is more random, which certainly plays into this discussion, although not to a degree that makes me think my greater point is incorrect.) But there’s no way that five games can tell us anything useful about McCown’s likelihood of sustaining success, no more than Rex Grossman’s various multi-game hot streaks told us about his ability to be consistently successful.

I think the fact that Marc Trestman has been unwavering in his support of Cutler should give you a good idea as to who should start. If you wanted to read into it further, you could probably look at it as insight into their organizational belief in Cutler. This is a team that is fighting for a playoff spot, coached by one of the more analytically minded men in the league, who works for one of the more analytically minded general managers. Neither of them was involved with acquiring Cutler (who will be a free agent after this year), and they know how well McCown has played. If there was ever a time for the Bears to make a move similar to San Francisco’s benching of Alex Smith from last season, this would be it. But all indications are that if Cutler can play, he’ll play. That should tell you more about the gulf between their abilities than anything I can say, and certainly more than anything six games worth of stats could tell you.

It’s also important to note that if Jay comes back and doesn’t play well, that doesn’t mean they made the wrong call. The results might not have been what they wanted, but if they believe that a healthy Cutler gives them a better chance to win than McCown would give them, they have to do it. Cutler not playing well wouldn’t automatically mean that McCown would have been better. But let’s hope we won’t have to worry about that particular narrative twist.

Anyway, it’s a good problem to have, isn’t it? Beats the hell out of debating Grossman vs. Orton.

Jay Rigdon is the editor and lead writer at Bleacher Nation Bears, and can also be found @BearsBN on Twitter.

36 responses to “Cutler or McCown? The NFL’s Sample Size Problem”

  1. Adam

    Sounds like Jay will start on Sunday. Probably the first start of his Bears career where his performance will determine his place on the team. Hopefully pressure from McCown will bring out the best in him.

  2. Kyle


  3. FarmerTanColin

    The small sample size has such an effect in the nfl its ridiculous. Mostly with QBs. Big contracts given to Rob Johnson and more recently Matt Flynn for one game and a few seasons on the bench.

    Probably why trades are less common because coaches don’t know the true talent of other teams. You can watch game film and assume a few things but players can look awfully good on film in a system that works. (Flynn).

    I would alway resort to talent and not whatever these fantasy football obsessed talking heads have to say. Bears know what they have in Jay and have said the same things all along. No drama with them…all the drama is with the sports media.

  4. CCunt

    “or if the refs don’t throw a flag for defensive holding on a ball that was actually intercepted.”

    What about if the defender didn’t commit the holding penalty that lead to the interception? Sorry, but I don’t see how that play can be construed as some kind of a black mark against McCown’s performance.

  5. JulioZuleta

    Really good stuff Jay.

  6. JulioZuleta

    I think a lot of the debate is manufactured by lazy writers who would rather play to the emotions of the meatheads and Cutler haters than do any real analysis into the team. Such an easy storyline for them. It’s really just an absurd assortment to say that McCown should start over Cutler. I think it’ll be clear how each player is viewed during this offseason when Cutler signs a 5 year, $75M contract somewhere and McCown gets something like 2/$6M. I know that’s not a perfect indicator, but you know what I mean.

    1. FarmerTanColin

      I agree. I’ve since stopped watching anything ESPN except the actual games or the other pregame shows. Just cannot stand their “Expert Analysis”. Then you have your sports buddies just regurgitating what they hear them say. Hate it.

  7. Matt

    One could argue that a pass is like an at-bat in baseball, which helps with sample size, but you’re right; it’s just really hard. Especially for a sport like football, where teamwork is probably more important than in any other sport.

    I will say two things about McCown, however: 1) it does feel like there’s been enough time to say he’s at least pretty good, and I don’t care who you’re playing, three straight weeks of almost 350 yards is impressive. 2) if it’s decided that McCown is indeed legit, the economics probably make him a smarter keep than Cutler. Even if the performance is, what, 80% to Cutler? he’d come at a fraction of Cutler’s cost.

  8. No More Home Plate Collisions and Other Bullets | Bleacher Nation | Chicago Cubs News, Rumors, and Commentary

    […] continues to kill it at the BN Bears page, including a thoughtful analysis of the Cutler/McCown issue, and the problem of small sample sizes in […]

  9. Bears Will Name Starting QB Today and Other Bullets | Bleacher Nation - Bears

    […] things changed after Josh McCown’s very good performance Monday night, even though I tried to outline a few reasons why that’d be a dangerous conclusion to draw in a post yesterday afternoon. Anyway, […]

  10. ssckelley

    I think it is highly unlikely the Bears resign Cutler next year. The Bears have already come out and said they are looking for a cap friendly deal with Cutler and with as many teams looking for a QB I think one of them gets stupid and gives him a huge contract. I would not blame Cutler for wanting to get top dollar.

    1. Jay

      Considering he’s already made $50 mil as a player AND he’s finally in a system that caters to QB’s and offense, I don’t see him running off to a wasteland like Tennessee just for a few more bucks. And I also don’t see the Bears letting him go. Starting him now nstead of McCown makes that pretty clear. They’ll franchise him if nothing else, even though Emery would rather not.

      1. ssckelley

        No way they franchise him, that would represent about 1/3rd of the cap that they have. I believe they will let him walk if Cutler demands top dollar and if I am Cutler why would I sign for anything less? This isn’t like baseball where someone that has made a ton of money signs for a hometown discount. This is football where life is not that comfortable after your playing days are over and after all the hits Cutler has taken he may not know who he is by the age of 40.

        Having said that it is tough to gauge what is going through Cutlers head about his contract and resigning with the Bears. He has not exactly been an open book over the years.

  11. jj

    It’s also important to recall who each QB played against. Cutler faced 3 top passing defenses, while McCown benefited from almost exclusively playing the bottom 10 passing defenses.

  12. ssckelley

    Another thing to keep in mind is Cutlers ability to read the defenses. Remember the 2 point conversion try against the Lions when the defense was showing man to man coverage and loaded the box. Had Cutler been in there he would have audibled out of the run play and thrown the ball to the corner of the end zone giving either Marshall or Jeffery a chance to go up and get it. Instead McCown handed the ball off to Forte who ran into a wall of Lion defenders.

    It is in my opinion Cutler is the best QB Chicago has had since Sid Luckman. It has been a long time since Chicago has had a real QB and I hope the Bears do not end up regretting allowing Cutler to sign somewhere else.

  13. Kirk

    It’s a well done piece, except flawed. You don’t mention the fumbles lost (Jay 3, Josh 1). Jay’s lost fumbles have led to touchdowns. Then you look at the stat line of 13 TD 11 TO for Cutler and it’s not very good.

    Also, Cutler has played poorly in more than one game this season. Both Detroit games he was not very good. He was bad against New Orleans and Washington as well.

    QBR is also not a better stat than Passer Rating. Case in point, in Week 1 Cutler had QBR of 85.7 where Peyton Manning posted an 83.6. Manning had 7TDs and no picks, whereas Cutler had 2 TDs and 1 pick. QBR is a useless stat.

    Football has a different set of numbers you have to go by. What sets it apart from baseball is that you need to combine the eye test with the statistics. In watching every game this year, the Eye test suggests McCown has been head an shoulders better than Cutler. The statistics follow suit. Sure it doesn’t predict what a player will do in the future, but McCown has been the better player this year.

    In my humble opinion, I think the Bears will be starting Cutler for the rest of the season so they can try to up his trade value in the offseason.

    1. JulioZuleta

      He’s a free agent. Sure they could tag and trade him, but those trades don’t bring much at all in return. If they were trying to use playing time to effect value they’d sit him the rest of the year and hope that drags his price down a little.

  14. MikeCubs

    Keep in mind Kristen K is from Chicago (Barrington). With a baby and another on the way, she wants to be near her mom. Cutler referenced this on the Waddle & Silvy show a few weeks ago. Motive for a home town discount.

  15. MikeCubs

    McCown seems to be more of a west coast offense QB – short quick throws (with some exceptions). Cutler makes longer more risky passes. While I agree Cutler has more talent, I’m not sure he’s a great fit for Trestman’s west coast offensive style.

  16. jmc

    the best since luckman? McMahon was awfully entertaining

  17. My Final Thoughts on Cutler and McCown | Bleacher Nation - Bears

    […] were good, a quarterback is going to perform at that level any given week. (A subject I touched on Wednesday.) There are many, many factors at play. Including the running game; as Kevin Fishbain noted on […]

  18. Bears “Optimistic” on Philadelphia Return for Briggs and Other Bullets | Bleacher Nation - Bears

    […] allowed an average of 204 yards on the ground. Again, all my statistical caveats from the QB-debate post apply here, and I don’t think you can blame that level of deterioration on the absence of any […]

  19. I Had to Look Up How to Spell “Schadenfreude” and Other Bullets | Bleacher Nation - Bears

    […] because his skill level is such that he often plays poorly. When the McCown debate was raging, I wrote this piece on the dangers of using small sample sizes, and how football as a sport really struggles with that […]