I’ve received a lot of feedback on the Uncertainty of Statistics article from a few days ago – thank you very much for your comments. Here are a couple of additional points that I probably should have made clearer in the original article:

The statistics themselves are not the issue
As I said in my opening paragraph, I am not calling the accuracy of the statistics provided by Opta into question at all, regarding Szczesny, Friedel or or any other statistic they provide. Those guys are fantastic.

The statistic chosen for the article is illustrative only
Szczesny vs Friedel is not the issue, it is just an example which I used to illustrate my point. Here’s another example: pass completion percentages. These are often produced after matches, particularly for midfielders, and it is commonly accepted that a higher percentage means a better performance, evidence that a particular player “ran the game”. So here are two identical stats from two different players on 21st December:

Both players completed 67 passes from 74 attempts, for a pass completion percentage of 90.5%. But even a cursory glance at the image shows that Carrick completed his passes much further from goal and much more frequently with his defence compared to Silva, whose passes were often attempted in the opponent’s half. This is the context which suggests that Silva was generally attempting more difficult passes than Carrick in these matches. This suggestion is partly backed up by the fact that Silva created two chances to Carrick’s one. The point is, given the context of the passes attempted by each player, I would expect Silva’s pass completion percentage to be lower than Carrick’s, but that is not necessarily a bad thing as I’d expect Silva’s passes to be more likely to lead to goal scoring opportunities than Carrick’s. A simple 90.5% pass completion percentage conceals all of this context, so it must be used in concert with other stats such as the location of each pass and the number of chances created to add the context which makes it more meaningful.

The original conclusion may indeed have been correct
I argued in the article that it is impossible to compare goalkeepers by their shots saved statistics alone. It may well be the case that Friedel is 18 percentage points better at saving shots than Szczesny i.e. the original conclusion is valid once shots have been controlled for. My point is that other conclusions could also be validly drawn from that same statistic; for example that Arsenal’s defence allow better goalscoring chances than Tottenham’s, meaning that Szczesny faces a higher proportion of shots which are better goalscoring chances (and therefore less likely to be saved by the goalkeeper) than Friedel. Without considering the context, it’s impossible to determine which conclusion is more likely to be correct. That is the point of the article.

As you know, this site makes use of the fantastic football statistics put together by Opta and delivered by the Stats Zone iPhone app. I am hugely in awe that while I sit on my sofa watching a match I can receive near-live statistics on each pass, shot and tackle taking place for less than the cost of a pint per season. This is the future and I’m glad to be a part of it. In fact, without Colm McMullan’s Total Football app (the precursor to Stats Zone), this website would probably not exist.

This increase in the availability of match statistics makes analysing football matches much easier, and has given rise to a number of armchair pundits such as myself. For example, want to prove who is the best goalkeeper? The shots saved percentages are right there. Which player is best at keeping possession? Just look at the pass completion percentages. It couldn’t be more simple with the facts available to all.

Unfortunately, it isn’t that simple. There is a reason why statistical analysis underpins most of science. Physicists at the Large Hadron Collider at CERN recently announced that they have effectively found the Higgs boson, but it will take a year to prove that the statistics suggesting it exists are genuine and that all other potential explanations have been ruled out. They are currently doing the same thing with the statistics “proving” that neutrinos travelled faster than the speed of light. Don’t worry, I won’t mention nuclear physics again, nor will I suggest that this rigour needs to be applied to Christopher Samba’s heading stats, but it illustrates how large and important a field statistical analysis is.

To take an example from a discussion I recently had on twitter: it was quoted that “Brad Friedel is far outperforming [Wojciech] Szczesny” as their shots saved percentages are 79% and 61% respectively. I believe that it is impossible to draw that conclusion from those statistics for reasons I will shortly explain, although before that I should note for full disclosure that I am an Arsenal fan, even though I would make exactly the same argument had the statistics been the other way around.

The reason why that conclusion is impossible to make (or to put it another way, why that statistic is meaningless as a method of evaluating goalkeepers) is because it suffers from a lack of context. Basically, it assumes that all shots are equal, when there is a very simple thought experiment which disproves this:

Picture two goalkeepers trialling for a place in a team you manage. You put them in a match on opposite sides and sit back to evaluate their performances. Team A has a midfield full of ‘Charlie Adam’s, constantly taking shots from 40 yards out which bobble towards goal. They get ten of these shots on target, and score three goals. Meanwhile Team B has wingers like Bale and Valencia, constantly firing dangerous crosses down the corridor of uncertainty for their strikers to shoot from six yards out. They also get ten of these shots on target and score seven goals. Which goalkeeper do you select?

Team B’s goalkeeper only let in three goals, while Team A’s let in seven. Team B’s shots saved percentage was 70%, while Team A’s was only 30%. However, I’d pick Team A’s goalkeeper all day long. Why? Because based on the context of the shots he had to face he wouldn’t be expected to save any of those point blank shots but he managed to keep three of them out, while Team B’s keeper should have thrown his cap on all ten of those weak long range shots but he somehow managed to let in three of them.

This is all very theoretical, so here’s an example from the games played on December 27th:

As you can see, Friedel faced two shots, saved them both, so had a clean sheet and a shots saved percentage of 100%. Szczesny on the other hand only faced one shot, let it in so had a shots saved percentage of 0%. Pretty clear cut who was the better keeper, no?

Well, no. Friedel faced two shots from outside the area, both of which he would be expected to save, and that’s exactly what he did. The goal Szczesny conceded was a deflected shot which Fletcher expertly steered back across goal into the far corner, wrongfooting the keeper, and realistically no goalkeeper would have been expected to keep that chance out. I’m assuming you’ve seen the goal but if not try the ESPN Goals app or Eurosport highlights:

The point is, 100% shots saved compared to 0% shots saved seems pretty damning, but when context is added, both percentages would be expected given the shots each keeper had to face, so using those statistics to compare the keepers is impossible. Note that I am not arguing that Szczesny is better than Friedel; this is not a defensive post and in my own personal opinion both are two of the best goalkeepers in the Premier League. All this post is claiming is that it is impossible to draw conclusions about goalkeepers based on shots saved percentages without context.

Is it possible to get around this problem and come up with a statistic which would allow genuine comparisons of goalkeepers? Well, if each shot was rated based on whether it should be saved or not, or perhaps given a score of 1-10 for how difficult a save it required, then that would provide the context needed to accurately compare keepers against each other. A goalkeeper making a higher proportion of difficult saves would therefore be unarguably better than a goalkeeper letting in a higher proportion of easier saves. Without this context, the statistic is meaningless.

You might suggest that these things even themselves out over a season, but this oft-used excuse seems to be accepted as conventional wisdom by many people in football without any proof whatsoever. In this instance, a solid back four will consistently lead to teams getting frustrated and shooting from distance over the course of a season, while a porous back four will consistently allow more point blank shots and one-on-ones. The number of shots taken against each side may be relatively consistent, as why would a team shoot from distance against a porous defence when they know they can create better chances by holding on to the ball? I refuse to believe that anything evens itself out over a season until somebody proves it with statistics, as there are a huge number of counterexamples, not least of which the equivalent cliché of “when you’re down there, these things go against you”. There doesn’t seem to be much “evening out over a season” going on there. Like “narrowly wide”, clichés can be oxymoronic. Or just moronic.

In conclusion, the reason for this post was just to make people stop and think when looking at the wonderful array of statistics on offer to us football fans these days. This applies as much to shot saved percentages as pass completion percentages (how easy or difficult were those passes?) or shots on target percentages (was that a free header or was he surrounded by defenders?). Context is the key to statistical analysis.