On Tracking Scoring Chances, Part I

Those that have been following the ongoing development of advanced statistical analysis in hockey are aware that a group of bloggers got together and began tracking scoring chances during games over the past two seasons. The project offered promise, including some insight into the ever-popular debate regarding the existence of shot quality. After the second season of tracking scoring chances, the group decided to shut the project down, as scoring chances seemed to tell us no more than the pure Corsi statistic was telling us.

While the statistically inclined fan community may have determined that there is no value added (for their purposes) by tracking scoring chances, NHL front offices have made and continue to make use of them. Recently, Ted Leonsis revealed that the Washington Capitals‘ coaching staff tracks scoring chances to evaluate players. Peter Laviolette has (possibly inadvertently) admitted that the Philadelphia Flyers also track scoring chances to evaluate his team’s play. Among a list of teams that openly use advanced statistical analysis, Adam Gretz has explicitly identified the Pittsburgh Penguins and Phoenix Coyotes as teams that use scoring chance data. Given how secretive teams are about what stats they use, it’s very likely that at least a handful of other NHL teams are tracking scoring chances.

So where is the disconnect? Why do teams find value in the scoring chance data, while the scoring chance project determined that scoring chances don’t tell us anything new? To get some insight into the process, J.P. and I have recruited some regulars from our comments section to do our own case study on scoring chance data. Those who venture into our comments will already recognize these contributors, but for everyone else let me welcome (and thank them for their help) Gould Old Days, Steckel Me Elmo, renstar, and D’ohboy. The six of us each independently charted scoring chances for one Capitals game, the game against the Pittsburgh Penguins on January 22, 2012. Join us after the jump for our initial thoughts on tracking scoring chances and the process in general.

Q1. Before we reveal to each other our specific observations on the game, what are your general thoughts after tracking scoring chances for the first time? Easier or harder than you’d thought? Did it do anything to inform or impact your decision on the utility of the scoring chance data you see out there? Any and all thoughts welcome…

RP: I think it’s harder than I expected. You can’t zone out or get caught up in the action too much because you always have to have your mind focused on tracking the scoring chances (which are generally the most exciting part of the game and thus easiest to get distracted from your task). I think it’s probably harder to track live (which I did) than on delay because when it’s live you still have the stress of not knowing the outcome of the game. On delay you’ve already seen the interesting parts, know the result, etc. so maybe it’s easier to focus on identifying scoring chances.

JP: I’m with you – it was harder than I expected, and that definitely relates to focus (and it made me realize how much I zone out at times watching the game unburdened by such responsibilities). I didn’t track the chances live (I went back and watched it on DVR a few days later; and I agree that not worrying about the outcome likely made it easier to focus on the chances themselves) and made liberal use of my “back a few seconds” button – there were plenty of shots from right around the perimeter of the scoring area that I wanted to re-watch. Granted, I likely obsessed over these more than I should have (and I’m sure I’d get better at the whole exercise with practice), but I wanted to try to be as sure as possible.

SME: It was pretty much what I expected. The only difference was having to stay more focused on the game and having a more one track mind for what you’re looking for. Two caveats come with it though. I paid less attention when the play was in the neutral zone, so I may not have seen the plays develop as much. I also watched it on a GCL replay, so I wasn’t worrying about the outcome, and, like JP, was afforded use of the back button.

renstar: I found it to be hard at first, but once I got into a zone it got much easier. In many ways, it felt the same as refereeing a game; I was completely dispassionate, yet I had a stronger feel for the flow of the game. In the first period, looking at my paper and seeing nothing but “P” in the first column hammered home how little control of the game the Caps had at that point. I also consciously watched the game with the sound off, as I didn’t want crowd or announcer noise to bias me either way. After tracking chances on delay with GCL, I don’t think I could track chances live. I rewound the stream far too many times to verify what I thought I was seeing. It took a lot of effort to be consistent, though any difficulty I had trying to be consistent about all of this probably goes away with more experience.

D’oh: At first, the coding of chances in itself seemed to be pretty easy. I went in with a clear image in my head of what I thought a scoring chance was, along with the factors that I planned to apply in the grey areas (Was there traffic? Was the traffic effective? Was the pass a good one? Was the shot on a backhand? Was the shot contested? Etc.). Still, there were some grey areas that I couldn’t resolve. For instance, was a scoring chance dependent on the players involved or was it purely based on the physics of the matter? That is, would a 45-foot wrister through traffic count the same if it came off Jeff Halpern’s stick or Alex Ovechkin’s? Ultimately, I went the route of common sense and made my chances dependent on the shooter. What really shocked me was the degree of focus it took to watch the game in this way. My mind started to wander in the second, and then it took a couple five-minute breaks in the third. I found myself rewinding and re-watching a lot of sequences. Even with all the rewinding, there was a chance that I forgot to log (which I later corrected). With practice and more focus, I think I could at least become consistent at logging chances, even if others might dispute my accuracy.

GOULD: I came into this with some vague sense that it would be pretty easy and straightforward — something you can do while watching a game you would otherwise be watching anyway. That was wrong. Everyone keeps using the same word — focus — and there’s a good reason for that. I’m used to tuning out hockey, maybe writing a comment on a gameday open thread or getting up to go do something else for a bit. You can’t do any of that and you can’t zone out when you’re scoring chances.

Q2. What was your general focus when you were trying to determine scoring chances? Was it a strict geographical focus, anything inside the “home plate” area was automatically a scoring chance? What were factors that informed your decision to discount home plate shots or to count a shot from further out?

RP: My focus was much more on dangerous shots (as Knee High has called it, “the pucker test”), or shots against that tend to make coaches angry. I did not automatically code every shot inside the home plate area as a scoring chance. One mitigating factor that would switch a home plate shot to a non-scoring chance is if the D was in position to block the shot and did block the shot. To me that indicates good defensive coverage and isn’t necessarily a breakdown that left the offense open to exploit. I can think of at least one specific example of that from this game. I don’t think I was counting any scoring chances from outside the home plate area. To be a scoring chance from that far out it would probably have to include both a cross-ice passing element and a hefty screen.

I also tried to evaluate things ex ante, not ex post. So you ask whether you would be comfortable giving up that shot again. I don’t have Letang’s goal as a scoring chance, for example. It was a good shot, there was net presence, but Neuvirth saw the puck and just guessed wrong. The fact that it was a snipe doesn’t change things, that’s just good execution and a good break for the Pens, but not the kind of shot I’d consider a true scoring chance.

SME: I did it pretty similarly, using “the pucker test” but I was a little more lenient. I counted Letang’s goal as a scoring chance; he got a good shot off from the middle of the ice, with traffic, and got it through. If it had gotten blocked or missed, I wouldn’t have counted it, but the fact that he made Neuvirth have to make a save (that he didn’t make) under tough circumstances made me code it as a chance (however, that is far from the only criteria). When coding the chances, I took into account goalie position, shooter, shot circumstance, and defensive positioning. In general, I looked at it as if I were the coach – would I be happy that my team just gave that shot (in most cases) up? If the answer is no, and I think my team could, and should, have played it differently, then I coded it as a chance, for the most part.

JP: I tried to do it more by-the-book – if it was in the “home plate” area, it was a chance; outside, no chance. There were certainly times I was thinking, “Well, the goalies in position and there’s a defender between the shooter and the netminder and backside pressure on the shooter… but it’s a “scoring chance.” Which, I suppose, is another general point to consider – perhaps not all scoring chances really are. Conversely, there may have been some from outside the “home plate” area that, due to the pass that set it up, the goalie’s position, and the shooter, might not have technically been a scoring chance… but certainly were. Yeah, I’m thinking of Alex Ovechkin’s goal – I had that outside the scoring chance area (barely), but it was a chance (and a goal). (And I agree on the Letang goal – I didn’t have it as a chance.)

SME: Another thing I looked for was a deliberate play, or attempt to make a play towards the net. If there was a pass to a player with an open net, and he mishandled or misshot the pass, I considered it a chance, depending on how close it was, of course. But if the puck took a flukey bounce towards the net in a manner clearly not intended by the player, it wasn’t a chance, regardless of how close it was. There had to be a certain level of intent to get it to the net.

GOULD: My standard was “if what that guy was trying had worked, that would have been a goal” or, in other words, “I’ve seen goals scored that way before, and it wasn’t a fluke.” I didn’t focus on home plate or anything like that. If it felt like an above average shot or an above average pass would get through, then I called it a chance. And yes, I said “pass.” A lot of what I called chances were plays where I thought two offensive players had isolated one defensive player, and they were one pass from having someone alone in front of the goalie with the puck on his stick. Like when there’s a guy behind the net, a guy in front of the net, and one defenseman between them. I think those bang-bang plays are among the most dangerous in hockey, and the fact that the puck didn’t get through doesn’t mean it couldn’t have. Obviously, anyone who looks at the stats will see that the result was that I was much more generous with chances than everyone else. Take this play: The Caps generate an offensive-zone turnover, and Brooks Laich controls the puck on the boards. He saucers a pass over to Alex Semin on the doorstep. A split-second difference in timing, and that might be a goal. Semin wasn’t ready for the pass, but if he had been he just needed to flick that one into the net. I think I scored a lot of passing plays like this one as chances. Because the way I figure, everyone else is scoring as chances events where with a better shot, there would have been a goal. But why not score plays where with a better pass there would have been an easy goal?

renstar: So my approach was to use the textbook definition as a starting point and count other situations that I thought could have reasonably resulted in a goal. I (very loosely) defined ‘reasonable’ to mean, even if there was no shot, “If a player on the offensive (defensive) team had done one thing differently for the better (worse), it could have resulted in a goal.” I also counted any goal that was a result of a direct offensive attempt by an offensive player. That is, a pass to the front of the net resulting in an accidental deflection off of a skate would have counted. Weird bounces from behind the net generally would not. If the shot was from outside of the “home plate,” I counted it. After all, a scoring chance is a chance to score, and if a score resulted from an offensive play, there was by definition a chance that it would happen. (Though I would probably not count a shot from the red line…subjectivity!) I started out skeptical of the “home plate” area, but as the game went on I started to see the merit of that zone as the primary scoring area. That said, I wasn’t strict about it. If a shot was a foot outside the dots or two or three feet from the top of the circles, depending on the ‘setup’ of the ice, I counted it. I also counted shots from way outside if the passing sequence led to the defense or goalie being off balance or out of position. Finally, I decided beforehand that I was going to look for situations where the puck was floating around in a dangerous area, e.g., in or around the crease. If an offensive player had a reasonable chance to play the puck, even if they whiffed or had their stick tied up, I counted it under the theory that the more the puck is in that area, the more likely the defense is to give up a dirty goal. If no one was nearby, I didn’t count it.

D’oh: During the Penguins’ first-period power play (around 3:50 left in the first), there was a sequence where Malkin took two one-timers from near the top of the right faceoff circle. I counted one as a scoring chance, but I didn’t count the other. The first shot was from slightly further away and from a worse angle. Moreover, the pass took longer to get to Malkin, which let Neuvirth get set. The second shot was from a better angle, and the pass got to Malkin’s stick much faster – as a result, Neuvirth didn’t come over square. Although Malkin put the shot high and wide, had he put it on net it had a great chance of going in. That sums up how I went about this exercise. I tried to take a holistic view of the play, not just the distance and angle of the shot relative to the “home-plate area.” Several big factors for me were: Was there effective traffic – not just players around the net, but unguarded players with their sticks ready for deflections or rebounds? How much time and space did a player have? Was there someone right on him? Was the player on their forehand or backhand? Did the player shoot right-handed or left-handed, and how did this affect the angle of their shot? Was the goalie in good position? Was the pass playable? There was a play in the first where Chimera threw a pass to the far post for Alex Semin. Had Semin got a stick on it, it was likely a goal. Nevertheless, the pass was a couple feet in front of Semin and he had no chance to get his stick on it – thus it wasn’t a scoring chance by my standards. Part of my standard was that “it takes two to tango;” in other words, you had to have both a good pass and a reasonable chance of reception. A terrible pass to a wide-open player wouldn’t cut it, nor would a great pass to a player whose stick was tied up.

Q3. One point touched on in response to that last question – should the shooter have any impact on whether something is a scoring chance or not?

RP: I definitely believe that the shooter matters, and the type of shot matters. One chance that I had noted as questionable (but didn’t count as a scoring chance) was a Mike Knuble backhand from the slot. He had defensive pressure and couldn’t get a lot on the shot. Geographically that’s in the heart of home plate, but he didn’t really get a clean shot off and it was on the backhand. Add in that Knuble isn’t known for his shooting prowess and I just couldn’t count that as a legitimate scoring chance. On the other hand, J.P. talked about how Ovechkin’s goal wasn’t a scoring chance because of where he shot from. I counted that as a scoring chance, but most shooters probably would not qualify from there. It makes no sense to me to treat shots from the same location as equal if the two players taking the shots are not equal (and this is really just a variation on what J.P. discussed above regarding not all scoring chances being equal). You can’t say that a shot by Ovechkin or Alexander Semin is the same kind of chance as a shot from Karl Alzner or Jeff Schultz from the same location (though a shot from the press box probably isn’t a scoring chance regardless). Logically that just makes sense. And if you take it a step further we can see that evaluating scoring chances by considering the shooter makes even more sense because it aligns with how the game is played. Coaches specifically put systems and checking lines together to limit certain players. A breakdown that allows Ovechkin or Stamkos a chance to get an open look is the kind of breakdown that makes coaches pull their hair out: that’s exactly the kind of play you want to prevent. A breakdown that allows a fourth liner to take a quality shot is not going to be nearly as aggravating (though obviously is not something the coaches want to see). Think of the Dmitri Orlov breakdown from a few weeks back, those were quality chances but they were 4th liners. I don’t think you can treat those chances the same as if they came against Neal and Malkin (and that’s probably why Orlov doesn’t see much ice against the Neals and Malkins of the league). In short, the shooter is a part of the breakdown, a part of the mistake. Without the shooter, you have an incomplete picture of the play as a whole and thus an incomplete picture of the putative scoring chance.

JP: I guess it was a bit of a trick question on my part. On the one hand, of course the shooter matters. That’s just common sense. But on the other hand, if you’re going to base what you consider a scoring chance on who’s shooting (and other factors), how useful can the data be to anyone else? It gets back to grading chances. I’m sure you could track chances in a manner wherein you’re taking the spot on the ice (objective), the shooter (subjective) and mitigating factors such as defenders, shot type, etc. (subjective). But again, the more subjectivity you introduce the less useful the data becomes to a larger community of would-be analysts. Essentially, what Rob’s describing probably mirrors what teams are doing internally, and what I’m talking about is how the rest of us are doing it. And that’s another major point – there’s no way that teams treat all shot attempts from inside the “home plate” area the same.

SME: The shooter definitely matters for all of the reasons that Rob mentioned. Teams are not going to play all players equally, and that is going to create opportunities that otherwise wouldn’t exist. I highly doubt any coach thinks his team is doing its job if it allows Malkin as many one timers from the dots as he got in the game. However, if they’re stopping Malkin from getting those, I’d have to think the coach is happy with that, even if they’re giving up that shot to someone like Adams. How many times have players like Malkin or Ovechkin scored (or rang the post) on shots that other players wouldn’t have even attempted? It makes it far more difficult to create a method of tracking that is highly useful across teams, but it would also be inaccurate to treat all shooters as equal. It’s a cruel reality of tracking scoring chances. Hockey isn’t a sport composed of events that are so easily separated from one another, like baseball. However, I still think there is utility in the ratio to show who had greater control in a game.

GOULD: I tended not to pay attention to shooter — at least not consciously. Since my standard was basically “have I seen non-fluky goals scored that way,” I didn’t care who was driving the play, only what was happening. Honestly, it didn’t occur to me to take the shooter into account. Would I do it differently next time? I don’t know — I’m as curious as the rest of you to read the other responses to this question. But I do worry that depending on shooter could become recursive in a bad way. “Player A is a great player because he has more scoring chances. Therefore, I will rate plays as scoring chances when they are made by Player A which I wouldn’t for other players.”

renstar: I think the shooter should definitely matter. That said, I had enough trouble deciding whether the situation should result in a counted chance and never even considered the shooter in my determination. Even with tape-delay and rewind, the task is difficult enough. Tracking the chances might have taken me an additional 30 minutes if I had looked out for the shooter (it took at least a couple of hours already). I think it would be near impossible to do live.

D’oh: Something that really vexed me was whether or not to change my judgement of a scoring chance based on which players were involved. At 17:38 of the third period, there was a scrum in front of the Caps’ net. The Pens had Neal and Kunitz there, both of whom I think are pretty good around the net. I scored it as a chance, but had that been Park and Vitale, I might not have. In the end, I came down pretty much exactly the way Rob did – I even recall going through the exact same thought process during both the Ovechkin goal and the Knuble backhand, and I scored them the same way (chance and not a chance, respectively). At the same time, I agree with JP – judging things by the shooter introduces another element of subjectivity to the process, and it’s another reason why I’m now less comfortable with the validity of scoring chance data than I might have been before.

Below is a table with each of our final results for each team during the game:

We also have a chart that shows how each player performed in the eyes of each scorer, with a column for the mean on the far right:

Tomorrow, in Part II, we’ll look at some specific examples and demonstrate how various factors resulted in our specific decisions.