HERO Charts – Frequently Asked Questions

Which seasons are included?

The charts encompass the last three NHL seasons (2014/15, 2015/16 and 2016/17). The seasons are weighted 44%-33%-22% based on temporal proximity. These weights were chosen based on findings in this piece on forecasting with Marcels.

Which skaters make the cut?

  • Top 420 Forwards (14 forwards * 30 teams) in weighted 5v5 time on ice to see at least 1 minute of NHL ice time this season
  • Top 210 Defensemen (7 defensemen * 30 teams) in weighted 5v5 time on ice to see at least 1 minute of NHL ice time this season

Why exclude second assists?

In 2011, Eric Tulsky wrote a piece in which he focused on forwards and more specifically, the persistence of goals, first assists and second assists. He not only found second assists to be less repeatable than goals and first assists from year-to-year, but he also demonstrated an even starker contrast when only looking at forwards who switched teams between seasons (thus removing the effects of team/linemate quality to a certain extent).

When recreating Tulsky’s experiment for defensemen, the results are very similar.

In addition, Michael Lopez published an article that highlighted another issue with second assists – scorekeeper bias. It turns out that scorekeepers tend to award home team stars with second assists at a rate that exceeds expectations given their second assist rates on the road.

To conclude, #SecondAssistsAreActuallyBad. They are no more than an ugly amalgamation of random noise and scorer bias – not exactly ideal when attempting to measure player talent.

Why only include 5v5 statistics when measuring performance?

The reason involves a combination of a few factors…

  • Special teams samples are relatively small (and therefore not very predictive of future results)
  • Special team stats are heavily influenced by systems/coaching
  • They are difficult to consolidate with even strength stats due to selection bias

How are performance tier cut-offs (ex. 1st liners, Middle-six forwards etc) determined?

The answer is simple. Of our 420 forwards, the tiers breakdown as follows for each metric…

1st Liner = Top 90 (3/team * 30 teams)

Middle 6 = Top 270 (9/team * 30 teams)

4th Liner = Top 360 (12/team * 30 teams)

And of our 210 defensemen…

Top-2 = Top 60 (2/team * 30 teams)

Top-4 = Top 120 (4/team * 30 teams)

Bot-2 = Top 180 (6/team * 30 teams)

Note: When choosing the maximum axis value for performance measures, skaters who manage results greater than 3 standard deviations above the mean are excluded.

Why doesn’t the COMPARE option work?

The “COMPARE” function is temporarily unavailable. It will be restored prior to the trade deadline.

What the heck is PERC?

“PERC” is just short for “percentile”. For example, a skater in the 97th percentile (PERC = 97) performed better than 97% of forwards in the sample in that specific metric.

What is pace?

Pace is a measure of a skater’s influence on the total number of shots that occur (for both teams) when he is on the ice (vs. league average). It is calculated by adding Shots Generated / Hour (vs. league average) and Shots Conceded / Hour. (vs. league average).

How are shot impacts measured and what exactly does “context-neutral” mean?

In the past, I have typically used relative measures (on-ice vs. on-bench) to quantify a skater’s impact on his teams’ shot differential. The issue with that method includes the following…

  • Relative stats do not account for teammate nor competition quality
  • They are difficult to compare across teams. For example, a player with a CF% Rel of +5% on a team with poor shot results is not equivalent to a skater managing the same mark on a team that controls well over 50% of shots. Also, a skater can play behind a wealth of depth in his particular position (see Nashville’s d corps over the last couple of years) and thus he is wrongly penalised by Corsi Relative.

At one point, NBA analysts were faced with this same issue. Eventually, they moved away from traditional Plus-Minus measures and adopted what is now known as Adjusted Plus-Minus. Among analysts currently active in the public sphere, Dawson Sprigings (@DTMAboutHeart) laid the groundwork for a hockey variant in this piece on CPM (Corsi Plus-Minus). He would ultimately manage to successfully apply the technique in hockey and is now a key cog in his WAR model. His series on WAR can be found at hockey-graphs.com – A Primer on @DTMAboutHeart’s WAR ModelIntroducing Expected Plus-Minus, Introducing Box Plus-Minus, Extras, Blending and Seasonal Adjustment, Testing and Final Remarks). I especially recommend checking out his piece on Expected Plus-Minus to gain a better understanding of regularization methods and the researchers who have published their own adaptions.

In HERO charts, I adopt some of the techniques outlined in the past but the focus is exclusively on measuring shot impact and simply accounting for score effects, venue, QoT and QoC.

The Methodology

In order to account for teammate and competition quality, we will apply a technique called ridge regression (or Tikhonov regularization). While it sounds fancy, it is just a linear model that introduces an appropriate amount of bias (or “shrinkage”) to prevent over-fitting and alleviate issues caused by multicollinearity present in the data (or in simpler terms – deal with the fact that multiple skaters spend significant amounts of ice time together, thus making it difficult to delineate their impacts).  Let’s use a simplified example to demonstrate how the procedure works…

Consider the following shift. It is 3v3 just for the sake of simplicity, but all shifts are 5v5 in practice. (Note: shot results are score and venue adjusted)…

ridgeline

So, we have a 75 second shift in which the Ottawa Senators (Home) managed only 48 shots/hour while the Pittsburgh Penguins (Away) managed 96 shots/hour. In our regression, we are going to have to frame these results in two ways…

  • From the perspective of Ottawa (Home) as the attacking team, Pittsburgh (Away) as the defending team with an outcome of 48 shots/hour
  • From the perspective of Pittsburgh (Away) as the attacking team, Ottawa (Home) as the defending team with an outcome of 96 shots/hour

Setting up situation (1)

sit1

Each skater now acts as a dummy variable. A value of 1 means that they were on the ice and a value of 0 means that they were on the bench or not playing at all. In this case, all other NHL skaters would get values of 0 since they were not on the ice for this given shift. Keep in mind that all skaters are split into two variables – one for offense (O) and one for defense (D). In this row of data, the defense components for Ottawa players receive a value of zero, as well as the offense components for Pittsburgh players. Since shots/hour is our response, a higher value will award attacking skaters (Turris, Ryan and Ceci) and penalize defending skaters (Crosby, Malkin and Letang).

Setting up situation (2)

sit2

Now we observe that the roles of offense and defense have been simply reversed.

We can now combine the two rows of data to get the following matrix (keep in mind that ALL other NHL skaters receive values of 0 for offense and defense since they are not on the ice)…

matrix

This is done for every shift that occurs in a season and the rows are bound together to form one (massive) matrix. We can then use the glmnet package in R to execute our regression. We first run a 10 fold cross validation to tune our lambda parameter (this is what controls the degree of shrinkage). Finally, we identify shots/hour as our response, shift lengths as our weights and our skater dummy variables as our predictors. The output gives us the estimated impact, in terms of both shots generated / hour vs. average (O) and shot conceded / hour vs. average (D), for all skaters in our sample. The beauty of the regression is that it simultaneously accounts for a skaters’ linemates and competition, thus the outputs are termed “context-neutral”. There are certainly other contextual factors to consider beyond teammates and competition (ex. zone starts, coaching etc) but for now, we will ignore them (perhaps integrating them into the procedure in the future).

Comparing the persistence of ridge regression coefficients to relative measures and raw rates when skaters switch teams (82 game seasons, 2007/08 to 2015/16, 30+ GP per season)…

teamchange

What we are aiming for is a metric that is not only persistent from year-to-year, but one that persists even if a skater changes teams. This stress test helps expose a reliance on contextual elements instead of what we want – skater talent. Observing the chart posted above, it is evident that raw measures are inferior (with respect to both positions) in comparison to the two other measures tested. Relative metrics slightly outperform ridge regression coefficients for forwards, but falter when the sample is limited to defensemen. To conclude, the ridge regression coefficients make the most sense from a theoretical standpoint and seem to produce the best results (in general) when considering how both positions fare under the circumstance of the experiment.

Raw data via Corsica.hockey or Emmanuel Perry’s DryScrape function

If you have a question that was not answered in this post, you can contact me at MimicoHero@yahoo.com or hit me up on twitter, @MimicoHero

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s