# Does it work?¶

## TL;DR¶

The attributes that make the underlying survival model more accurate and useful from a game-level have a significant impact on how the metric rates players across the league; the metric becomes *contextual*, in that the same event under the same in-game conditions will be viewed differently based on exogenous factors such as schedule and team or opponent net rating. This effect is particularly visible with players on teams that have an extreme net rating.

The metric is limited by the gaps in traditional play-by-play data. We can only account for events that are explicitly logged; the effect of ball denial, “hockey” assists, individual defense, and other visible but implicit events will be severely underestimated.

## Review methodology¶

First, we will examine the relationship between SPA and the team net rating. If the metric is working as expected, we should see that “winning teams” have high impact players.

### Team-season aggregation¶

First, we will load the team net ratings for each team between 2006-07 and 2020-21. We will bin the team-seasons to deciles, and plot the distribution of SPA ratings.

Interestingly, there is a sharp decline in total SPA for teams with extreme net rating values. Teams that have more neutral net ratings seem to be viewed more favourably by the metric.

### Player-season aggregation¶

Next, let’s aggregate the game-level ratings to the player-season level to understand how players are viewed based on the context of their team quality.

This plot shows that players on teams with a net rating between -1.2 and 2.9 have the highest median SPA rating; somewhat surprisingly, players on elite teams (net rating 6.6-12.1) have a median SPA rating lower than players on terrible teams (net rating between -15.5 and -7).

However, team net rating is an input to the underlying time-varying Cox proportional hazards model. This means that team net rating is likely to influence how the model rates the impact of specific events. To analyze this, we will look at the distribution of pre-game win probability for each team.

We can see that there is a clear relationship between the team net rating and the pre-game win probability.

### Examining the model sensitivity to net rating¶

Let’s examine how the model reacts to net rating. To do this, we will create a simulated game with fixed exogenous variables and vary the net rating.

Based on the graph above, we can infer that net rating impacts both the initial win probability for a game as well as how the model rates the impact of a given event. Generally speaking, the shifts in win probability will be less drastic as net rating rises, and this is especially true when the win probability is low. As the win probability rises, the shifts from event to event will be similar between good and great teams.

Additionally, a high net rating implies that a given team is playing more minutes with a large lead. Let’s repeat the experiment above, but instead of varying home net rating we will vary the margin of the game.

Intuitively, the model discounts the impact of events in games with a high scoring margin. So, players will be evaluated differently based on how good their team is as well as the margin of the game; as a result, players on good teams will have a lower SPA rating than players on more neutral teams.

### Adjusting the raw SPA ratings by removing contextual information¶

So, we know that SPA ratings will be sensitive to the contextual variables for each game; a basket scored by a player on a well-rested, highly rated team will be viewed as less impactful than a basket scored by a player on a mediocre team that played last night. However, we should ask: how would the ratings change if we isolated them to the direct actions players take during the game? To answer this question, we regenerated the play-by-play survival probability for every game with no contextual information – just time and score.

In our view, the graph above is more balanced. Players on elite teams have the highest peak total impact, although the sum total might still be affected by playing in low-leverage, high margin games. In the end, we are satisfied with the insight this adjustment gives.