<commit_before>\section{A Data-driven Rating System}\label{sec:club_rating}

As an alternative to direct human assessment ratings (whether self-assessment or captain assessment), we advocate here for a data-driven rating system that considers each player's history within BUDA club leagues. Unlike recreational leagues in which players are randomly assigned to teams according to skill level, players in club leagues form their own teams and select a division of competition which they believe to be appropriate to their team-wide skill level.  There are four divisions ranging from casual (Division 4) to highly competitive (Division 1).

Each club team uses a vetting process to determine whether or not a player can join their team. This is often as simple as a friend's recommendation, but it can also be as involved as a multi-day tryout. The key point is that in order to join a team, a player must have exhibited evidence that they will be competent at the skill level of the division in which that team competes.

Based on our experience of playing in different BUDA club league divisions, we begin by assigning the following ratings to each division in the style of the Elo system (reference): Division 1: 1800, Division 2: 1400, Division 3: 1100, and Division 4: 900.  The idea is that a Division 1 team is expected to beat a Division 2 team 90\% of the time, a Division 2 team wins 85\% of the time against a Division 3 team, and a Division 3 team beats a Division 4 team 75\% of the time.

The next step is to acknowledge that within each division, there is a wide range of team-wide skill level. Again, our experience with BUDA suggests that a weak Division 1 team is comparable to a strong division 2 team.  With this in mind, we calculate team ratings as the following:

\begin{equation}
{\rm Rating}_{\rm team} = {\rm Rating}_{\rm division} + 60 \times {\rm AGD},
\end{equation}

\noindent where ${\rm AGD}$ is the average goal differential per game for that team. Under this definition, a Division 1 team that loses on average by 3.3 goals per game has an equal rating to a Division 2 team that wins on average by 3.3 goals per game.

This methodology yields a team rating for every club team in the BUDA database. The club rating for a given player is the arithmetic mean of all of their team ratings leading up to that point in time. Here is a concrete example. Suppose a player has experience on an average Division 1 team $({\rm AGD} = 0)$, a strong Division 2 team $({\rm AGD} = 3)$, and an average Division 2 team $({\rm AGD} = 0)$. In this scenario, their club rating will be $\frac{1}{3}(1800 + 1580 + 1400) = 1593$.

The last step that is unique to assigning a club rating to a player is the conversion to the same 0 to 100 scale used by BUDA for self-assessment ratings and captain ratings. The simple philosophy guiding this process is (once again) our own personal experience within BUDA, which suggests that an average Division 1 player will have a club rating of 80, while an average Division 2 player will have a club rating of 60. An average Division 3 player gets a club rating of 45 and an average Division 4 player gets a club rating of 30. We use interpolation to fill in the gaps for ratings between these values.

Figure~\ref{fig:correlation_club} shows the relationship between team performance and club rating. The correlation between club rating and team performance is stronger than between captain rating and team performance ($r = 0.43$ for club rating, $r = 0.31$ for captain rating). This demonstrates that club rating is a better predictor of team performance than captain rating---and a much better predictor than self-assessment rating. If a player has a high club rating, that is a very strong indicator that they are a good player (and vice versa).

One of the limitations of the club rating method is that there are players without any club league experience.  On average, a roster of 16 players can expect to have 3 players without club experience. Although this is better than the average of 4 players that typically do not have a captain rating--see Section~\ref{sec:self_rating}--it's still a gap to account for. Some of these are players new to ultimate or simply new to BUDA. Individual self-assessment ratings are the only means by which to rate these players. Others have significant experience in recreational leagues, but zero experience in club leagues. For this latter subset of players, we use a captain rating if available. 
<commit_msg>edited A_Data_driven_Rating_System.tex<commit_after>\section{A Data-driven Rating System}\label{sec:club_rating}

As an alternative to direct human assessment ratings (whether self-assessment or captain assessment), we advocate here for a data-driven rating system that considers each player's history within BUDA club leagues. Unlike recreational leagues in which players are randomly assigned to teams according to skill level, players in club leagues form their own teams and select a division of competition which they believe to be appropriate to their team-wide skill level.  There are four divisions ranging from casual (Division 4) to highly competitive (Division 1).

Each club team uses a vetting process to determine whether or not a player can join their team. This is often as simple as a friend's recommendation, but it can also be as involved as a multi-day tryout. The key point is that in order to join a team, a player must have exhibited evidence that they will be competent at the skill level of the division in which that team competes.

Based on our experience of playing in different BUDA club league divisions, we begin by assigning the following ratings to each division in the style of the Elo system (reference): Division 1: 1800, Division 2: 1400, Division 3: 1100, and Division 4: 900.  The idea is that a Division 1 team is expected to beat a Division 2 team 90\% of the time, a Division 2 team wins 85\% of the time against a Division 3 team, and a Division 3 team beats a Division 4 team 75\% of the time.

The next step is to acknowledge that within each division, there is a wide range of team-wide skill level. Again, our experience with BUDA suggests that a weak Division 1 team is comparable to a strong division 2 team.  With this in mind, we calculate team ratings as the following:

\begin{equation}
{\rm Rating}_{\rm team} = {\rm Rating}_{\rm division} + 60 \times {\rm AGD},
\end{equation}

\noindent where ${\rm AGD}$ is the average goal differential per game for that team. Under this definition, a Division 1 team that loses on average by 3.3 goals per game has an equal rating to a Division 2 team that wins on average by 3.3 goals per game.

This methodology yields a team rating for every club team in the BUDA database. The club rating for a given player is the arithmetic mean of all of their team ratings leading up to that point in time. Here is a concrete example. Suppose a player has experience on an average Division 1 team $({\rm AGD} = 0)$, a strong Division 2 team $({\rm AGD} = 3)$, and an average Division 2 team $({\rm AGD} = 0)$. In this scenario, their club rating will be $\frac{1}{3}(1800 + 1580 + 1400) = 1593$.

The last step that is unique to assigning a club rating to a player is the conversion to the same 0 to 100 scale used by BUDA for self-assessment ratings and captain ratings. The simple philosophy guiding this process is (once again) our own personal experience within BUDA, which suggests that an average Division 1 player will have a club rating of 80, while an average Division 2 player will have a club rating of 60. An average Division 3 player gets a club rating of 45 and an average Division 4 player gets a club rating of 30. We use interpolation to fill in the gaps for ratings between these values.

Figure~\ref{fig:correlation_club} shows the relationship between team performance and club rating. The correlation between club rating and team performance is stronger than between captain rating and team performance ($r = 0.43$ for club rating, $r = 0.31$ for captain rating). This demonstrates that club rating is a better predictor of team performance than captain rating---and a much better predictor than self-assessment rating. If a player has a high club rating, that is a very strong indicator that they are a good player (and vice versa).

One of the limitations of the club rating method is that there are players without any club league experience.  On average, a roster of 16 players can expect to have 3 players without club experience. Although this is better than the average of 4 players that typically do not have a captain rating---see Section~\ref{sec:self_rating}---it's still a gap to account for. Some of these are players new to ultimate or simply new to BUDA. Individual self-assessment ratings are the only means by which to rate these players. Others have significant experience in recreational leagues, but zero experience in club leagues. For this latter subset of players, we use a captain rating if available. 

{'fraction_non_alphanumeric': 0.044161593801881574, 'fraction_numerical': 0.02058660763696735, 'mean_word_length': 4.293497363796134, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2111732', 'n_tokens_mistral': 2248, 'n_tokens_neox': 2012, 'n_words': 1507}