<jupyter_start><jupyter_text>A Formula for Success: Predicting How to Win in Baseball Notebook 1  I. The ProblemBaseball is a sport of numbers and statistics. Because of this, there are all sorts of measurements of what happened in a game, as well as team and specific player performance. Over the years, and made more popular by fantasy sports and books/movies such as Moneyball, statistics have gotten even better at determining the value of players. These advanced statistics are commonly referred to as sabermetrics.But what is the formula for a team's success? Obviously, it's winning, because a team must win in order to become championship contenders. And obviously, winning is accomplished by scoring more runs than your opponent. But if you were a general manager of a baseball franchise, you would want to go beyond that to determine a more precise equation to field a consistently winning team. Then, based on this formula, a general manager should be able to determine a player's value in terms of contributing to a team's winning chances. Research QuestionsAs such, this project focuses on answering the following questions:* What features most significantly impact the winning percentage of a given Major League Baseball (MLB) team?* For those given features, to what extent do they impact a team's winning percentage?* Based on this, who has been the most valuable player in contributing to a team's winning percentage from 2015-2020?    By answering these questions, a general manager can build a roster to maximize it's odds of winning. As such, a general manager could target particular players in free agency or through trades to acquire them. Furthermore, since spending in baseball is limited based on what a particular team can afford in its market, as well as a tax beyond a particular salary threshold, a general manager could use this formula to determine if a given player is an improvement over a player currently rostered at that position on the team.This particular notebook will work to answer the first two of the research questions.  II. Data AcquisitionTo answer these questions, we will need to find data over the last few seasons from MLB. We will need both team statistics to determine the equation for winning and player statistics to determine who has contributed the most to winning. To acquire this data, we will use the MLB statistic repository Fangraphs.com. This data will be pulled specifically from the [database found here](https://www.fangraphs.com/leaders.aspx?pos=all&stats=bat&lg=all&qual=y&type=8&season=2020&month=0&season1=2020&ind=0). We will use multiple requests, then export each of them. For the team stats, you click on the "Team Stats" panel, select multiple seasons, split them, hit submit, then export. Do this for the Batting, Pitching, and Fielding tabs. We split the seasons because a given team will vary in its performance--and its roster--from year to year, adding to our sample size. To build an adequate sample size, we will pull data for each season beginning in 2006. For the individual stats, you click on the "Player Stats" panel; because we are viewing individual players stats over the past few seasons, we will select "Multiple Seasons" from 2015 to 2020.While the years selected for this sample seem arbitrary, they are not. Baseball is known to have time periods that stylize play, commonly known as eras. From the late 1980s into most of the 2000s, power hitting had a surprising jolt due to the prevalence of steroids and other performance-enhancing drugs. Long-standing home run records were broken multiple times during this period. The 2006 season is when MLB began to test and discipline players for performance-enhancing drugs, and therefore is a good starting point for our dataset.This process can easily be repeated by others who wish to replicate or customize the data pulled for their own usage. To begin, we will need to pull the various dataframes, then combine them into a single dataframe.<jupyter_code># import needed tools and libraries
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
# Retrieve CSVs into dataframes
team_bat = pd.read_csv(r'~/Github/DATA601_FinalProject/Data/TeamStats_Batting.csv')
team_pitch = pd.read_csv(r'~/Github/DATA601_FinalProject/Data/TeamStats_Pitching.csv')
team_field = pd.read_csv(r'~/Github/DATA601_FinalProject/Data/TeamStats_Fielding.csv')

# Update team names with year to prevent confusion (So 2019 Orioles and 2020 Orioles are clear in Team column)
team_bat['TeamYear'] = team_bat['Season'].astype(str) + '_' + team_bat['Team']
team_pitch['TeamYear'] = team_pitch['Season'].astype(str) + '_' + team_pitch['Team']
team_field['TeamYear'] = team_field['Season'].astype(str) + '_' + team_field['Team']

# Move new TeamYear column to front
team_bat = team_bat[['TeamYear'] + [col for col in team_bat.columns if col != 'TeamYear']]
team_pitch = team_pitch[['TeamYear'] + [col for col in team_pitch.columns if col != 'TeamYear']]
team_field = team_field[['TeamYear'] + [col for col in team_field.columns if col != 'TeamYear']]

# Merge dataframes
teamstats = pd.merge(team_bat, team_field, how = 'outer', on = 'TeamYear')
teamstats = pd.merge(teamstats, team_pitch, how = 'outer', on = 'TeamYear')

# Drop repeat columns and rearrange columns
teamstats.drop(['Season_x', 'Team_x', 'Season_y', 'Team_y'], axis=1, inplace = True)
move_cols = ['TeamYear', 'Season', 'Team']
teamstats = teamstats[move_cols + [col for col in teamstats.columns if col not in move_cols]]

# Check dataframe
teamstats

# Save dataframe to CSV
teamstats.to_csv(r'~/Github/DATA601_FinalProject/Data/TeamStats_Combined.csv', index = None)<jupyter_output><empty_output><jupyter_text>III. Data Exploration - Part 1Why part one. Well, as you will see below, we will actually need to do some data cleaning and feature engineering in order to properly explore the data.<jupyter_code># Copy teamstats dataframe by pulling from newly saved CSV
df = pd.read_csv(r'~/Github/DATA601_FinalProject/Data/TeamStats_Combined.csv')
df.head()
# Look at the stats of the data
df.describe()
# Look at details of the data
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 450 entries, 0 to 449
Data columns (total 64 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   TeamYear  450 non-null    object 
 1   Season    450 non-null    int64  
 2   Team      450 non-null    object 
 3   G_x       450 non-null    int64  
 4   PA        450 non-null    int64  
 5   HR        450 non-null    int64  
 6   R         450 non-null    int64  
 7   RBI       450 non-null    int64  
 8   SB        450 non-null    int64  
 9   BB%       450 non-null    object 
 10  K%        450 non-null    object 
 11  ISO       450 non-null    float64
 12  BABIP_x   450 non-null    float64
 13  AVG       450 non-null    float64
 14  OBP       450 non-null    float64
 15  SLG       450 non-null    float64
 16  wOBA      450 non-null    float64
 17  wRC+      450 non-null    int64  
 18  EV_x      180 non-null    float64
 19  BsR       450 non-null    float64
 20  Off       450 non-null    float6[...]<jupyter_text>At first glance, we see a mixture of different datatypes, including strings/objects, integers, and floats. However, aside from the "Team" column, all of our data should be numerical. This means that the other columns labeled as objects are incorrectly categorized. If we look back at the header (first five rows) of the table, we see that this is due to the percentage sign (%) being used in the column. We will need to correct this during our data cleaning. We also see some columns have many null values. We will need to clean these as well.<jupyter_code># Check the number of unique values
df.nunique(axis = 0)<jupyter_output><empty_output><jupyter_text>One thing that stands out is that there are 31 unique entries for the Team feature. Considering there are only 30 MLB teams, we know that there must be something we need to alter here. No teams/franchises have moved; however, the Tampa Bay Devil Rays did rename themselves the Rays for the 2008 season. That could be the cause. Let's check if that is what we are seeing here.<jupyter_code># Look for team name including string 'Rays'
df[df['Team'].str.contains('Rays')]<jupyter_output><empty_output><jupyter_text>As we see above, the Tampa Bay franchise has two names in our dataset: the Devil Rays and the Rays. We will need to clean this up for simplicity in the data cleaning section. Since our target variable would be wins, let's take a look at it more closely.<jupyter_code># Box plot for wins
sns.boxplot(y = df['W'])<jupyter_output><empty_output><jupyter_text>We see a ton of outliers here, which is something worth exploring. Let's view this distribution in a histogram.<jupyter_code># Histogram of wins
x = df['W']
num_bins = 10
n, bins, patches = plt.hist(x, num_bins, facecolor='blue', alpha=0.5)

# Histogram settings
plt.xlabel('Wins')
plt.ylabel('Frequency')
plt.title('Histogram of Frequency of Wins in a Season')
plt.show()<jupyter_output><empty_output><jupyter_text>We would expect to see a rather normal distribution here, and we can see a portion of the histogram between approximately 50 and 105 wins appears normally distributed. So what could explain this?It is very likely the result of the shortened 2020 season. The standard MLB season is 162 games; due to the COVID-19 pandemic, the 2020 season was shortened to only 60 games.This leaves us with a decision to make: do we remove the 2020 season altogether, or do we still keep it as part of our dataset? III. Data Cleaning - Part 1The solution we will use will be to view wins as a percentage of total games played rather than raw total wins. This is otherwise known as winning percentage. We will need to create this as a new feature.<jupyter_code># Create new Winning Percentage column
# Win percentage = Wins / Games Started
df['W%'] = (df['W']/df['GS'])

# Check with header
df[['Team', 'W', 'L', 'W%']]<jupyter_output><empty_output><jupyter_text>Now let's check the box plot and histogram again.<jupyter_code># Box plot for win percentage
sns.boxplot(y = df['W%'])
# Histogram of win percentage
x = df['W%']
num_bins = 10
n, bins, patches = plt.hist(x, num_bins, facecolor='blue', alpha=0.5)

# Histogram settings
plt.xlabel('Win Percentage')
plt.ylabel('Frequency')
plt.title('Histogram of Frequency of Win %')
plt.savefig('WPHistogram.png', bbox_inches = 'tight')
plt.show()<jupyter_output><empty_output><jupyter_text>We see that the win percentage is evenly distributed, which shows that the shortened 2020 season was the issue. However, by extension, we can assume that other columns with raw total data as opposed to ratio data is similarly affected by the shortened season. This means that we must update these columns as well prior to exploring the data.<jupyter_code># Create list of columns to be converted
col_change = ['PA', 'HR', 'R', 'RBI', 'SB', 'Inn', 'BIZ', 'Plays','IP']

# Rewrite columns with their "per game" data
df[col_change] = df[col_change].div(df['GS'].values, axis = 0)

# Check work
df.head(10)<jupyter_output><empty_output><jupyter_text>Thankfully, this worked, as we see our data has been converted to a per-game basis. We will also need to change the string columns with percentages in them to floats. We will do this next.<jupyter_code># Create list of columns with percentages to convert to floats
col_change2 = ['BB%', 'K%', 'LOB%', 'GB%', 'HR/FB']

# Convert columns
df[col_change2] = df[col_change2].replace('%', '', regex = True).astype('float')/100

# Check results
df.head()<jupyter_output><empty_output><jupyter_text>By cross-referencing with the previous header, we see that we have successfully converted those percentage-objects to floats.Another large decision we will need to make is what to do with our null values. We saw that some of our columns have null values for greater tha 10% of the rows. Some are missing more than half. Given that the all columns with null values have at least 60 out of a total of 450 rows, each of these columns will be dropped. We can also drop the wins ('W') and losses ('L') columns because they are unnecessary now that we have created win percentage.<jupyter_code># Drop columns with null values
df.dropna(axis=1, inplace = True)

# Drop wins and losses
df.drop(['W', 'L'], axis=1, inplace = True)<jupyter_output><empty_output><jupyter_text>Now that we've dropped those, let's also update the Tampa Bay Rays/Devil Rays. We will rename the Devil Rays to the Rays (just as in real life), so that it does not appear that there are 31 franchises in MLB.<jupyter_code># Replace Devil Rays with Rays
df['Team'].replace('Devil Rays', 'Rays', inplace = True)

# Correct TeamYear column as well
df['TeamYear'] = df['Season'].astype(str) + '_' + df['Team']

# Check work
df[df['Team'].str.contains('Rays')]<jupyter_output><empty_output><jupyter_text>IV. Data Exploration - Part 2We can now proceed with more data exploration. Since our target variable is win percentage, let's take a look at the comparison of different teams' win percentage.<jupyter_code># Graph win percentage by team
plt.figure(figsize = (30, 20))
aplot = sns.boxplot(x = 'Team', y = 'W%', data = df)
aplot.axes.set_title('Box Plot of Win Percentage by MLB Team, 2006-2020', fontsize=30)
aplot.set_xlabel('Team', fontsize=20)
aplot.set_ylabel('Win Percentage (W%)', fontsize=20)
aplot.figure.savefig('BoxPlotWPTeam.png', bbox_inches = 'tight')<jupyter_output><empty_output><jupyter_text>Some interesting trends we see here:* Most teams have had a wide range of win percentages, indicating they had some good seasons and some bad seasons.* The Yankees, Dodgers, and Cardinals were more consistent, as they have smaller spreads than most teams.    * They also were consistently good teams, as they rarely had a win percentage below 0.5. The Dodgers even had an outlier season with a win percentage above 0.7.* The Astros, Rays, Tigers, and Orioles have very large spreads, indicating their season win percentage has had more variability.    * As an Orioles fan, I can sadly vouch for this. From 2006-2011, watching the Orioles was painful indeed, but 2012-2016 was much better until the team regressed again in 2017 and beyond. Let's look at the box plots and histograms of more features.<jupyter_code># Graphs settings for multiple features
plt.figure(figsize=(20, 15))
# Graphs of features
plt.subplot(3,3,1)
sns.boxplot(y = df['PA'], palette = 'Set1')
plt.subplot(3,3,2)
sns.boxplot(y = df['HR'], palette = 'Set2')
plt.subplot(3,3,3)
sns.boxplot(y = df['R'], palette = 'Set1_r')
plt.subplot(3,3,4)
sns.boxplot(y = df['RBI'], palette = 'Set2_r')
plt.subplot(3,3,5)
sns.boxplot(y = df['SB'], palette = 'cividis')
plt.subplot(3,3,6)
sns.boxplot(y = df['BB%'], palette = 'winter_r')
plt.subplot(3,3,7)
sns.boxplot(y = df['K%'],palette = 'Spectral')
plt.subplot(3,3,8)
sns.boxplot(y = df['ISO'], palette = 'cool')
plt.subplot(3,3,9)
sns.boxplot(y = df['BABIP_x'], palette = 'coolwarm')
# Graphs settings for multiple features
plt.figure(figsize=(20, 15))
# Graphs of features
plt.subplot(3,3,1)
sns.distplot(df['PA'], bins = 10)
plt.subplot(3,3,2)
sns.distplot(df['HR'], bins = 10)
plt.subplot(3,3,3)
sns.distplot(df['R'], bins = 10)
plt.subplot(3,3,4)
sns.distplot(df['RBI'], bins = 10)
plt.subplot(3,3,5)
sns.distplot(df['SB'], bins = 10)
plt.subplot(3,3,6)
sns.distplot(df['BB%'], bins = 10)
plt.subplot(3,3,7)
sns.distplot(df['K%'],bins = 10)
plt.subplot(3,3,8)
sns.distplot(df['ISO'], bins = 10)
plt.subplot(3,3,9)
sns.distplot(df['BABIP_x'], bins = 10)<jupyter_output><empty_output><jupyter_text>From the box plots above, we see a number of interesting trends.<jupyter_code># Graphs settings for multiple features
plt.figure(figsize=(20, 15))
# Graphs of features
plt.subplot(3,3,1)
sns.boxplot(y = df['AVG'], palette = 'Set1')
plt.subplot(3,3,2)
sns.boxplot(y = df['OBP'], palette = 'Set2')
plt.subplot(3,3,3)
sns.boxplot(y = df['SLG'], palette = 'Set1_r')
plt.subplot(3,3,4)
sns.boxplot(y = df['wOBA'], palette = 'Set2_r')
plt.subplot(3,3,5)
sns.boxplot(y = df['wRC+'], palette = 'cividis')
plt.subplot(3,3,6)
sns.boxplot(y = df['BsR'], palette = 'winter_r')
plt.subplot(3,3,7)
sns.boxplot(y = df['Off'],palette = 'Spectral')
plt.subplot(3,3,8)
sns.boxplot(y = df['Def_x'], palette = 'cool')
plt.subplot(3,3,9)
sns.boxplot(y = df['WAR_x'], palette = 'coolwarm')
# Graphs settings for multiple features
plt.figure(figsize=(20, 15))
# Graphs of features
plt.subplot(3,3,1)
sns.distplot(df['AVG'], bins = 10)
plt.subplot(3,3,2)
sns.distplot(df['OBP'], bins = 10)
plt.subplot(3,3,3)
sns.distplot(df['SLG'], bins = 10)
plt.subplot(3,3,4)
sns.distplot(df['wOBA'], bins = 10)
plt.subplot(3,3,5)
sns.distplot(df['wRC+'], bins = 10)
plt.subplot(3,3,6)
sns.distplot(df['BsR'], bins = 10)
plt.subplot(3,3,7)
sns.distplot(df['Off'],bins = 10)
plt.subplot(3,3,8)
sns.distplot(df['Def_x'], bins = 10)
plt.subplot(3,3,9)
sns.distplot(df['WAR_x'], bins = 10)
# Graphs settings for multiple features
plt.figure(figsize=(20, 15))
# Graphs of features
plt.subplot(3,3,1)
sns.boxplot(y = df['AVG'], palette = 'Set1')
plt.subplot(3,3,2)
sns.boxplot(y = df['OBP'], palette = 'Set2')
plt.subplot(3,3,3)
sns.boxplot(y = df['SLG'], palette = 'Set1_r')
plt.subplot(3,3,4)
sns.boxplot(y = df['wOBA'], palette = 'Set2_r')
plt.subplot(3,3,5)
sns.boxplot(y = df['wRC+'], palette = 'cividis')
plt.subplot(3,3,6)
sns.boxplot(y = df['BsR'], palette = 'winter_r')
plt.subplot(3,3,7)
sns.boxplot(y = df['Off'],palette = 'Spectral')
plt.subplot(3,3,8)
sns.boxplot(y = df['Def_x'], palette = 'cool')
plt.subplot(3,3,9)
sns.boxplot(y = df['WAR_x'], palette = 'coolwarm')
# Graphs settings for multiple features
plt.figure(figsize=(20, 15))
# Graphs of features
plt.subplot(3,3,1)
sns.distplot(df['AVG'], bins = 10)
plt.subplot(3,3,2)
sns.distplot(df['OBP'], bins = 10)
plt.subplot(3,3,3)
sns.distplot(df['SLG'], bins = 10)
plt.subplot(3,3,4)
sns.distplot(df['wOBA'], bins = 10)
plt.subplot(3,3,5)
sns.distplot(df['wRC+'], bins = 10)
plt.subplot(3,3,6)
sns.distplot(df['BsR'], bins = 10)
plt.subplot(3,3,7)
sns.distplot(df['Off'],bins = 10)
plt.subplot(3,3,8)
sns.distplot(df['Def_x'], bins = 10)
plt.subplot(3,3,9)
sns.distplot(df['WAR_x'], bins = 10)<jupyter_output><empty_output><jupyter_text>Let's view correlations with win percentage. We will also use a correlation heatmap to display this.<jupyter_code># Check correlations with win percentage
cor_matrix = df.corr()
print(cor_matrix['W%'].sort_values(ascending = False))
# Create object for graph
win_cor = df.corr()

# Remove half of heatmap
mask = np.zeros(win_cor.shape, dtype = bool)
mask[np.triu_indices(len(mask))] = True

# Graph correlation heatmap
plt.subplots(figsize = (25,20))
cor_heatmap = sns.heatmap(win_cor, xticklabels = win_cor.columns, yticklabels = win_cor.columns, cmap = 'coolwarm', mask = mask)
cor_heatmap.set_title('Correlation Heatmap of Original Dataset', fontdict={'fontsize':18}, pad=12)
plt.savefig('Heatmap1.png', bbox_inches = 'tight')<jupyter_output><empty_output><jupyter_text>We see a number of strong correlations between features here, such as slugging percentage (SLG) with runs (R) and runs batted in (RBI), which makes sense. If you hit the ball well and get on base, you'll score more runs. We see a strong negative correlation between strike outs per 9 innings (K/9) with balls in zone (BIZ) and plays made (Plays). This also makes sense, as a team that strikes batters out will have to field fewer balls hit by batters.In terms of correlations with win percentage (W%), we see the strongest positive correlation with offensive wins above replacement (WAR_x). This is expected, because if your players are batting farther above a given level (an average replacement player), you will likely have more wins. We see the strongest negative correlation with earned run average (ERA). This is also expected, because if you allow opposing teams to score fewer runs against you, you have a better chance of outscoring them for a win.  V. Data Cleaning - Part 2Given that we have explored the data as well as some correlations, we should start to remove some features to assist us in developing our model. We will do this through a few ways: first, we will choose to remove features that describe elements of the game rather than aspects which contribute to scoring or a lack thereof. As an example, plate appearances (PA) is a feature that describes how many at-bats a team has. Its positive correlation with win percentage would suggest that with more plate appearances, you are more likely to win. However, correlation does not equal causation. You earn more plate appearances through getting on base and avoiding outs. That's why plate appearances also has a strong positive correlation with batting average (AVG) and on base percentage (OBP). Thus, plate appearances is likely not useful in determining how a team wins, but is a byproduct of other features which contribute to winning. Similarly, earned run average (ERA) is not helpful in explaining how to stymie an opponent's scoring; it is a metric of the extent to which an opponent is able/unable to score against that team.We should remove features like plate appearances and earned run average, as well as other features that explain elements of the game, such as innings played (Inn) and innings pitched (IP).<jupyter_code># Drop unnecessary columns
df.drop(['PA', 'ERA', 'FIP', 'Inn', 'IP', 'LOB%'], axis=1, inplace = True)<jupyter_output><empty_output><jupyter_text>The second way we will narrow down the dataset is through recursive feature elimination (RFE). RFE essentially builds a model with every feature, then weighs each feature's importance. This allows us, through a process of elimination, to more easily narrow down the features that maximize the strength of the model while minimizing its cost.<jupyter_code># import tools
from sklearn.feature_selection import RFE
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
# Create dataset for Y
y_data = df['W%']

# Create dataset for X, removing target variable and non-numerical features
x_data = df.drop(['W%', 'TeamYear', 'Team'], axis = 1)

# Separate data into train and test groups using 25% of samples in test group
x_train, x_test, y_train, y_test = train_test_split(x_data, y_data, test_size = 0.25, random_state = 100)

# Report number of samples in each group
print('The number of training samples is:', x_train.shape[0])
print('The number of test samples is:', x_test.shape[0])

# Create model for RFE
lr = LinearRegression()

# Fit the model for RFE
lr.fit(x_train, y_train)

# Use Recursive Feature Elimination
rfe = RFE(lr, 15)
rfe = rfe.fit(x_train, y_train)

# Print list of features that show if they are in the top 15 features with boolean test
print('Is a given feature in the best 15 as determined by RFE?:')
print(list(zip(x_train.columns, rfe.support_, rfe.ranking_)))<jupyter_output>The number of training samples is: 337
The number of test samples is: 113
Is a given feature in the best 15 as determined by RFE?:
[('Season', False, 15), ('G_x', False, 35), ('HR', True, 1), ('R', False, 2), ('RBI', False, 4), ('SB', False, 12), ('BB%', True, 1), ('K%', False, 5), ('ISO', True, 1), ('BABIP_x', True, 1), ('AVG', True, 1), ('OBP', True, 1), ('SLG', True, 1), ('wOBA', True, 1), ('wRC+', False, 20), ('BsR', False, 25), ('Off', False, 26), ('Def_x', False, 21), ('WAR_x', False, 14), ('rSZ', False, 30), ('rCERA', False, 16), ('rSB', False, 10), ('rGDP', False, 31), ('rARM', False, 27), ('rGFP', False, 19), ('rPM', False, 28), ('DRS', False, 29), ('BIZ', True, 1), ('Plays', True, 1), ('RZR', True, 1), ('OOZ', False, 34), ('ARM', False, 18), ('DPR', False, 22), ('RngR', False, 23), ('ErrR', False, 24), ('UZR', False, 17), ('UZR/150', False, 9), ('Def_y', False, 33), ('SV', False, 8), ('G_y', False, 32), ('GS', False, 11), ('K/9', False, 6), ('BB/9', False, 3), ('HR/9', True, [...]<jupyter_text>Let's make this more presentable by identifying the 15 columns selected by RFE.<jupyter_code># Best 15 features as determined by RFE
sig_cols = x_train.columns[rfe.support_]
sig_cols<jupyter_output><empty_output><jupyter_text>Now that we have identified the 15 features we will examine further in the modeling notebook, let's turn them into a new dataset and save it. We should also include the target variable and the identifying features.<jupyter_code>df2 = df[['TeamYear', 'Season', 'Team', 'W%', 'HR', 'BB%', 'ISO', 'BABIP_x', 'AVG', 'OBP', 'SLG', 'wOBA', 'BIZ',
       'Plays', 'RZR', 'HR/9', 'BABIP_y', 'GB%', 'HR/FB']]
df2.head()
# Save cleaned dataset to CSV
df2.to_csv(r'~/Github/DATA601_FinalProject/Data/TeamStats_Combined_Cleaned.csv', index = None)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11227942953831278, 'fraction_numerical': 0.027918781725888325, 'mean_word_length': 4.9857728478418135, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 80, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '382838', 'license': 'MIT', 'n_tokens_mistral': 8462, 'n_tokens_neox': 7706, 'n_words': 3154, 'path': 'Notebooks/Notebook 1 - Data Exploration & Cleaning.ipynb', 'repo_name': 'cmszip/DATA601-FinalProject', 'size': '24822'}