<jupyter_start><jupyter_text>First let us read in the processed dataframes, and generate appropriate dataframes for some simple exploratory data analysis.<jupyter_code># read all 14-19 season data
path_list = glob.glob('{}*.csv'.format(IN_PATH))
df_list = [pd.read_csv(path) for path in path_list]

# identify EPL teams and relagated teams for each season
epl_list = [pd.concat([df[:17], df[20:]]).reset_index(drop=True) for df in df_list]

# concat seasons 14-18 for analysis
df_tot = pd.concat(df for df in epl_list[:-1])
features = [col for col in df_tot.columns if col not in ('lead_pts', 'team')]
res  = ['lead_pts']<jupyter_output><empty_output><jupyter_text>The below scatter plots display the relationship between each feature in our dataframes and the target variable lead_pts.<jupyter_code># configure plot
fig, axes = plt.subplots(3, 3, figsize=(16,16),sharey=True)
arr = np.array(features).reshape(3, 3)

# iterate over all features
for row, col_arr in enumerate(arr):
    # plot each feature against lead_pts
    for col, feature in enumerate(col_arr):
        axes[row, col].scatter(df_tot[feature], df_tot['lead_pts'])
        if col == 0:
            axes[row, col].set(xlabel=feature, ylabel='lead_pts')
        else:
            axes[row, col].set(xlabel=feature)

fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>It is evident that all the features with the exception of number of free kicks, yellow cards and red cards a fairly strong linear relathionship with the leading season's points. The number of yellow/red cards per season seems to be more random.We know that even within the same division, the performance of the teams are vastly different, especially between the teams that consistently compete in the Champion's League (i.e. top 4 each season) and the teams that below average performance.  To see if the different groups affect the above relationships, let us observe the relationship between shots and leading season's points for two groups: the top 4 teams each season, and the teams that rank the bottom 3 of the teams that successfully remained in the premier league (i.e. ranks 14-17).Note: To really see the difference, it would be ideal to compare top 4 vs the relegated teams each season. But since we are looking at the leading season's points and not the current season's points, relegated teams cannot be considered because they are not in the premier league the next season.<jupyter_code># configure plot
fig, axes = plt.subplots(1, 2, figsize=(12,4),sharey=True)

# select teams compeating in the Champions League/relegated teams
df_top = pd.concat(df.loc[:4] for df in epl_list)
df_bottom = pd.concat(df.loc[13:16] for df in epl_list)
df = [df_top, df_bottom]

# populate subplots
for i, ax in enumerate(axes):
    if i == 0:
        ax.set_ylabel('lead_pts')
    ax.set_xlabel('shots')
    ax.scatter(df[i].shots, df[i].lead_pts)

fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>It is interesting to see that for the top tier teams, the leading season's points increases linearly as the number of shots taken increases, but in the other group, the leading season's points remains relatively constant. This may be something we can refer back to if it turns out that our model predictions for teams with weak performances have issues. Now moving on to the feature selection process, we will start by examining the univariate correlation of each input feature with the target variable.<jupyter_code># calculate univariate feature importance 
feat_importance = df_tot.drop(['team', 'lead_pts'], axis=1).apply(lambda x: x.corr(df_tot.lead_pts))
order = np.argsort(feat_importance)
feat_importance[order]

# generate feature importance plot
plt.barh(range(len(order)), feat_importance[order], align='center')
plt.yticks(range(len(order)), [features[i] for i in order])
plt.xlabel('Relative Importance')
plt.show()

# drop unwanted features
features = [features[i] for i in order if feat_importance[i] > 0.4]
features.insert(0, 'team')<jupyter_output><empty_output><jupyter_text>We will drop features with the correlation coefficient less than 0.4, and thus the number of free kicks, yellow cards, and red cards will be dropped from the set of features. We can now observe the correlation heatmap to check correlation between remaining features.<jupyter_code># generate correlation heatmap for seasons 14-18
fig, ax = plt.subplots(1, figsize=(15,10))
ax.set_title('Total Corr. Heatmap', size=15)

# correlation matrix
corr = df_tot[features].corr()

# Generate a mask for the upper triangle
mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True

# Generate a custom diverging colormap
cmap = sns.diverging_palette(220, 10, as_cmap=True)

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr, mask=mask, cmap=cmap, annot=True, 
            cbar=False, square=True, linewidths=.5, ax=ax)
plt.show()<jupyter_output><empty_output><jupyter_text>As expected, many of the features have high correlation amongst one another. However, we will not check for multicollinearity because we are only interested in the predictive accuracy, not the impact of each feature on the target.We are now ready for fitting our model, we will start with the linear regression model to see how it performs.<jupyter_code># divide dataframe into design matrix and output vector
df_tot = df_tot.reset_index(drop=True)
X = df_tot[features].drop(columns=['team'])
y = df_tot.lead_pts
# generate linear model
lr_model = LinearRegression()

# evaluate model
lr_cv_scores = cross_val_score(lr_model, X, y, cv=5, scoring='neg_mean_absolute_error')*-1
print('MAE of {:.2f} points (+/-) {:.2f} points'.format(np.mean(lr_cv_scores), np.std(lr_cv_scores)))<jupyter_output>MAE of 7.95 points (+/-) 0.56 points<jupyter_text>The linear regression model produced a mean absolute error of ~8 points. We can see how significant 8 points really is by comparing the total points earned in a season between teams. We will take the last season (18/19) as an example:<jupyter_code># last season (i.e. 18/19)
df = df_list[-1].sort_values(by=['season_pts'], ascending=False)

# add barchart to compare season points
fig = plt.figure(figsize=(15,10))
ax = fig.add_subplot(111)
ax.bar(df.team, df.season_pts)
plt.setp(ax.get_xticklabels(), rotation=30, horizontalalignment='right')
plt.ylabel('Season Points')

plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the 1st and 2nd place were quite close (in fact they were only off by 1 point), and there is a large gap between the 2nd and the 3rd placed teams. The 3rd to 5th place teams all earned similar number of points (again, only off by 2 points), and there is another drop, and so on.It is evident that 8 points is a significant enough number, meaning that the predicted standings can be quite different from the actual standings. Let us explore other models to see if we can lower the MAE. It may be the case that there are non-linearities that cannot be captured by the linear regression model, so we will now try using a neural network model. Let us first normalize the data, and split the data into training and testing sets.<jupyter_code># for reproducibility
np.random.seed(43)

# normalize data
scaler = MinMaxScaler()
scaler.fit(X)
X = scaler.transform(X)

# train test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
print('{} training samples'.format(X_train.shape[0]))
print('{} testing samples'.format(X_test.shape[0]))<jupyter_output>208 training samples
52 testing samples<jupyter_text>We can now perform a simple parameter estimation through grid search cross validation. We will attempt to find hyperparameters that result in model performance equal to or better than the linear regression model.<jupyter_code>def create_model():
    """ create_model
        Output: 
            <keras Sequential model> model : keras sequential model with 2 hidden layers 
                                             and 1 dropout layer
    """   
    # define model
    model = Sequential()
    model.add(Dense(36, input_dim=6, activation='relu'))
    model.add(Dropout(0.1))
    model.add(Dense(36, activation='relu'))
    model.add(Dense(1, activation='linear'))

    # compile model
    model.compile(loss='mean_absolute_error', optimizer = 'adam')
    
    return model
# hyperparameters
batch_size = 1
epochs = 10

model_cv = KerasRegressor(build_fn=create_model, verbose=0)

# define the grid search parameters
batch_sizes = [1, 105, 208]
epochs      = [10, 50, 100]

param_grid = dict(epochs=epochs, batch_size=batch_sizes)

# perform grid search for initializer, number of epochs, and batch size
grid = GridSearchCV(estimator=model_cv, param_grid=param_grid, cv=10, scoring='neg_mean_absolute_error')
grid_results = grid.fit(X_train, y_train, verbose=0)
# print results
print('Best MAE of {:.2f} using parameters {}'.format(grid_results.best_score_*-1, 
                                                     grid_results.best_params_))
# fit model using the best parameters
model_cv.fit(X_train, y_train, epochs=50, batch_size=1, verbose=0)
pred = model_cv.predict(X_test)

# compute the MAE
score = mean_absolute_error(y_test, pred)
print('MAE of {:.2f}'.format(score))<jupyter_output>MAE of 7.33<jupyter_text>The performance of the neural network model is comparable to the linear regression model, with a lack of a significant improvement. In reality, there is so much more going on in the premier league that the previous season's performance alone cannot adequately capture the performance of the current season (i.e. for instance, the manager of each club, foreign money invested in club, buying/selling franchise stars, and the list is endless).Just for fun, we will predict the outcome of the current (19/20) season, where only 22 of the 38 rounds have been played, using the linear regression model. We compare the predictions to the current league standings, and see if there are some realistic predictions that were made.<jupyter_code># current standings for roudn 22 of 38
curr_standings = ['Liverpool', 'Man City', 'Leicester', 'Chelsea', 'Man United', 'Wolves',
                  'Sheffield', 'Tottenham', 'Crystal Palace', 'Arsenal', 'Everton', 'Newcastle',
                  'Southampton', 'Burnley', 'Brighton', 'West Ham', "Watford", 'Aston Villa',
                  'Bournmouth', 'Norwich']
curr_pts = [64, 48, 45, 39, 34, 34, 33, 31, 30, 29, 29, 29, 28, 27, 25, 23, 23, 22, 20, 17]

# current season matrix
X_curr = epl_list[-1][features]


# fit model
lr_model.fit(X,y)

# generate predictions
pred_lr = lr_model.predict(X_curr.drop(columns=['team']))

res = pd.DataFrame({'teams_predicted': X_curr.team, 'predictions': pred_lr.round().astype(int)}) \
                    .sort_values(by=['predictions'], ascending=False).reset_index(drop=True) 
res['curr_standing'] = curr_standings
res['curr_pts'] = curr_pts
res<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09473876173976475, 'fraction_numerical': 0.018965988875718064, 'mean_word_length': 4.824747742963356, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '141283', 'license': 'MIT', 'n_tokens_mistral': 3494, 'n_tokens_neox': 3239, 'n_words': 1261, 'path': 'src/models.ipynb', 'repo_name': 'jimmywoo1/epl_predictions', 'size': '10967'}