<jupyter_start><jupyter_text>Picking an error metric:- The average rating column contains continuous values.- Choosing a regression based metric.- The histogram 'looks' Gaussian.- So picking Mean squared error as it the easiest one to start with.- Referring to the [sklearn regression metrics help] for more info.[sklearn regression metrics help]:http://scikit-learn.org/stable/modules/model_evaluation.htmlregression-metrics  Clustering with KMeans: Steps:- Filtering out non-numerical values from the DataFrame.- Next step is to find the cluster the'distances' between rows of numerical data.- Picking 10 clusters, one for each rating value.- Fitting the [KMeans] model to obtain the cluster labels.- Finding the mean and standard deviation of each row.- Plotting the clusters.[KMeans]:http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html<jupyter_code>#Filtering 

print(boardgames.dtypes != object)

                    #Excluding id column
filtered_columns = boardgames.columns[1:].to_series()[boardgames.dtypes != object]
numeric_columns = boardgames[filtered_columns]
numeric_columns.head(3)
kmeans_model = KMeans(n_clusters=10)
kmeans_model.fit(numeric_columns)
labels = kmeans_model.labels_
labels.shape
game_mean = numeric_columns.mean(axis=1)
game_std = numeric_columns.std(axis=1)
game_mean.shape<jupyter_output><empty_output><jupyter_text>Matplotlib plotting with legend:https://stackoverflow.com/questions/26558816/matplotlib-scatter-plot-with-legend26559256<jupyter_code>rating_labels = list(np.unique(labels))

plt.figure(figsize=(12,12))
plt.xlim((-100,12000))
plt.ylim((-100,50000))
plt.xlabel('game_mean')
plt.ylabel('game_std')
plt.scatter(game_mean,game_std,c=labels,cmap='rainbow')
plt.show()<jupyter_output><empty_output><jupyter_text>Notes: - The user_rated and yearpublished columns are relatively large compared to other values.- Larger the user base larger the game_mean value and consequently larger game_std value.- This means that games with larger user base will have a greater mean value and standard deviation.<jupyter_code>l = numeric_columns.columns.tolist()
l.remove('bayes_average_rating')
correlations = numeric_columns[l].corr()
print(correlations.average_rating)<jupyter_output>yearpublished     0.108461
minplayers       -0.032701
maxplayers       -0.008335
playingtime       0.048994
minplaytime       0.043985
maxplaytime       0.048994
minage            0.210049
users_rated       0.112564
average_rating    1.000000
total_owners      0.137478
total_traders     0.119452
total_wanters     0.196566
total_wishers     0.171375
total_comments    0.123714
total_weights     0.109691
average_weight    0.351081
Name: average_rating, dtype: float64<jupyter_text>Notes:- The average_weight column depicting [subjective weights] has the highest correlation.- The total_* columns have a high correlations with average rating. - Owners, wanters and wishers of a game to have relatively high correlation with the average rating.- The minage column has a surprisingly high correlation. This might indicate an age bias in the dataset.[subjective weights]:https://boardgamegeek.com/wiki/page/Weight<jupyter_code>reg = LinearRegression()
l = numeric_columns.columns.tolist()
l.remove('bayes_average_rating')
l.remove('minplayers')
l.remove('maxplayers')
l.remove('minplaytime')
l.remove('maxplaytime')
l.remove('average_rating')
#data = numeric_columns[l]
data = numeric_columns[l]
X_tr, X_test, y_tr ,y_test = train_test_split(data,numeric_columns.average_rating,test_size=0.7,random_state=123)
reg.fit(X_tr,y_tr)
predictions = reg.predict(X_test)
predictions.shape
mse = mean_squared_error(y_test,predictions)
print("Mean square error : {0:.3f}".format(mse))
print("Standard deviation of Average ratings: {0:.3f}".format(numeric_columns['average_rating'].std()))<jupyter_output>Standard deviation of Average ratings: 1.579
{'fraction_non_alphanumeric': 0.09726522187822498, 'fraction_numerical': 0.04411764705882353, 'mean_word_length': 5.78984238178634, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '310894', 'license': 'MIT', 'n_tokens_mistral': 1347, 'n_tokens_neox': 1186, 'n_words': 307, 'path': 'boardgame_reviews.ipynb', 'repo_name': 'ganprad/jupyter_notebooks', 'size': '3876'}