<jupyter_start><jupyter_text>Movies Example - Predicing the "NaN"-values Author: <NAME> In this example I used a dataset which contains the following columns:   Rank | Title | Year | Score | Metascore | Genre | Vote | Director | Runtime | **Revenue** | Description | RevCat My aim was to predict the **Revenue** based on the other information. There are some "NaN"-values in the column "Revenue" and instead of filling them with an assumption (e.g. median-value) as I did in another Jupiter-Notebook, I wanted to predict these values. Therefore I did the following:- I separated the rows with "NaN"-values in column "Revenue"- I drew a stratified sample (based on "Revenue") on this remaining dataset and I received a training dataset and testing dataset:<jupyter_code>import matplotlib.pyplot as plt
fig, axs = plt.subplots()
axs.pie([747, 6726, 2527], labels=['training data', 'testing data', 'Revenue=NaN'])
plt.show<jupyter_output><empty_output><jupyter_text>- I created a pipeline to fill the "NaN"-value in other columns (e.g. "Metascore", "Score"). - used the training dataset and fittet it with the "DecisionTreeRegressor" model- verified with a cross-validation, how good this model/parameters are- did a prediction on a subset of the testing dataset and did a side-by-side comparison of prediction and true value- performed a prediction on the testing dataset and calculated the mean-squared error<jupyter_code>import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedShuffleSplit
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler
#from sklearn.preprocessing import OneHotEncoder  we will need this later
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import cross_val_score
PATH = os.path.join("datasets", "movies")
def format_plot(ax, title, feature1, feature2):
    #ax.xaxis.set_major_formatter(plt.NullFormatter())
    #ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.set_xlabel(feature1, color='gray')
    ax.set_ylabel(feature2, color='gray')
    ax.set_title(title, color='gray')<jupyter_output><empty_output><jupyter_text>Importing the data The movies dataset can be found here: https://www.kaggle.com/isaactaylorofficial/imdb-10000-most-voted-feature-films-041118<jupyter_code>def load_data(path=PATH):
    csv_path = os.path.join(path, "movies.csv")
    return pd.read_csv(csv_path)
movies = load_data()
movies.head()
movies.dtypes
movies.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 11 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   Rank         10000 non-null  int64  
 1   Title        10000 non-null  object 
 2   Year         10000 non-null  int64  
 3   Score        10000 non-null  float64
 4   Metascore    6781 non-null   float64
 5   Genre        10000 non-null  object 
 6   Vote         10000 non-null  int64  
 7   Director     9999 non-null   object 
 8   Runtime      10000 non-null  int64  
 9   Revenue      7473 non-null   float64
 10  Description  10000 non-null  object 
dtypes: float64(3), int64(4), object(4)
memory usage: 859.5+ KB<jupyter_text>"NaN"-values in column "Revenue" In column "Revenue" there are 7473 "non-null" values, and 2527 "null" values:<jupyter_code>10000-7473<jupyter_output><empty_output><jupyter_text>These are the datarows, where column "Revenue" is null:<jupyter_code>movies_RevenueNaN = movies[movies["Revenue"].isnull()]
movies_RevenueNaN.head()
len_movies_RevenueNaN = len(movies_RevenueNaN)
len_movies_RevenueNaN
movies_NotNull = movies[movies["Revenue"].notnull()]
movies_NotNull.head()
len_movies_NotNull = len(movies_NotNull)
len_movies_NotNull
len_movies_RevenueNaN + len_movies_NotNull
fig, axs = plt.subplots()
axs.pie([len_movies_RevenueNaN, len_movies_NotNull], 
        labels=['Revenue=NaN', 'Revenue NotNull'], 
        colors  = ['green', 'yellow']
       )
plt.show<jupyter_output><empty_output><jupyter_text>Visualisation of the data<jupyter_code>movies_NotNull['Revenue'].hist()
movies_RevenueNaN['Revenue'].hist()<jupyter_output><empty_output><jupyter_text>Some scatter plots to see, how "Revenue", "Year" and "Score" behave:<jupyter_code>fig, ax = plt.subplots(figsize=(8, 6))
point_style = dict(cmap='Paired', s=50)
pts = ax.scatter(movies_NotNull['Revenue'], movies_NotNull['Year'], c=movies_NotNull['Score'], s=10, alpha=0.8)
cb = fig.colorbar(pts, ax=ax)

# format plot
format_plot(ax, 'Input Data', 'Revenue', 'Year')
cb.set_ticks([])
cb.set_label('Latent Variable: Score', color='gray')

fig.savefig('images/movies/movies_revenue_year_score.png')<jupyter_output><empty_output><jupyter_text>The Revenue increased during the years from 1920 to 2020.<jupyter_code>movies.hist(bins=50, figsize=(20,15))<jupyter_output><empty_output><jupyter_text>As the "Rank" is a continuous ascending index, the histogramm looks like an rectangle. The "Runtime" of an average movie is about 100 minutes, which makes sense. The shape of the "Revenue" histogramm also seems so make sense. The "Metascore" and "Score" have some interesting jags, which is due to the discrete values (1,2,3... but not 1.24 or 3.73), which these values can take.<jupyter_code>movies_NotNull.hist(bins=50, figsize=(20,15))<jupyter_output><empty_output><jupyter_text>The difference in the "Rank" histogramm is because the row "NaN"-values in column "Revenue" are not included here.  Examine how "Year", "Score" and "Revenue" behave:<jupyter_code>fig, ax = plt.subplots(figsize=(8, 6))
point_style = dict(cmap='Paired', s=50)
pts = ax.scatter(movies_NotNull['Year'], movies_NotNull['Score'], c=movies_NotNull['Revenue'], s=10, alpha=0.4)
cb = fig.colorbar(pts, ax=ax)

# format plot
format_plot(ax, 'Input Data', 'Year', 'Score')
cb.set_ticks([])
cb.set_label('Latent Variable: Score', color='gray')

fig.savefig('images/movies/movies_year_score_revenue.png')<jupyter_output><empty_output><jupyter_text>In 1920 most movies hat a score of 8. In 2020 most movies had a score between about 4 and 9. I think this is a bit strange, because I would have expected, that there are also movies with a lower score in 1920.   Stratified Sampling<jupyter_code>movies_NotNullC = movies_NotNull[:].copy(deep=True)
movies_NotNullC['RevCat']=pd.cut(movies_NotNullC['Revenue'], 
                        bins=[-1,100,200,300,np.inf], 
                        labels=[1, 2, 3, 4])
movies_NotNullC['RevCat'].value_counts()
movies_NotNullC["RevCat"].value_counts() / len(movies_NotNullC)
len(movies_NotNullC)
split = StratifiedShuffleSplit(n_splits=1, test_size=0.9, random_state=42)
for train_index, test_index in split.split(movies_NotNullC, movies_NotNullC["RevCat"]):
    strat_train_set = movies_NotNullC.iloc[train_index]
    strat_test_set = movies_NotNullC.iloc[test_index]
strat_train_set["RevCat"].value_counts() / len(strat_train_set)
len(strat_train_set)
strat_test_set["RevCat"].value_counts() / len(strat_test_set)
len(strat_test_set)
def revenue_cat_proportions(data):
    return data["RevCat"].value_counts() / len(data)

train_set, test_set = train_test_split(movies_NotNullC, test_size=0.9, random_state=42)

compare_props = pd.DataFrame({
    "Overall": revenue_cat_proportions(movies_NotNullC),
    "Stratified": revenue_cat_proportions(strat_test_set),
    "Random": revenue_cat_proportions(test_set),
}).sort_index()

compare_props["Rand. %error"] = 100 * compare_props["Random"] / compare_props["Overall"] - 100
compare_props["Strat. %error"] = 100 * compare_props["Stratified"] / compare_props["Overall"] - 100

print(compare_props)

for set_ in (strat_train_set, strat_test_set):
    set_.drop("RevCat", axis=1, inplace=True)
movies_train = strat_train_set.drop('Revenue', axis=1)
movies_train_labels = strat_train_set['Revenue'].copy()
len_movies_train = len(movies_train)
len_movies_train
movies_test = strat_test_set.drop('Revenue', axis=1)
movies_test_labels = strat_test_set['Revenue'].copy()
len_movies_test = len(movies_test)
len_movies_test
len_movies_train + len_movies_test
len_movies_NotNull
len_movies_RevenueNaN
len_movies_train + len_movies_test + len_movies_RevenueNaN<jupyter_output><empty_output><jupyter_text>The whole dataset of 10000 rows has been split up into - a training dataset ("movies_train"), - a testing dataset ("movies_test") and- a dataset, where "Revenue"="NaN"<jupyter_code>fracs = [len_movies_train, len_movies_test, len_movies_RevenueNaN]
labels = ['training data (movies_train)', 'testing data (movies_test)', 'Revenue=NaN']
fig, axs = plt.subplots()
axs.pie(fracs, labels=labels)
plt.show
plt.show<jupyter_output><empty_output><jupyter_text>Pipeline Now I want to implement a pipeline for doing all the datapreparation work quicker when I am testing it later. Only numerical-columns will be taken. The other will be thrown away (for the moment, I will change this later):<jupyter_code>imputer = SimpleImputer(strategy="median")
movies_train_num = movies_train.select_dtypes(include=[np.number])
imputer.fit(movies_train_num)
X = imputer.transform(movies_train_num) # Transform the training set:
movies_tr = pd.DataFrame(X, columns=movies_train_num.columns,index=movies_train.index)
movies_tr.head()
num_attribs = list(movies_train_num)
num_attribs
num_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')), 
])

num_attribs = ['Rank', 'Year', 'Score', 'Metascore', 'Vote', 'Runtime'] #list(movies_train_num)

full_pipeline = ColumnTransformer([
    ('num', num_pipeline, num_attribs)
])<jupyter_output><empty_output><jupyter_text>Apply now the full pipeline on the training-dataset "movies_train". But before we do this, we have a look into the column "Metascore" and count how many "nan"-values we have there in the training dataset "movies_train":<jupyter_code>tmp = movies_train[movies_train["Metascore"].isnull()]
tmp.head()<jupyter_output><empty_output><jupyter_text>Now apply the Pipeline:<jupyter_code>movies_train_prepared = full_pipeline.fit_transform(movies_train)<jupyter_output><empty_output><jupyter_text>Now let's count the "nan" values in the new prepared datast "movies_train_prepared". I have to transform it back to a Pandas-Dataframe format first:<jupyter_code>tmp_num = movies_train.select_dtypes(include=[np.number])
tmp_prep = pd.DataFrame(movies_train_prepared, columns=tmp_num.columns,index=movies_train.index)
tmp = tmp_prep[tmp_prep["Metascore"].isnull()]
tmp<jupyter_output><empty_output><jupyter_text>Zero, as we wanted! All "nan"-values in "movies_train_prepared" have been removed by the "median" value (this was how the pipeline was built). Great, this was part of the job, the Pipline should have done.  The other part was to eliminate some columns. We now have only 6 remaining instead of 10 columns.   Model "DecisionTreeRegressor"<jupyter_code>tree_reg = DecisionTreeRegressor(random_state=42)
tree_reg.fit(movies_train_prepared, movies_train_labels)

movies_predictions = tree_reg.predict(movies_train_prepared)<jupyter_output><empty_output><jupyter_text>Verifiy if training labels ("movies_train_labels") and predicted labels ("movies_predictions"). Here for the first 10 datasets:<jupyter_code>list(movies_train_labels[0:10])
list(movies_predictions[0:10])<jupyter_output><empty_output><jupyter_text>The training-labels and the predicted labels match for the first 10 datasets. The fitting of the model was successful.How big is the error for all training-datasets?<jupyter_code>trainmse = mean_squared_error(movies_train_labels, movies_predictions)
trainrmse = np.sqrt(trainmse)
trainrmse<jupyter_output><empty_output><jupyter_text>Cross-Validation<jupyter_code>scores = cross_val_score(tree_reg, 
                         movies_train_prepared,
                         movies_train_labels,
                         scoring="neg_mean_squared_error", 
                         cv=10)

rmse_scores = np.sqrt(-scores)

def display_scores(scores):
    print("Scores:", scores)
    print("Mean:", scores.mean())
    print("Standard deviation:", scores.std())

display_scores(rmse_scores)<jupyter_output>Scores: [ 69.66166984  62.59977724  62.6450061  112.98391756  47.99491086
  52.16787811  42.65104005  83.21059338  41.09199207  63.84158945]
Mean: 63.88483746577791
Standard deviation: 20.447326452253154<jupyter_text>Test the model We take on arbitrary row in the testing dataset "movies_test". Take for example row number 322:<jupyter_code>some_data = movies_test.iloc[0:20]
some_data_label = movies_test_labels.iloc[0:20]
some_data<jupyter_output><empty_output><jupyter_text>As we didn't apply the pipeline on the testing dataset (we only did on the training dataset "movies_train"), there might still some "nan" values in columns "Metascore". The true value for the "Revenue" is:<jupyter_code>some_data_label<jupyter_output><empty_output><jupyter_text>This row is the same as the following dataset from the original "movies"-database:<jupyter_code>some_movies = movies.iloc[some_data.index[0:len(some_data)]]
some_movies<jupyter_output><empty_output><jupyter_text>Lets predict the revenue for this dataset:<jupyter_code>some_data_prepared = full_pipeline.fit_transform(some_data)
some_data_predictions = tree_reg.predict(some_data_prepared)
list(some_data_predictions)<jupyter_output><empty_output><jupyter_text>Compare this prediction to the value "some_data_label".<jupyter_code>list(some_data_label)<jupyter_output><empty_output><jupyter_text>Compare the true value ("some_data_labels") and the predicted value ("some_data_predicitons") side-by-side. Left side is the true value from the original movies dataset. Right side is the predicted value based on the tree model:<jupyter_code>side_by_side = [(true, pred) for true, pred in zip(list(some_data_label), list(some_data_predictions))]
side_by_side<jupyter_output><empty_output><jupyter_text>The mean-squared-error is as follows:<jupyter_code>mse = mean_squared_error(some_data_label, some_data_predictions)
rmse = np.sqrt(mse)
rmse<jupyter_output><empty_output><jupyter_text>Taking now the whole testing dataset:<jupyter_code>movies_test_prepared =full_pipeline.fit_transform(movies_test)
movies_test_predictions = tree_reg.predict(movies_test_prepared)
lin_mse = mean_squared_error(movies_test_labels, movies_test_predictions)
lin_rmse = np.sqrt(lin_mse)
lin_rmse
movies_test_labels.mean()
movies_test_labels.std()<jupyter_output><empty_output><jupyter_text>A side-by-side comparison off the testing dataset. Left side is the true value from the original movies dataset. Right side is the predicted value based on the tree model:<jupyter_code>side_by_side = [(true, pred) for true, pred in zip(list(movies_test_labels), list(movies_test_predictions))]
side_by_side<jupyter_output><empty_output><jupyter_text>Plotting the true labels and the predicted labels on the testing dataset:<jupyter_code>fig, ax = plt.subplots(figsize=(8, 8))
pts = ax.scatter(movies_test_predictions, movies_test_labels, s=10, alpha=0.8)<jupyter_output><empty_output><jupyter_text>Looks confusing. I would have expected something a bit more similar to the following plot.Plotting the same for the training dataset:<jupyter_code>movies_train_prepared =full_pipeline.fit_transform(movies_train)
movies_train_predictions = tree_reg.predict(movies_train_prepared)
fig, ax = plt.subplots(figsize=(8, 6))
pts = ax.scatter(movies_train_predictions, movies_train_labels, s=10, alpha=0.8)<jupyter_output><empty_output><jupyter_text>Now I calculate the "Revenue" where it has "NaN"-values:<jupyter_code>movies_RevenueNaN_prepared =full_pipeline.fit_transform(movies_RevenueNaN)
movies_RevenueNaN_predictions = tree_reg.predict(movies_RevenueNaN_prepared)<jupyter_output><empty_output><jupyter_text>These are the predictions:<jupyter_code>movies_RevenueNaN_predictions<jupyter_output><empty_output><jupyter_text>I will insert the prediction into the dataset:<jupyter_code>movies_RevenueNaN.loc[:,"Revenue"] = movies_RevenueNaN_predictions
movies_RevenueNaN<jupyter_output><empty_output><jupyter_text>Future amendments The movies dataset has the following columns (including "Revenue" which I wanted to predict):<jupyter_code>movies.head(0)<jupyter_output><empty_output><jupyter_text>But the model is only taking the follwing columns into consideration:<jupyter_code>num_attribs<jupyter_output><empty_output><jupyter_text>Remember, that the dataset, which is used for the model looks as follows (all non-numeric columns disappeared):<jupyter_code>movies_tr<jupyter_output><empty_output><jupyter_text>Therefore lot's of information, which is in the dataset has not yet been used, e.g. "Genre", "Director". These information could have an positive impact on the correctness of the predictions.  But as "Genre" has 486 different values it is a bit more complicated to treat them as "categorial" values:<jupyter_code>movies['Genre'].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10243703358208955, 'fraction_numerical': 0.024836753731343284, 'mean_word_length': 6.647347302719572, 'pattern_counts': {'":': 4, '<': 153, '<?xml version=': 0, '>': 153, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '361989', 'license': 'Apache-2.0', 'n_tokens_mistral': 6133, 'n_tokens_neox': 5681, 'n_words': 1392, 'path': 'Movies Machine Learning - Predict NaNs.ipynb', 'repo_name': 'AndreasTraut/Machine-Learning-with-Python', 'size': '17159'}