<jupyter_start><jupyter_text>1.1 Target encoding Replacing each category with a blend of the mean target value for that category and the mean target value over all the training data.Advantages:* compared to one-hot encoding: significant reduction of dimensionality of the data, helps avoiding the curse of dimensionality* flexibility: by adjusting min_sample_leaf we can choose how many categories we want in ecoded dataDisadvantages:* possible data leakage and consequently overfitting (may be reduced by regularization - increasing smoothing parameter)* additional hyperparameter to tune Before target encoding we have over ten thousand unique locations<jupyter_code>df.it_location.nunique()<jupyter_output><empty_output><jupyter_text>Many values with low count are not very informative or are different spellings of the same place<jupyter_code>cities_count = df.groupby("it_location").size()

cities_count.sort_values(ascending=True)[:40]<jupyter_output><empty_output><jupyter_text>Many locations appear only once<jupyter_code>cities_count.where(cities_count == 1).sum()
X = df.it_location

y = df.price

encoder = ce.TargetEncoder(cols='it_location')

encoder.fit(X, y)<jupyter_output><empty_output><jupyter_text>After target encoding number of unique values decreased significantly<jupyter_code>X_cleaned = encoder.transform(X)

X_cleaned.it_location.nunique()
unique_values = [ce.TargetEncoder(cols='it_location', min_samples_leaf=i).fit_transform(X, y).nunique() for i in range(1, 1000, 10)]

unique_values = [unique_values[i].values[0] for i in range(len(unique_values))]
import matplotlib as mpl
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)<jupyter_output><empty_output><jupyter_text>By manipulating number of minimum samples to take category average into account while encoding, we can adjust granularity of our data<jupyter_code>plt.figure(figsize=(10,5))
ax = sns.lineplot(x = range(1, 1000, 10), y = unique_values)
ax.set(xlabel='min_samples_leaf', ylabel='Number of unique values' )
plt.show()<jupyter_output><empty_output><jupyter_text>1.2 One Hot Encoding Creating  one binary  feature per category.Advantages:* easily interpretable results with models that output coefficients estimates for each feature* memory eficient if used with sparse matrix Disadvantages:* encoding high cardinality features results in too many aditional featues that can hurt model performance due to high dimensionality<jupyter_code>df.main_category.unique()
df.main_category.nunique()
X = df.main_category

encoder = ce.OneHotEncoder(cols='main_category', use_cat_names=True)
encoder.fit(X)

X_cleaned = encoder.transform(X)
X_cleaned.head()<jupyter_output><empty_output><jupyter_text>1.3 Binary encodingSimilar to one hot encoding, but stores categories as binary bitstrings.Advantages:* results in fewer additional featuresDisadvantages:* not interpretable coefficients<jupyter_code>encoder = ce.BinaryEncoder(cols='main_category')
encoder.fit(X, y)

X_cleaned = encoder.transform(X)
X_cleaned.head()<jupyter_output><empty_output><jupyter_text>1.4 Leave One Out EncodingSimilar to target encoding but excludes the current row’s target when calculating the mean target for a level to reduce the effect of outliers.Advantages:* robust to outliers* option to add noise into training data in order to decrease overfitting (sigma parameter)Disadvantages:* possibility of data leakage* additional hyperparameter to tune<jupyter_code>encoder = ce.LeaveOneOutEncoder(cols='main_category')
encoder.fit(X, y)

X_cleaned = encoder.transform(X)
X_cleaned.head()<jupyter_output><empty_output><jupyter_text>2 Data imputation<jupyter_code>from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
from sklearn.metrics import mean_squared_error
imputer = IterativeImputer()
numerical_data = df[['price', 'it_seller_rating', 'it_quantity']]
def compute_imputation_RMSE(data, columns_to_impute, imputer, perc_missing, num_iterations):
      
    RMSEs = []
    
    remove_number = int(data.shape[0] * perc_missing)
    
    for i in range(num_iterations):

        data_impute = data.copy()

        # Removing data
        for column in columns_to_impute:
            
            drop_indices = np.random.choice(data.index, remove_number, replace=False)

            data_impute.loc[drop_indices, column] = None
        
        # Imputation
        data_impute = pd.DataFrame(imputer.fit_transform(data_impute), columns=data.columns)
        
        RMSEs.append(mean_squared_error(data, data_impute, squared=False))
        
    return RMSEs<jupyter_output><empty_output><jupyter_text>2.1 Imputation vs number of features with missing data<jupyter_code>RMSEs_one_columm = compute_imputation_RMSE(data=numerical_data, 
                                columns_to_impute=['it_seller_rating'],
                                imputer=imputer, 
                                perc_missing=0.1, 
                                num_iterations=10)
RMSEs_two_columns = compute_imputation_RMSE(data=numerical_data, 
                                columns_to_impute=['it_seller_rating', 'it_quantity'],
                                imputer=imputer, 
                                perc_missing=0.1, 
                                num_iterations=10)
RMSEs_three_columns = compute_imputation_RMSE(data=numerical_data, 
                                columns_to_impute=['it_seller_rating', 'it_quantity', 'price'],
                                imputer=imputer, 
                                perc_missing=0.1, 
                                num_iterations=10)<jupyter_output><empty_output><jupyter_text>RMSE is significanlty smaller when data is removed only from one feature, suggesting that there is loss of information when more data is removed.Interestingly enough, removing values from two and three featues have similar effect on RMSE (but with higher SD when removing 3 features)<jupyter_code>print(f' Mean RMSE: {round(np.mean(RMSEs_one_columm), 1)}, SD: {round(np.std(RMSEs_one_columm))}')
print(f' Mean RMSE: {round(np.mean(RMSEs_two_columns), 1)}, SD: {round(np.std(RMSEs_two_columns))}')
print(f' Mean RMSE: {round(np.mean(RMSEs_three_columns), 1)}, SD: {round(np.std(RMSEs_three_columns))}')
RMSEs = pd.DataFrame({'one_column_removed' : RMSEs_one_columm,
                      'two_columns_removed' : RMSEs_two_columns,
                      'three_columns_removed' : RMSEs_three_columns,
                      'iterations' : range(1, 11)})
plt.figure(figsize=(12, 8))

ax = sns.lineplot(x='iterations', y='value', hue='variable', style='variable',
                  markers=True, dashes=False, data=pd.melt(RMSEs, id_vars='iterations'))

handles, labels = ax.get_legend_handles_labels()
ax.legend(reversed(handles[1:]), reversed(labels[1:]), title= '', 
          bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
ax.set(ylabel='RMSE', title="Result of number of columns with missing data on quality of imputation")
plt.show()<jupyter_output><empty_output><jupyter_text>2.2 Imputation vs % of data missing<jupyter_code>data_missing = [compute_imputation_RMSE(data=numerical_data, 
                                columns_to_impute=['it_seller_rating'],
                                imputer=IterativeImputer(), 
                                perc_missing=0.05 * i, 
                                num_iterations=10) for i in range(1, 11)]
data_missing = pd.DataFrame(np.asanyarray(data_missing).T, columns=[f'{0.05 * i:.2f}%' for i in range(1, 11)])
data_missing['iterations'] = range(1, 11)
plt.figure(figsize=(12, 8))

ax = sns.lineplot(x='iterations', y='value', hue='variable',
                  markers=True, dashes=False, palette='gist_earth_r', data=pd.melt(data_missing, id_vars='iterations'))

handles, labels = ax.get_legend_handles_labels()
ax.legend(reversed(handles[1:]), reversed(labels[1:]), title='% of missing data', 
          bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
ax.set(ylabel='RMSE', title="Result of % of missing data on quality of imputation")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0995686999383857, 'fraction_numerical': 0.012322858903265557, 'mean_word_length': 4.229381443298969, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '286314', 'license': 'Apache-2.0', 'n_tokens_mistral': 2656, 'n_tokens_neox': 2476, 'n_words': 586, 'path': 'Prace_domowe/Praca_domowa2/Grupa2/PingielskiJakub/Praca_domowa_02.ipynb', 'repo_name': 'niladrem/2020L-WUM', 'size': '8115'}