<jupyter_start><jupyter_text>Column Description● store_id - The unique identifier of a store.● product_id - The unique identifier of a product.● date - Sales date (YYYY-MM-DD)● sales - Sales quantity● revenue - Daily total sales revenue● stock - End of day stock quantity● price - Product sales price● promo_type_1 - Type of promotion applied on channel 1● promo_bin_1 - Binned promotion rate for applied promo_type_1● promo_type_2 - Type of promotion applied on channel 2● promo_bin_2 - Binned promotion rate for applied promo_type_2● promo_discount_2 - Discount rate for applied promo type 2● promo_discount_type_2 - Type of discount applied● product_length - Length of product● product_depth - Depth of product● product_width - Width of product● hierarchy1_id● hierarchy2_id● hierarchy3_id● hierarchy4_id● hierarchy5_id● storetype_id● store_size● city_id● train_or_test - rows with train tag will be used to train KNNRegressor and rows with testtag will be used for accuracy calculation  Code<jupyter_code>sales = pd.read_csv("sales.csv")
ph = pd.read_csv("product_hierarchy.csv")
store_cities = pd.read_csv("store_cities.csv")
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
print("Sales size: ", sales.shape)
print("Product Hierarchy size: ", ph.shape)
print("Store Cities size: ", store_cities.shape)
sales.head()
sales.isna().sum()
print("Min Date: ", sales.date.min() )
print("Max Date: ", sales.date.max() )
ph.head()
store_cities.head()<jupyter_output><empty_output><jupyter_text>Q3 - Classification 2 When there is a new product in the system, every business user has a business experience to guess what will be the seasonality curve in the future. Can we replace a business user's experience with an algorithm?<jupyter_code>q3_submissions = pd.read_csv("Q3/Q3_submission.csv")
predictions= list(q3_submissions.product_id.values)
sales_products = sales.merge(ph,on='product_id', how='inner')
sales_products.head()
sales.shape
sales_products.shape
print("Unique Products in ph: ",len(ph.product_id.unique()))
print("Unique Products in sales: ",len(sales.product_id.unique()))
q3_train = ph[~ph.product_id.isin(predictions)]
q3_submission = ph[ph.product_id.isin(predictions)]
q3_train.head()
q3_submission.head()
q3_train.sort_values(by="cluster_id",inplace=True)
q3_train.index = q3_train.product_id
q3_train.drop("product_id",axis=1, inplace=True)

q3_submission.index = q3_submission.product_id
q3_submission.drop("product_id",axis=1, inplace=True)
q3_train.head()
q3_submission.head()
q3_train.cluster_id.value_counts()<jupyter_output><empty_output><jupyter_text>**Label Encoding**<jupyter_code>le = preprocessing.LabelEncoder()
le.fit(q3_train.cluster_id)
q3_train.cluster_id = le.transform(q3_train.cluster_id)
sales_products.head()
sales_by_4 = sales_products.groupby("hierarchy4_id").sum()
sales_by_3 = sales_products.groupby("hierarchy3_id").sum()
sales_by_2 = sales_products.groupby("hierarchy2_id").sum()
sales_by_1 = sales_products.groupby("hierarchy1_id").sum()
sales_by_4.drop(sales_by_4.columns[5:],axis=1,inplace=True)
sales_by_3.drop(sales_by_3.columns[5:],axis=1,inplace=True)
sales_by_2.drop(sales_by_2.columns[5:],axis=1,inplace=True)
sales_by_1.drop(sales_by_1.columns[5:],axis=1,inplace=True)
sales_by_4.head()
q3_train.head()
q3_submission.head()
train = q3_train.merge(sales_by_4,how="inner",on="hierarchy4_id")
train = train.merge(sales_by_3,how="inner",on="hierarchy3_id")
train = train.merge(sales_by_2,how="inner",on="hierarchy2_id")
train = train.merge(sales_by_1,how="inner",on="hierarchy1_id")
train.head()
train.shape
test = q3_submission.merge(sales_by_4,how="inner",on="hierarchy4_id")
test = test.merge(sales_by_3,how="inner",on="hierarchy3_id")
test = test.merge(sales_by_2,how="inner",on="hierarchy2_id")
test = test.merge(sales_by_1,how="inner",on="hierarchy1_id")
test.head()
test.shape
len(train.columns)
len(test.columns)
train.shape
ph.head()
train.head()
train.shape<jupyter_output><empty_output><jupyter_text>**Label Encoding**<jupyter_code>le = preprocessing.LabelEncoder()
le.fit(train.hierarchy5_id)
train.hierarchy5_id = le.transform(train.hierarchy5_id)
le = preprocessing.LabelEncoder()
le.fit(train.hierarchy4_id)
train.hierarchy4_id = le.transform(train.hierarchy4_id)
le = preprocessing.LabelEncoder()
le.fit(train.hierarchy3_id)
train.hierarchy3_id = le.transform(train.hierarchy3_id)
le = preprocessing.LabelEncoder()
le.fit(train.hierarchy2_id)
train.hierarchy2_id = le.transform(train.hierarchy2_id)
le = preprocessing.LabelEncoder()
le.fit(train.hierarchy1_id)
train.hierarchy1_id = le.transform(train.hierarchy1_id)
train.head()
le = preprocessing.LabelEncoder()
le.fit(test.hierarchy5_id)
test.hierarchy5_id = le.transform(test.hierarchy5_id)
le = preprocessing.LabelEncoder()
le.fit(test.hierarchy4_id)
test.hierarchy4_id = le.transform(test.hierarchy4_id)
le = preprocessing.LabelEncoder()
le.fit(test.hierarchy3_id)
test.hierarchy3_id = le.transform(test.hierarchy3_id)
le = preprocessing.LabelEncoder()
le.fit(test.hierarchy2_id)
test.hierarchy2_id = le.transform(test.hierarchy2_id)
le = preprocessing.LabelEncoder()
le.fit(test.hierarchy1_id)
test.hierarchy1_id = le.transform(test.hierarchy1_id)
test.head()
test.isna().sum()
train["product_length"] =train["product_length"].fillna(train["product_length"].median())
train.product_depth.fillna(0,inplace=True)
train.product_width.fillna(0,inplace=True)
test.product_length.fillna(0,inplace=True)
test.product_depth.fillna(0,inplace=True)
test.product_width.fillna(0,inplace=True)
train.isna().sum()
len(train.columns)
len(test.columns)
train.shape
test.shape
train.tail()
test.tail()
# Missing of the entry can also be a valuable information
# So we will create a column that is False when value is missing
# We encoded missingness in categorical columns so we will just create _na columns for numerical types
def fix_missing(df, col, name):
    if is_numeric_dtype(col):
        if pd.isnull(col).sum():
            df[name+"_na"] = pd.isnull(col)
        df[name] = col.fillna(col.median())
        
# We will have codes starting from 0 (for missing)
# def numericalize(df, col, name):
#     if not is_numeric_dtype(col):
#         print(df[name].cat.categories)
#         print(col.cat.codes+1)
#         df[name] = col.cat.codes+1
        
# dic = dict()
# def numericalize(df, col, name):
#     if not is_numeric_dtype(col):
#         if name == 'hierarchy4_id':
#             for i in range(len(df[name].cat.categories)):
#                 dic[str(df[name].cat.categories[i])] = list(set(df[name].cat.codes))[i]+1
#         df[name] = col.cat.codes+1

    
def proc_df(df, y_fld):
    
#     y = df[y_fld].values
#     df.drop([y_fld], axis = 1, inplace = True)
    
    for n, c in df.items():
        fix_missing(df, c, n)
        
#     for n, c in df.items():
#         numericalize(df, c, n)
    
    y = df[y_fld].values
    df.drop([y_fld], axis = 1, inplace = True)
    
    res = [df, y]
    
    return res<jupyter_output><empty_output><jupyter_text>Model<jupyter_code>X_train = train.drop("cluster_id",axis=1)[:500]
y_train = train.cluster_id[:500].values
X_test = test.drop("cluster_id",axis=1)
y_test = test.cluster_id.values
# Fitting Random Forest Classification to the Training set
classifier = RandomForestClassifier(n_estimators = 10, criterion = 'entropy', random_state = 42)
classifier.fit(X_train, y_train)
rfc_pred = classifier.predict(X_test)
print ("Random Forest Train Accuracy Baseline:", metrics.accuracy_score(y_train, classifier.predict(scaled_X_train)))
print ("Random Forest Test Accuracy Baseline:", metrics.accuracy_score(y_test, classifier.predict(scaled_X_test)))
#print('Accuracy of random forest classifier on test set: {:.2f}'.format(rfc.score(X_test, y_test)))
# Feature Scaling
scaler = StandardScaler()
scaled_X_train = scaler.fit_transform(X_train)
scaled_X_test = scaler.transform(X_test)

# Fitting Random Forest Classification to the Training set
classifier = RandomForestClassifier(n_estimators = 10, criterion = 'entropy', random_state = 42)
classifier.fit(scaled_X_train, y_train)

rfc_pred = classifier.predict(X_test)

print ("Random Forest Train Accuracy Baseline:", metrics.accuracy_score(y_train, classifier.predict(scaled_X_train)))
print ("Random Forest Test Accuracy Baseline:", metrics.accuracy_score(y_test, classifier.predict(scaled_X_test)))
#print('Accuracy of random forest classifier on test set: {:.2f}'.format(rfc.score(X_test, y_test)))<jupyter_output><empty_output><jupyter_text>**Dividing the datasets**<jupyter_code>train.merge(ph,on="hierarchy5_id",how="inner").head()
train = train.drop(train.columns[30:],axis=1)
test = test.merge(ph,on="hierarchy5_id",how="inner")
test = test.drop(test.columns[30:],axis=1)
test.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13266583229036297, 'fraction_numerical': 0.017635680964842415, 'mean_word_length': 6.374161073825503, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '495577', 'license': 'MIT', 'n_tokens_mistral': 3248, 'n_tokens_neox': 3159, 'n_words': 586, 'path': 'Q3 - XGBoost.ipynb', 'repo_name': 'berkayalan/AlgoRun-Datathon', 'size': '8789'}