<jupyter_start><jupyter_text>Feature lab<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import sklearn as sk
import matplotlib.pyplot as plt
from tqdm import tqdm
dataset_path = "../datasets/1.0v/"
!ls ../datasets/1.0v<jupyter_output>Features.pdf  infos.csv  items.csv  orders.csv	README.md  Task_2020.pdf<jupyter_text>Loading our datasets...<jupyter_code>infos = pd.read_csv(dataset_path + 'infos.csv', sep='|')
items = pd.read_csv(dataset_path + 'items.csv', sep='|')
orders = pd.read_csv(dataset_path + 'orders.csv', sep='|')<jupyter_output><empty_output><jupyter_text>FIle heads<jupyter_code>orders.head()
items.head()
infos.head()<jupyter_output><empty_output><jupyter_text>Basic stats **Conclusion**: Our items have a high ```simulationPrice``` standard deviation, which affects directly our approach.<jupyter_code>infos.describe().round(1)
items.describe().round(1)
orders.describe().round(1)<jupyter_output><empty_output><jupyter_text>Extracting the dates of promotion This may be useful later on the competition. For now, let's just understand we have access to this data.<jupyter_code>promotions = infos['promotion'].value_counts()
infos['promotion'].unique()<jupyter_output><empty_output><jupyter_text>Removing the items that were never sold<jupyter_code># Taking the ids of the items that were sold at least once...
sold_items_id = orders['itemID'].unique()
sold_items_id
# Items that were sold at least once will be stored in 'sold_items'
sold_items = items[items.itemID.isin(sold_items_id)]
sold_items<jupyter_output><empty_output><jupyter_text>Just making a sanity check to be sure if we really have kept only the items that had been sold at least once...<jupyter_code>fact_check = np.array(sold_items.itemID)
fact_check == sorted(sold_items_id)
# We'll extract only the infos about the items that were sold at least once...
sold_infos = infos[infos.itemID.isin(sold_items_id)]
sold_infos<jupyter_output><empty_output><jupyter_text>Another sanity check...<jupyter_code>fact_check_2 = np.array(sold_infos.itemID)
fact_check_2 == sorted(sold_items_id)<jupyter_output><empty_output><jupyter_text>Most revenue items It's very strange that some items are sold below the 'recommended retail price'<jupyter_code># Concatenating the itemID column with the revenue column...
sold_items_revenue = pd.DataFrame(pd.concat([sold_items.itemID, sold_infos.simulationPrice -  sold_items.recommendedRetailPrice], axis=1, names=['itemID', 'revenue']))
items_revenue = pd.concat([items.itemID, infos.simulationPrice -  items.recommendedRetailPrice], axis=1, names=['itemID', 'revenue'])
items_revenue<jupyter_output><empty_output><jupyter_text>Demystifying the categories Ordering the items by how much they were sold and displaying it's categories (the first position has the id of the most sold item, and so goes on).<jupyter_code>category = items[['itemID','category1','category2','category3']]
category_and_orders = pd.merge(orders, category, on='itemID')
category_and_orders_stats = category_and_orders.groupby('itemID').aggregate({'order' : ['sum','count'],'category1': ['mean'], 'category2': ['mean'], 'category3': ['mean']})
category_and_orders_stats = category_and_orders_stats.sort_values(('order', 'sum'), ascending=False)
category_and_orders_stats<jupyter_output><empty_output><jupyter_text>How well distribuited are the categories in the orders?<jupyter_code>cat1_order_amount = category_and_orders.groupby('category1').aggregate({'order' : ['sum']})
cat1_order_amount.sort_values(('order', 'sum'), ascending=False)
plt.plot(cat1_order_amount)
cat2_order_amount = category_and_orders.groupby('category2').aggregate({'order' : ['sum']})
cat2_order_amount.sort_values(('order', 'sum'), ascending=False)
plt.plot(cat2_order_amount)
cat3_order_amount = category_and_orders.groupby('category3').aggregate({'order' : ['sum']})
cat3_order_amount.sort_values(('order', 'sum'), ascending=False)
plt.plot(cat3_order_amount)<jupyter_output><empty_output><jupyter_text>Customer rating evaluation Almost 70% of our customer ratings consist of zeros, so this feature might not be as useful as I expected at first glance<jupyter_code>sold_items.customerRating.hist()
print(f'{len(sold_items.customerRating[sold_items.customerRating == 0])/len(sold_items) * 100}%')<jupyter_output>67.19512195121952%
{'fraction_non_alphanumeric': 0.12595508219495252, 'fraction_numerical': 0.012734429266033805, 'mean_word_length': 8.473684210526315, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '492697', 'license': 'MIT', 'n_tokens_mistral': 1507, 'n_tokens_neox': 1429, 'n_words': 317, 'path': 'dora/pre-processing-features/feature_lab.ipynb', 'repo_name': 'joaopedromattos/DMC2020', 'size': '4319'}