<jupyter_start><jupyter_text>W4995-AML Project team 32 This note book represents team project of group 32 for W4995-AML course  Imports<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import pandas as pd
import warnings
#warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Data Loading<jupyter_code>train_data = pd.read_csv('./data/train.csv')
test_data = pd.read_csv('./data/test.csv')
meal_info = pd.read_csv('./data/meal_info.csv')
center_info = pd.read_csv('./data/fulfilment_center_info.csv')<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>train_data.head()
meal_info.head()
center_info.head()
train_data.info()
meal_info.info()
center_info.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 77 entries, 0 to 76
Data columns (total 5 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   center_id    77 non-null     int64  
 1   city_code    77 non-null     int64  
 2   region_code  77 non-null     int64  
 3   center_type  77 non-null     object 
 4   op_area      77 non-null     float64
dtypes: float64(1), int64(3), object(1)
memory usage: 3.1+ KB<jupyter_text>**Exciting, there is no obvious missing data**<jupyter_code>train_data = pd.merge(train_data,meal_info,on='meal_id',how='left')
train_data = pd.merge(train_data,center_info, on='center_id', how='left')
train_data = train_data.drop(columns=["center_id", "meal_id"])
train_data.head()<jupyter_output><empty_output><jupyter_text>Features: Checkout_price, base_price and region code<jupyter_code>features = ['checkout_price','base_price','region_code']
target = train_data['num_orders']
print(f'missing values in checkout price: ', any(train_data['checkout_price'].isnull()))
print(f'missing values in base price: ', any(train_data['base_price'].isnull()))
print(f'missing values in region code: ', any(train_data['region_code'].isnull()))
print(f'missing values in number of orders: ', any(train_data['num_orders'].isnull()))<jupyter_output>missing values in checkout price:  False
missing values in base price:  False
missing values in region code:  False
missing values in number of orders:  False<jupyter_text>It seems there's no missing valuse in checkout price, base price and region code. Also no missing values in target.<jupyter_code>ind = 0
plt.figure(figsize=(15,5))
for i in features:
    ind+=1
    plt.subplot(1,3,ind)
    plt.hist(train_data[i])
    plt.title(i)<jupyter_output><empty_output><jupyter_text>For check out and base price, the distributions are a bit skew to the right. For region code distribution, it looks like the data is gathered from several specific regions.<jupyter_code>print(train_data['region_code'].unique())<jupyter_output>[56 85 77 34 35 71 93 23]<jupyter_text>It's based on 8 regions.<jupyter_code>ind = 0
plt.figure(figsize=(18,5))
for i in features:
    ind+=1
    plt.subplot(1,3,ind)
    plt.scatter(train_data[i],target)
    plt.title(f'num orders vs '+i)
    plt.xlabel(i)
    plt.ylabel('num orders')<jupyter_output><empty_output><jupyter_text>It doesn't look like there's relationship between check out price, base price, region code and our target. However, at some specific values of the three features, number of orders are higher than others.<jupyter_code>plt.scatter(train_data['checkout_price'],train_data['base_price'])
plt.xlabel('check out price')
plt.ylabel('base price')
plt.show()<jupyter_output><empty_output><jupyter_text>There's a linear relationship between check out price and base price (since they are related).<jupyter_code>plt.scatter(train_data['checkout_price'],train_data['region_code'])
plt.xlabel('check out price')
plt.ylabel('region code')
plt.show()<jupyter_output><empty_output><jupyter_text>There's no relationship between check out price and region code, same for base price and region code. features_1:'city_code','center_type','op_area'<jupyter_code>features_1 = ['city_code','center_type','op_area']
print(f'missing values in city code: ', any(train_data['city_code'].isnull()))
print(f'missing values in center type: ', any(train_data['center_type'].isnull()))
print(f'missing values in op area: ', any(train_data['op_area'].isnull()))<jupyter_output>missing values in city code:  False
missing values in center type:  False
missing values in op area:  False<jupyter_text>no missing values<jupyter_code>print(train_data['city_code'].unique())
print(train_data['center_type'].unique())<jupyter_output>[647 614 679 659 526 590 599 685 461 649 541 478 703 576 628 702 579 596
 648 632 522 615 577 517 651 683 713 609 675 553 593 654 693 658 515 556
 561 680 620 638 700 676 604 456 562 695 699 602 485 698 473]
['TYPE_C' 'TYPE_B' 'TYPE_A']<jupyter_text>There are 51 citys. and 3 center types<jupyter_code>fig, ax = plt.subplots(figsize=(15,10))
sns.countplot(x=train_data.city_code)
plt.title('city_code')<jupyter_output><empty_output><jupyter_text>Most of the cities have around 5000-7000 data points, expect for a few, such as 252 and 959, which have more than 40,000 data point.<jupyter_code>fig, ax = plt.subplots(figsize=(15,10))
sns.countplot(x=train_data.center_type)
plt.title('center_type')<jupyter_output><empty_output><jupyter_text>The distribution of center_type is unbalanced with more TYPE_A<jupyter_code>fig, ax = plt.subplots(figsize=(15,10))
plt.hist(x=train_data.op_area)
plt.title('op_area')<jupyter_output><empty_output><jupyter_text>op_area is roughly normally distributed.<jupyter_code>fig, ax = plt.subplots(3,1,figsize=(15,30))

for var, subplot in zip(['city_code','center_type','op_area'],ax.flatten()):
    subplot.set_yscale("log")
    sns.boxplot(x=var, y=target, data=train_data, palette="vlag", ax=subplot)
    subplot.set_title('target vs ' + var)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12591958939264328, 'fraction_numerical': 0.04721984602224123, 'mean_word_length': 6.344221105527638, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '233652', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2250, 'n_tokens_neox': 2004, 'n_words': 496, 'path': 'index.ipynb', 'repo_name': 'jihuali/aml-project', 'size': '5845'}