<jupyter_start><jupyter_text>The Data Split the dataset in to train and test. We are using training data only for EDA.<jupyter_code>df = pd.read_csv('train.tsv', sep = '\t')
msk = np.random.rand(len(df)) < 0.8
train = df[msk]
test = df[~msk]
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>EDA<jupyter_code>train.head()
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1185866 entries, 0 to 1482533
Data columns (total 8 columns):
train_id             1185866 non-null int64
name                 1185866 non-null object
item_condition_id    1185866 non-null int64
category_name        1180783 non-null object
brand_name           679496 non-null object
price                1185866 non-null float64
shipping             1185866 non-null int64
item_description     1185863 non-null object
dtypes: float64(1), int64(3), object(4)
memory usage: 81.4+ MB<jupyter_text>Price<jupyter_code>train.price.describe()
plt.subplot(1, 2, 1)
(train['price']).plot.hist(bins=50, figsize=(12, 6), edgecolor = 'white', range = [0, 250])
plt.xlabel('price', fontsize=12)
plt.title('Price Distribution', fontsize=12)

plt.subplot(1, 2, 2)
np.log(train['price']+1).plot.hist(bins=50, figsize=(12,6), edgecolor='white')
plt.xlabel('log(price+1)', fontsize=12)
plt.title('Price Distribution', fontsize=12)<jupyter_output><empty_output><jupyter_text>The price of items are left skewed, vast majority of items priced at 10-20. However, the most expensive item priced at 2009. So we make log-transformation on the price.  Shipping<jupyter_code>train['shipping'].value_counts() / len(train)<jupyter_output><empty_output><jupyter_text>Over 55% of items shipping fee were paid by the buyers. How shipping related to price?<jupyter_code>shipping_fee_by_buyer = train.loc[df['shipping'] == 0, 'price']
shipping_fee_by_seller = train.loc[df['shipping'] == 1, 'price']

fig, ax = plt.subplots(figsize=(18,8))
ax.hist(shipping_fee_by_seller, color='#8CB4E1', alpha=1.0, bins=50, range = [0, 100],
       label='Price when Seller pays Shipping')
ax.hist(shipping_fee_by_buyer, color='#007D00', alpha=0.7, bins=50, range = [0, 100],
       label='Price when Buyer pays Shipping')
plt.xlabel('price', fontsize=12)
plt.ylabel('frequency', fontsize=12)
plt.title('Price Distribution by Shipping Type', fontsize=15)
plt.tick_params(labelsize=12)
plt.legend()
plt.show()
print('The average price is {}'.format(round(shipping_fee_by_seller.mean(), 2)), 'if seller pays shipping');
print('The average price is {}'.format(round(shipping_fee_by_buyer.mean(), 2)), 'if buyer pays shipping')<jupyter_output>The average price is 22.58 if seller pays shipping
The average price is 30.11 if buyer pays shipping<jupyter_text>We compare again after log-transformation on the price.<jupyter_code>fig, ax = plt.subplots(figsize=(18,8))
ax.hist(np.log(shipping_fee_by_seller+1), color='#8CB4E1', alpha=1.0, bins=50,
       label='Price when Seller pays Shipping')
ax.hist(np.log(shipping_fee_by_buyer+1), color='#007D00', alpha=0.7, bins=50,
       label='Price when Buyer pays Shipping')
plt.xlabel('log(price+1)', fontsize=12)
plt.ylabel('frequency', fontsize=12)
plt.title('Price Distribution by Shipping Type', fontsize=15)
plt.tick_params(labelsize=12)
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>It is obvious that the average price is higher when buyer pays shipping.  Category Names  Category names<jupyter_code>print('There are', train['category_name'].nunique(), 'unique values in category name column')<jupyter_output>There are 1265 unique values in category name column<jupyter_text>Top 10 most common category names<jupyter_code>train['category_name'].value_counts()[:10]<jupyter_output><empty_output><jupyter_text>Item condition vs. Price<jupyter_code>sns.boxplot(x = 'item_condition_id', y = np.log(train['price']+1), data = train, palette = sns.color_palette('RdBu',5))<jupyter_output><empty_output><jupyter_text>There seems to be various on the average price between each item condition id. Therefore, we are going to use all the features to build our model.  LightGBM  Settings<jupyter_code>NUM_BRANDS = 4000
NUM_CATEGORIES = 1000
NAME_MIN_DF = 10
MAX_FEATURES_ITEM_DESCRIPTION = 50000<jupyter_output><empty_output><jupyter_text>There are missing values in the columns.<jupyter_code>print('There are %d items that do not have a category name.' %train['category_name'].isnull().sum())
print('There are %d items that do not have a brand name.' %train['brand_name'].isnull().sum())
print('There are %d items that do not have a description.' %train['item_description'].isnull().sum())
def handle_missing_inplace(dataset): 
    dataset['category_name'].fillna(value='missing', inplace=True) 
    dataset['brand_name'].fillna(value='missing', inplace=True) 
    dataset['item_description'].replace('No description yet,''missing', inplace=True) 
    dataset['item_description'].fillna(value='missing', inplace=True)
def cutting(dataset):
    pop_brand = dataset['brand_name'].value_counts().loc[lambda x: x.index != 'missing'].index[:NUM_BRANDS]
    dataset.loc[~dataset['brand_name'].isin(pop_brand), 'brand_name'] = 'missing'
    pop_category = dataset['category_name'].value_counts().loc[lambda x: x.index != 'missing'].index[:NUM_CATEGORIES]
def to_categorical(dataset):
    dataset['category_name'] = dataset['category_name'].astype('category')
    dataset['brand_name'] = dataset['brand_name'].astype('category')
    dataset['item_condition_id'] = dataset['item_condition_id'].astype('category')
df = pd.read_csv('train.tsv', sep = '\t')
msk = np.random.rand(len(df)) < 0.8
train = df[msk]
test = df[~msk]
test_new = test.drop('price', axis=1)
y_test = np.log1p(test["price"])<jupyter_output><empty_output><jupyter_text>Drop rows where price = 0<jupyter_code>train = train[train.price != 0].reset_index(drop=True)
nrow_train = train.shape[0]
y = np.log1p(train["price"])
merge: pd.DataFrame = pd.concat([train, test_new])
handle_missing_inplace(merge)
cutting(merge)
to_categorical(merge)
merge.head()<jupyter_output><empty_output><jupyter_text>Count vectorize name and category name columns.<jupyter_code>cv = CountVectorizer(min_df=NAME_MIN_DF)
X_name = cv.fit_transform(merge['name'])
cv = CountVectorizer()
X_category = cv.fit_transform(merge['category_name'])<jupyter_output><empty_output><jupyter_text>TFIDF Vectorize item_description column.<jupyter_code>tv = TfidfVectorizer(max_features=MAX_FEATURES_ITEM_DESCRIPTION, ngram_range=(1, 3), stop_words='english')
X_description = tv.fit_transform(merge['item_description'])<jupyter_output><empty_output><jupyter_text>Label binarize brand_name column.<jupyter_code>lb = LabelBinarizer(sparse_output=True)
X_brand = lb.fit_transform(merge['brand_name'])<jupyter_output><empty_output><jupyter_text>Create dummy variables for item_condition_id and shipping columns.<jupyter_code>X_dummies = csr_matrix(pd.get_dummies(merge[['item_condition_id', 'shipping']], sparse=True).values)<jupyter_output><empty_output><jupyter_text>Create sparse merge.<jupyter_code>sparse_merge = hstack((X_dummies, X_description, X_brand, X_category, X_name)).tocsr()
sparse_merge.shape<jupyter_output><empty_output><jupyter_text>Remove features with document frequency <=1.<jupyter_code>mask = np.array(np.clip(sparse_merge.getnnz(axis=0) - 1, 0, 1), dtype=bool)
sparse_merge = sparse_merge[:, mask]
sparse_merge.shape<jupyter_output><empty_output><jupyter_text>Separate train and test data from sparse merge.<jupyter_code>X = sparse_merge[:nrow_train]
X_test = sparse_merge[nrow_train:]
train_X = lgb.Dataset(X, label=y)
params = {
        'learning_rate': 0.75,
        'application': 'regression',
        'max_depth': 3,
        'num_leaves': 100,
        'verbosity': -1,
        'metric': 'RMSE',
    }<jupyter_output><empty_output><jupyter_text>TrainingTraining a model requires a parameter list and data set. And training will take a while.<jupyter_code>gbm = lgb.train(params, train_set=train_X, num_boost_round=3200, verbose_eval=100)<jupyter_output><empty_output><jupyter_text>Prediction<jupyter_code>y_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration)
from sklearn.metrics import mean_squared_error
print('The rmse of prediction is:', mean_squared_error(y_test, y_pred) ** 0.5)<jupyter_output>The rmse of prediction is: 0.46164222941613137
{'fraction_non_alphanumeric': 0.13660939565060676, 'fraction_numerical': 0.0342424606512075, 'mean_word_length': 7.050290135396518, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 82, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '515064', 'license': 'MIT', 'n_tokens_mistral': 3111, 'n_tokens_neox': 2878, 'n_words': 595, 'path': 'Mercari Price Suggestion Lightgbm.ipynb', 'repo_name': 'BeeChainFoundation/Brainstorming', 'size': '8323'}