<jupyter_start><jupyter_text>Categories1. Are they broad enough?2. Are they descriptive enough?<jupyter_code>itemCategories.head()<jupyter_output><empty_output><jupyter_text>too descriptive. At first lookwe may try to decouple broad categories by `split('-')`<jupyter_code>test.head()
train = train.merge(items.drop('item_name', axis=1), on='item_id', how='left')<jupyter_output><empty_output><jupyter_text>Obvious features: - ~~day~~- month- quarter- year- ~~dayofweek~~- ~~is workingday~~- ~~delta from every 22th day of month (~salary day in Rus)~~- ~~delta from every 7th day of month (~pre-salary day in Rus)~~I considered that in test set there is only a month that we know from a datetime features. It means that we can get only month+ features<jupyter_code># train['day'] = train['date'].dt.day
train['month'] = train['date'].dt.month
train['year'] = train['date'].dt.year
train['quarter'] = train['date'].dt.quarter
# train['dayofweek'] = train['date'].dt.weekday
# train['is_weekend'] = (train['dayofweek'] >= 5).astype('int8')
train.head()<jupyter_output><empty_output><jupyter_text>Sales & Price features: (think more...)**H1**:! People buy things when the price goes down- 1quart price in previous month (shop-item)- 3quart price in previous month (shop-item)- price delta in prev month (shop-item)- price delta over 2 months (shop-item)- monthly sales by {shop_id, category_id}- monthly sales by {shop_id, item_id}- monthly sales by {category_id}Probably it worth to make a **H2**:! People buy when a shop signals about sales (no matter which item)<jupyter_code>shop_item = train.groupby(['date_block_num', 'shop_id', 'item_id'])
q25 = shop_item.item_price.quantile(.25)
q75 = shop_item.item_price.quantile(.75)
q25.reset_index()
q25.reset_index().date_block_num + 1
train.merge(
    q25.reset_index().rename({'item_price': 'mon_price_1qrt'}, axis=1), 
    on=['date_block_num', 'shop_id', 'item_id'])
q25.reset_index().rename({'item_price': 'mon_price_1qrt'}, axis=1)
pd.DataFrame.rename?
shop_item.item_price.quantile(.75)
((shop_item.item_price.quantile(.25) - shop_item.item_price.quantile(.75)) != 0)<jupyter_output><empty_output><jupyter_text>Geographical features1. Get the city from the shop info2. Try to get the lat lon of the shop with map api ?-------------- what's next?<jupyter_code>def extract_city(value):
    return value.split()[0].strip('!')

def yandex_latlong(city):
    return geocoder.google(city).latlng


shops['city'] = shops['shop_name'].apply(extract_city)
# because of the error with geocoder.yandex, i took ready latlong from here:
# https://www.kaggle.com/bourov/yada-yet-another-e-da-by-geo-location-of-shops/data?select=coordinates.csv
coordinates = pd.read_csv('../input/coordinates.csv', index_col=0)
coordinates.head()
shops = shops.merge(coordinates, left_on='shop_id', right_index=True)
shops
shops.groupby('city').size().describe()<jupyter_output><empty_output><jupyter_text>C&C: Cleaning &ClusteringLet's try to cleanup the NaNs, indicate them with a new column, and add clusters with kNN (aka Regions)<jupyter_code>train.groupby(['date_block_num', 'shop_id', 'item_id']).item_cnt_day.sum()<jupyter_output><empty_output><jupyter_text>Validation- is it just a timewise split?- do we have new shops / items in test set? (if yes - we have to mimic this in validation strategy also)<jupyter_code># shops in test but not in train (unseen)
_mask = ~ np.isin(test.shop_id.unique(), train.shop_id.unique())
test.shop_id.unique()[_mask]
# items in test but not in train (unseen)
_mask = ~ np.isin(test.item_id.unique(), train.item_id.unique())
test.item_id.unique()[_mask].shape,  test.item_id.unique().shape<jupyter_output><empty_output><jupyter_text>Let's do the same with a validation set<jupyter_code>_mask = ~ np.isin(valid.shop_id.unique(), train[train.date_block_num < 33].shop_id.unique())
valid.shop_id.unique()[_mask]
_mask = ~ np.isin(valid.item_id.unique(), train[train.date_block_num < 33].item_id.unique())
valid.item_id.unique()[_mask].shape, valid.item_id.unique().shape<jupyter_output><empty_output><jupyter_text>Well, the test set is roughly the same as validation. We can use validation set as `date_block_num == 33` for sureProbably it's a good idea to exclude `shop_id == 36` to mimic a test set (the last has no new shops)One more **TODO**:check if we can use 2 folds: 33's and 32's blocks<jupyter_code>valid = train[train['date_block_num'] == (train['date_block_num'].max())]
(test['shop_id'].isin(train['shop_id']) == False).sum(), \
(test['item_id'].isin(train['item_id']) == False).sum()
test.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15565217391304348, 'fraction_numerical': 0.012608695652173913, 'mean_word_length': 6.3147853736089035, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '144062', 'license': 'Apache-2.0', 'n_tokens_mistral': 1644, 'n_tokens_neox': 1577, 'n_words': 390, 'path': 'notebooks/EDA_and_baseline.ipynb', 'repo_name': 'Codealist/future-sales-competition', 'size': '4600'}