<jupyter_start><jupyter_text>Loading datasetsNote that the seperator of these `csv` files are `\t` instead of `,` used in common.<jupyter_code>user_dtype = dict(
    age='category',
    gender='category',
    country='category',
    province='category',
    city='category',
    city_level='category',
    device_name='category'
)

user_file  = '../datasets/raw/traindata/user_features_data.csv'
df = pd.read_csv(user_file, sep='\t', index_col='user_id', dtype=user_dtype)<jupyter_output>/home/wupj/miniconda3/envs/pytorch17/lib/python3.7/site-packages/numpy/lib/arraysetops.py:580: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  mask |= (ar1 == a)<jupyter_text>View & Quality viewHead view of *user* dataset, which entirely consists of discrete numbers. There are `5910800` users' record in the dataset.The *user* dataset consists entirely of numbers, it's good.<jupyter_code>df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>Quality- [x] No duplicte record.- [x] No `nan` record.<jupyter_code># check duplicte
print(f'duplicate num: {df.index.nunique() - df.shape[0]}')
# check nan
print('nan numbers:')
pd.DataFrame(df.isnull().sum()).T<jupyter_output>duplicate num: 0
nan numbers:<jupyter_text>Statistics* mean* variance* information entropyBased on the following analysis (mostly according to information entropy), we select `age`, `gender`, `province` (weak suggested), `city_level`, `device_name` (weak suggested), as user's features.<jupyter_code>def info_entropy(count):
    n = count.sum()
    prob = count.values / n
    return np.sum(prob * -np.log(prob))
stats = dict(mean=[], std=[], entropy=[])
for col in df.columns:
    count = df[col].value_counts()
    stats['mean'].append(count.mean())
    stats['std'].append(count.std())
    stats['entropy'].append(info_entropy(count))
pd.DataFrame(stats, index=df.columns).T<jupyter_output><empty_output><jupyter_text>Device nameAccording to the analysis results, the devices with counts below **3000** (or device label greater than **250**) should be integrated into one label.<jupyter_code>count = df.device_name.value_counts()
print(f'number mean: {count.mean()}')
print(f'number std: {count.std()}')
print(f'info entropy: {info_entropy(count)}')
fig, ax1 = plt.subplots(1,1)
ax2 = ax1.twinx()
ax1.grid()
ax1.plot(count.values, c='C0', label='count')
ax2.plot(count.cumsum().values, c='C1', label='cumsum')
ax1.set_yscale('log')
ax1.set_xlabel('Device')
ax1.set_ylabel('Count')
ax2.set_ylabel('Cumsum')
fig.legend(loc=1, bbox_to_anchor=(1, 1), bbox_transform=ax1.transAxes)
fig.show()<jupyter_output>number mean: 3237.0208105147863
number std: 11621.459303297364
info entropy: 5.271148343003711<jupyter_text>Since there is a significant unbalance in the number of devices, we should keep first $k$ devices and integrate last devices into one label. We discuss $k$ by the infomation entropy of device feature.<jupyter_code>keeps = np.arange(50, 400, 50)
entropy = list()
for k in keeps:
    top_k = df.device_name.value_counts().iloc[:k]
    top_k = top_k.append(pd.Series(df.shape[0] - top_k.sum(), index=[k]))
    entropy.append(info_entropy(top_k))
fig, ax = plt.subplots(1,1)
ax.grid()
ax.plot(keeps, entropy)
ax.set_xlabel('Keep top-$k$')
ax.set_ylabel('Infomation entropy')
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14949674363528714, 'fraction_numerical': 0.03049141503848431, 'mean_word_length': 6.158898305084746, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '19824', 'license': 'MIT', 'n_tokens_mistral': 1240, 'n_tokens_neox': 1183, 'n_words': 274, 'path': 'eda/user.ipynb', 'repo_name': 'Pangjing-Wu/multi-objective-video-recommendation', 'size': '3378'}