<jupyter_start><jupyter_text>Loading datasetsNote that the seperator of these `csv` files are `\t` instead of `,` used in common.<jupyter_code>video_dtype = dict(
    video_name='object',
    video_tags='object',
    video_description='object',
    video_release_date='object',
    video_director_list='object',
    video_actor_list='object',
    video_score='float',
    video_second_class='object',
    video_duration='int'
)

video_file = '../datasets/raw/traindata/video_features_data.csv'
df = pd.read_csv(video_file, sep='\t', index_col='video_id', dtype=video_dtype, quoting=csv.QUOTE_NONE)<jupyter_output><empty_output><jupyter_text>View & Quality viewHead view of *user* dataset, which consists of continuous/discrete numbers and Chinese characters. There are `50355` videos' record in the dataset.<jupyter_code>len(df)
df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>Quality- [x] No duplicte record.- [ ] No `nan` record.Damn, this dataset shows low quality that more than one third of it records contain `nan`.<jupyter_code># check duplicte
print(f'duplicate num: {df.index.nunique() - df.shape[0]}')
# check nan
print('nan numbers:')
pd.DataFrame(df.isnull().sum()).T<jupyter_output>duplicate num: 0
nan numbers:<jupyter_text>`Nan` process* video_name: not a valid feature.* video_tages: replace `nan` by string `'nan'`* video_description: not a valid feature (candidate feature).* video_release_date: replace `nan` by `1850-01-01`.* video_director_list: replace `nan` by string `'nan'`.* video_actor_list: replace `nan` by string `'nan'`.* video_score: replace `nan` by `0.0`.* video_second_class: replace `nan` by string `'nan'`.<jupyter_code>df.video_name.fillna('nan', inplace=True)
df.video_tags.fillna('nan', inplace=True)
df.video_description.fillna('nan', inplace=True)
df.video_release_date.fillna('1850-01-01', inplace=True)
df.video_director_list.fillna('nan', inplace=True)
df.video_actor_list.fillna('nan', inplace=True)
df.video_score.fillna(0, inplace=True)
df.video_second_class.fillna('nan', inplace=True)
print('nan numbers:')
pd.DataFrame(df.isnull().sum()).T<jupyter_output>nan numbers:<jupyter_text>Statistics* mean* variance* information entropyBased on the following analysis, we select `video_release_date`, `video_score`, `video_second_class`, and `video_duration` as video's features.  Analysis tools<jupyter_code>def extract_terms(series:pd.Series) -> list:
    terms = list()
    for row in series.values:
        term = row.split(',')
        term = [t.strip() for t in term]
        terms.append(term)
    return terms

def list_flatten(x):
    flatten = list()
    for i in x:
        if isinstance(i, list):
            flatten += list_flatten(i)
        else:
            flatten.append(i)
    return flatten

def info_entropy(count:np.array):
    n = count.sum()
    prob = count.values / n
    return np.sum(prob * -np.log(prob))<jupyter_output><empty_output><jupyter_text>video_tagsThe tags of videos are more than `21436`, which seemed to be specific to individual movies rather than a category. It is not a representative feature.<jupyter_code>video_tags = extract_terms(df.video_tags)
count = Counter(list_flatten(video_tags))
count = pd.Series(dict(count)).sort_values(ascending=False)
print(f'n tags: {count.shape[0]}')
print(f'number mean: {count.mean()}')
print(f'number std: {count.std()}')
print(f'infomation entropy: {info_entropy(count)}')
fig, ax1 = plt.subplots(1,1)
ax2 = ax1.twinx()
ax1.grid()
ax1.plot(count.values, c='C0', label='count')
ax2.plot(count.cumsum().values, c='C1', label='cumsum')
ax1.set_yscale('log')
ax1.set_xlabel('Film tag')
ax1.set_ylabel('Count')
ax2.set_ylabel('Cumsum')
fig.legend(loc=1, bbox_to_anchor=(1, 1), bbox_transform=ax1.transAxes)
fig.show()
pd.DataFrame(count).head()<jupyter_output>n tags: 21674
number mean: 7.899280243609855
number std: 143.20538427386785
infomation entropy: 5.988537999580134<jupyter_text>video_second_classWe analyzed the second class tags of videos, which nicely represent film categories.<jupyter_code>video_second_class   = extract_terms(df.video_second_class)
video_second_class_n = [len(film) for film in video_second_class]
count = Counter(list_flatten(video_second_class))
count = pd.Series(dict(count)).sort_values(ascending=False)
print(f'n tags: {count.shape[0]}')
print(f'max num per film: {max(video_second_class_n)}')
print(f'mean num per film: {sum(video_second_class_n) / len(video_second_class_n)}')
print(f'total tags num mean: {count.mean()}')
print(f'total tags num std: {count.std()}')
print(f'infomation entropy: {info_entropy(count)}')
fig, ax1 = plt.subplots(1,1)
ax2 = ax1.twinx()
ax1.grid()
ax1.plot(count.values, c='C0', label='count')
ax2.plot(count.cumsum().values, c='C1', label='cumsum')
ax1.set_yscale('log')
ax1.set_xlabel('Film tag')
ax1.set_ylabel('Count')
ax2.set_ylabel('Cumsum')
fig.legend(loc=1, bbox_to_anchor=(1, 1), bbox_transform=ax1.transAxes)
fig.show()
pd.DataFrame(count).head()<jupyter_output>n tags: 145
max tag num per film: 9
mean tag num per film: 2.1433422698838247
number mean: 744.3310344827586
number std: 2679.4772112795217
infomation entropy: 2.873241390586258<jupyter_text>Video Release DateWe simplify this feature to the corresponding decade. The year of release is evenly distributed, and most films released after the 1950s.<jupyter_code>df.video_release_date = pd.to_datetime(df.video_release_date)
year_format = lambda date: date.year // 10 * 10
df['video_release_year'] = df.video_release_date.apply(year_format)
count = df.video_release_year.value_counts()
print(f'number mean: {count.mean()}')
print(f'number std: {count.std()}')
print(f'info entropy: {info_entropy(count)}')
pd.DataFrame(count.sort_index(ascending=False))<jupyter_output>number mean: 3147.1875
number std: 5952.878947408556
info entropy: 1.764389316114186<jupyter_text>Video ScoreIt's an excellent feature with a good ability of representation and does not need to be processed.<jupyter_code>count = df.video_score.value_counts()
print(f'number mean: {count.mean()}')
print(f'number std: {count.std()}')
print(f'info entropy: {info_entropy(count)}')
fig, ax = plt.subplots(1,1)
ax.set_xlabel('Film score')
ax.set_ylabel('Count')
df.video_score.hist(bins=20, ax=ax)
plt.show()<jupyter_output>number mean: 1007.1
number std: 979.9606249899085
info entropy: 3.4362176886296667<jupyter_text>Video DurationThe scale of this feature is larger than others, therefore $z$-score normalization should be applied to it.<jupyter_code>count = df.video_duration.value_counts()
print(f'number mean: {count.mean()}')
print(f'number std: {count.std()}')
print(f'info entropy: {info_entropy(count)}')
fig, ax = plt.subplots(1,1)
ax.set_yscale('log')
ax.set_xlabel('Film duration')
ax.set_ylabel('Count')
df.video_duration.hist(bins=50, ax=ax)
plt.show()<jupyter_output>number mean: 7.306297156123041
number std: 15.204418578793913
info entropy: 8.002389315751271
{'fraction_non_alphanumeric': 0.14137033824804857, 'fraction_numerical': 0.04856895056374675, 'mean_word_length': 6.628445424476295, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '251213', 'license': 'MIT', 'n_tokens_mistral': 2700, 'n_tokens_neox': 2493, 'n_words': 540, 'path': 'eda/film.ipynb', 'repo_name': 'Pangjing-Wu/multi-objective-video-recommendation', 'size': '6918'}