<jupyter_start><jupyter_text>- 该笔记主要是对数据做初步分析并进行处理，为后续模型训练准备好数据- 主要利用了两个数据    - ml-latest: movielens 数据集 (http://files.grouplens.org/datasets/movielens/ml-20m.zip),评分数据在20M左右    - movie_detail.csv，这部分数据是信息比较全的电影数据，为了项目上线，ml-latest只取该数据集里面的电影数据分析，数据来源：https://github.com/LuckyZXL2016/Movie_Recommend<jupyter_code>import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
import re
import os
import matplotlib.pyplot as plt
from IPython.core.display import Image, display
%matplotlib inline
def show_image(url):
    display(Image(url, width=200, unconfined=True))<jupyter_output><empty_output><jupyter_text>将数据库导出的表中电影名重名、空值的记录过滤掉<jupyter_code>df_movie = pd.read_csv('../../data/movie_detail.csv', engine='python', index_col=0)
df_movie.shape
df_movie['moviename'] = df_movie['moviename'].map(lambda x : x.strip() if type(x) is str else np.nan)
df_movie = df_movie.drop_duplicates(subset=['moviename'])
df_movie = df_movie.dropna(subset=['moviename'])
df_movie.shape<jupyter_output><empty_output><jupyter_text>将movielens的数据集中title重名为空的记录过滤掉<jupyter_code>df_movie_latest = pd.read_csv('../../data/ml-latest/movies.csv')
def get_movie_name_and_year(title):
    title = title.strip()
    if '(' not in title:
        return title, ""
    regex = r'^(.*)\((\d+)\)$'
    result = re.match(regex, title)
    if result is None:
        #print(title)
        return title, ""
    else:
        return result.group(1).strip(), result.group(2).strip()
df_movie_latest['moviename'] = df_movie_latest['title'].map(lambda x : get_movie_name_and_year(x)[0])
df_movie_latest['showyear'] = df_movie_latest['title'].map(lambda x : get_movie_name_and_year(x)[1])
df_movie_latest.shape
df_movie_latest = df_movie_latest.drop_duplicates(subset=['moviename'])
df_moive_latest = df_movie_latest.dropna(subset=['moviename'])
df_movie_latest.shape
df_movie_clean = pd.merge(df_movie, df_movie_latest, on='moviename')
df_movie_clean['showyear'] = df_movie_clean['showyear_x']
df_movie_clean = df_movie_clean[['moviename', 'director',
       'leadactors', 'screenwriter', 'picture', 'averating', 'numrating',
       'description', 'typelist', 'backpost', 'movieId', 'title', 'genres',
       'showyear']]
df_movie_clean = df_movie_clean.reset_index(drop=True)
if not os.path.exists("../../data/ml-clean/"):
    os.makedirs('../../data/ml-clean/')
df_movie_clean.to_csv('../../data/ml-clean/movies.csv')
df_ratings = pd.read_csv('../../data/ml-latest/ratings.csv')
df_ratings_clean = pd.merge(df_movie_clean[['movieId']], df_ratings, on='movieId')
df_ratings_clean.shape
df_ratings.shape
df_ratings_clean = df_ratings_clean.reset_index(drop=True)
df_ratings_clean.to_csv('../../data/ml-clean/ratings.csv')
df_ratings_clean.shape<jupyter_output><empty_output><jupyter_text>分析数据并作处理<jupyter_code>df_movies = pd.read_csv('../../data/ml-clean/movies.csv', index_col=0)
print('电影数量：{}'.format(df_movies.shape[0]))
df_ratings = pd.read_csv('../../data/ml-clean/ratings.csv', index_col=0)
df_ratings.columns
print('用户的数量：{}'.format(df_ratings['userId'].nunique()))
print('评论数量：{}'.format(df_ratings.shape[0]))<jupyter_output>评论数量：16205120<jupyter_text>电影被用户评论的情况<jupyter_code>df_ratings['movieId'].value_counts().describe()<jupyter_output><empty_output><jupyter_text>- 一半的电影评论的用户数不足6个，$\frac{3}{4}$的用户被不到50个用户打分- 这种很少有用户评论的电影在UserCF中由于过于稀疏，商品之间难以产生关系，需要将评论数小于等于6个的电影去掉<jupyter_code>df_valid_movies = df_ratings['movieId'].value_counts().reset_index()
df_valid_movies.columns = ['movieId', 'Count']
df_valid_movies = df_valid_movies[df_valid_movies['Count'] > 6]<jupyter_output><empty_output><jupyter_text>用户评论电影的情况<jupyter_code>df_ratings['userId'].value_counts().describe()
df_ratings['userId'].value_counts().clip(5000, 20000).describe()<jupyter_output><empty_output><jupyter_text>- 差不多1/4的用户评论数在10个以下，过少评论的用户提供的信息也比较少，需要去掉。<jupyter_code>df_valid_users = df_ratings['userId'].value_counts().reset_index()

df_valid_users.columns = ['userId', 'Count']

df_valid_users = df_valid_users[df_valid_users['Count'] > 10]<jupyter_output><empty_output><jupyter_text>多次评论的情况<jupyter_code>pd.to_datetime(df_ratings['timestamp'].min(), unit='s')
pd.to_datetime(df_ratings['timestamp'].max(), unit='s')
df_ratings['movie_user'] = df_ratings['movieId'].map(str) + '_' + df_ratings['userId'].map(str)
df_ratings['movie_user'].value_counts().describe()
df_valid_ratings = df_ratings['movie_user'].value_counts().reset_index()
df_valid_ratings.columns = ['userId', 'Count']
df_valid_ratings = df_valid_ratings[df_valid_ratings['Count'] <= 1]
df_valid_ratings['userId'] = df_valid_ratings['userId'].map(lambda x : int(x.split('_')[0]))
df_valid_ratings = df_valid_ratings.drop_duplicates(subset=['userId'])
df_valid_ratings.shape<jupyter_output><empty_output><jupyter_text>- 有将近1/10的评论是用户对电影的重复评论- 将有重复评论的用户筛掉  筛选数据<jupyter_code>df_ratings_clean = pd.merge(df_ratings, df_valid_movies[['movieId']], on='movieId')
df_ratings_clean = pd.merge(df_ratings_clean, df_valid_users[['userId']], on='userId')
print("筛选前的评论数：{}".format(df_ratings.shape[0]))
print("筛选后的评论数：{}".format(df_ratings_clean.shape[0]))
print("筛选前用户数：{}, 筛选后用户数：{}".format(df_ratings['userId'].nunique(), df_ratings_clean['userId'].nunique()))
print("筛选前电影数：{}, 筛选后电影数：{}".format(df_ratings['movieId'].nunique(), df_ratings_clean['movieId'].nunique()))
df_movies_clean = pd.merge(df_movies, df_valid_movies[['movieId']], on='movieId')
df_valid_movies.shape
df_movies_clean.shape
df_movies.shape
df_movies['movieId'].nunique()<jupyter_output><empty_output><jupyter_text>将筛选后的数据进行保存，供后续模型训练使用<jupyter_code>df_movies_clean.to_csv('../../data/ml-clean/movies_v2.csv')
df_ratings_clean.to_csv('../../data/ml-clean/ratings_v2.csv')<jupyter_output><empty_output><jupyter_text>筛选出需要导入微信小程序后台的数据<jupyter_code>df_movies_import = pd.read_csv('../../data/ml-clean/movies_v2.csv', index_col=0)
df_movies_import['name'] = df_movies_import['moviename']
df_movies_import['picUrl'] = df_movies_import['picture'].map(lambda x : x if x is not np.nan and 'jpg' in x else np.nan)
df_movies_import['score'] = df_movies_import['averating']
cols = ['movieId', 'name', 'score', 'picUrl', 'director', 'leadactors', 'genres', 'showyear']
df_movies_import = df_movies_import[cols].dropna(subset=['picUrl']).copy()
df_movies_import['imdbPicUrl'] = df_movies_import['picUrl']
df_movies_import['smallPicUrl'] = df_movies_import['picUrl']
df_movies_import.head()
df_movies_import['imdbPicUrl'].head().values.tolist()
df_movies_import.to_csv('../../data/wxcloud/import_movies_data.csv', index=None)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17265636947545496, 'fraction_numerical': 0.009328643523474537, 'mean_word_length': 8.732142857142858, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 50, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '641382', 'license': 'MIT', 'n_tokens_mistral': 2871, 'n_tokens_neox': 2880, 'n_words': 255, 'path': 'code/algorithm/prepare-data.ipynb', 'repo_name': 'aierwiki/favor-movies-public', 'size': '6539'}