<jupyter_start><jupyter_text>2014-2015 season for player_stat<jupyter_code>player_stat_14_15 = player_stat[player_stat['Season'] == '2014-15']
player_stat_14_15.shape
player_stat_14_15['Player'].value_counts()<jupyter_output><empty_output><jupyter_text>- Comment: no duplicated names for player_stat_14_15  Get 2014-2015 data for player_cv<jupyter_code>player_cv_14_15 = player_cv[(player_cv['From']<=2015) & (player_cv['To']>=2014)]
player_cv_14_15.shape<jupyter_output><empty_output><jupyter_text>Merge these two<jupyter_code>merge_stat_cv = pd.merge(player_stat_14_15, player_cv_14_15, left_on='Player', right_on='Player', how='inner')
merge_stat_cv.shape[0] == player_stat_14_15.shape[0]<jupyter_output><empty_output><jupyter_text>- Comment: $\{$player $P$: $P \in$ player_stat_14_15$\}$  $\subseteq$ \{ player $P$: $P \in$ player_cv_14_15 \}  Get 2014-2015 data for player_salary<jupyter_code>player_salary_14_15 = player_salary[player_salary['SEASON'] == '2014-2015']
player_salary_14_15.shape
player_salary_14_15.head(3)<jupyter_output><empty_output><jupyter_text>preprocess the name<jupyter_code># all names contain ","
player_salary_14_15['NAME'].str.contains(",").sum() == player_salary_14_15.shape[0]
# all names contain one and only one ","
for l in player_salary_14_15['NAME'].str.split(",").to_list():
    if len(l)!=2:
        print(l)<jupyter_output><empty_output><jupyter_text>- Comment: We noticed that in player_salary_14_15, these four players' names are not in good format: <NAME>, <NAME>, <NAME>, <NAME>. For example, in the table merge_stat_cv, <NAME> is called <NAME>.<jupyter_code>false_names = ['<NAME>', '<NAME>', '<NAME>', '<NAME>']
true_names = ['<NAME>', '<NAME>', '<NAME>', '<NAME>']
def preprocess_name(name):
    if ',' in name:
        ind = name.find(',')
        name = name[:ind]
    ### We remove the Jr.
    if "Jr." in name:
        name = name.replace(" Jr.", "")
    if "Jr" in name:
        name = name.replace(" Jr.", "")
    if "III" in name:
        name = name.replace(" III", "")
    while '.' in name:
        ind = name.find('.')
        name = name.replace('.','')
        
    name = name.lower()
    name = name.strip()
    
    if name in false_names:
        name = true_names[false_names.index(name)]
    
    return name<jupyter_output><empty_output><jupyter_text>Final dataframe to work on for 2014-2015<jupyter_code>player_salary_14_15['Player'] = player_salary_14_15['NAME'].apply(lambda s: preprocess_name(s))
merge_stat_cv['Player'] = merge_stat_cv['Player'].apply(lambda s: preprocess_name(s))
merge_14_15 = pd.merge(merge_stat_cv, player_salary_14_15, left_on='Player', right_on='Player', how='inner')
merge_14_15.shape
merge_14_15.head()
np.sort(merge_14_15.TEAM.unique())
merge_14_15.to_csv(data_path + "merge_14_15.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16113074204946998, 'fraction_numerical': 0.05901060070671378, 'mean_word_length': 5.708530805687204, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '206683', 'license': 'MIT', 'n_tokens_mistral': 1149, 'n_tokens_neox': 1051, 'n_words': 189, 'path': '.ipynb_checkpoints/merge_preprocessing_2014_2015-checkpoint.ipynb', 'repo_name': 'nhatminh-96/datacamp-project', 'size': '2946'}