<jupyter_start><jupyter_text>Recommendations with IBM - Data Cleaning  Preparations

Install necessary libraries and local scripts.<jupyter_code>!pip install -e ../
import pandas as pd

from ibm_recommend.cleaning import email_mapper<jupyter_output><empty_output><jupyter_text>Articles<jupyter_code>articles = pd.read_csv("../data/raw/articles_community.csv")
articles.head()
articles.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1056 entries, 0 to 1055
Data columns (total 6 columns):
 #   Column           Non-Null Count  Dtype 
---  ------           --------------  ----- 
 0   Unnamed: 0       1056 non-null   int64 
 1   doc_body         1042 non-null   object
 2   doc_description  1053 non-null   object
 3   doc_full_name    1056 non-null   object
 4   doc_status       1056 non-null   object
 5   article_id       1056 non-null   int64 
dtypes: int64(2), object(4)
memory usage: 49.6+ KB<jupyter_text>> NOTES

> Remove `Unnamed: 0`

> Convert `article_id` as string

> Reorder/rename columns (`article_id`, `name`, `decription`, `body`)<jupyter_code>articles[articles.doc_body.isna()]
articles[articles.doc_description.isna()]<jupyter_output><empty_output><jupyter_text>> NOTES

> Descriptions and body is not that important as long as the name/title exist. Leaving as is.<jupyter_code>articles.doc_status.value_counts()<jupyter_output><empty_output><jupyter_text>> NOTES

> Drop `doc_status` as it has no other value than `Live`<jupyter_code>articles[articles.article_id.duplicated(keep=False)].sort_values(by="article_id")<jupyter_output><empty_output><jupyter_text>> NOTES

> Safe to drop duplicates on `article_id`  Perform Cleaning<jupyter_code># Remove `Unnamed: 0` and `doc_status`

articles = articles.drop(columns=["Unnamed: 0", "doc_status"])
articles.head()
# drop duplicates on `article_id`

articles = articles.drop_duplicates(subset="article_id", keep="first")
articles[articles.article_id.duplicated(keep=False)].sort_values(by="article_id")
# Reorder/rename columns (`article_id`, `name`, `decription`, `body`)

articles = articles.rename(columns={ "doc_full_name": "name", "doc_description": "description", "doc_body": "body" })
articles = articles[["article_id", "name", "description", "body"]]
articles
articles.info()
articles.to_csv("../data/processed/articles.csv", index=False)<jupyter_output><empty_output><jupyter_text>User Interaction<jupyter_code>interactions = pd.read_csv("../data/raw/user-item-interactions.csv")
interactions.head()
interactions.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 45993 entries, 0 to 45992
Data columns (total 4 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  -----  
 0   Unnamed: 0  45993 non-null  int64  
 1   article_id  45993 non-null  float64
 2   title       45993 non-null  object 
 3   email       45976 non-null  object 
dtypes: float64(1), int64(1), object(2)
memory usage: 1.4+ MB<jupyter_text>> NOTES

> Drop `Unnamed: 0`

> Convert article_id to `int64`

> Email is GUID encoded, we can simplify this to integer for better view.<jupyter_code>interactions[~interactions.article_id.astype("int64").isin(articles.article_id.unique())][["article_id", "title"]].drop_duplicates()<jupyter_output><empty_output><jupyter_text>> NOTES

> There are 277 articles that are not currently on articles table. Keeping the title for possibility of including them in the articles table.<jupyter_code>interactions[interactions.email.isna()]<jupyter_output><empty_output><jupyter_text>> NOTES

> Although we have null on users (email) in the table, it might not matter when we are fetching the "popular" options. Leaving as is.

> Additional information that null emails are from a single user, map these to a single user id later. **Perform Cleaning**<jupyter_code># Drop `Unnamed: 0`

interactions = interactions.drop(columns=["Unnamed: 0"])
interactions.head()
# Convert article_id to `string`

interactions.article_id = interactions.article_id.astype("int64")
print(interactions.article_id.dtype)
interactions.head()
# map `email` to numeric `user_id`, keep the `email` column for additional analysis later

interactions["user_id"] = email_mapper(interactions.email)
interactions.head()
# reorder columns

interactions = interactions[["user_id", "article_id", "title", "email"]]
interactions.head()
interactions.info()
interactions.to_csv("../data/processed/interactions.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12952677182848257, 'fraction_numerical': 0.025994223505887582, 'mean_word_length': 4.97875166002656, 'pattern_counts': {'":': 3, '<': 41, '<?xml version=': 0, '>': 60, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '80745', 'license': 'MIT', 'n_tokens_mistral': 1585, 'n_tokens_neox': 1467, 'n_words': 354, 'path': 'notebooks/01 - Data Wrangling.ipynb', 'repo_name': 'netxph/ibm-recommend', 'size': '4501'}