<jupyter_start><jupyter_text>Prepare the AI Spam Classifier DatasetWe'll be combining 2 open source datasets curated by [The University of California, Irvine (UCI)](https://archive.ics.uci.edu):- Spam SMS ([source](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection))- YouTube Spam ([source](https://archive.ics.uci.edu/ml/datasets/YouTube+Spam+Collection)) Requirements- Python- Jupyter (Setup with [this video](https://www.youtube.com/watch?v=9tPS-7TWjq0))- Pandas  Step 1. Download Datasets Create destination folders<jupyter_code>import pathlib

USE_PROJECT_ROOT = True
BASE_DIR = pathlib.Path(".").resolve()
if USE_PROJECT_ROOT:
    BASE_DIR = BASE_DIR.parent.parent
DATASET_DIR = BASE_DIR / "datasets"
ZIPS_DIR = DATASET_DIR / 'zips'
EXPORT_DIR = DATASET_DIR / "exports"
SMS_SPAM_DIR = DATASET_DIR / 'imports' / 'sms-spam'
YOUTUBE_SPAM_DIR = DATASET_DIR / 'imports' / 'youtube-spam'
ZIPS_DIR.mkdir(exist_ok=True, parents=True)

EXPORT_DIR.mkdir(exist_ok=True, parents=True)

SMS_SPAM_DIR.mkdir(exist_ok=True, parents=True)

YOUTUBE_SPAM_DIR.mkdir(exist_ok=True, parents=True)<jupyter_output><empty_output><jupyter_text>You could also create the directories using:```!mkdir -p $DATASET_DIR/zips/!mkdir -p $SMS_SPAM_DIR!mkdir -p $YOUTUBE_SPAM_DIR!mkdir -p $EXPORT_DIR```  UCI Spam SMSSource: https://archive.ics.uci.edu/ml/datasets/sms+spam+collection<jupyter_code>!curl https://archive.ics.uci.edu/ml/machine-learning-databases/00228/smsspamcollection.zip -o $ZIPS_DIR/uci-sms-spam.zip
!unzip -o $ZIPS_DIR/uci-sms-spam.zip -d $SMS_SPAM_DIR<jupyter_output>% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  198k  100  198k    0     0   422k      0 --:--:-- --:--:-- --:--:--     0-:--:-- --:--:-- --:--:--  422k
Archive:  /Users/cfe/Dev/ai-microservice/datasets/zips/uci-sms-spam.zip
  inflating: /Users/cfe/Dev/ai-microservice/datasets/imports/sms-spam/SMSSpamCollection  
  inflating: /Users/cfe/Dev/ai-microservice/datasets/imports/sms-spam/readme<jupyter_text>YouTube SpamSource: https://archive.ics.uci.edu/ml/datasets/YouTube+Spam+Collection<jupyter_code>!curl https://archive.ics.uci.edu/ml/machine-learning-databases/00380/YouTube-Spam-Collection-v1.zip -o $ZIPS_DIR/uci-youtube-spam.zip
!unzip -o $ZIPS_DIR/uci-youtube-spam.zip -d $YOUTUBE_SPAM_DIR<jupyter_output>% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  159k  100  159k    0     0   359k      0 --:--:-- --:--:-- --:--:--  359k
Archive:  /Users/cfe/Dev/ai-microservice/datasets/zips/uci-youtube-spam.zip
  inflating: /Users/cfe/Dev/ai-microservice/datasets/imports/youtube-spam/Youtube01-Psy.csv  
   creating: /Users/cfe/Dev/ai-microservice/datasets/imports/youtube-spam/__MACOSX/
  inflating: /Users/cfe/Dev/ai-microservice/datasets/imports/youtube-spam/__MACOSX/._Youtube01-Psy.csv  
  inflating: /Users/cfe/Dev/ai-microservice/datasets/imports/youtube-spam/Youtube02-KatyPerry.csv  
  inflating: /Users/cfe/Dev/ai-microservice/datasets/imports/youtube-spam/__MACOSX/._Youtube02-KatyPerry.csv  
  inflating: /Users/cfe/Dev/ai-microservice/datasets/imports/youtube-spam/Youtube03-LMFAO.csv  
  inflating: /Users/cfe/Dev/ai-microservice/datasets/imports/youtube-spam/__MACOSX/._Youtube03-LM[...]<jupyter_text>Step 2. Load Datasets into a Pandas DataFrame<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>**Load the `sms-spam` dataset into a pandas dataframe**<jupyter_code>sms_path = SMS_SPAM_DIR / 'SMSSpamCollection'
sms_df = pd.read_csv(str(sms_path), sep='\t', header=None)<jupyter_output><empty_output><jupyter_text>Now set the headers<jupyter_code>sms_df.columns = ['label', 'text']
sms_df['source'] = 'uci-spam-sms'<jupyter_output><empty_output><jupyter_text>**Load the `youtube-spam` datasets into a pandas dataframe**The youtube-spam dataset is stored across multiple csvs. Let's combine them into 1 big file.<jupyter_code>location = YOUTUBE_SPAM_DIR
csvs = list(location.glob("*.csv"))
new_dfs = []
for csv in csvs:
    csv_df = pd.read_csv(str(csvs[0]), usecols=['CLASS', 'CONTENT'])
    csv_df.rename(columns={'CLASS': 'class', "CONTENT": 'text'}, inplace=True)
    csv_df['label'] = csv_df['class'].apply(lambda x: "spam" if str(x) == "1" else "ham")
    sub_df = csv_df.copy()[['label', 'text']] 
    new_dfs.append(sub_df)

yt_df = pd.concat(new_dfs)
yt_df['source'] = 'uci-youtube-spam'<jupyter_output><empty_output><jupyter_text>**Combine the `sms-spam` dataset and the `youtube-spam` dataset**<jupyter_code>df = pd.concat([sms_df, yt_df])<jupyter_output><empty_output><jupyter_text>Step 3. Export Complete Dataset<jupyter_code>df.to_csv(EXPORT_DIR / 'spam-dataset.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14832339024891997, 'fraction_numerical': 0.01563464307755606, 'mean_word_length': 4.518728717366629, 'pattern_counts': {'":': 1, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '252473', 'license': 'Apache-2.0', 'n_tokens_mistral': 2017, 'n_tokens_neox': 1900, 'n_words': 282, 'path': 'guides/spam-classifier/1 - Prepare the AI Spam Classifier Dataset.ipynb', 'repo_name': 'DataStax-Examples/ai-as-an-api', 'size': '4861'}