<jupyter_start><jupyter_text>Step 1: Load Dataset<jupyter_code>USE_PROJECT_ROOT = True
BASE_DIR = pathlib.Path().resolve()
if USE_PROJECT_ROOT:
    BASE_DIR = BASE_DIR.parent
    
DATASET_DIR = BASE_DIR / "datasets"
EXPORT_DIR = DATASET_DIR / "exports"
DATASET_CSV_PATH = EXPORT_DIR / 'spam-dataset.csv'
TRAINING_DATA_PATH = EXPORT_DIR / 'spam-training-data.pkl'
print("BASE_DIR is", BASE_DIR)
RUN_DATASET_PREPARE = False
if RUN_DATASET_PREPARE:
    # if active, this will download and prepare the dataset.
    SOURCE_NB = pathlib.Path('01-Prepare_AI_Spam_Classifier_Dataset.ipynb')
    if SOURCE_NB.exists():
        %run './{SOURCE_NB}'
    else:
        print("01-Prepare_AI_Spam_Classifier_Dataset.ipynb does not exist.")
if not DATASET_CSV_PATH.exists():
    raise Exception(f"You must download or create the spam-dataset.csv \n{DATASET_CSV_PATH} not found.")
df = pd.read_csv(str(DATASET_CSV_PATH))
df.head()<jupyter_output><empty_output><jupyter_text>Step 2: Convert Dataset to Lists<jupyter_code>texts = df['text'].tolist()
labels = df['label'].tolist()<jupyter_output><empty_output><jupyter_text>Map our labels from being text values to being integer values. It's pretty simple:<jupyter_code>labels_legend = {'ham': 0, 'spam': 1}
labels_legend_inverted = {f"{v}":k for k,v in labels_legend.items()}<jupyter_output><empty_output><jupyter_text>The inverted legend is there to help us when we need to add a label to our predictions later.What's cool, is you can create this legend automatically with:```pythonlegend = {f"{x}": i for i, x in enumerate(list(set(labels)))}legend_inverted = {f"{v}":k for k,v in legend.items()}```<jupyter_code>labels_as_int =  [labels_legend[str(x)] for x in labels]
random_idx = random.randint(0, len(texts))
print('Random Index', random_idx)

assert texts[random_idx] == df.iloc[random_idx].text
assert labels[random_idx] == df.iloc[random_idx].label
assert labels_legend_inverted[str(labels_as_int[random_idx])] == labels[random_idx]<jupyter_output>Random Index 5264<jupyter_text>Step 3: Tokenize TextsThe Keras Tokenizer will convert our raw text into vectors. Converting texts to vectors is a required step for any machine learning model (not just keras).<jupyter_code>MAX_NUM_WORDS = 280
MAX_SEQUENCE_LENGTH = 300<jupyter_output><empty_output><jupyter_text>> MAX_NUM_WORDS is set to the current max length of any given post (tweet) on Twitter. This max number of words is likely to exceed all of our sms text size (typically 160 characters).<jupyter_code>tokenizer = Tokenizer(num_words=MAX_NUM_WORDS)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
word_index = tokenizer.word_index
print('Found %s unique tokens.' % len(word_index))
assert len(sequences) == len(texts) == len(labels_as_int)<jupyter_output><empty_output><jupyter_text>Step 4: Create X, y training setsIn machine learning, it's common to denote the training inputs as X and their corresponding labels (the outputs) as y.Let's start with the X data (aka the text) by padding all of our tokenized sequences. This ensures all training inputs are the same shape (aka size).Each sentence in each paragraph in every conversation you have is rarely the same length. It is almost certainly sometimes the same length, but rarely all the time. With that in mind, we want to categorize every sentence (or paragraph) as either spam or ham -- an arbitrary length of data into known length of data.This means we have two challenges:- Matrix multiplication has strict rules- Spoken or written language rarely adheres to strict rules.What to do?X as new representation for the text from our raw dataset. As stated above, there's a very small chance that all data in this group is the exact same length so we'll use the built-in tool called pad_sequences to correct for the inconsistent length. This length is actually called shape because of it's roots in linear algebra (matrix multiplication).<jupyter_code>X = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)
y = to_categorical(np.asarray(labels_as_int))<jupyter_output><empty_output><jupyter_text>Step 5: Split our Training DataIf we trained on all of our data, our model will fit very well to that training data but it will not perform well on new data; aka it will be mostly useless.Since we have the X and y designations, we split the data into at least 2 corresponding sets: training data and validation data for each designation resulting in X_train, X_test, y_train, y_test.An easy way (but not the only way) is to use scikit-learn for this:<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)<jupyter_output><empty_output><jupyter_text>Step 6: Export our Training DataFor this step, we'll use Python's built-in pickle module. Pickle is not secure so only open pickles that you create yourself. I am doing it as a way to pass data between jupyter notebooks.<jupyter_code>training_data = {
    'X_train': X_train,
    'X_test': X_test,
    'y_train': y_train,
    'y_test': y_test,
    'max_words': MAX_NUM_WORDS,
    'max_sequence': MAX_SEQUENCE_LENGTH,
    'legend': labels_legend,
    'labels_legend_inverted': labels_legend_inverted,
    "tokenizer": tokenizer,
}
training_data
with open(TRAINING_DATA_PATH, 'wb') as f:
    pickle.dump(training_data, f)
data = {}

with open(TRAINING_DATA_PATH, 'rb') as f:
    data = pickle.load(f)
print('Done creating tokenized train & test data...')<jupyter_output>Done creating tokenized train & test data...
{'fraction_non_alphanumeric': 0.08926944798688285, 'fraction_numerical': 0.005829841501184187, 'mean_word_length': 5.736196319018405, 'pattern_counts': {'":': 4, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '78149', 'license': 'MIT', 'n_tokens_mistral': 1806, 'n_tokens_neox': 1742, 'n_words': 597, 'path': 'guides/02-Convert_Dataset_Into_Vectors.ipynb', 'repo_name': 'KewJS/spam_classification', 'size': '5489'}