<jupyter_start><jupyter_text>Multi-label binarizationLet's preprocess our labels using the [StringLookup](https://keras.io/api/layers/preprocessing_layers/categorical/string_lookup) layer.<jupyter_code>features = tf.ragged.constant(train_df['targets'].values)
lookup = tf.keras.layers.StringLookup(output_mode='multi_hot')
lookup.adapt(features)
vocab = lookup.get_vocabulary()
def invert_multi_hot(encoded_labels):
    '''Reverse a single multi-hot encoded label to a tuple of vocab terms.'''
    hot_indices = np.argwhere(encoded_labels == 1.0)[...,0]
    return np.take(vocab, hot_indices)

print('Vocabulary:\n')
print(vocab)<jupyter_output>Vocabulary:

['[UNK]', 'female', 'nausea', '35 year', 'male', 'post prandial bloating or fullness with meals', 'nsaid use or nonsteroidal anti inflammatory drug use', 'no blood in stool', 'minimal to no change with tums', 'intermittent', 'getting worse or progressive or symptoms now daily', 'fhx of pud or family history of peptic ulcer disease', 'epigastric discomfort', 'duration 2 months', 'darker bowel movements', 'burning or gnawing or burning and gnawing', 'awakens at night', '2 to 3 beers a week', 'recent visit to emergency department with negative workup', 'onset 5 years ago', 'no illicit drug use', 'no chest pain', 'no caffeine use', 'increased stress', 'increased frequency recently', 'feels hot or feels clammy', 'fatigue or difficulty concentrating', 'episodes of heart racing', 'episodes last 15 to 30 minutes', 'episode of hand numbness or episode of finger numbness', 'associated throat tightness', 'associated sob or associated shortness of breath', 'associated nausea', 'associ[...]<jupyter_text>Separate the individual targets from the label pool and then use it to represent a given label set with 0's and 1's<jupyter_code>sample_label = train_df['targets'].iloc[0]
print(f'Original label: {sample_label}')

label_binarized = lookup([sample_label])
print(f'Label-binarized representation: {label_binarized}')<jupyter_output>Original label: ['no relief with motrin or no relief with tylenol', '20 year', '1 day duration or 2 days duration', 'myalgias', 'global headache or diffuse headache', 'neck pain', 'vomiting', 'no rash', 'nausea', 'viral symptoms or rhinorrhea or scratchy throat', 'shares an apartment', 'meningococcal vaccine status unknown', 'family history of migraines', 'female', 'photophobia', 'no known illness contacts', 'subjective fever']
Label-binarized representation: [[0. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 1.
  1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]<jupyter_text>Data preprocessing and [tf.data.Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) objects<jupyter_code>train_df['clean'].apply(lambda x: len(x.split(" "))).describe()
max_seqlen = 98
batch_size = 128
padding_token = '<pad>'
auto = tf.data.AUTOTUNE

def make_dataset(dataframe, is_train=True):
    labels = tf.ragged.constant(dataframe['targets'].values)
    label_binarized = lookup(labels).numpy()
    dataset = tf.data.Dataset.from_tensor_slices(
        (dataframe['clean'].values, label_binarized)
    )
    dataset = dataset.shuffle(batch_size * 10) if is_train else dataset
    return dataset.batch(batch_size)<jupyter_output><empty_output><jupyter_text>Prepare the [tf.data.Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) objects.<jupyter_code>train_dataset = make_dataset(train_df, is_train=True)
validation_dataset = make_dataset(val_df, is_train=False)
test_dataset = make_dataset(test_df, is_train=False)<jupyter_output><empty_output><jupyter_text>Preview the dataset<jupyter_code>text_batch, label_batch = next(iter(train_dataset))

for i, text in enumerate(text_batch[:5]):
    label = label_batch[i].numpy()[None, ...]
    print(f'Student note: {text}')
    print(f'Targets: {invert_multi_hot(label[0])}')
    print(' ')<jupyter_output>Student note: b'45 yo f complaint feelings nervousness constant worsening past weeks 3 weeks ago started giving lectures thinking giving lectures makes anxiety worse negatively impacting life causing decreased appetite mostly sunday nights monday morning prior lectures caused decreased libido ros heart racing pounding palpitations lightheadedness dizziness chest pain sob abd pain diarrhea constipations fever chills change hair cold heat intolerance agoraphobia pmh none lmp 1 week ago occurs regularly every 4 weeks heavy fhx father died heart attack age 65 shx lives 2 kids husband mother feels safe home denies tobacco illicit drug use drinks 1 2 alcoholic beverages month'
Targets: ['female' 'weight stable' 'stress due to caring for elderly parents'
 'no depressed mood' 'lack of other thyroid symptoms' 'insomnia'
 'heavy caffeine use' 'decreased appetite' 'anxious or nervous' '45 year']
 
Student note: b'hpi 67 yo f presents c inability sleep x 3wks pt reports difficulty falling asleep s[...]<jupyter_text>VectorizationVectorize the text to represent it as a quantitative value. We will use [TextVectorization layer](https://keras.io/api/layers/preprocessing_layers/text/text_vectorization)<jupyter_code># Get unique words in student notes.
vocabulary = set()
train_df['clean'].str.split().apply(vocabulary.update)
vocabulary_size = len(vocabulary)
print(vocabulary_size)<jupyter_output>42110<jupyter_text>Now we create our vectorization layer and map() to the [tf.data.Datasets](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) created earlier.<jupyter_code>text_vectorizer = layers.TextVectorization(
    max_tokens=vocabulary_size, ngrams=2, output_mode='tf_idf'
)

with tf.device('/CPU:0'):
    text_vectorizer.adapt(train_dataset.map(lambda text, label: text))

train_dataset = train_dataset.map(
    lambda text, label: (text_vectorizer(text), label), num_parallel_calls=auto).prefetch(auto)
validation_dataset = validation_dataset.map(
    lambda text, label: (text_vectorizer(text), label), num_parallel_calls=auto).prefetch(auto)
test_dataset = test_dataset.map(
    lambda text, label: (text_vectorizer(text), label), num_parallel_calls=auto).prefetch(auto)
text_batch, label_batch = next(iter(train_dataset))
len(text_batch[0]), len(label_batch[0])<jupyter_output><empty_output><jupyter_text>Create a text classification model<jupyter_code>def make_model():
    shallow_mlp_model = keras.Sequential(
        [
            layers.Dense(512, activation="relu"),
            layers.Dense(256, activation="relu"),
            layers.Dense(lookup.vocabulary_size(), activation="softmax"),
        ]
    )
    return shallow_mlp_model<jupyter_output><empty_output><jupyter_text>Train our model<jupyter_code>epochs = 2

shallow_mlp_model = make_model()
shallow_mlp_model.compile(
    loss="binary_crossentropy", optimizer="adam", metrics=["categorical_accuracy"]
)

history = shallow_mlp_model.fit(
    train_dataset, validation_data=validation_dataset, epochs=epochs
)


def plot_result(item):
    plt.plot(history.history[item], label=item)
    plt.plot(history.history["val_" + item], label="val_" + item)
    plt.xlabel("Epochs")
    plt.ylabel(item)
    plt.title("Train and Validation {} Over Epochs".format(item), fontsize=14)
    plt.legend()
    plt.grid()
    plt.show()


plot_result("loss")
plot_result("categorical_accuracy")<jupyter_output>Epoch 1/2
297/297 [==============================] - 29s 96ms/step - loss: 0.0202 - categorical_accuracy: 0.2570 - val_loss: 1.1700e-04 - val_categorical_accuracy: 0.1546
Epoch 2/2
297/297 [==============================] - 29s 98ms/step - loss: 2.3018e-05 - categorical_accuracy: 0.2510 - val_loss: 1.0444e-04 - val_categorical_accuracy: 0.2116<jupyter_text>Evaluate the model<jupyter_code>train_dataset
validation_dataset
test_dataset
_, categorical_acc = shallow_mlp_model.evaluate(test_dataset)
print(f"Categorical accuracy on the test set: {round(categorical_acc * 100, 2)}%.")
# Create a model for inference.
model_for_inference = keras.Sequential([text_vectorizer, shallow_mlp_model])

# Create a small dataset just for demoing inference.
inference_dataset = make_dataset(test_df.sample(100), is_train=False)
text_batch, label_batch = next(iter(inference_dataset))
predicted_probabilities = model_for_inference.predict(text_batch)

# Perform inference.
for i, text in enumerate(text_batch[:10]):
    label = label_batch[i].numpy()[None, ...]
    print(f"Student notes: {text}")
    print(f"Targets(s): {invert_multi_hot(label[0])}")
    predicted_proba = [proba for proba in predicted_probabilities[i]]
    all_labels = [
        x
        for _, x in sorted(
            zip(predicted_probabilities[i], lookup.get_vocabulary()),
            key=lambda pair: pair[0],
            reverse=True,
            
        )
    ][:len(invert_multi_hot(label[0]))]
    print(f"Predicted Targets(s): ({', '.join([label for label in all_labels])})")
    print(" ")<jupyter_output>Student notes: b'ms moore 45 female presenting generalized feelings nervousness last 3 4 weeks lots stress past couple months regarding work family 2 weeks ago workload increased feels anxious night giving lecture also reports decrease appetite however change weight also reports difficulty falling asleep last couple weeks thinking things falls asleep stays asleep gets 7 hours drinks 5 6 cups coffee day done years nothing makes nervousness better worse denies palpitations chest pain constipation fatigue meds otc tylenol needed allergies nkda pmhx none family hx dad sudden heart attack 61 social history occupation english professor etoh 1 2 drinks month tobacco never drugs never'
Targets(s): ['female' 'weight stable' 'stress due to caring for elderly parents'
 'no depressed mood' 'lack of other thyroid symptoms' 'insomnia'
 'heavy caffeine use' 'decreased appetite' 'anxious or nervous' '45 year']
Predicted Targets(s): (insomnia, stress due to caring for elderly parents, heavy caffeine us[...]
{'fraction_non_alphanumeric': 0.12165762210162802, 'fraction_numerical': 0.028613714849531326, 'mean_word_length': 4.987005316007088, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '45705', 'license': 'MIT', 'n_tokens_mistral': 3569, 'n_tokens_neox': 3206, 'n_words': 991, 'path': 'notebooks/brent/multi-label-classification-model.ipynb', 'repo_name': 'codeup-nlp-capstone/nlp-capstone', 'size': '10135'}