<jupyter_start><jupyter_text>Parts Distributor SKU classifier, part 2: Explore the modelIn [part 1](parts-distributor-sku-classifier-part-1.ipynb) we built a model that can classify strings like `SN74LVC541APWR` or `296-8521-1-ND` as either manufacturer part numbers, Mouser SKUs or Digi-Key SKUs. In part 2, we will load that model and run it on various inputs to get a feel for how it works. Load the modelFirst let's load the model and the training data back into memory.<jupyter_code>import pandas as pd
import numpy as np
import json
from IPython.display import Markdown, display
from keras.models import model_from_json
import graphviz
from IPython.display import SVG
# Load the model architecture
with open("data/trained_model_layers.json", 'r') as json_file:
    model = model_from_json(json_file.read())

# Load the model weights
model.load_weights("data/trained_model_weights.h5")

# Compile the model
batch_size = 32
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

# Load the data set
df = pd.read_json("data/cleaned_training_data.json")

# Load the dictionary
with open("data/char_dictionary.json", 'r') as json_file:
    partnum_dict = json.load(json_file)
    
class_names = ['MPN', 'Mouser SKU', 'Digi-Key SKU']

def d(col, ds, class_filter=None, df=df):
    if class_filter is not None:
        return list(df[(df['dataset'] == ds) & (df['class'] == class_filter)][col])
    else:
        return list(df[df['dataset'] == ds][col])
    
df.head()
# Test the model to make sure it's loaded correctly
score, acc = model.evaluate(d('x', 'val'), d('y', 'val'), batch_size=batch_size)
display(Markdown('### Accuracy of the model: {:.2f}%'.format(acc * 100.0)))
res = []
for c in sorted(df['class'].unique()):
    score, acc = model.evaluate(d('x', 'val', class_filter=c), d('y', 'val', class_filter=c), batch_size=batch_size)
    res.append([class_names[c], '{:.2f}%'.format(acc*100.0)])
pd.DataFrame(res, columns=['class', 'accuracy'])<jupyter_output>2720/2789 [============================>.] - ETA: 0s<jupyter_text>The model is miscategorizing some of the MPNs. Let's visualize the confusion.<jupyter_code>def cross_category_bleeding(model, x_test, y_test, batch_size):
    preds = model.predict(x_test)
    cat_count = np.shape(y_test)[1]
    r = np.zeros((cat_count, cat_count))
    for x, y, pred in zip(x_test, y_test, preds):
        class_being_tested = np.argmax(y)
        class_predicted = np.argmax(pred)
        r[class_being_tested][class_predicted] += 1
    # split array into rows
    # normalize each row
    # stack rows back into one array
    return np.stack([v/np.sum(a) if v else 0 for v in a] for a in np.split(r.flatten(), cat_count))

def graphviz_cross_category_diagram(res_matrix, class_labels=[]):
    graph = graphviz.Digraph()
    label = lambda c: class_labels[c] if len(class_labels) > c else 'Category {}'.format(c)
    node_name = lambda c: 'C{}'.format(c)
    for true_c in range(len(res_matrix)):
        graph.node(node_name(true_c), label(true_c))
        preds = res_matrix[true_c]
        for pred_c in range(len(preds)):
            acc = res_matrix[true_c][pred_c] * 100.0
            if acc > 0:
                acc_str = '{:.3f}%'.format(acc)
                graph.edge(node_name(true_c), node_name(pred_c), acc_str)
    return graph

r = cross_category_bleeding(model, d('x', 'val'), d('y', 'val'), batch_size)
g = graphviz_cross_category_diagram(r, class_names)
SVG(g.pipe(format='svg'))<jupyter_output><empty_output><jupyter_text>(Arrows indicate results of categorization, i.e. `Mouser SKU -> MPN` means that some samples that we know for sure are Mouser SKUs got categorized as MPNs by the model). Mouser SKU false-positivesThe main thing this diagram is telling us is that the Mouser SKU classifier has a bit of a "false positives" problem with MPNs. The MPN class is a noisy catch-all without discernible patterns, so it's not entirely realistic to expect this kind of a model to learn something as nebulous - the best we can hope for is that the model learns all well-defined classes really well, and the one noisy "leftovers" class takes care of itself. ([More on this "I don't know this data" problem here.](https://towardsdatascience.com/making-your-neural-network-say-i-dont-know-bayesian-nns-using-pyro-and-pytorch-b1c24e6ab8cd))To better understand what's happening under the hood, let's run some examples through the model one character at a time. Let's also define a bunch of extra columns on the data frame to save keystrokes later.<jupyter_code>preds = model.predict(list(df['x']))
df['predicted_class'] = list(np.argmax(y) for y in preds)
df['predicted_values'] = list(y for y in preds)
df['true_class_mpn'] = (df['class'] == 0)
df['true_class_mouser'] = (df['class'] == 1)
df['true_class_digikey'] = (df['class'] == 2)
df['pred_class_mpn'] = (df['predicted_class'] == 0)
df['pred_class_mouser'] = (df['predicted_class'] == 1)
df['pred_class_digikey'] = (df['predicted_class'] == 2)
df['dataset_train'] = (df['dataset'] == 'train')
df['dataset_val'] = (df['dataset'] == 'val')

df[df.true_class_mpn & df.pred_class_mouser].head(10)[['partnum', 'predicted_values']]<jupyter_output><empty_output><jupyter_text>Let's take a look at `DF14-15P-1.25H(55)` - this is very clearly not a Mouser SKU because those usually begin with a group of digits followed by a dash, never with characters. We are going to run this through the model one character at a time and plot the classifier output along the way.(As a reminder, all Mouser SKUs start with a few digits followed by a dash, like `595-TPS65986ABZQZR` or `576-SP4024-01FTG-C` and DON'T END with "`-ND`" because that's Digi-Key's territory. Like I said, this is a bit of a toy problem).<jupyter_code>def char_by_char_classification_plot(partnum):
    x = df[df['partnum'] == partnum].iloc[0]['x']
    x_steps = [np.array(x[:i]) for i in range(1, len(x))]
    step_preds = []
    for i, x_step in enumerate(x_steps):
        c = partnum[i] if i < len(partnum) else ''
        pred = model.predict(np.array(x_step, ndmin=2))
        step_preds.append([c] + [v for v in pred.flat])
    df_res = pd.DataFrame(step_preds, columns=['char'] + class_names)
    return df_res.plot.line(x='char', grid=True, fontsize=15, xticks=range(len(x)-1), yticks=[0.1*v for v in range(11)], figsize=(15, 5))

char_by_char_classification_plot('DF14-15P-1.25H(55)')<jupyter_output><empty_output><jupyter_text>The classifier starts off well with a healthy dose of pessimism. But then its misguided confidence in this being a Mouser SKU  grows by leaps and bounds, especially around less common characters like '.' and '('<jupyter_code>char_by_char_classification_plot('N2514-6V0C-RB-WD')<jupyter_output><empty_output><jupyter_text>Here we begin an not-entirely-unreasonable bet on the Digi-Key SKU class, and pretty high probability of this NOT being Mouser. It continues this way all the way until several "end of sequence" characters past the end, where perhaps the model realizes that this is not Digi-Key (because it doesn't end "-ND"), and Mouser ends up winning, somehow.<jupyter_code>char_by_char_classification_plot('NPA-700B-015A')<jupyter_output><empty_output><jupyter_text>Something similar to the previous case here.  Digi-Key SKU false-positivesIt's a bit of a nit-pick, but I'm seeing a consistent 0.763% false-positive rate on Digi-Key SKUs coming from the MPNs. I suspect a mistake in the data. Let's check it.<jupyter_code>df[df.true_class_mpn & df.pred_class_digikey][['dataset', 'partnum', 'predicted_values']]<jupyter_output><empty_output><jupyter_text>Looks like both our training and validation data contains some Digi-Key SKUs disguised as MPNs. We will re-label those before we train the next model.  MPN false-positivesSome Mouser SKUs got recognized as MPNs. Let's look into that.<jupyter_code>df[df.true_class_mouser & df.pred_class_mpn][['dataset', 'partnum', 'predicted_values']]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1265632816408204, 'fraction_numerical': 0.015632816408204103, 'mean_word_length': 4.87582659808964, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 36, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '457668', 'license': 'MIT', 'n_tokens_mistral': 2824, 'n_tokens_neox': 2679, 'n_words': 822, 'path': 'parts-distributor-sku-classifier-part-2-explore.ipynb', 'repo_name': 'pcbng/parts-distributor-sku-classifier', 'size': '7996'}