<jupyter_start><jupyter_text>**Anomaly Infused Classification**This notebook tests an idea of mine: Can you use anomaly detection to improve a classification. If this works (and it seems like it does, currently at ~2 sigma), you can improve any classification network (at least like here with some symmetry in the data (aka convolutions, gnns), but I guess this should also work without) without any other input.To do this, I use an anomaly detection algorithm I invented (and call oneoff networks), to define anomality for each point on the image. Afterwards I just add this anomality to the input of a convolutional network and compare the result to the same network trained the same way, but without a second input. The first part is the comparison network, so just a copy of https://keras.io/examples/vision/mnist_convnet/I dont change much, as I guess you can assume that the usual keras tutorial is well optimized. This suggests that the final result (anomaly infused network) is not as good as it could be.<jupyter_code>import numpy as np
from tensorflow import keras
from tensorflow.keras import layers
# Model / data parameters
num_classes = 10
input_shape = (28, 28, 1)

# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)
print("x_train shape:", x_train.shape)
print(x_train.shape[0], "train samples")
print(x_test.shape[0], "test samples")


# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
model = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Flatten(),
        layers.Dropout(0.5),
        layers.Dense(num_classes, activation="softmax"),
    ]
)

model.summary()
batch_size = 128
epochs = 15

model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])

hn=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1)
score = model.evaluate(x_test, y_test, verbose=0)
print("Test loss:", score[0])
print("Test accuracy:", score[1])<jupyter_output>Test loss: 0.026425588876008987
Test accuracy: 0.9900000095367432<jupyter_text>**now for the oneoff network**here I train an image to image network resulting in 1 while removing biases<jupyter_code>modeloo = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Conv2D(32, kernel_size=(7, 7), activation="relu",padding="same",use_bias=False),
        #layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Conv2D(64, kernel_size=(4, 4), activation="relu",padding="same",use_bias=False),
        layers.Conv2D(16, kernel_size=(4, 4), activation="relu",padding="same",use_bias=False),
        layers.Conv2D(4, kernel_size=(2, 2), activation="relu",padding="same",use_bias=False),
        #layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Conv2D(1,kernel_size=(1,1),activation="linear",padding="same",use_bias=False),
    ]
)

modeloo.summary()
batch_size = 128
epochs = 5

modeloo.compile(loss="mse", optimizer="adam", metrics=[])

modeloo.fit(x_train, np.ones_like(x_train), batch_size=batch_size, epochs=epochs, validation_split=0.1)
oo=modeloo.predict(x_train)
oot=modeloo.predict(x_test)
print(oo.shape)
of=oo.flatten()
print(of.shape)
import matplotlib.pyplot as plt
plt.hist(of,bins=50)
plt.show()
delta=np.abs(of-1)
plt.hist(delta,bins=50)
plt.yscale("log",nonposy="clip")
plt.show()
#some statistics processing
print(np.mean(delta))
print(np.std(delta))
print(np.max(delta))
stat_mean=np.mean(delta)
stat_sigma=np.std(delta)
def corr(q):
  return np.exp(0.5*(np.abs(q-1)-stat_mean)/(stat_sigma))
print(corr(np.max(delta+1)))
print(corr(1))
oc=corr(oo)
oct=corr(oot)
oct-=np.mean(oc)
oct/=np.std(oct)
oc-=np.mean(oc)
oc/=np.std(oc)
print(oc.shape)
print(x_train.shape)
print(np.mean(oct))
print(np.std(oct))<jupyter_output>-0.0012162012
1.0000002<jupyter_text>**Now lets use this for anomaly infused training**<jupyter_code>#at first concat this as new inputs
cc=np.concatenate((x_train,oc),axis=-1)
print(cc.shape)
cct=np.concatenate((x_test,oct),axis=-1)
print(cct.shape)
modelc = keras.Sequential(
    [
        keras.Input(shape=cc.shape[1:]),
        layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Flatten(),
        layers.Dropout(0.5),
        layers.Dense(num_classes, activation="softmax"),
    ]
)

modelc.summary()
batch_size = 128
epochs = 15

modelc.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])

hc=modelc.fit(cc, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1)
scorec = modelc.evaluate(cct, y_test, verbose=0)
print("Test loss:", scorec[0])
print("Test accuracy:", scorec[1])
acn=hn.history["accuracy"]
acc=hc.history["accuracy"]
print(acn,acc)<jupyter_output>[0.8845555782318115, 0.9658703804016113, 0.9730555415153503, 0.9769814610481262, 0.9795185327529907, 0.9813703894615173, 0.983296275138855, 0.9838888645172119, 0.9854629635810852, 0.986037015914917, 0.98674076795578, 0.9873889088630676, 0.9878703951835632, 0.9880555272102356, 0.9887592792510986] [0.88355553150177, 0.9658148288726807, 0.9740740656852722, 0.9771666526794434, 0.9800370335578918, 0.9824629426002502, 0.9842963218688965, 0.9850000143051147, 0.9860000014305115, 0.9864259362220764, 0.9863148331642151, 0.9881666898727417, 0.9876111149787903, 0.9891852140426636, 0.9896296262741089]<jupyter_text>sadly the images are not very decisive<jupyter_code>plt.plot(acn,label="normal training")
plt.plot(acc,label="anomaly infused")
plt.legend()
plt.show()
plt.plot([a-b for a,b in zip(acn,acc)])
plt.show()
lon=hn.history["loss"]
loc=hc.history["loss"]
print(lon,loc)
plt.plot(lon,label="normal training",alpha=0.5)
plt.plot(loc,label="anomaly infused",alpha=0.5)
plt.legend()
plt.yscale("log")
plt.show()<jupyter_output><empty_output><jupyter_text>Quick compare:<jupyter_code>print("without",score)
print("with   ",scorec)
print("both should be positive")
print("loss",score[0]-scorec[0])
print("acc",scorec[1]-score[1])<jupyter_output>both should be positive
loss 0.004345474764704704
acc 0.0015000104904174805<jupyter_text>Rerunning results in both should be positiveloss -0.0016080308705568314acc 0.00010001659393310547 both should be positiveloss 0.00017721019685268402acc -0.000299990177154541 both should be positiveloss 0.005704481154680252acc 0.00279998779296875 both should be positiveloss 0.0008180048316717148acc 0.000599980354309082 both should be positiveloss 0.0008532833307981491acc -0.00010001659393310547 both should be positiveloss 0.0019453037530183792acc 0.0006000399589538574 both should be positiveloss 0.005755022168159485acc 0.002200007438659668 both should be positiveloss 3.5393983125686646e-05acc 0.00019997358322143555 both should be positiveloss -0.0015629995614290237acc -0.000800013542175293 14/20 >0, sigma~2, so about 2 sigma significance Having done this quick and dirty, lets do this more proffesional:<jupyter_code>ll=[[0.02173512801527977,0.0233431588858366],
    [0.023866165429353714,0.02368895523250103],
    [0.0278194360435009,0.022114954888820648],
    [0.023399805650115013,0.0225818008184433],
    [0.025487083941698074,0.024633800610899925],
    [0.02620108425617218,0.0242557805031538],
    [0.027982046827673912,0.022227024659514427],
    [0.026208026334643364,0.026172632351517677],
    [0.023979149758815765,0.02554214932024479],
    [0.026425588876008987,0.022080114111304283]]<jupyter_output><empty_output><jupyter_text>I look only at losses, because there less random. First collumn: without, second with<jupyter_code>fr=len([1 for zw in ll if zw[0]>zw[1]])/len(ll)
print("fraction of improvements",fr)
ln=[zw[0] for zw in ll]
lc=[zw[1] for zw in ll]

import numpy as np
mn=np.mean(ln)
mc=np.mean(lc)
sn=np.std(ln)/np.sqrt(len(ln))
sc=np.std(lc)/np.sqrt(len(lc))

diff=mn-mc
sigma=np.sqrt(sn**2+sc**2)
print("Difference",diff)
print("With error",sigma)
print("This means a significance of")
print(diff/sigma)<jupyter_output>Difference 0.0016463143751025214
With error 0.0007480448084267025
This means a significance of
2.200823208124485<jupyter_text>So still about a 2 sigma improvement<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13211819313936374, 'fraction_numerical': 0.1736669308275784, 'mean_word_length': 6.264802631578948, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '496370', 'license': 'MIT', 'n_tokens_mistral': 4240, 'n_tokens_neox': 3305, 'n_words': 669, 'path': 'main.ipynb', 'repo_name': 'psorus/aic', 'size': '8833'}