<jupyter_start><jupyter_text>Overlap reproductionThis notebook should reproduce some results of the Amit's book (Attractor Neural Networks) in the section 4.1<jupyter_code>from __future__ import print_function
import sys
sys.path.append('../')

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from hopfield import Hopfield

%matplotlib inline
sns.set(font_scale=2.0)<jupyter_output><empty_output><jupyter_text>Symmetric mix of attractorsAccording to Amit with the Hebbian rule of this model a very particular type of **spurios states** appears: symmetric mixtures. Symmetric mixtures are called like that because they overlaps with the intended attractors become more or less same. In the asynchronous case **only symmetric mixtures of odd number of states are stable**. We show a symmetric mixture here First we build the network with very low temperature (noise)<jupyter_code>n_dim = 400
n_store = 7
T = 0.0
prng = np.random.RandomState(seed=10000)
N = 2000

nn = Hopfield(n_dim=n_dim, T=T, prng=prng)
list_of_patterns = nn.generate_random_patterns(n_store)
nn.train(list_of_patterns)<jupyter_output><empty_output><jupyter_text>Then we run the network<jupyter_code>overlaps = np.zeros((N, n_store))
for i in range(N):
    nn.update_async()
    overlaps[i, :] = nn.calculate_overlap()<jupyter_output><empty_output><jupyter_text>Ploting<jupyter_code># Plot this thing
fig = plt.figure(figsize=(16, 12))
ax = fig.add_subplot(111)

ax.set_xlabel('Iterations')
ax.set_ylabel('Overlap')
ax.axhline(y=0, color='k')

for pattern_n, overlap in enumerate(overlaps.T):
    ax.plot(overlap, '-', label='m' +str(pattern_n))

ax.legend()
ax.set_ylim(-1.1, 1.1)

plt.show()<jupyter_output><empty_output><jupyter_text>Here we see a symmetric mixture of three states  Effect of temperature in symmetric mixturesAs Amit discusses we can increase the temperature to a **sweet spot** where the spurious states are destroyed and only the real attractors are preserved. We can try to repeat the same process with higher temperature<jupyter_code>n_dim = 400
n_store = 7
T = 0.8
prng = np.random.RandomState(seed=10000)
N = 2000

nn = Hopfield(n_dim=n_dim, T=T, prng=prng)
list_of_patterns = nn.generate_random_patterns(n_store)
nn.train(list_of_patterns)<jupyter_output><empty_output><jupyter_text>Then we run the network<jupyter_code>overlaps = np.zeros((N, n_store))
for i in range(N):
    nn.update_async()
    overlaps[i, :] = nn.calculate_overlap()<jupyter_output><empty_output><jupyter_text>Ploting<jupyter_code># Plot this thing
fig = plt.figure(figsize=(16, 12))
ax = fig.add_subplot(111)

ax.set_xlabel('Iterations')
ax.set_ylabel('Overlap')
ax.axhline(y=0, color='k')

for pattern_n, overlap in enumerate(overlaps.T):
    ax.plot(overlap, '-', label='m' +str(pattern_n))

ax.legend()
ax.set_ylim(-1.1, 1.1)

plt.show()<jupyter_output><empty_output><jupyter_text>We can appreciate here that with higher noises all the other overlaps become close to 0 and one state the symmetric reflexion of state m0 is being recalled  Effect of very high noiseFinally if the noise is very high only the state where all the overlaps vanish will be stable.<jupyter_code>n_dim = 400
n_store = 7
T = 3.0

nn = Hopfield(n_dim=n_dim, T=T, prng=prng)
list_of_patterns = nn.generate_random_patterns(n_store)
nn.train(list_of_patterns)<jupyter_output><empty_output><jupyter_text>Then we run the network<jupyter_code>overlaps = np.zeros((N, n_store))
for i in range(N):
    nn.update_async()
    overlaps[i, :] = nn.calculate_overlap()<jupyter_output><empty_output><jupyter_text>Ploting<jupyter_code># Plot this thing
fig = plt.figure(figsize=(16, 12))
ax = fig.add_subplot(111)

ax.set_xlabel('Iterations')
ax.set_ylabel('Overlap')
ax.axhline(y=0, color='k')

for pattern_n, overlap in enumerate(overlaps.T):
    ax.plot(overlap, '-', label='m' +str(pattern_n))

ax.legend()
ax.set_ylim(-1.1, 1.1)

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11326037159582591, 'fraction_numerical': 0.019852379740391956, 'mean_word_length': 6.318435754189944, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '617988', 'license': 'MIT', 'n_tokens_mistral': 1496, 'n_tokens_neox': 1414, 'n_words': 366, 'path': 'notebooks/2016-11-29(Overlap reproduction).ipynb', 'repo_name': 'h-mayorquin/hopfield_sequences', 'size': '3929'}