<jupyter_start><jupyter_text>Train a  Bernoulli-Bernoulli RBM (BBRBM) to fit XOR calculations<jupyter_code>import numpy as np
import timeit
from tfrbm.bbrbm import BBRBM
bm = BBRBM(n_visible=3,n_hidden=6)<jupyter_output><empty_output><jupyter_text>dataset[i][0] xor dataset[i][1] = dataset[i][2]<jupyter_code>dataset = np.array([[0,0,0],[0,1,1],[1,0,1],[1,1,0]])<jupyter_output><empty_output><jupyter_text>Train and predict same data in same iterationsx is the correct data to train the model  x2 has wrong xor answer ( the third digit )  x2 would be reconstructed to x after training<jupyter_code>for i in range(4):
    x = dataset[i:i+1].copy()
    x2 = dataset[i:i+1].copy()
    x2[0,2] = 1-x2[0,2]
    print "x:",x
    print "x2:",x2
    print "Training model with x"
    err = bm.fit(x,n_epoches=1000)
    print "model reconstructed x:",np.round(bm.reconstruct(x))
    print "model reconstructed x2:",np.round(bm.reconstruct(x2))
    print "Training done"

    positions_to_predict = [2]
    prediction = bm.predict(x,positions_to_predict=positions_to_predict) # best result, energy for all possibilities
    print "model predicting x third digit to be:",prediction[0].tolist()
    prediction = bm.predict(x2,positions_to_predict=positions_to_predict)
    print "model predicting x2 third digit to be:",prediction[0].tolist()
    print "------------------------------------"<jupyter_output>x: [[0 0 0]]
x2: [[0 0 1]]
Training model with x
model reconstructed x: [[0. 0. 0.]]
model reconstructed x2: [[0. 0. 0.]]
Training done
model predicting x third digit to be: [0]
model predicting x2 third digit to be: [0]
------------------------------------
x: [[0 1 1]]
x2: [[0 1 0]]
Training model with x
model reconstructed x: [[0. 1. 1.]]
model reconstructed x2: [[0. 1. 1.]]
Training done
model predicting x third digit to be: [1]
model predicting x2 third digit to be: [1]
------------------------------------
x: [[1 0 1]]
x2: [[1 0 0]]
Training model with x
model reconstructed x: [[1. 0. 1.]]
model reconstructed x2: [[1. 0. 1.]]
Training done
model predicting x third digit to be: [1]
model predicting x2 third digit to be: [1]
------------------------------------
x: [[1 1 0]]
x2: [[1 1 1]]
Training model with x
model reconstructed x: [[1. 1. 0.]]
model reconstructed x2: [[1. 1. 0.]]
Training done
model predicting x third digit to be: [0]
model predicting x2 third digit to be: [0]
-----[...]<jupyter_text>All x variations would be reconstructured to the last training data [1, 1, 0]<jupyter_code>for i in range(4):
    x = dataset[i:i+1].copy()
    x2 = dataset[i:i+1].copy()
    x2[0,2] = 1-x2[0,2]
    print "x:",x
    print "x2:",x2
    
    positions_to_predict = [2]
    print "model reconstructed x:",np.round(bm.reconstruct(x))
    print "model reconstructed x2:",np.round(bm.reconstruct(x2))
    print "------------------------------------"<jupyter_output>x: [[0 0 0]]
x2: [[0 0 1]]
model reconstructed x: [[1. 1. 0.]]
model reconstructed x2: [[1. 1. 0.]]
------------------------------------
x: [[0 1 1]]
x2: [[0 1 0]]
model reconstructed x: [[1. 1. 0.]]
model reconstructed x2: [[1. 1. 0.]]
------------------------------------
x: [[1 0 1]]
x2: [[1 0 0]]
model reconstructed x: [[1. 1. 0.]]
model reconstructed x2: [[1. 1. 0.]]
------------------------------------
x: [[1 1 0]]
x2: [[1 1 1]]
model reconstructed x: [[1. 1. 0.]]
model reconstructed x2: [[1. 1. 0.]]
------------------------------------<jupyter_text>Majority vote data would be the data reconstruction targetnew fit() on top of the already trained model will fit the model to the new training data<jupyter_code>training_data = np.array([[0,0,0],[0,0,1]])
training_data = np.repeat(training_data, [9,1],axis=0)
print "training_data:",training_data
start = timeit.default_timer()
err = bm.fit(training_data,n_epoches=1000)
stop = timeit.default_timer()
print 'Time: ', stop - start, "sec"
print "model reconstructed [0,0,0]:",np.round(bm.reconstruct(training_data[0:1]))
print "model reconstructed [0,0,1]:",np.round(bm.reconstruct(training_data[-2:-1]))
positions_to_predict = [2]
prediction = bm.predict(training_data[0:1],positions_to_predict=positions_to_predict)
print "model predicting [0,0,0] third digit to be:",prediction[0].tolist()
prediction = bm.predict(training_data[-2:-1],positions_to_predict=positions_to_predict)
print "model predicting [0,0,1] third digit to be:",prediction[0].tolist()<jupyter_output>training_data: [[0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 1]]
Time:  2.04646015167 sec
model reconstructed [0,0,0]: [[0. 0. 0.]]
model reconstructed [0,0,1]: [[0. 0. 0.]]
model predicting [0,0,0] third digit to be: [1]
model predicting [0,0,1] third digit to be: [1]<jupyter_text>model would be overwritten with new training dataModel may not reconstruct to the next new majority [0,0,1] because of the previous trainings.  But still can predict 0 xor 0 to be 1 from free energy comparison  May need more iterations (3000 here) than before to reverse the previous memory, and also learn the new dataNote here that the training process is stochastic, so different runs may lead to different results  See tfrbm/bbrbm.py `line 12` `... sample_bernoulli ...`<jupyter_code>training_data = np.array([[0,0,0],[0,0,1]])
training_data = np.repeat(training_data, [2,8],axis=0)
print "training_data:",training_data
start = timeit.default_timer()
err = bm.fit(training_data,n_epoches=3000)
stop = timeit.default_timer()
print 'Time: ', stop - start, "sec"
print "model reconstructed [0,0,0]:",np.round(bm.reconstruct(training_data[0:1]))
print "model reconstructed [0,0,1]:",np.round(bm.reconstruct(training_data[-2:-1]))
positions_to_predict = [2]
prediction = bm.predict(training_data[0:1],positions_to_predict=positions_to_predict)
print "model predicting [0,0,0] third digit to be:",prediction[0].tolist()
prediction = bm.predict(training_data[-2:-1],positions_to_predict=positions_to_predict)
print "model predicting [0,0,1] third digit to be:",prediction[0].tolist()<jupyter_output>training_data: [[0 0 0]
 [0 0 0]
 [0 0 1]
 [0 0 1]
 [0 0 1]
 [0 0 1]
 [0 0 1]
 [0 0 1]
 [0 0 1]
 [0 0 1]]
Time:  7.17365694046 sec
model reconstructed [0,0,0]: [[0. 0. 0.]]
model reconstructed [0,0,1]: [[0. 0. 1.]]
model predicting [0,0,0] third digit to be: [1]
model predicting [0,0,1] third digit to be: [1]<jupyter_text>Try all xor configurations<jupyter_code>bm = BBRBM(n_visible=3,n_hidden=6)
training_data = np.array([[0,0,0],[1,0,1],[0,1,1],[1,1,0]])

# print "training_data:",training_data
start = timeit.default_timer()

training_data_ = np.repeat(training_data, [1,1,1,1],axis=0)
err = bm.fit(training_data_,n_epoches=20000)

stop = timeit.default_timer()
print 'Time: ', stop - start, "sec"
print "model reconstructed [0,0,0]:",np.round(bm.reconstruct(training_data[0:1]))
print "model reconstructed [0,0,1]:",np.round(bm.reconstruct([[0,0,1]]))
print "model reconstructed [1,0,1]:",np.round(bm.reconstruct(training_data[1:2]))
print "model reconstructed [1,0,0]:",np.round(bm.reconstruct([[1,0,0]]))
print "model reconstructed [0,1,1]:",np.round(bm.reconstruct(training_data[2:3]))
print "model reconstructed [0,1,0]:",np.round(bm.reconstruct([[0,1,0]]))
print "model reconstructed [1,1,1]:",np.round(bm.reconstruct(training_data[3:4]))
print "model reconstructed [0,1,0]:",np.round(bm.reconstruct([[1,1,1]]))
                                              
positions_to_predict = [2]
prediction = bm.predict(training_data[0:1],positions_to_predict=positions_to_predict)
print "model predicting [0,0,0] third digit to be:",prediction[0].tolist()
prediction = bm.predict(training_data[1:2],positions_to_predict=positions_to_predict)
print "model predicting [1,0,1] third digit to be:",prediction[0].tolist()
prediction = bm.predict(training_data[2:3],positions_to_predict=positions_to_predict)
print "model predicting ",training_data[2:3]," third digit to be:",prediction[0].tolist()
prediction = bm.predict(training_data[-1:],positions_to_predict=positions_to_predict)
print "model predicting ",training_data[-1:]," third digit to be:",prediction[0].tolist()
bm.get_weights()<jupyter_output>Time:  42.9245870113 sec
model reconstructed [0,0,0]: [[0. 0. 0.]]
model reconstructed [0,0,1]: [[0. 0. 1.]]
model reconstructed [1,0,1]: [[1. 0. 1.]]
model reconstructed [1,0,0]: [[1. 0. 0.]]
model reconstructed [0,1,1]: [[0. 1. 1.]]
model reconstructed [0,1,0]: [[0. 1. 0.]]
model reconstructed [1,1,1]: [[1. 1. 0.]]
model reconstructed [0,1,0]: [[1. 1. 1.]]
model predicting [0,0,0] third digit to be: [0]
model predicting [1,0,1] third digit to be: [1]
model predicting  [[0 1 1]]  third digit to be: [1]
model predicting  [[1 1 0]]  third digit to be: [0]
{'fraction_non_alphanumeric': 0.19537186276767213, 'fraction_numerical': 0.06435643564356436, 'mean_word_length': 4.217417417417417, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '575493', 'license': 'MIT', 'n_tokens_mistral': 3442, 'n_tokens_neox': 3101, 'n_words': 901, 'path': 'rbm_calc_xor.ipynb', 'repo_name': 'shitian-ni/tensorflow-rbm', 'size': '8686'}