<jupyter_start><jupyter_text>Load and Process the Data<jupyter_code>data = np.loadtxt('data/data.csv', delimiter=',')<jupyter_output><empty_output><jupyter_text>The last 13 rows of the data represent my output<jupyter_code>X = np.transpose(data[:-13,:])
Y = np.transpose(data[-13:,:])<jupyter_output><empty_output><jupyter_text>Every channel has a background of around ~150. We don't want to factor that in. Output should always be positive, so these have been scaled in the range $\left[0,1\right]$.<jupyter_code>inscale = MinMaxScaler()
outscale = MinMaxScaler()
xs = inscale.fit_transform(X)
ys = outscale.fit_transform(Y)<jupyter_output><empty_output><jupyter_text>Split the data into **train**, **test**, and **val**idation segments. The data are in blocks of 20 measurements for each sample. Since the samples should be independent of each other, we can randomly select from each of the 7 samples. I have elected for a 70/15/15 train/test/validate split.<jupyter_code># Choose 6 random numbers from 1-20.
c = np.random.choice(range(20), 6, replace=False)

# Make these the indexes we will use for our test/validation
while c.shape[0] <= (xs.shape[0] - 6) * 0.3:
    c = np.concatenate((c, c[-6:]+20))
xs_test = np.array([xs[i, :] for i in c[::2]])
ys_test = np.array([ys[i, :] for i in c[::2]])
xs_val = np.array([xs[i, :] for i in c[1::2]])
ys_val = np.array([ys[i, :] for i in c[1::2]])
xs_train = np.delete(xs, c, 0)
ys_train = np.delete(ys, c, 0)
print(xs_test.shape)
print(ys_test.shape)
print(xs_val.shape)
print(ys_val.shape)
print(xs_train.shape)
print(ys_train.shape)<jupyter_output>(21, 11725)
(21, 13)
(21, 11725)
(21, 13)
(98, 11725)
(98, 13)<jupyter_text>We now have 98 training records, 21 testing records, and 21 validation records. Each record is a vector of 11725 input values and 13 output values.<jupyter_code>ys_train.shape<jupyter_output><empty_output><jupyter_text>Classifier<jupyter_code>tf.__version__
from tflearn.datasets import titanic
titanic.download_dataset('titanic_dataset.csv')
from tflearn.data_utils import load_csv
data, labels = load_csv('data/tflearn_data.csv', target_column=0, categorical_labels=True, n_classes=7)
len(data[0])<jupyter_output><empty_output><jupyter_text>Build a Deep Neural Network<jupyter_code>net = tflearn.input_data(shape=[None, 11725])
net = tflearn.fully_connected(net, 20000)
net = tflearn.fully_connected(net, 20000)
net = tflearn.fully_connected(net, 20000)
net = tflearn.fully_connected(net, 20000)
net = tflearn.fully_connected(net, 20000)
net = tflearn.fully_connected(net, 13, activation='softmax')
net = tflearn.regression(net)
model = tflearn.DNN(net, tensorboard_verbose=3, tensorboard_dir='./')
model.fit(xs_train, ys_train, validation_set=(xs_val, ys_val),
          n_epoch=10, batch_size=16, show_metric=True)
model.predict_label(np.array(data)[:]).shape
plt.hist(model.predict(np.array(data)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14035087719298245, 'fraction_numerical': 0.04437564499484004, 'mean_word_length': 6.672823218997362, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '188484', 'license': 'MIT', 'n_tokens_mistral': 1156, 'n_tokens_neox': 1061, 'n_words': 258, 'path': 'tflearn.ipynb', 'repo_name': 'piovere/LibsAnalysis', 'size': '2907'}