<jupyter_start><jupyter_text>`STARSKØPE-2`**Building a Cyberoptic Neural Network Telescope for Astrophysical Object Classification**> Flatiron School Capstone Project* `<NAME>`* `Instructor: <NAME> PhD`* `Data Science Full-Time Program`* `Blog post URL:`* `Non-Technical Presentation`:  ImportOnce again let's import the requisite libraries and our dataset.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib as mpl
%matplotlib inline
from matplotlib.colors import LogNorm

import seaborn as sns
sns.set_style('whitegrid')
import matplotlib.pyplot as plt
plt.style.use('seaborn-bright')
 

font_dict={'family':'"Titillium Web", monospace','size':16}
mpl.rc('font',**font_dict)


#ignore pink warnings
import warnings
warnings.filterwarnings('ignore')
# Allow for large # columns
pd.set_option('display.max_columns', 0)
# pd.set_option('display.max_rows','')<jupyter_output>/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm<jupyter_text>Import dataset which has already been split into train and test sets, `exoTrain.csv.zip` and `exoTest.csv.zip` (I compressed them from their original csv format since the training set is > 240 MB so we'll to unzip them).<jupyter_code># Use this instead if using Jupyter notebook/local
# import os, glob, sys
# home = os.path.abspath(os.curdir)
# os.listdir(home)
# %cd ../
# %cd data
# %ls


# navigating to dataset on google colabs - step 1 mount drive
from google.colab import drive
drive.mount('/gdrive',force_remount=True)

%cd /gdrive
# google colabs step 2 - change directory to `data`
%cd 'My Drive/Colab Notebooks/starskope/data'/
%ls
# uncomment below if you need to unzip the data files
# !unzip -q 'exoTrain.csv.zip'
# !unzip -q 'exoTest.csv.zip'
# %ls
# read in train and test files
train = pd.read_csv('exoTrain.csv')
test = pd.read_csv('exoTest.csv')
# cd backto home / root directory
%cd ../
# We will want to do our own train-test split rather than use the default splitting
# So let's combine the train and test datasets into one
# Both indexes begin with 0 so we need to adjust the index to avoid duplicates/overlap

idx2 = list(range(5088,5658)) # the train dataset index ends with 5087
idx2 = pd.Series(idx2)
idx2
test = test.rename(index=idx2)
test.tail()
# Now we can combine training and test datasets 

exodata = train.append(test)
exodata.info()
# create datasets for spectographs
planets = exodata.loc[exodata['LABEL'] == 2]
negatives = exodata.loc[exodata['LABEL'] == 1]
# Prep data for ML : Separate target and feature datasets
# TARGETS
y_train = train['LABEL']
y_test = test['LABEL']

# FEATURES
x_train = train.copy().drop(columns=['LABEL'], axis=1)
x_test = test.copy().drop(columns=['LABEL'], axis=1)

# create datasets for spectographs
train_planets = train.loc[train['LABEL'] == 2]
train_negatives = train.loc[train['LABEL'] == 1]
test_planets = test.loc[test['LABEL'] == 2]
test_negatives = test.loc[test['LABEL'] == 1]<jupyter_output><empty_output><jupyter_text>Spectographs<jupyter_code>signal = np.array(x_train.iloc[0])
signal
spec, freqs, t, m = plt.specgram(signal, NFFT=256,Fs=2, cmap='magma')
plt.colorbar()
plt.xlabel('Time MJD')
plt.ylabel('PDC_SAP Flux (e-/s)')
# create folders for storing training spectographs
%mkdir train_planets
%mkdir train_negatives

# create folders for storing test spectographs
%mkdir test_planets
%mkdir test_negatives
# setup paths for storing images in folders created above 
from os import path
train_planets_outpath = "./train_planets"
test_planets_outpath = "./test_planets"

train_negatives_outpath = "./train_negatives"
test_negatives_outpath = "./test_negatives"
# create function for generating and saving spectograph figs
def make_specgram(df):
    fig, ax = plt.subplots()
    fig, freqs, t, m = plt.specgram(signal, Fs=2, NFFT=256, cmap='magma')
    plt.colorbar()
    plt.xlabel('Time MJD')
    plt.ylabel('PDC_SAP Flux (e-/s)')
    return fig, freqs, t, m 
# create spectographs for training set - planets
for i in train_planets.index:
    signal = np.array(train_planets.iloc[i])
    make_specgram(train_planets)
    plt.savefig(path.join(train_planets_outpath,'train_planets_{0}.png'.format(i)))
# create spectographs for training set - NO planets
for i in train_negatives.index:
    signal = np.array(train_negatives.iloc[i])
    make_specgram(train_negatives)
    plt.savefig(path.join(train_negatives_outpath,'train_negatives_{0}.png'.format(i)))
# create spectographs for test set - planets
for i in test_planets.index:
    signal = np.array(test_planets.iloc[i])
    make_specgram(test_planets)
    plt.savefig(path.join(test_planets_outpath,'spec_{0}.png'.format(i)))
# create spectographs for test set - NO planets
for i in test_negatives.index:
    signal = np.array(test_negatives.iloc[i])
    make_specgram(test_negatives)
    plt.savefig(path.join(test_negatives_outpath,'spec_{0}.png'.format(i)))<jupyter_output><empty_output><jupyter_text>Preparing Images using Flow<jupyter_code>!pip install pillow
!pip install opencv-contrib-python<jupyter_output>Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (7.0.0)
Requirement already satisfied: opencv-contrib-python in /usr/local/lib/python3.6/dist-packages (4.1.2.30)
Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from opencv-contrib-python) (1.18.2)<jupyter_text>Prep images using .flow<jupyter_code>import cv2,glob,os
import pandas as pd
train_planet_files = glob.glob('train_planets')
train_negative_files = glob.glob('train_negatives')
all_train_files = [*train_planet_files,*train_negative_files]

test_planet_files = glob.glob('test_planets')
test_negative_files = glob.glob('test_negatives')
all_test_files = [*test_planet_files,*test_negative_files]


all_filename_vars = [train_planet_files, train_negative_files,
                        test_planet_files, test_negative_files]
# converting an image with the Keras API

from PIL import Image
from keras.preprocessing import image
from imageio import imread
from skimage.transform import resize
import cv2
from tqdm import tqdm

from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.preprocessing.image import array_to_img
# From <NAME> : https://colab.research.google.com/drive/1fwXPY3IDHxNiv7YgOpt3p5BvUaO4VruB#scrollTo=L5CNHxnagDjo

def load_img_cv2(filename, RGB=True):
  import cv2
  IMG = cv2.imread(filename)
  if RGB: cmap = cv2.COLOR_BGR2RGB
  else:
    cmap=cv2.COLOR_BGR2GRAY
  return cv2.cvtColor(IMG, cmap)


# defining a function to read images and convert to array
def read_img(img_path, target_size=(64,64,3)):
  img = image.load_img(img_path, target_size=target_size)
  img = image.img_to_array
  return img
# Read in training and test filenames to produce X and y data splits.

def load_train_test(train_planet_files, train_negative_files,
                        test_planet_files, test_negative_files, img_size=(64,64,3), val_size=0.1):
  # y labels are encoded as 1=negatives, 2=planets
  # returns X_train, X_test, y_train, y_test, y_val

  display('[i] LOADING IMAGES')

  train_img = []
  train_label = []

  for img_path in tqdm(train_planet_files):
    train_img.append(read_img(img_path, target_size=img_size))
    train_label.append(2)

  for img_path in tqdm(train_negative_files):
    train_img.append(read_img(img_path, target_size=img_size))
    train_label.append(1)

  test_img = []
  test_label = []

  for img_path in tqdm(test_planet_files):
    test_img.append(read_img(img_path, target_size=img_size))
    test_label.append(2)

  for img_path in tqdm(test_negative_files):
    test_img.append(read_img(img_path, target_size=img_sie))
    test_label.append(1)

  from sklearn.model_selection import train_test_split
  X = np.array(train_img, np.float32)
  y = np.array(train_label)

  X_test = np.array(test_img, np.float32)
  y_test = np.array(test_label)
  X_train,X_val, y_train, y_val = train_test_split(X,y,test_size=0.1)
  print('\n[i] Length of Splits:')
  print(f"X_train={len(X_train)}, X_test={len(X_test)}, X_val={len(X_val)}")
  return X_train, X_test, X_val, y_train, y_test, y_val

# create function for ImageDataGenerators for train, test and val data.
# returns training_set, test_set, val_set
def train_test_val_datagens(X_train, X_test, X_val, y_train, y_test, y_val,
                            BATCH_SIZE = 32, train_datagen_kws=dict(
                                shear_range=0.2,
                                zoom_range=0.2,
                                horizontal_flip=True
                            )):
  from keras.preprocessing.image import ImageDataGenerator

  train_datagen = ImageDataGenerator(rescale = 1./255, **train_datagen_kws)
  test_datagen = ImageDataGenerator(rescale = 1./255)
  val_datagen = ImageDataGenerator(rescale = 1./255)

  training_set = train_datagen.flow(X_train, y=y_train, batch_size=BATCH_SIZE)
  test_set = test_datagen.flow(X_test, y=y_test, batch_size=BATCH_SIZE)
  val_set = val_datagen.flow(X_val, y=y_val, batch_size=BATCH_SIZE)

  return training_set, test_set, val_set
def get_shapes_dict(training_set, verbose=True):
  shapes = ["Batchsize", "img_width", "img_height", "img_dim"]
  SHAPES = dict(zip(shapes, training_set[0][0].shape))
  if verbose:
    print('SHAPES DICT:')
    print(SHAPES)
    print(training_set[0][0].shape)
    print('\n[i] Labels for batch (1=negative, 2=planet')
    print(training_set[0][1])
  return SHAPES 
  
## USING FUNCTIONS TO LOAD IN IMAGES 
X_train,X_test,X_val,y_train,y_test,y_val = load_train_test(*all_filename_vars,
                                                  val_size=0.1,img_size=(64,64,3))
train_test_val_vars = [X_train,X_test,X_val,y_train,y_test,y_val ]
training_set,test_set,val_set = train_test_val_datagens(*train_test_val_vars,
                                                        BATCH_SIZE=128)
    # X_train,X_test,X_val,y_train,y_test,y_val, BATCH_SIZE=128)    

SHAPES = get_shapes_dict(training_set)
# # helper functions for reading in the images

# # load the image
# img = load_img('bondi_beach.jpg')
# print(type(img))


# # convert to numpy array
# img_array = img_to_array(img)
# print(img_array.dtype)
# print(img_array.shape)
# # convert back to image
# img_pil = array_to_img(img_array)
# print(type(img))

# # example of saving an image with the Keras API
# from keras.preprocessing.image import load_img
# from keras.preprocessing.image import save_img
# from keras.preprocessing.image import img_to_array
# # load image as as grayscale
# img = load_img('bondi_beach.jpg', grayscale=True)
# # convert image to a numpy array
# img_array = img_to_array(img)
# # save the image with a new filename
# save_img('bondi_beach_grayscale.jpg', img_array)
# # load the image to confirm it was saved correctly
# img = load_img('bondi_beach_grayscale.jpg')
# print(type(img))
# print(img.format)
# print(img.mode)
# print(img.size)
# img.show()
def complete_image_processing(*train_test_filenames,img_size=(64,64,3),val_size=0.1,
                              BATCH_SIZE = 32, train_datagen_kws= dict(
                                shear_range = 0.2, 
                                zoom_range = 0.2,
                                horizontal_flip = True),verbose=True):
    """Calls all 3 image prep functions and returns training,test,val datagens,
    plus SHAPES dict."""    
    # img_params = list(locals())
    # print(img_params[1:])
    # if verbose:
        # print('\n[i] Creating training ImageDataGenerator using:')
        # [print(f"{img_params[i]}") for i in range(len(img_params))];


    # print(len(train_test_filenames))
    if len(train_test_filenames) != 4:
        raise Exception('Must provide 4 filenames dog_train_files, cat_train_files,dog_test_files,cat_test_files')
    train_test_val_vars = load_train_test_val(
        *train_test_filenames, val_size=0.1,img_size=img_size)

    training_set,test_set,val_set = train_test_val_datagens(*train_test_val_vars,
                                                            BATCH_SIZE=BATCH_SIZE)

    SHAPES = get_shapes_dict(training_set)

    # print(pd.Series())

    return training_set,test_set,val_set,SHAPES



training_set,test_set,val_set,SHAPES = \
complete_image_processing(*all_filename_vars, img_size=(64,64,3),BATCH_SIZE=64)<jupyter_output><empty_output><jupyter_text>Model  CNNFor the first iteration, we'll construct a CNN model, after which we will build an RBM.<jupyter_code># # Part 1 - Building the CNN
# clock = fs.jmi.Clock()
# clock.tic('')
# # Importing the Keras libraries and packages
# from keras.models import Sequential
# from keras.layers import Conv2D
# from keras.layers import MaxPooling2D
# from keras.layers import Flatten
# from keras.layers import Dense

# # Initialising the CNN
# classifier = Sequential()

# # Step 1 - Convolution
# classifier.add(Conv2D(SHAPES['Batchsize'], (3, 3),
#                              input_shape = (SHAPES['img_width'],
#                                             SHAPES['img_height'],
#                                             SHAPES['img_dim']),
#                              activation = 'relu'))

# classifier.add(Conv2D(SHAPES['Batchsize'], (3, 3),
#                       input_shape = (SHAPES['img_width'], 
#                                      SHAPES['img_height'],
#                                      SHAPES['img_dim']),
#                        activation = 'relu'))
# # Step 2 - Pooling
# classifier.add(MaxPooling2D(pool_size = (2, 2)))

# # Adding a second convolutional layer
# classifier.add(Conv2D(SHAPES['Batchsize'], (3, 3),
#                       activation = 'relu'))
# classifier.add(MaxPooling2D(pool_size = (2, 2)))

# # Step 3 - Flattening
# classifier.add(Flatten())

# # Step 4 - Full connection
# classifier.add(Dense(units = SHAPES['Batchsize'], activation = 'relu'))

# classifier.add(Dense(units = 1, activation = 'sigmoid'))

# # Compiling the CNN
# classifier.compile(optimizer = 'adam', 
#                    loss = 'binary_crossentropy',
#                    metrics = ['accuracy'])
# print()
# display(classifier.summary())
# # Part 2 - Fitting the CNN to the images

# history =classifier.fit_generator(training_set,
#                          steps_per_epoch = 500,
#                          epochs = 3,
#                          validation_data = val_set,#test_set,
#                          validation_steps =100,workers=-1)

# clock.toc('')

# y_hat_test = classifier.predict_classes(X_test).flatten()
# # print(pd.Series(y_hat_test).value_counts(normalize=True))
# evaluate_model(y_test,y_hat_test,history)
def cd_gdrive_mkdirs(model_subfolder='Datasets/Models/cat_vs_dog/'):
    """cd to /gdrive/My Drive/ to allow for saving files to google drive
    Also makes all subfolders in 'model_subfolder'"""
    
    import os
    ## To save to Gdrive, must first chdir to My Drive (so there's no spaces in fpath)
    curdir = os.path.abspath(os.curdir)
    gdrive_folder =r'/gdrive/My Drive/'

    try:
        os.chdir(gdrive_folder)
    except Exception as e:
        print(f'ERROR: {e}')

    try:
        os.makedirs(model_subfolder,exist_ok=True)
        print('Directories created.')
    except:
        print('Error making directories')

    return print(os.path.abspath(os.curdir))

model_subfolder='Datasets/Models/cat_vs_dog/'
cd_gdrive_mkdirs(model_subfolder=model_subfolder)

print('\n',model_subfolder)
os.listdir(model_subfolder)
# checkpoint
from keras.callbacks import ModelCheckpoint, EarlyStopping, CSVLogger

def create_csvlogger(filename):
    return CSVLogger(filename, separator=',', append=False)

def create_checkpoint(monitor='val_acc',model_subfolder='Datasets/Models/cat_vs_dog/'):
    filepath=model_subfolder+"weights-improvement-{epoch:02d}-{"+monitor+":.2f}.hdf5"
    checkpoint = ModelCheckpoint(filepath, monitor=monitor, verbose=1, save_best_only=True, mode='max')
    return checkpoint

def create_early_stopping(monitor = 'val_acc',min_delta = 0, patience = 1,
                          verbose = 1, restore_best_weights = True):

    args = locals()
    earlystop = EarlyStopping(**args)
    return earlystop
# def get_callbacks():
model_subfolder='Datasets/Models/cat_vs_dog/'
cd_gdrive_mkdirs(model_subfolder=model_subfolder)
callbacks_list = [create_checkpoint('val_acc'),
                  create_early_stopping(),
                  create_csvlogger(model_subfolder+'callback_log.csv')]
callbacks_list
model_subfolder='Datasets/Models/cat_vs_dog/'
cd_gdrive_mkdirs(model_subfolder=model_subfolder)
callbacks_list = [create_checkpoint('val_acc'),
                  create_early_stopping(),
                  create_csvlogger(model_subfolder+'callback_log.csv')]
callbacks_list
# Part 1 - Building the CNN
clock = fs.jmi.Clock()
clock.tic('')
# Importing the Keras libraries and packages
from keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense

# Initialising the CNN
classifier = Sequential()

# Step 1 - Convolution
classifier.add(Conv2D(SHAPES['Batchsize'], (5, 5),
                             input_shape = (SHAPES['img_width'],
                                            SHAPES['img_height'],
                                            SHAPES['img_dim']),
                             activation = 'relu'))

classifier.add(Conv2D(SHAPES['Batchsize'], (3, 3),
                      input_shape = (SHAPES['img_width'], 
                                     SHAPES['img_height'],
                                     SHAPES['img_dim']),
                       activation = 'relu'))
# Step 2 - Pooling
classifier.add(MaxPooling2D(pool_size = (2, 2)))

# Adding a second convolutional layer
classifier.add(Conv2D(SHAPES['Batchsize'], (3, 3),
                      activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))

# Step 3 - Flattening
classifier.add(Flatten())

# Step 4 - Full connection
classifier.add(Dense(units = SHAPES['Batchsize'], activation = 'relu'))

classifier.add(Dense(units = 1, activation = 'sigmoid'))

# Compiling the CNN
classifier.compile(optimizer = 'adam', 
                   loss = 'binary_crossentropy',
                   metrics = ['accuracy'])
print()
display(classifier.summary())
# Part 2 - Fitting the CNN to the images

history =classifier.fit_generator(training_set,
                         steps_per_epoch = 1000,
                         epochs = 3,
                         validation_data = val_set,#test_set,
                         validation_steps = 500,workers=-1,
                         callbacks=callbacks_list)

clock.toc('')

y_hat_test = classifier.predict_classes(X_test).flatten()
# print(pd.Series(y_hat_test).value_counts(normalize=True))
evaluate_model(y_test,y_hat_test,history)
# from keras.utils import 
model_loaded = classifier
model_loaded.load_weights('Datasets/Models/cat_vs_dog/weights-improvement-01-0.72.hdf5')
model_loaded.weights
def save_model(model,model_subfolder = 'Datasets/Models/cat_vs_dog/',
               base_modelname = 'CNN_cat_dog_02142020', as_json=True,
               return_fpaths=True,verbose=True):
    import os
    ## To save to Gdrive, must first chdir to My Drive (so there's no spaces in fpath)
    curdir = os.path.abspath(os.curdir)

    gdrive_folder =r'/gdrive/My Drive/'
    model_subfolder = 'Datasets/Models/cat_vs_dog/'

    try:
        os.chdir(gdrive_folder)
        os.makedirs(model_subfolder,exist_ok=True)
    except Exception as e:
        print(f'ERROR: {e}')

    # os.listdir(model_subfolder)
    # https://jovianlin.io/saving-loading-keras-models/
    try:
        weight_fpath = model_subfolder+base_modelname+'_weights.h5'
        model.save_weights(weight_fpath, overwrite=True)

        if as_json:
            model_fpath = model_subfolder+base_modelname+'_model.json'
            # Save the model architecture
            with open(model_fpath, 'w') as f:
                f.write(model.to_json())
        else:
            model_fpath = model_subfolder+base_modelname+'_model.h5'
            model.save(model_fpath)
        if verbose: 
            print(f"[io] Model architecture saved as {model_fpath}")
            print(f"[io] Model weights saved as {weight_fpath}")
        else:
            print(f"[io] Successfully saved model.")

    except Exception as e:
        import warnings
        warnings.warn(f"ERROR SAVING: {e}")
    if return_fpaths:
        return model_fpath, weight_fpath



def load_model(model_fpath,weight_fpath=None,as_json=True):
    from keras.models import model_from_json
    if (as_json == True) & (weight_fpath is None):
        raise Exception('If using as_json=True, must provide ')

    # Model reconstruction from JSON file
    with open(model_fpath, 'r',encoding="utf8") as f:
        model2 = model_from_json(f.read())

    # Load weights into the new model
    model2.load_weights(weight_fpath)
    display(model2.summary())
    return model2

# model_fpath,weight_fpath = save_model(model)
# model_loaded = load_model(model_fpath,weight_fpath)<jupyter_output><empty_output><jupyter_text>Tuning with Gridsearch<jupyter_code>def build_model(SHAPES,filter_size=(3,3), pool_size=(2,2),dropout=True): 
    vars_ = locals()
    print(f'[i] MODEL BUILT USING:\n\t{vars_}')
    # Part 1 - Building the CNN

    # Importing the Keras libraries and packages
    from keras.models import Sequential
    from keras.layers import Conv2D
    from keras.layers import MaxPooling2D
    from keras.layers import Flatten
    from keras.layers import Dense

    # Initialising the CNN
    classifier = Sequential()

    # Step 1 - Convolution
    classifier.add(Conv2D(SHAPES['Batchsize'], filter_size,
                                input_shape = (SHAPES['img_width'], SHAPES['img_height'], SHAPES['img_dim']),
                                activation = 'relu'))

    classifier.add(Conv2D(SHAPES['Batchsize'], filter_size,
                        input_shape = (SHAPES['img_width'], SHAPES['img_height'], SHAPES['img_dim']), activation = 'relu'))

    # Step 2 - Pooling
    classifier.add(MaxPooling2D(pool_size = pool_size))
    if dropout:
        classifier.add(Dropout(0.2))

    # Adding a second convolutional layer
    classifier.add(Conv2D(SHAPES['Batchsize'], filter_size, activation = 'relu'))
    classifier.add(MaxPooling2D(pool_size = pool_size))

    # Step 3 - Flattening
    classifier.add(Flatten())

    # Step 4 - Full connection
    classifier.add(Dense(units = SHAPES['Batchsize'], activation = 'relu'))
    classifier.add(Dense(units = 1, activation = 'sigmoid'))

    # Compiling the CNN
    classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy',
                       metrics = ['accuracy'])
    display(classifier.summary())
    return classifier
    # Part 2 - Fitting the CNN to the images



def train_model(classifier,training_set, test_set, 
                params=dict(steps_per_epoch = 2000,
                            epochs = 3, validation_steps = 500,
                            workers=-1)):
    vars = locals()
    print(f'[i] Training model using\n\t{vars}\n')
    clock = Timer()
    
    history_ = classifier.fit_generator(training_set,
                                        validation_data = test_set,
                                        validation_steps=validation_steps,
                                        **params)
                        # workers=workers)

    clock.stop()
    return history_


model_=build_model(SHAPES)
history = train_model(model_,training_set,test_set)
y_hat_val = model_.predict_classes(X_val).flatten()
evaluate_model(y_val,y_hat_val,history=history)<jupyter_output><empty_output><jupyter_text>Saving/Loading<jupyter_code>## To save to Gdrive, must first chdir to My Drive (so there's no spaces in fpath)
curdir = os.path.abspath(os.curdir)

gdrive_folder =r'/gdrive/My Drive/'
model_subfolder = 'Datasets/Models/cat_vs_dog/'

try:
    os.chdir(gdrive_folder)
    os.makedirs(model_subfolder,exist_ok=True)
except Exception as e:
    print(f'ERROR: {e}')
os.listdir(model_subfolder)<jupyter_output><empty_output><jupyter_text>Tuning with RandomSearchTODO<jupyter_code><jupyter_output><empty_output><jupyter_text>Restricted Boltzmann MachinesTODO Bernoulli Restricted Boltzmann Machine (RBM).    A Restricted Boltzmann Machine with binary visible units and    binary hidden units. Parameters are estimated using Stochastic Maximum    Likelihood (SML), also known as Persistent Contrastive Divergence (PCD)    [2].    The time complexity of this implementation is ``O(d ** 2)`` assuming    d ~ n_features ~ n_components.<jupyter_code>from scipy.ndimage import convolve
from sklearn import linear_model, datasets, metrics
from sklearn.model_selection import train_test_split
from sklearn.neural_network import BernoulliRBM
from sklearn.pipeline import Pipeline
from sklearn.base import clone
# Models we will use
logistic = linear_model.LogisticRegression(solver='lbfgs', max_iter=10000,
                                           multi_class='multinomial')

rbm = BernoulliRBM(random_state=0, verbose=True)

rbm_features_classifier = Pipeline(steps=[('rbm', rbm), ('logistic', logistic)])

# Hyperparameters
rbm.learning_rate = 0.06
rbm.n_iter = 20
# More components tend to give better prediction performance, but larger
# fitting time
rbm.n_components = 100
logistic.C = 6000
# Training RBM-Logistic Pipeline
rbm_features_classifier.fit(X_train, Y_train)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10877617061094, 'fraction_numerical': 0.012593083551015634, 'mean_word_length': 3.5681211041852183, 'pattern_counts': {'":': 1, '<': 42, '<?xml version=': 0, '>': 45, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '285089', 'license': 'MIT', 'n_tokens_mistral': 8785, 'n_tokens_neox': 8378, 'n_words': 1815, 'path': 'starskope-2-colab.ipynb', 'repo_name': 'jirvingphd/starskope', 'size': '25662'}