<jupyter_start><jupyter_text>Cheat sheetThis is a cheat sheet for the neural network implementations used in the workshop. First I import libraries and define some functions for visualization.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np

def make_fig(name, x, y):
    fig = plt.figure(figsize=(8,6))
    plt.plot(x,y)
    plt.title(name)
    plt.show()<jupyter_output><empty_output><jupyter_text>Layers  DenseFully connected layer. Each neuron connected to each neuron in the previous layer. Most commonly used layer type.  FlattenCreating a 1D vector from a 2D matrix. Commonly used to feed an image to a dense layer.  Con2DUsed for image feature extraction. Applies a 2 dimentional kerner on the image. The kernel weights are trainable.  MaxPooling2DReduces the resolution of the image.  Activation functions  Linear$g(z)=z$Only used in output layer, in the case of regression when the labels are not bounded.<jupyter_code>x = np.arange(-4,4,0.01)
make_fig('linear', x, x)<jupyter_output><empty_output><jupyter_text>Sigmoid$g(z)=\frac{1}{1+e^{-z}}$Transforms parameters between 0 and 1. Used in output layer if the label is a probability.<jupyter_code>x = np.arange(-4,4,0.01)
make_fig('sigmoid', x, 1/(1+np.exp(-x)))<jupyter_output><empty_output><jupyter_text>TanH$g(z)=\frac{e^x-e^{-z}}{e^{-z}+e^z}$Transforms parameters between -1 and 1. Used for bounding.<jupyter_code>x = np.arange(-4,4,0.01)
make_fig('tanh', x, (np.exp(x)-np.exp(-x))/(np.exp(-x)+np.exp(x)))<jupyter_output><empty_output><jupyter_text>ReLu$g(z)=\text{max}(0,z)$Most common in hidden layers.<jupyter_code>x = np.arange(-2,2,0.01)
make_fig('ReLu', x, np.clip(x,0,None))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1347517730496454, 'fraction_numerical': 0.02186761229314421, 'mean_word_length': 5.938524590163935, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '575263', 'license': 'MIT', 'n_tokens_mistral': 629, 'n_tokens_neox': 604, 'n_words': 162, 'path': 'Cheat_sheet.ipynb', 'repo_name': 'tillaczel/Machine-learning-workshop', 'size': '1692'}