<jupyter_start><jupyter_text>Neuromatch Academy: Week 2, Day 1, Tutorial 3 Day 1: Parameter Sharing (CNNs and RNNs)__Content creators:__ <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME> __Content reviewers:__ <NAME>, <NAME>, <NAME>, <NAME>__Production editors:__ <NAME>, <NAME> __Based on material from:__ <NAME>, <NAME>, <NAME>, <NAME> --- Tutorial ObjectivesAt the end of this tutorial, we will be able to:- Understand the structure of a Recurrent Neural Network (RNN)- Build a simple RNN model<jupyter_code>#@markdown Tutorial slides
# you should link the slides for all tutorial videos here (we will store pdfs on osf)

from IPython.display import HTML

HTML('<iframe src="https://docs.google.com/presentation/d/1jrKnXGoCXovB5pMGtnsF0ICxr1Cu_vDk/edit#slide=id.p1" frameborder="100" width="960" height="569" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe>')
#@title Dependencies
!pip install livelossplot --quiet
# Imports
import torch
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import HTML

from tqdm.notebook import tqdm, trange
from time import sleep

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
device, torch.get_num_threads()

!mkdir images
!wget "https://raw.githubusercontent.com/dem1995/generally_available_files/main/chicago_skyline_shrunk_v2.bmp"
!mv *.bmp images
# @title Figure Settings
%config InlineBackend.figure_format = 'retina'
%matplotlib inline

fig_w, fig_h = (8, 6)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})

plt.rcParams["mpl_toolkits.legacy_colorbar"] = False

import warnings
warnings.filterwarnings("ignore", category=UserWarning, module="matplotlib")


plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/"
              "course-content/master/nma.mplstyle")
# @title Set seed for reproducibility
seed = 2021
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.cuda.manual_seed(seed)
np.random.seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)

print ('Seed has been set.')<jupyter_output><empty_output><jupyter_text>Section 1: Recurrent Neural Networks (RNNs)<jupyter_code>#@title Video 1: Intro to RNNs
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="0Majex-aF0E", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video<jupyter_output><empty_output><jupyter_text>RNNs are compact models that operate over timeseries, and have the ability to remember past input. They also save parameters by using the same weights at every time step.  If you've heard of Transformers, those models dont' have this kind of temporal weight sharing, and so they are *much* larger.The code below is adapted from https://github.com/spro/char-rnn.pytorch<jupyter_code>#@title RNN framework (run me)

!pip install unidecode

!wget --output-document=/content/sample_data/twain.txt https://github.com/amfyshe/amfyshe.github.io/blob/master/twain.txt

# https://github.com/spro/char-rnn.pytorch

import torch
import torch.nn as nn
from torch.autograd import Variable

class CharRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, model="gru", n_layers=1):
        super(CharRNN, self).__init__()
        self.model = model.lower()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.n_layers = n_layers

        self.encoder = nn.Embedding(input_size, hidden_size)
        if self.model == "gru":
            self.rnn = nn.GRU(hidden_size, hidden_size, n_layers)
        elif self.model == "lstm":
            self.rnn = nn.LSTM(hidden_size, hidden_size, n_layers)
        elif self.model == "rnn":
            self.rnn = nn.RNN(hidden_size, hidden_size, n_layers)
        self.decoder = nn.Linear(hidden_size, output_size)

    def forward(self, input, hidden):
        batch_size = input.size(0)
        encoded = self.encoder(input)
        output, hidden = self.rnn(encoded.view(1, batch_size, -1), hidden)
        output = self.decoder(output.view(batch_size, -1))
        return output, hidden

    def init_hidden(self, batch_size):
        if self.model == "lstm":
            return (Variable(torch.zeros(self.n_layers, batch_size, self.hidden_size)),
                    Variable(torch.zeros(self.n_layers, batch_size, self.hidden_size)))
        return Variable(torch.zeros(self.n_layers, batch_size, self.hidden_size))



#@title Helpers (run me)
# https://github.com/spro/char-rnn.pytorch

import unidecode
import string
import random
import time
import math
import torch

# Reading and un-unicode-encoding data

all_characters = string.printable
n_characters = len(all_characters)

def read_file(filename):
    file = unidecode.unidecode(open(filename).read())
    return file, len(file)

# Turning a string into a tensor

def char_tensor(string):
    tensor = torch.zeros(len(string)).long()
    for c in range(len(string)):
        try:
            tensor[c] = all_characters.index(string[c])
        except:
            continue
    return tensor

# Readable time elapsed

def time_since(since):
    s = time.time() - since
    m = math.floor(s / 60)
    s -= m * 60
    return '%dm %ds' % (m, s)

def generate(decoder, prime_str='A', predict_len=100, temperature=0.8, cuda=False):
    hidden = decoder.init_hidden(1)
    prime_input = Variable(char_tensor(prime_str).unsqueeze(0))

    if cuda:
        hidden = hidden.cuda()
        prime_input = prime_input.cuda()
    predicted = prime_str

    # Use priming string to "build up" hidden state
    for p in range(len(prime_str) - 1):
        _, hidden = decoder(prime_input[:,p], hidden)

    inp = prime_input[:,-1]

    for p in range(predict_len):
        output, hidden = decoder(inp, hidden)

        # Sample from the network as a multinomial distribution
        output_dist = output.data.view(-1).div(temperature).exp()
        top_i = torch.multinomial(output_dist, 1)[0]

        # Add predicted character to string and use as next input
        predicted_char = all_characters[top_i]
        predicted += predicted_char
        inp = Variable(char_tensor(predicted_char).unsqueeze(0))
        if cuda:
            inp = inp.cuda()

    return predicted
# https://github.com/spro/char-rnn.pytorch
!ls
!pwd

batch_size = 50
chunk_len = 200
model = "rnn" # other options: lstm, gru

#hyperparams
n_layers = 2
hidden_size = 200
n_epochs = 2000
learning_rate = 0.01
print_every = 25

def train(inp, target):
  hidden = decoder.init_hidden(batch_size)
  decoder.zero_grad()
  loss = 0

  for c in range(chunk_len):
    output, hidden = decoder(inp[:,c], hidden)
    loss += criterion(output.view(batch_size, -1), target[:,c])

  loss.backward()
  decoder_optimizer.step()
  return loss.item() / chunk_len

file, file_len = read_file('/content/sample_data/twain.txt')

def random_training_set(chunk_len, batch_size):
  inp = torch.LongTensor(batch_size, chunk_len)
  target = torch.LongTensor(batch_size, chunk_len)
  for bi in range(batch_size):
    start_index = random.randint(0, file_len - chunk_len - 1)
    end_index = start_index + chunk_len + 1
    chunk = file[start_index:end_index]
    inp[bi] = char_tensor(chunk[:-1])
    target[bi] = char_tensor(chunk[1:])
  inp = Variable(inp)
  target = Variable(target)
  return inp, target

decoder = CharRNN(
    n_characters,
    hidden_size,
    n_characters,
    model=model,
    n_layers=n_layers,
)
decoder_optimizer = torch.optim.Adagrad(decoder.parameters(), lr=learning_rate)
criterion = nn.CrossEntropyLoss()

start = time.time()
all_losses = []
loss_avg = 0


print("Training for %d epochs..." % n_epochs)
for epoch in tqdm(range(1, n_epochs + 1), position=0, leave=True):
  loss = train(*random_training_set(chunk_len, batch_size))
  loss_avg += loss

  if epoch % print_every == 0:
    print('[%s (%d %d%%) %.4f]' % (time_since(start), epoch,\
                                   epoch / n_epochs * 100, loss))
    print(generate(decoder, 'Wh', 100), '\n')<jupyter_output><empty_output><jupyter_text>Section 2: Power consumption in Deep LearningTraining NN models can be incredibly costly, both in actual money but also in power consumption.<jupyter_code>#@title Video 2: Carbon Footprint of AI
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="2upwdK3bcXU", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video<jupyter_output><empty_output><jupyter_text>Take a few moments to chat with your pod about the following points:* Which societal costs of training do you find most compelling?* When is training an AI model worth the cost?  Who should make that decision?* Should there be additional taxes on energy costs for compute centers?   Section 3: Wrap upWhat a day!  We've learned a lot!  The basics of CNNs and RNNs, and how changes to architecture that allow models to parameter share can greatly reduce the size of the model.  We learned about convoliution and pooling, as well as the basic idea behind RNNs.  To wrap up we thought about the impact of training large NN models.<jupyter_code>#@title Video 3: Wrap-up
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="Ikb4hwR4pU0", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11129550876465097, 'fraction_numerical': 0.012965460014521315, 'mean_word_length': 4.1424, 'pattern_counts': {'":': 4, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '49932', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3308, 'n_tokens_neox': 3149, 'n_words': 862, 'path': 'tutorials/W2D1_ConvnetsAndRecurrentNeuralNetworks/student/W2D1_Tutorial3.ipynb', 'repo_name': 'haltakov/course-content-dl', 'size': '9768'}