<jupyter_start><jupyter_text>MNIST SGD Get the 'pickled' MNIST dataset from http://deeplearning.net/data/mnist/mnist.pkl.gz. We're going to treat it as a standard flat dataset with fully connected layers, rather than using a CNN.[UPDATE]  Since deeplearning.net stop providing datasets anymore, I host datasets on my website.  But I'm not sure how long I can keep it.<jupyter_code># データをダウンロードします。
# ! wget http://shabelab.com/datasets/mnist.pkl.tar.gz
# データを解凍します。
# ! tar xvzf mnist.pkl.tar.gz
# データを解凍して、4つの変数に格納します。
# データはnumpy形式になっています。
with open('mnist.pkl', 'rb') as f:
    ((x_train, y_train), (x_valid, y_valid), _) = pickle.load(f, encoding='latin-1')
# Pathを設定します。
path = Path("~/.fastai/data/mnist/")
# 一つ表示してみます。
plt.imshow(x_train[0].reshape((28,28)), cmap="gray")
x_train.shape
# インプットとアウトプットのカタチを確認しておきます。
x_train, x_train.shape
y_train, y_train.shape
# numpyからPyTorchのtensorへ変換

x_train,y_train,x_valid,y_valid = map(
    torch.tensor, 
    (x_train,y_train,x_valid,y_valid))

n,c = x_train.shape

x_train.shape, y_train.min(), y_train.max()
# # ロス関数を定義しておきます。
# # おなじみの差を2乗して平均をとる奴です。

# def mse(y_hat, y):
#     return ((y_hat-y)**2).mean()<jupyter_output><empty_output><jupyter_text>In lesson2-sgd we did these things ourselves:```pythonx = torch.ones(n,2) def mse(y_hat, y): return ((y_hat-y)**2).mean()y_hat = x@a```Now instead we'll use PyTorch's functions to do it for us, and also to handle mini-batches (which we didn't do last time, since our dataset was so small).<jupyter_code>x_train.shape
y_train.shape
# PyTorchのDataset、及びTensorDatasetクラスを準備しておきます。
from torch.utils.data import Dataset, TensorDataset
# インプットとアウトプットをぶっ込んでDatasetを作ってくれるのがTensorDatasetです。
train_ds = TensorDataset(x_train, y_train)
valid_ds = TensorDataset(x_valid, y_valid)

# バッチサイズは64にしておきます。
bs=64
# 64個ずつデータをもってこれる用DataLoaderを作成します。

train_dl = DataLoader(train_ds, batch_size=bs)
valid_dl = DataLoader(valid_ds, batch_size=bs)

# DataLoaders was called DataBunch in ver 1.x
# fastai的にはまとめたいので、まとめます。
dls = DataLoaders(train_dl, valid_dl)
# ちゃんとできているか、最初のバッチを取得してみます。
x, y = dls.one_batch()
x.shape, y.shape
class Mnist_Logistic(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(784, 10, bias=True)

    def forward(self, xb):
        return self.lin(xb)
model = Mnist_Logistic().cuda()
model
model.lin
x.shape
y, y.shape

for p in model.parameters():
    print(p.shape)
[p.shape for p in model.parameters()]
lr=2e-2
loss_func = nn.CrossEntropyLoss()
device = torch.device("cuda")
def update(x,y,lr):
    
    wd = 1e-5
    
    x = x.to(device)
    y = y.to(device)
    
    y_hat = model(x)
    
    # weight decay
    w2 = 0.
    for p in model.parameters():
        w2 += (p**2).sum()
    
    # add to regular loss
    loss = loss_func(y_hat, y) + w2*wd
    
    loss.backward()
    
    with torch.no_grad():
        
        for p in model.parameters():
            
            # p = p - lr * p.grad
            p.sub_(lr * p.grad)
            
            p.grad.zero_()
    
    return loss.item()
losses = [update(x,y,lr) for x,y in dls.train]

# losses = []
# for x, y in data.train_dl:
#     loss = update(x, y, lr)
#     losses.append(loss)
plt.plot(losses);
class Mnist_NN(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = nn.Linear(784, 105, bias=True)
        self.lin2 = nn.Linear(105, 50, bias=True)
        self.lin3 = nn.Linear(50, 10, bias=True)

    def forward(self, xb):
        
        x = self.lin1(xb)
        x = F.relu(x)
        
        x = self.lin2(x)
        x = F.relu(x)
        
        x = self.lin3(x)
        
        return x
model = Mnist_NN().cuda()
model
losses = [update(x,y,lr) for x,y in dls.train]
plt.plot(losses);
# lr=
model = Mnist_NN().cuda()
def update(x,y,lr):
    
    x = x.to(device)
    y = y.to(device)
    
    opt = optim.Adam(model.parameters(), lr)
    
    # opt = optim.SGD(model.parameters(), lr)
    
    y_hat = model(x)
    
    loss = loss_func(y_hat, y)
    
    loss.backward()
    
    
    opt.step()
    
    opt.zero_grad()
    
    
    
    return loss.item()
# losses = [update(x,y,2e-2) for x,y in data.train_dl]
from torch import optim
# sgd
# losses = [update(x,y,1e-4) for x,y in dls.train]

# adam
losses = [update(x,y,1e-4) for x,y in dls.train]
plt.plot(losses);
# fastai 的にやる

learn = Learner(dls, Mnist_NN(), loss_func=loss_func, metrics=accuracy)
learn.lr_find()

learn.fit_one_cycle(1, 1e-2)
learn.recorder.plot_loss()
learn.recorder.plot_sched()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13902331720193578, 'fraction_numerical': 0.013858336999560053, 'mean_word_length': 3.194649446494465, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '278036', 'license': 'MIT', 'n_tokens_mistral': 2017, 'n_tokens_neox': 1938, 'n_words': 381, 'path': 'nbs/v1_lesson5-sgd-mnist_ja.ipynb', 'repo_name': 'akiraenduo/mytutorials', 'size': '4546'}