<jupyter_start><jupyter_text>Neural Networks in Python - 3 ways to get you started!In this tutorial, we will implement a multi-layered perceptron (a type of a feed-forward neural network) in Python using three different libraries. We’ll start off with the most basic example possible, going to more complex and flexible frameworks with the aim of increasing our understanding of how to implement neural networks in Python.For an indepth explanation see the blog post at Medium.com:https://medium.com/@mattvonrohr/neural-networks-in-python-3-ways-to-get-started-with-multi-layered-perceptrons-20c44e22ae05  scikit-learn<jupyter_code>!pip install scikit-learn
from sklearn.datasets import load_boston

boston = load_boston()
X = boston.data
y = boston.target

NUM_EPOCHS = 1000
NUM_HIDDEN = 500
LR = 0.01
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPRegressor
from sklearn.metrics import mean_squared_error, r2_score

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

def report(y_pred, y_true):
  rmse = np.sqrt(mean_squared_error(y_pred, y_true))
  r2 = r2_score(y_pred, y_true)
  print(f'RMSE: {rmse:.3f}\n R^2: {r2:.3f}')
reg = MLPRegressor(hidden_layer_sizes=[NUM_HIDDEN], max_iter=NUM_EPOCHS, learning_rate_init=LR, random_state=42)
reg.fit(X_train, y_train)
y_pred = reg.predict(X_test)

report(y_pred, y_test)<jupyter_output>RMSE: 3.237
 R^2: 0.841<jupyter_text>Keras<jupyter_code>!pip install keras
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# make random numbers reproducible
tf.random.set_seed(42)

model = keras.Sequential(
    [
        layers.Dense(13, activation="relu"),
        layers.Dense(NUM_HIDDEN, activation="relu"),
        layers.Dense(1),
    ]
)

optimizer = keras.optimizers.Adam(LR)
model.compile(loss='mse', optimizer=optimizer)

model.fit(X_train, y_train, epochs=NUM_EPOCHS, verbose=0)
y_pred = model.predict(X_test)

report(y_pred, y_test)<jupyter_output>RMSE: 3.461
 R^2: 0.836<jupyter_text>PyTorch<jupyter_code>!pip install torch
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(13, NUM_HIDDEN)
        self.fc2 = nn.Linear(NUM_HIDDEN, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

# make random numbers reproducible
torch.manual_seed(0)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print('Training on', device)
model = Net().to(device)

X_train_pt = torch.FloatTensor(X_train).to(device)
X_test_pt = torch.FloatTensor(X_test).to(device)
y_train_pt = torch.FloatTensor(y_train).reshape(-1, 1).to(device)

loss_function = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=LR)

for t in range(NUM_EPOCHS):
    optimizer.zero_grad()
    y_pred_pt = model(X_train_pt)
    loss = loss_function(y_pred_pt, y_train_pt)
    loss.backward()   
    optimizer.step()

    if t % 200 == 99:
        y_pred_pt = model(X_test_pt)
        print(t, loss.cpu().item())
        model.zero_grad()

y_pred = model(X_test_pt).reshape(-1).detach().cpu().numpy()
report(y_pred, y_test)<jupyter_output>Training on cpu
99 9.056133270263672
299 4.458338737487793
499 2.8459088802337646
699 1.7787655591964722
899 1.095325231552124
RMSE: 3.965
 R^2: 0.790<jupyter_text>Just for some fun, let's ensemble our predictions together with a GBM (which does have good performance out of the box) and further reduce the RMSE of our predictions:<jupyter_code>from sklearn.ensemble import GradientBoostingRegressor
reg = GradientBoostingRegressor(n_estimators=1000, learning_rate=0.025)
reg.fit(X_train, y_train)
y_pred_gbr = reg.predict(X_test)
report(y_pred_gbr, y_test)
y_pred_ensemble = (y_pred_gbr + y_pred) / 2
report(y_pred_ensemble, y_test)
import pandas as pd
pd.DataFrame({
    'ensemble': y_pred_ensemble,
    'y_test': y_test
})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10718548579613273, 'fraction_numerical': 0.044163284793506806, 'mean_word_length': 5.125730994152047, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '605532', 'license': 'CC0-1.0', 'n_tokens_mistral': 1721, 'n_tokens_neox': 1564, 'n_words': 350, 'path': 'neural_networks_in_python.ipynb', 'repo_name': 'twiddles/neural-networks-starter', 'size': '4189'}