<jupyter_start><jupyter_text>Full ANN (artificial nueral network) regression code - how to appyly an artifical neural network to any tabular data set- feature engineering- feature encoding- embeddings- dropout layers Steps- Read in the data- feature engineering    - calculate distance    - derive useful date and time stats- deal with categorical dta    - embedding- create tabular model class    -use tabular model class to work with continuous and categorical data- add in loss function and optimizer- train/test split- train the model- evaluate on test data- predict on brand new data<jupyter_code>import torch
import torch.nn as nn
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
df = pd.read_csv('../Data/NYCTaxiFares.csv')
#use .. to go up one level in the directory
df.head()
df['fare_amount'].describe()
#copy and pasted from library notebook
def haversine_distance(df, lat1, long1, lat2, long2):
    """
    Calculates the haversine distance between 2 sets of GPS coordinates in df
    """
    r = 6371  # average radius of Earth in kilometers
       
    phi1 = np.radians(df[lat1])
    phi2 = np.radians(df[lat2])
    
    delta_phi = np.radians(df[lat2]-df[lat1])
    delta_lambda = np.radians(df[long2]-df[long1])
     
    a = np.sin(delta_phi/2)**2 + np.cos(phi1) * np.cos(phi2) * np.sin(delta_lambda/2)**2
    c = 2 * np.arctan2(np.sqrt(a), np.sqrt(1-a))
    d = (r * c) # in kilometers

    return d
df.columns
#feature engineering
df['dist_km'] = haversine_distance(df,'pickup_latitude', 'pickup_longitude', 'dropoff_latitude', 'dropoff_longitude')
df.head()<jupyter_output><empty_output><jupyter_text>Feature engineering is taking existing features and creating a new feature that is more useful than the feautres you already have- we took long and lat to feature engineer distacne in kmNow we will do it on the date time object<jupyter_code>df['pickup_datetime'] = pd.to_datetime(df['pickup_datetime'])
df.info()
df.head()
my_time = df['pickup_datetime'][0]
#my_time. tab will show you all the things you can grab
my_time.hour
#change to correct time zone
df['EDTdate'] = df['pickup_datetime'] - pd.Timedelta(hours=4)
df['Hour'] = df['EDTdate'].dt.hour
df['AMorPM'] = np.where(df['Hour'] < 12, 'AM', 'PM')
df['Weekday'] = df['EDTdate'].dt.strftime("%a")
# %a returns weekday
# can also use dt.dayofweek which returns number of day of week
# could then map to day of the week
df.head()<jupyter_output><empty_output><jupyter_text>Separate categorical from continuous variables<jupyter_code>cat_cols = ['Hour', 'AMorPM', 'Weekday']
# hours could be continuous or categorical, we will do categorical, 24 categories
cont_cols = ['pickup_longitude',
       'pickup_latitude', 'dropoff_longitude', 'dropoff_latitude',
       'passenger_count', 'dist_km']
y_col = ['fare_amount']
df.dtypes
#use pandas to change data type to category
for cat in cat_cols:
    df[cat] = df[cat].astype('category')
df.dtypes
df['Hour'].head()
df['AMorPM'].head()
df['Weekday'].head()<jupyter_output><empty_output><jupyter_text>There are category codes and categories<jupyter_code>df['AMorPM'].cat.categories
df['AMorPM'].cat.codes
df['Weekday'].cat.categories
df['Weekday'].cat.codes
#adding .values makes it a numpy array
df['Weekday'].cat.codes.values
hr = df['Hour'].cat.codes.values
ampm = df['AMorPM'].cat.codes.values
wkdy = df['Weekday'].cat.codes.values
cats = np.stack([hr,ampm,wkdy], axis=1)
cats
# can do it all as three lines of code, the above just breaks it down to show what is happening

#cat_cols = ['Hour', 'AMorPM', 'Weekday']
#for cat in cat_cols:
    #df[cat] = df[cat].astype('category')
#cats = np.stack([df[col].cat.codes.values for col in cat_cols], 1)
cats = torch.tensor(cats, dtype = torch.int64)
# now do for continuous variables
conts = np.stack([df[col].values for col in cont_cols], axis = 1)
conts
conts = torch.tensor(conts, dtype = torch.float)
conts<jupyter_output><empty_output><jupyter_text>Now both continuous and categorical variables are pytorch tensors<jupyter_code>y = torch.tensor(df[y_col].values, dtype = torch.float)
y
cats.shape
conts.shape
y.shape<jupyter_output><empty_output><jupyter_text>Embedding- a simple lookup table that stores embeddings of a fixed dictinoary and size- esentially one hot encoding (dummy coding)<jupyter_code>cat_szs = [len(df[col].cat.categories) for col in cat_cols]
cat_szs
# divide number of unqiue entries in each column by two, max out at about 50
emb_szs = [(size, min(50, (size+1)//2)) for size in cat_szs]
#for 24, 2, 7 it with either take 50 or divide it by 2
emb_szs<jupyter_output><empty_output><jupyter_text>Define a tabular model and set up network<jupyter_code>catz = cats[:2]
catz
selfembeds = nn.ModuleList(nn.Embedding(ni,nf) for ni,nf in emb_szs)
selfembeds
embeddingz = []

for i,e in enumerate(selfembeds):
    embeddingz.append(e(catz[:,i]))
embeddingz<jupyter_output><empty_output><jupyter_text>We have 24 items in the first tensor, 2 in the second, 7 in the last<jupyter_code>z = torch.cat(embeddingz, 1)
z
selfembdrop = nn.Dropout(0.4) 
#randomly zero some of the elmenets of the input tensor with whatever probability you provide
#that way you dont overfit the model
z = selfembdrop(z)
z<jupyter_output><empty_output><jupyter_text>What we did- during the forward method we build out the embedding layer- each embedding layer corresponds to one of the categorical variables- then we assign numerical value to the one hot encoding- put them all together in one tensor- pass through a dropout layer and get assigned values<jupyter_code>class TabularModel(nn.Module):

    def __init__(self, emb_szs, n_cont, out_sz, layers, p=0.5):
        # this setup allows us to use Layers = [n, n1, n2]
        super().__init__()
        self.embeds = nn.ModuleList([nn.Embedding(ni, nf) for ni,nf in emb_szs])
        self.emb_drop = nn.Dropout(p)
        self.bn_cont = nn.BatchNorm1d(n_cont) #normalize
        
        layerlist = []
        n_emb = sum((nf for ni,nf in emb_szs))
        n_in = n_emb + n_cont
        
        for i in layers:
            layerlist.append(nn.Linear(n_in,i)) 
            layerlist.append(nn.ReLU(inplace=True))
            layerlist.append(nn.BatchNorm1d(i))
            layerlist.append(nn.Dropout(p))
            n_in = i
        layerlist.append(nn.Linear(layers[-1],out_sz))
            
        self.layers = nn.Sequential(*layerlist)
    
    def forward(self, x_cat, x_cont):
        embeddings = []
        for i,e in enumerate(self.embeds):
            embeddings.append(e(x_cat[:,i]))
        x = torch.cat(embeddings, 1)
        x = self.emb_drop(x)
        
        x_cont = self.bn_cont(x_cont)
        x = torch.cat([x, x_cont], 1)
        x = self.layers(x)
        return x
torch.manual_seed(33)
model = TabularModel(emb_szs, conts.shape[1], 1, [200,100], p=0.4)
model
criterion = nn.MSELoss() #np.sqrt(MSE) -> RMSE
optimizer = torch.optim.Adam(model.parameters(),lr=0.001)
#train/test split
batch_size = 60000
test_size = int(batch_size*0.2) #make an integer rather than a float
# Data shuffled already
cat_train = cats[:batch_size-test_size]
cat_test = cats[batch_size-test_size:batch_size]
con_train = conts[:batch_size-test_size]
con_test = conts[batch_size-test_size:batch_size]
y_train = y[:batch_size-test_size]
y_test = y[batch_size-test_size:batch_size]
len(cat_train)
len(con_train)
len(cat_test)
import time

start_time = time.time()

epochs = 300

losses = []

for i in range(epochs):
    i += 1
    
    y_pred = model(cat_train,con_train)
    loss = torch.sqrt(criterion(y_pred, y_train))#RMSE
    losses.append(loss)
    
    if i%10 == 1:
        print(f'epoch: {i} loss is {loss}')
        
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
duration = time.time() - start_time
print(f'Training took {duration/60} minutes')
plt.plot(range(epochs),losses)
#validate model on test set
with torch.no_grad():
    
    y_val = model(cat_test,con_test)
    
    loss = torch.sqrt(criterion(y_val,y_test))
loss<jupyter_output><empty_output><jupyter_text>similar to the training set<jupyter_code>#predicted vs actual
for i in range(10):
    diff = np.abs(y_val[i].item()-y_test[i].item())
    print(f'{i}.) PREDICTED:{y_val[i].item():8.2f} TRUE: {y_test[i].item():8.2f} DIFF: {diff:8.2f}')
    
    #8.2f just formats the floating point<jupyter_output>0.) PREDICTED:    3.67 TRUE:     2.90 DIFF:     0.77
1.) PREDICTED:   25.67 TRUE:     5.70 DIFF:    19.97
2.) PREDICTED:    6.29 TRUE:     7.70 DIFF:     1.41
3.) PREDICTED:   13.34 TRUE:    12.50 DIFF:     0.84
4.) PREDICTED:    4.72 TRUE:     4.10 DIFF:     0.62
5.) PREDICTED:    5.46 TRUE:     5.30 DIFF:     0.16
6.) PREDICTED:    4.25 TRUE:     3.70 DIFF:     0.55
7.) PREDICTED:   17.55 TRUE:    14.50 DIFF:     3.05
8.) PREDICTED:    7.20 TRUE:     5.70 DIFF:     1.50
9.) PREDICTED:   11.45 TRUE:    10.10 DIFF:     1.35<jupyter_text>some of the raw differences are far off but the overall rmse is doing good as shown in loss<jupyter_code>#save the model
torch.save(model.state_dict(),'TaxiModel.pt')<jupyter_output><empty_output><jupyter_text>CLASSIFICATION EXAMPLE<jupyter_code>#difference column to use
df['fare_class'].value_counts()<jupyter_output><empty_output><jupyter_text>Then follow all the same steps as above for the regression until we get to creating the model - Change in the output size to 2- this would be set to however many classes you had (categories)<jupyter_code>#torch.manual_seed(33)
#model = TabularModel(emb_szs, conts.shape[1], 2, [200,100], p=0.4) # out_sz = 2<jupyter_output><empty_output><jupyter_text>Then when we get to the cirterion and optimizer, replace MSE loss function with cross entropy loss<jupyter_code>#criterion = nn.CrossEntropyLoss()
#optimizer = torch.optim.Adam(model.parameters(), lr= 0.001)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12066686998068517, 'fraction_numerical': 0.023889397173935144, 'mean_word_length': 4.118626430801249, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '332949', 'license': 'MIT', 'n_tokens_mistral': 3626, 'n_tokens_neox': 3468, 'n_words': 959, 'path': 'Unit 7 Notes 2.ipynb', 'repo_name': 'Python-Deep-Learning-Bootcamp/Artificial-Neural-Networks', 'size': '9837'}