<jupyter_start><jupyter_text>Predicting Student Admissions with Neural NetworksIn this notebook, we predict student admissions to graduate school at UCLA based on three pieces of data:- GRE Scores (Test)- GPA Scores (Grades)- Class rank (1-4)The dataset originally came from here: http://www.ats.ucla.edu/ Loading the dataTo load the data and format it nicely, we will use two very useful packages called Pandas and Numpy. You can read on the documentation here:- https://pandas.pydata.org/pandas-docs/stable/- https://docs.scipy.org/<jupyter_code># Importing pandas and numpy
import pandas as pd
import numpy as np

# Reading the csv file into a pandas DataFrame
data = pd.read_csv('student_data.csv')

# Printing out the first 10 rows of our data
display(data[:10])
data.shape<jupyter_output><empty_output><jupyter_text>Plotting the dataFirst let's make a plot of our data to see how it looks. In order to have a 2D plot, let's ingore the rank.<jupyter_code># Importing matplotlib
import matplotlib.pyplot as plt

# Function to help us plot
def plot_points(data):
    X = np.array(data[["gre","gpa"]])
    y = np.array(data["admit"])
    admitted = X[np.argwhere(y==1)]
    rejected = X[np.argwhere(y==0)]
    plt.scatter([s[0][0] for s in rejected], [s[0][1] for s in rejected], s = 25, color = 'red', edgecolor = 'k')
    plt.scatter([s[0][0] for s in admitted], [s[0][1] for s in admitted], s = 25, color = 'cyan', edgecolor = 'k')
    plt.xlabel('Test (GRE)')
    plt.ylabel('Grades (GPA)')
    
# Plotting the points
plot_points(data)
plt.show()<jupyter_output><empty_output><jupyter_text>Roughly, it looks like the students with high scores in the grades and test passed, while the ones with low scores didn't, but the data is not as nicely separable as we hoped it would. Maybe it would help to take the rank into account? Let's make 4 plots, each one for each rank.<jupyter_code># Separating the ranks
data_rank1 = data[data["rank"]==1]
data_rank2 = data[data["rank"]==2]
data_rank3 = data[data["rank"]==3]
data_rank4 = data[data["rank"]==4]

# Plotting the graphs
plot_points(data_rank1)
plt.title("Rank 1")
plt.show()
plot_points(data_rank2)
plt.title("Rank 2")
plt.show()
plot_points(data_rank3)
plt.title("Rank 3")
plt.show()
plot_points(data_rank4)
plt.title("Rank 4")
plt.show()<jupyter_output><empty_output><jupyter_text>This looks more promising, as it seems that the lower the rank, the higher the acceptance rate. Let's use the rank as one of our inputs. In order to do this, we should one-hot encode it. TODO: One-hot encoding the rankUse the `get_dummies` function in numpy in order to one-hot encode the data.<jupyter_code># TODO:  Make dummy variables for rank
data['rank'] = data['rank'].astype(str)
one_hot_data = pd.get_dummies(data)
display(one_hot_data[:10])
# TODO: Drop the previous rank column
#one_hot_data = one_hot_data.drop(['rank'],axis=1)

# Print the first 10 rows of our data
one_hot_data[:10]<jupyter_output><empty_output><jupyter_text>TODO: Scaling the dataThe next step is to scale the data. We notice that the range for grades is 1.0-4.0, whereas the range for test scores is roughly 200-800, which is much larger. This means our data is skewed, and that makes it hard for a neural network to handle. Let's fit our two features into a range of 0-1, by dividing the grades by 4.0, and the test score by 800.<jupyter_code># Making a copy of our data
processed_data = one_hot_data[:]

# TODO: Scale the columns
processed_data['gre'] = processed_data[['gre']] /800
processed_data['gpa'] = processed_data[['gpa']] / 4
# Printing the first 10 rows of our procesed data
processed_data[:10]<jupyter_output><empty_output><jupyter_text>Splitting the data into Training and Testing In order to test our algorithm, we'll split the data into a Training and a Testing set. The size of the testing set will be 10% of the total data.<jupyter_code>sample = np.random.choice(processed_data.index, size=int(len(processed_data)*0.9), replace=False)
train_data, test_data = processed_data.iloc[sample], processed_data.drop(sample)

print("Number of training samples is", len(train_data))
print("Number of testing samples is", len(test_data))
print(train_data[:10])
print(test_data[:10])<jupyter_output>Number of training samples is 360
Number of testing samples is 40
     admit    gre     gpa  rank_1  rank_2  rank_3  rank_4
21       1  0.825  0.9075       0       1       0       0
362      0  0.850  0.7850       0       1       0       0
290      1  0.775  0.8625       0       1       0       0
151      0  0.500  0.8450       0       1       0       0
97       0  0.600  0.8925       0       1       0       0
10       0  1.000  1.0000       0       0       0       1
0        0  0.475  0.9025       0       0       1       0
70       0  0.800  1.0000       0       0       1       0
222      1  0.600  0.7550       1       0       0       0
165      0  0.875  1.0000       1       0       0       0
    admit    gre     gpa  rank_1  rank_2  rank_3  rank_4
25      1  1.000  0.9150       1       0       0       0
35      0  0.500  0.7625       0       1       0       0
44      0  0.875  0.7350       0       1       0       0
62      0  0.800  0.9175       0       0       1       0
71      0  [...]<jupyter_text>Splitting the data into features and targets (labels)Now, as a final step before the training, we'll split the data into features (X) and targets (y).<jupyter_code>features = train_data.drop('admit', axis=1)
targets = train_data['admit']
features_test = test_data.drop('admit', axis=1)
targets_test = test_data['admit']

print(features[:10])
print(targets[:10])<jupyter_output>gre     gpa  rank_1  rank_2  rank_3  rank_4
21   0.825  0.9075       0       1       0       0
362  0.850  0.7850       0       1       0       0
290  0.775  0.8625       0       1       0       0
151  0.500  0.8450       0       1       0       0
97   0.600  0.8925       0       1       0       0
10   1.000  1.0000       0       0       0       1
0    0.475  0.9025       0       0       1       0
70   0.800  1.0000       0       0       1       0
222  0.600  0.7550       1       0       0       0
165  0.875  1.0000       1       0       0       0
21     1
362    0
290    1
151    0
97     0
10     0
0      0
70     0
222    1
165    0
Name: admit, dtype: int64<jupyter_text>Training the 2-layer Neural NetworkThe following function trains the 2-layer neural network. First, we'll write some helper functions.<jupyter_code># Activation (sigmoid) function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
def sigmoid_prime(x):
    return sigmoid(x) * (1-sigmoid(x))
def error_formula(y, output):
    return - y*np.log(output) - (1 - y) * np.log(1-output)<jupyter_output><empty_output><jupyter_text>TODO: Backpropagate the errorNow it's your turn to shine. Write the error term. Remember that this is given by the equation $$ -(y-\hat{y}) \sigma'(x) $$<jupyter_code># TODO: Write the error term formula
def error_term_formula(y, output):
    return (y - output) * output * (1 -output)
# Neural Network hyperparameters
epochs = 1000
learnrate = 0.5

# Training function
def train_nn(features, targets, epochs, learnrate):
    
    # Use to same seed to make debugging easier
    np.random.seed(42)

    n_records, n_features = features.shape
    last_loss = None

    # Initialize weights
    weights = np.random.normal(scale=1 / n_features**.5, size=n_features)
    print(weights)
    for e in range(epochs):
        del_w = np.zeros(weights.shape)
        for x, y in zip(features.values, targets):
            # Loop through all records, x is the input, y is the target

            # Activation of the output unit
            #   Notice we multiply the inputs and the weights here 
            #   rather than storing h as a separate variable 
            output = sigmoid(np.dot(x, weights))

            # The error, the target minus the network output
            error = error_formula(y, output)

            # The error term
            #   Notice we calulate f'(h) here instead of defining a separate
            #   sigmoid_prime function. This just makes it faster because we
            #   can re-use the result of the sigmoid function stored in
            #   the output variable
            error_term = error_term_formula(y, output)

            # The gradient descent step, the error times the gradient times the inputs
            del_w += error_term * x

        # Update the weights here. The learning rate times the 
        # change in weights, divided by the number of records to average
        weights += learnrate * del_w / n_records

        # Printing out the mean square error on the training set
        if e % (epochs / 10) == 0:
            out = sigmoid(np.dot(features, weights))
            loss = np.mean((out - targets) ** 2)
            print("Epoch:", e)
            if last_loss and last_loss < loss:
                print("Train loss: ", loss, "  WARNING - Loss Increasing")
            else:
                print("Train loss: ", loss)
            last_loss = loss
            print("=========")
    print("Finished training!")
    return weights
    
weights = train_nn(features, targets, epochs, learnrate)<jupyter_output>[ 0.2027827  -0.05644616  0.26441774  0.62177434 -0.09559271 -0.09558601]
Epoch: 0
Train loss:  0.2738355672541657
=========
Epoch: 100
Train loss:  0.21402975039208713
=========
Epoch: 200
Train loss:  0.21274061599853877
=========
Epoch: 300
Train loss:  0.21206898533027688
=========
Epoch: 400
Train loss:  0.21166119941618322
=========
Epoch: 500
Train loss:  0.2113716354844602
=========
Epoch: 600
Train loss:  0.21113847380975226
=========
Epoch: 700
Train loss:  0.21093464719900842
=========
Epoch: 800
Train loss:  0.21074775281229216
=========
Epoch: 900
Train loss:  0.2105718023522165
=========
Finished training!<jupyter_text>Calculating the Accuracy on the Test Data<jupyter_code># Calculate accuracy on test data
tes_out = sigmoid(np.dot(features_test, weights))
predictions = tes_out > 0.5
accuracy = np.mean(predictions == targets_test)
print("Prediction accuracy: {:.3f}".format(accuracy))
x_test_eval = [20,2.2,0,0,0,1]
out_test = sigmoid(np.dot(x_test_eval, weights))
display(out_test)<jupyter_output><empty_output><jupyter_text>Another Implementation<jupyter_code>import numpy as np
from data_prep import features, targets, features_test, targets_test


def sigmoid(x):
    """
    Calculate sigmoid
    """
    return 1 / (1 + np.exp(-x))

# TODO: We haven't provided the sigmoid_prime function like we did in
#       the previous lesson to encourage you to come up with a more
#       efficient solution. If you need a hint, check out the comments
#       in solution.py from the previous lecture.
def sigmoid_prime(x):
    sig = sigmoid(x)
    return sig * (1-sig)
    
# Use to same seed to make debugging easier
np.random.seed(42)

n_records, n_features = features.shape
last_loss = None

# Initialize weights
weights = np.random.normal(scale=1 / n_features**.5, size=n_features)

# Neural Network hyperparameters
epochs = 10000
learnrate = 0.05

for e in range(epochs):
    del_w = np.zeros(weights.shape)
    for x, y in zip(features.values, targets):
        # Loop through all records, x is the input, y is the target

        # Note: We haven't included the h variable from the previous
        #       lesson. You can add it if you want, or you can calculate
        #       the h together with the output

        # TODO: Calculate the output
        output = sigmoid(np.dot(x, weights))

        # TODO: Calculate the error
        error = y - output

        # TODO: Calculate the error term
        error_term = error * output * (1 - output)

        # TODO: Calculate the change in weights for this sample
        #       and add it to the total weight change
        del_w += error_term * x

    # TODO: Update weights using the learning rate and the average change in weights
    weights += learnrate *del_w /n_records

    # Printing out the mean square error on the training set
    if e % (epochs / 10) == 0:
        out = sigmoid(np.dot(features, weights))
        loss = np.mean((out - targets) ** 2)
        if last_loss and last_loss < loss:
            print("Train loss: ", loss, "  WARNING - Loss Increasing")
        else:
            print("Train loss: ", loss)
        last_loss = loss


# Calculate accuracy on test data
tes_out = sigmoid(np.dot(features_test, weights))
predictions = tes_out > 0.5
accuracy = np.mean(predictions == targets_test)
print("Prediction accuracy: {:.3f}".format(accuracy))<jupyter_output>Train loss:  0.2640432346990608
Train loss:  0.20950679783595352
Train loss:  0.20090206502342442
Train loss:  0.1986432137900258
Train loss:  0.19780830304192185
Train loss:  0.197430844922458
Train loss:  0.19723793934327805
Train loss:  0.1971311721534873
Train loss:  0.19706873643267633
Train loss:  0.19703074987657476
Prediction accuracy: 0.725
{'fraction_non_alphanumeric': 0.09717382888114595, 'fraction_numerical': 0.07859078590785908, 'mean_word_length': 2.7383502170767002, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '446698', 'license': 'MIT', 'n_tokens_mistral': 4776, 'n_tokens_neox': 4157, 'n_words': 1462, 'path': 'student-admissions/StudentAdmissions.ipynb', 'repo_name': 'AlAlaa-Tashkandi/deep-learning', 'size': '12915'}