<jupyter_start><jupyter_text>First Attempt implementing FC neural network in PyTorch<jupyter_code>import torch # God please write code for me<jupyter_output><empty_output><jupyter_text>Problem DescriptionThe aim of the model is to estimate the [IMDB](https://www.imdb.com) score of the a movie, based on:- Name- Release Date- Production Budget- Genre- Worldwide Gross- Run TimeWe then try to identify this relationship.$$\left.\begin{array}{l}\text{Film Name} \\\text{Release Date} \\\text{Production Budget} \\\text{Genre} \\\text{Worldwide Gross} \\\text{Run Time}\end{array}\right\}\Rightarrow\text{IMDB Rating}$$  Data SourceThe data source comes from [here](https://vega.github.io/editor/data/movies.json), licensed under MIT License. PreprocessingFor preprocessing, I used a combination of Lua and Python. FilterThe original JSON file has more than 3000 films, but many of which has missing information.The `preprocess.lua` **filters out all entries without any one of the following parameters**:- Film Name (`Title`)- Release Date (`Release Date`)- Production Budget (`Production Budget`)- Genre (`Main Genre`)- Worldwide Gross (`Worldwide Gross`)- Run Time (`Running Time min`)- IMDB Rating (`IMBD Rating`)After filtering, **1141 films** survived with **12 different genres**:```json["Action","Black Comedy","Comedy","Adventure","Drama","Romantic Comedy","Horror","Thriller/Suspense","Musical","Documentary","Western","Concert/Performance"]```The list of movies that survived is stored in `movies_processed.json`.The list of genres are stored in `genres.json`.  EncodingBefore feeding the data into the FC neural network, we first need to find a way to encode the data into a list of numbers.<jupyter_code>import json, os

with open("genres.json") as f:
    Genres = json.loads(f.read())
with open("movies_processed.json") as f:
    Movies = json.loads(f.read())
print("%d movies found with %d genres." % (len(Movies), len(Genres)))<jupyter_output>1140 movies found with 12 genres.<jupyter_text>Input and OutputWe'll input all information of a movie into a FC network. But before doing so, proper encoding is necessary. Encoding<jupyter_code>import numpy as np
EncodedInput = []

# Titles: stats
maxLen = -1
maxAscii = -1
minAscii = np.Infinity
for i in Movies:
    t = str(i['Title'])
    if len(t) > maxLen:
        maxLen = len(t)
    for char in t:
        if ord(char) < minAscii:
            minAscii = ord(char)
            print("MIN @", char, "({0})".format(ord(char)))
        if ord(char) > maxAscii:
            maxAscii = ord(char)
            print("MAX @", char, "({0})".format(ord(char)))
print('maxLen = {0}\nmaxAscii = {1}\nminAscii = {2}'.format(maxLen, maxAscii, minAscii))<jupyter_output>MIN @ B (66)
MAX @ B (66)
MAX @ r (114)
MIN @   (32)
MAX @ w (119)
MAX @ z (122)
MAX @ È (200)
MAX @ ‡ (8225)
maxLen = 58
maxAscii = 8225
minAscii = 32<jupyter_text>In the above snippet we found that the maximum length for a title is 58. Several strange symbols are found in addition to regular `(space)` to `z` in ASCII representations. To encode the title, any character with `ord(char) > 122 or ord(char) < 32` will be **clamped** into the range.The following code implements such encoding.<jupyter_code># Clamping function
def clamp(val, valMin, valMax):
    if val > valMax:
        return valMax, True
    elif val < valMin:
        return valMin, True
    else:
        return val, False

# ASCII encode
TitleEncoded = list(list(clamp(ord(char), 32, 122)[0] for char in str(i["Title"])) for i in Movies)
# Extending to maxLen
TitleEncoded = list(i + [0] * (maxLen - len(i)) for i in TitleEncoded)

# Confirmation
print(Movies[0]["Title"], TitleEncoded[0])<jupyter_output>Broken Arrow [66, 114, 111, 107, 101, 110, 32, 65, 114, 114, 111, 119, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]<jupyter_text>The title in its ASCII form will be given as input to the neural network in **an array of 58 integers** ranging from 32 to 122.---Before moving on, define the following normalization function first. It maps an array of data into an array of float between `normMin` and `normMax`.<jupyter_code>def normalize(arr, normMin, normMax, inMin = -np.inf, inMax = np.inf):
    if inMax == np.inf and inMin == -np.inf:
        inMin, inMax = min(arr), max(arr)
    return list(map(lambda val: (val - inMin) / (inMax - inMin) * (normMax - normMin) + normMin, arr))

print(TitleEncoded[0])
TitleEncoded = list(normalize(titleEncoded, 0, 1, 0, 122) for titleEncoded in TitleEncoded)
print(Movies[0]["Title"], TitleEncoded[0])
# Add to EncodedInput
EncodedInput.append(TitleEncoded)<jupyter_output>[66, 114, 111, 107, 101, 110, 32, 65, 114, 114, 111, 119, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Broken Arrow [0.5409836065573771, 0.9344262295081968, 0.9098360655737705, 0.8770491803278688, 0.8278688524590164, 0.9016393442622951, 0.26229508196721313, 0.5327868852459017, 0.9344262295081968, 0.9344262295081968, 0.9098360655737705, 0.9754098360655737, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]<jupyter_text>Then we'll move on to release dates.<jupyter_code>import time

# Example: Jul 25 2008
Timestamps = list(time.mktime(time.strptime(i["Release Date"], "%b %d %Y")) / 100 for i in Movies)
print("Release Date Timestamps ranges from", min(Timestamps), "to", max(Timestamps))<jupyter_output>Release Date Timestamps ranges from 3296736.0 to 22074912.0<jupyter_text>In the above snippet we found that the time starts with a big integer and has a rather big range. To normalize the time (foundamentally logical because time is consecutive), we set a linear normalization function to clamp all ranges into $[0,\,1]$.<jupyter_code>NormalizedTimestamps = normalize(Timestamps, 0, 1)
# Add to EncodedInput
EncodedInput.append(list([i] for i in NormalizedTimestamps))<jupyter_output><empty_output><jupyter_text>The timestamps takes **1** float value when given as input to the neural network.<jupyter_code>NormalizedProductionBudgets = normalize(list(i["Production Budget"] for i in Movies), 0, 1)
print(NormalizedProductionBudgets[0], Movies[0]["Production Budget"])
EncodedInput.append(list([i] for i in NormalizedProductionBudgets))
def getGenreId(str):
    return Genres.index(str)
GenreId = list(map(getGenreId, list(i["Major Genre"] for i in Movies)))
print(GenreId[0], Movies[0]["Major Genre"])
EncodedInput.append(list([i / len(Genres)] for i in GenreId))
NormalizedWorldwideGross = normalize(list(i["Worldwide Gross"] for i in Movies), 0, 1)
print(NormalizedWorldwideGross[0], Movies[0]["Worldwide Gross"])
EncodedInput.append(list([i] for i in NormalizedWorldwideGross))
# EncodedInput.append(list([i["Running Time min"]] for i in Movies))
NormalizedRunTime = normalize(list(i["Running Time min"] for i in Movies), 0, 1)
print(NormalizedRunTime[0], Movies[0]["Running Time min"])
EncodedInput.append(list([i] for i in NormalizedRunTime))<jupyter_output>0.3522727272727273 108<jupyter_text>Before finishing, let's check the final results in `EncodedInput`:<jupyter_code>print(len(EncodedInput))
print(list(i[0] for i in EncodedInput), Movies[0])<jupyter_output>6
[[0.5409836065573771, 0.9344262295081968, 0.9098360655737705, 0.8770491803278688, 0.8278688524590164, 0.9016393442622951, 0.26229508196721313, 0.5327868852459017, 0.9344262295081968, 0.9344262295081968, 0.9098360655737705, 0.9754098360655737, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.26313610011962824], [0.21664838846239745], [0.0], [0.08049054258816266], [0.3522727272727273]] {'IMDB Votes': 33584, 'Production Budget': 65000000, 'IMDB Rating': 5.8, 'MPAA Rating': 'R', 'Rotten Tomatoes Rating': 55, 'Distributor': '20th Century Fox', 'Director': '<NAME>', 'Creative Type': 'Contemporary Fiction', 'US Gross': 70645997, 'Major Genre': 'Action', 'Worldwide Gross': 148345997, 'Source': 'Original Screenplay', 'Release Date': 'Feb 09 1996', 'Running Time min': 108, 'Title': 'Broken Arrow'}<jupyter_text>Now we can generate the encoded input for all movies.<jupyter_code>from functools import reduce

def concat(list1, list2):
    list1.extend(list2)
    return list1

EncodedInput = list(reduce(concat, list(i[j] for i in EncodedInput)) for j in range(0, len(Movies)))
pointCount = len(EncodedInput)
print(EncodedInput[0], Movies[0])
inputDimension = len(EncodedInput[0])<jupyter_output>[0.5409836065573771, 0.9344262295081968, 0.9098360655737705, 0.8770491803278688, 0.8278688524590164, 0.9016393442622951, 0.26229508196721313, 0.5327868852459017, 0.9344262295081968, 0.9344262295081968, 0.9098360655737705, 0.9754098360655737, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.26313610011962824, 0.21664838846239745, 0.0, 0.08049054258816266, 0.3522727272727273] {'IMDB Votes': 33584, 'Production Budget': 65000000, 'IMDB Rating': 5.8, 'MPAA Rating': 'R', 'Rotten Tomatoes Rating': 55, 'Distributor': '20th Century Fox', 'Director': '<NAME>', 'Creative Type': 'Contemporary Fiction', 'US Gross': 70645997, 'Major Genre': 'Action', 'Worldwide Gross': 148345997, 'Source': 'Original Screenplay', 'Release Date': 'Feb 09 1996', 'Running Time min': 108, 'Title': 'Broken Arrow'}<jupyter_text>GroundtruthsThe groundtruth in this case is the IMDB rating of the 1140 movies. Simply compile the groundtruths based on the movies.<jupyter_code>GroundTruth = normalize(list(round(i["IMDB Rating"]) for i in Movies), 0, 10, 0, 10)
assert(len(EncodedInput) == len(GroundTruth))

def toOutTensor(result):
    t = [0] * 10
    t[int(result)] = 1
    return t

GroundTruth = list(map(toOutTensor, GroundTruth))
Source = list((EncodedInput[i], GroundTruth[i]) for i in range(len(GroundTruth)))<jupyter_output><empty_output><jupyter_text>Constructing Neural NetworkNow it's time to make a neural network using basic PyTorch functions!Among the 1140 movies, ID 1-1000 is selected as *training set*, with batch size 25 (40 batches). ID 1001-1100 is *comfirmation set*, with batch size 100 (1 batch). ID 1101-1140 is the *test set*.<jupyter_code># Shuffle sets
def shuffleSource():
    np.random.shuffle(Source)
shuffleSource()

# Configure batch size and count
batchSize = 25
batchCount = 40

def getBatch(i):
    if 25 * i > 1000:
        assert()
    else:
        return Source[batchSize * i:batchSize * (i + 1)]

def separateTensorsFromBatch(batch):
    return torch.tensor(list(i[0] for i in batch)), torch.tensor(list(i[1] for i in batch))
dtype = torch.float
device = torch.device("cpu")

# D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
D_in, H, D_out = inputDimension, 100, 10

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)
w2 = torch.randn(H, D_out, device=device, dtype=dtype)

learning_rate = 7.1e-5

import torch.nn.functional as F

for epoch in range(300):
#     print("------- EPOCH", epoch, "-------")
    shuffleSource()
    for batchId in range(batchCount):
        trainIn, trainOut = separateTensorsFromBatch(getBatch(batchId))
        # Forward pass: compute predicted y
        h = trainIn.mm(w1)
        h_relu = h.clamp(min=0)
        y_pred = h_relu.mm(w2)
        y_pred_sm = F.softmax(y_pred, dim=1)
        
#       print("Output at case 0:", y_pred[0])
#       print("Expected output at case 0:", trainOut[0])
#       Compute and print loss
        loss = (y_pred_sm - trainOut).pow(2).sum().item()
        
        # Backprop to compute gradients of w1 and w2 with respect to loss
        grad_y_pred = 2.0 * (y_pred - trainOut)
        grad_w2 = h_relu.t().mm(grad_y_pred)
        grad_h_relu = grad_y_pred.mm(w2.t())
        grad_h = grad_h_relu.clone()
        grad_h[h < 0] = 0
        grad_w1 = trainIn.t().mm(grad_h)

        # Update weights using gradient descent
        w1 -= learning_rate * grad_w1
        w2 -= learning_rate * grad_w2
def pred(inputArr):
    inputTensor = torch.tensor([inputArr])
    # Forward pass: compute predicted y
    h = inputTensor.mm(w1)
    h_relu = h.clamp(min=0)
    y_pred = h_relu.mm(w2)
    y_pred_sm = F.softmax(y_pred, dim=1)
    arr = list(float(i) for i in y_pred_sm[0])
    return arr.index(max(arr)) + 1, max(arr)

correct = 0
case = 0

for i in range(1000, 1140):
    case += 1
    p, q = pred(Source[i][0])
    a = Source[i][1].index(1)
    correct += (p==a)
    if case != 0:
        print("%d: %d (%d) Confidence %.2f%% -> %.2f%%" % (i, p, a, q*100, correct/case * 100))<jupyter_output>1000: 7 (7) Confidence 12.07% -> 100.00%
1001: 7 (7) Confidence 12.25% -> 100.00%
1002: 7 (4) Confidence 11.85% -> 66.67%
1003: 7 (7) Confidence 11.94% -> 75.00%
1004: 8 (7) Confidence 12.63% -> 60.00%
1005: 7 (6) Confidence 16.31% -> 50.00%
1006: 7 (8) Confidence 11.74% -> 42.86%
1007: 7 (6) Confidence 12.59% -> 37.50%
1008: 7 (6) Confidence 12.19% -> 33.33%
1009: 7 (8) Confidence 13.60% -> 30.00%
1010: 8 (7) Confidence 13.40% -> 27.27%
1011: 7 (5) Confidence 15.84% -> 25.00%
1012: 7 (7) Confidence 12.77% -> 30.77%
1013: 7 (4) Confidence 13.02% -> 28.57%
1014: 7 (7) Confidence 11.86% -> 33.33%
1015: 7 (8) Confidence 12.31% -> 31.25%
1016: 7 (6) Confidence 11.66% -> 29.41%
1017: 7 (7) Confidence 12.71% -> 33.33%
1018: 7 (8) Confidence 13.44% -> 31.58%
1019: 7 (6) Confidence 11.46% -> 30.00%
1020: 7 (7) Confidence 10.91% -> 33.33%
1021: 7 (7) Confidence 11.61% -> 36.36%
1022: 7 (6) Confidence 12.55% -> 34.78%
1023: 7 (6) Confidence 13.91% -> 33.33%
1024: 7 (6) Confidence 12.59% -> 32.00[...]
{'fraction_non_alphanumeric': 0.15516505406943654, 'fraction_numerical': 0.1368099032441662, 'mean_word_length': 4.202442635085122, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 73, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '293538', 'license': 'MIT', 'n_tokens_mistral': 6686, 'n_tokens_neox': 5340, 'n_words': 1615, 'path': 'PROJECT_IMDB/main.ipynb', 'repo_name': 'fanqiNO1/SJTU-AI', 'size': '14060'}