<jupyter_start><jupyter_text>> With Factorization Machines, multi-dimensional, feature-rich databases can be used and meta data can be used for finding out useful features as well.>Our dataset is fairly sparse as even bigger datasets have been reduced so as to make them easier to work with.  Collaborative Filtering for generating User and Item Features using Matrix Decomposition<jupyter_code>import numpy as np
import pandas as pd
import tensorflow as tf
import matplotlib.pyplot as plt
from IPython.display import display, HTML<jupyter_output><empty_output><jupyter_text>Data[MovieLens 100K Movie Recommendation Data](https://grouplens.org/datasets/movielens/100k/) Second Order Factorization Model$$ \widehat{y}\left ( x \right ) = w_{0}  + \sum_{i=1}^{n}w_{i}x_{i} + \sum_{i=1}^{n}\sum_{j=i+1}^{n}x_{i}x_{j} $$>v : k dimensional latent vectors associated with each variable>w0 : global bias>wi : user-item specific bias>xi xj : user-item interaction  Ratings$$ \hat{r} = \vec{u}^t \vec{p} $$>The ratings in this model have been taken as the inner product of the two vectors. The User Vector 'u' and the Vector with the Latent Values.  Get the Dataset<jupyter_code>MovieLensData = pd.read_csv("../Datasets/MovieLens/ml-100k/ml-100k/u.data", delimiter="\t",names = ["user","item","rate","time"])<jupyter_output><empty_output><jupyter_text>>Here we are using the GoupLens MovieLens Dataset which contains data on different users, and different movies and the ratings each of the users have given to individual movies.   Instantiate Session for running the TensorFlow Graph<jupyter_code>sess = tf.Session()
init = tf.initialize_all_variables()<jupyter_output>WARNING:tensorflow:From <ipython-input-3-c3c256f2e5cc>:2 in <module>.: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.<jupyter_text>Create Sample for Training the Model > Here we are considering the training size as 70% of the datasets on top of which the predictions would take place.> We are taking 70% of the values withouit replacement from the dataset at random.<jupyter_code>train_size = np.random.rand(len(MovieLensData)) < 0.7
_train = MovieLensData[train_size]<jupyter_output><empty_output><jupyter_text>Create lists for the User Ids and Item Ids and the respective ratings attributed<jupyter_code>user_indecies = [x-1 for x in _train.user.values]
item_indecies = [x-1 for x in _train.item.values]
rates = _train.rate.values<jupyter_output><empty_output><jupyter_text>Defining the maximum nuber of features to be extracted<jupyter_code>feature_len = 10<jupyter_output><empty_output><jupyter_text>Defining New User and Item Matrices with their vectors as the feature vectors.>In the case of the User Matrix we have the Feature Column.>In the case of the Item Matrix we have the Feature Row.> tf.truncated_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)>Outputs random values from a truncated normal distribution.>The truncated normal distribution is the probability distribution of a normally distributed random variable whose value is either bounded below or above (or both).<jupyter_code>U = tf.Variable(initial_value=tf.truncated_normal([943,feature_len]), name='user')
P = tf.Variable(initial_value=tf.truncated_normal([feature_len,1682]), name='item')<jupyter_output><empty_output><jupyter_text>Defining the result >This yields us the ratings obtained after defining the features.<jupyter_code>result = tf.matmul(U, P)<jupyter_output><empty_output><jupyter_text>Flattening the "result" Matrix> Flattening Example>tensor 't' [[[1, 1, 1], [2, 2, 2]], [[3, 3, 3],[4, 4, 4]],[[5, 5, 5],[6, 6, 6]]]>tensor 't' has shape [3, 2, 3]>pass '[-1]' to flatten 't'>reshape(t, [-1]) ==> [1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6] tf.gather(params, indices, validate_indices=None, name=None)>Gather slices from params according to indices>indices must be an integer tensor of any dimension (usually 0-D or 1-D). Produces an output tensor with shape indices.shape + params.shape[1:]<jupyter_code>result_flatten = tf.reshape(result, [-1])

R = tf.gather(result_flatten, user_indecies * tf.shape(result)[1] + item_indecies, name='extracting_user_rate')<jupyter_output><empty_output><jupyter_text>Regularization and the Loss Function$$ min\sum_{i=1}^{n}V(f(\hat{x}_i)f(\hat{y}_i)) + \lambda P(f) $$for a loss function $$ {\displaystyle V} $$ that describes the cost of predicting $$ {\displaystyle f(x)} $$ when the label is $$ {\displaystyle y} $$, such as the square loss or hinge loss, and for the term $$ {\displaystyle \lambda } $$ which controls the importance of the regularization term. $$ {\displaystyle P(f)} $$ is typically a penalty on the complexity of $$ {\displaystyle f} $$, such as restrictions for smoothness or bounds on the vector space norm. Difference between the Random Values and the Actual Values<jupyter_code>diff_op = tf.sub(R, rates, name='trainig_diff')
diff_op_squared = tf.abs(tf.square(diff_op, name="squared_difference"))<jupyter_output><empty_output><jupyter_text>Computing Base Cost$$  min\sum_{i=1}^{n}V(f(R)f(rates)) $$<jupyter_code>base_cost = tf.reduce_sum(diff_op_squared, name="sum_squared_error")<jupyter_output><empty_output><jupyter_text>Computing Norm and defining lambdaNorm : $$  f $$ Lambda: $$ \lambda$$<jupyter_code>lda = tf.constant(.001, name='lambda') #lambda
norm_sums = tf.add(
    tf.reduce_sum(tf.abs(U, name='user_abs'), name='user_norm'), 
    tf.reduce_sum(tf.abs(P, name='item_abs'), name='item_norm')) # sum of the U and P Matrices taken as norms<jupyter_output><empty_output><jupyter_text>Computing Regularization Function $$ \lambda P(f) $$<jupyter_code>regularizer = tf.mul(norm_sums, lda, 'regularizer') #nultiply the P(f) and lambda<jupyter_output><empty_output><jupyter_text>Obtaining the Final Loss Function$$\sum_{i=1}^{n}V(R)f(rates) + \lambda P(f)$$<jupyter_code>cost = tf.add(base_cost, regularizer)<jupyter_output><empty_output><jupyter_text>Learning/Training Model Defining the rate at which the model learns>Here, we can see that the exponential decay function is used for the learning rate as we don't want the model to keep learning continuously.>We have tuned the decay model such that the model decays for the next 10000 steps with a decay rate of 0.96 from the base learning rate of 0.001<jupyter_code>lr = tf.constant(.001, name='learning_rate')
global_step = tf.Variable(0, trainable=False)
learning_rate = tf.train.exponential_decay(lr, global_step, 10000, 0.96, staircase=True)<jupyter_output><empty_output><jupyter_text>Defining the Optimization Method for the Learning Model> Gradient Descent Optimizer>Gradient descent is an optimization algorithm used to find the values of parameters (coefficients) of a function (f) that minimizes a cost function (cost).>It is best used when the parameters cannot be calculated analytically (e.g. using linear algebra) and must be searched for by an optimization algorithm.<jupyter_code>optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_step = optimizer.minimize(cost, global_step=global_step)<jupyter_output><empty_output><jupyter_text>Final Optimized Cost Function$$ min\sum_{i=1}^{n}V(f(R)f(rates)) + \lambda P(f) $$After going through all the above steps we have arrived at the preferred equation which would be an instance of training for our Learning Model.  Initializing all the Variables>In Tensorflow none of the objects or variables output anyting till they have been initialized. After they are initialized, only then are they stored as TF operations and only then after that they can be evaluated within the session.<jupyter_code>init = tf.initialize_all_variables()
sess.run(init)<jupyter_output>WARNING:tensorflow:From <ipython-input-17-8b3cf5fafc87>:1 in <module>.: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.<jupyter_text>Displaying Matrices > Here we can see that the U and P, the User and Item Matrices respectively, which have been formed through Matrix Decomposition have inner vectors which are of size 10.> These 10 vectors are the 10 feature vectors which define 10 quantified features for each individual Item in the Item Matrix and each individual User in the User Matrix.<jupyter_code>display(sess.run(U))
display(sess.run(tf.shape(U)))

display(sess.run(P))
display(sess.run(tf.shape(P)))

display(sess.run(result))
display(sess.run(tf.shape(result)))<jupyter_output><empty_output><jupyter_text>Running the Training Model<jupyter_code>costs = []
costs2 = []
actual_rate = []
predicted_rates = []
for epochs in range(100):
    for i in range(epochs):
        sess.run(training_step)
    
    u, p, r = MovieLensData[['user', 'item', 'rate']].values[7]
    rhat = tf.gather(tf.gather(result, u-1), p-1)
    print( "rating for user " + str(u) + " for item " + str(p) + 
          " is " + str(r) + " and our prediction is: " + str(sess.run(rhat)))
    actual_rate.append(r)
    predicted_rates.append(sess.run(rhat))

    df_test = MovieLensData[~train_size]
    user_indecies_test = [x-1 for x in df_test.user.values]
    item_indecies_test = [x-1 for x in df_test.item.values]
    rates_test = df_test.rate.values
    R_test = tf.gather(result_flatten, user_indecies_test * tf.shape(result)[1] + item_indecies_test, name='extracting_user_rate_test')
    diff_op_test = tf.sub(R_test, rates_test, name='test_diff')
    diff_op_squared_test = tf.abs(diff_op, name="squared_difference_test")

    cost_test = tf.div(tf.reduce_sum(tf.square(diff_op_squared_test, name="squared_difference_test"), name="sum_squared_error_test"), df_test.shape[0] * 2, name="average_error")
    costs.append(sess.run(cost_test))
    cost_test2 = tf.div(tf.reduce_sum((diff_op_squared_test), name="sum_squared_error_test"), df_test.shape[0] * 2, name="average_error")
    costs2.append(sess.run(cost_test2))
    print(sess.run(cost_test))<jupyter_output>rating for user 253 for item 465 is 5 and our prediction is: 1.91077
22.8756
rating for user 253 for item 465 is 5 and our prediction is: 1.81087
18.146
rating for user 253 for item 465 is 5 and our prediction is: 2.13749
15.0613
rating for user 253 for item 465 is 5 and our prediction is: 3.99948
6.01734
rating for user 253 for item 465 is 5 and our prediction is: 4.13264
2.40418
rating for user 253 for item 465 is 5 and our prediction is: 3.69467
2.73345
rating for user 253 for item 465 is 5 and our prediction is: 2.88029
4.24694
rating for user 253 for item 465 is 5 and our prediction is: 3.56331
3.63945
rating for user 253 for item 465 is 5 and our prediction is: 3.64076
3.27738
rating for user 253 for item 465 is 5 and our prediction is: 3.01023
5.03861
rating for user 253 for item 465 is 5 and our prediction is: 3.02325
4.13409
rating for user 253 for item 465 is 5 and our prediction is: 3.58138
2.33654
rating for user 253 for item 465 is 5 and our prediction is: 3.62383
2.26133
[...]<jupyter_text>Plotting Results for Evaluation<jupyter_code>plt.plot(costs2)

plt.xlabel("No. of Epochs Trained for")
plt.ylabel("Cost")
plt.show()

plt.plot(costs)

plt.xlabel("No. of Epochs Trained for")
plt.ylabel("Squared Cost")
plt.show()

plt.plot(actual_rate)
plt.plot(predicted_rates)

plt.xlabel("No. of Epochs Trained for")
plt.ylabel("Rating Prediction")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10840438489646773, 'fraction_numerical': 0.03662780581172786, 'mean_word_length': 5.842261904761905, 'pattern_counts': {'":': 0, '<': 83, '<?xml version=': 0, '>': 114, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '574029', 'license': 'Apache-2.0', 'n_tokens_mistral': 4079, 'n_tokens_neox': 3688, 'n_words': 1203, 'path': 'Notebooks/Factorization Machines.ipynb', 'repo_name': 'Vibzy19/recomsys', 'size': '11494'}