<jupyter_start><jupyter_text>Kaggle Competition Important Note: Try to make sure in each cell, if you are modifying/altering the variables, a copy of the previous cell's variables should be used instead. This is so that when something needs to be done again midway, you don't have to run the whole notebook from the beginning. All Imports<jupyter_code>import pandas as pd
import numpy as np
import re<jupyter_output><empty_output><jupyter_text>Input<jupyter_code>#movies.txtz
movies = pd.read_csv("./movies.txt", sep="::", names=["MovieID", "Title", "Genre"], engine = "python")
print(movies.shape)
movies.head()
#Code reserve, for preprocessing.
dict_age = {"1": "Under 18",
            "18": "18-24",
            "25": "25-34",
            "35": "35-44",
            "45": "45-49",
            "50": "50-55",
            "56": "56+"
           }
dict_gender = {"M": "Male",
               "F": "Female"
              }
dict_occupation = {
    "0": "other or not specified",
    "1": "academic/educator",
    "2": "artist",
    "3": "clerical/admin",
    "4": "college/grad student",
    "5": "customer service",
    "6": "doctor/health care",
    "7": "executive/managerial",
    "8": "farmer",
    "9": "homemaker",
    "10": "K-12 student",
    "11": "lawyer",
    "12": "programmer",
    "13": "retired",
    "14": "sales/marketing",
    "15": "scientist",
    "16": "self-employed",
    "17": "technician/engineer",
    "18": "tradesman/craftsman",
    "19": "unemployed",
    "20": "writer"
}
#users.txt
users = pd.read_csv("./users.txt", sep="::", names=["UserID", "Gender", "Age", "Occupation", "Zip-code"], engine = "python")
print(users.shape)
users.head()
#training data
train = pd.read_csv("./training.txt", names=["UserID", "MovieID", "Rating", "Timestamp"]) #sep = ','
print(train.shape)
train.head()
#testing data
test = pd.read_csv("./testing.txt", names=["UserID", "MovieID", "Timestamp"]) #sep = ','
print(test.shape)
test.head()<jupyter_output><empty_output><jupyter_text>Data Exploration and Manipulation In this section, we explore the kinds of data, make meaningful (or meaningless) analysis, and do a bit of changing with the data. Preprocessing is for later, where we modify the structure of data and/or alter the rows in data to make it suitable for feeding to our prediction algorithms. Will use dummytrain variables as to not change the original dataframe, unless necessary. Change Timestamp to Meaningful Format<jupyter_code>dummytrain1 = train.copy()
dummytrain1['Timestamp'] = pd.to_datetime(train['Timestamp'], unit='s')
dummytrain1.head()
dummytrain2 = dummytrain1.copy()
dummytrain2['year'] = pd.DatetimeIndex(dummytrain2['Timestamp']).year
dummytrain2['month'] = pd.DatetimeIndex(dummytrain2['Timestamp']).month
dummytrain2['day'] = pd.DatetimeIndex(dummytrain2['Timestamp']).day
dummytrain2['hour'] = pd.DatetimeIndex(dummytrain2['Timestamp']).hour
dummytrain2 = dummytrain2.drop(columns = ['Timestamp'])
dummytrain2.head()<jupyter_output><empty_output><jupyter_text>Seperating Genres, then one-hot encoding<jupyter_code>movievariable = movies.copy()
#After turning the column 'Genre' into a list of genres, uncomment code below
'''
movies_genre = pd.concat([movievariable,pd.get_dummies(movievariable['Genre'].apply(pd.Series).stack()).sum(level=0)],axis=1)
movies_genre.drop('Genre',axis=1,inplace=True)
movies_genre.head()
'''
print()<jupyter_output><jupyter_text>List of movies unrated/unwatched<jupyter_code>result_set = set(map(int, movies['MovieID'].unique().tolist()))
movies_set = set(range(1, 3883))
print(sorted(movies_set - result_set))<jupyter_output>[91, 221, 323, 622, 646, 677, 686, 689, 740, 817, 883, 995, 1048, 1072, 1074, 1182, 1195, 1229, 1239, 1338, 1402, 1403, 1418, 1435, 1451, 1452, 1469, 1478, 1481, 1491, 1492, 1505, 1506, 1512, 1521, 1530, 1536, 1540, 1560, 1576, 1607, 1618, 1634, 1637, 1638, 1691, 1700, 1712, 1736, 1737, 1745, 1751, 1761, 1763, 1766, 1775, 1778, 1786, 1790, 1800, 1802, 1803, 1808, 1813, 1818, 1823, 1828, 1838, 3815]<jupyter_text>[Insert additional data exploration here that seems needed for preprocessing]<jupyter_code>#insert any more exploration codes and notes here<jupyter_output><empty_output><jupyter_text>Preprocessing In this section, we now alter the data and/or its structure. Handle Incomplete Data<jupyter_code>#insert code to handle missing data here, be it drop or alter<jupyter_output><empty_output><jupyter_text>Handle Inconsistent Data<jupyter_code>#insert code to handle data that is most likely incorrect, from result of Data Exploration<jupyter_output><empty_output><jupyter_text>Handle Noise / Outliers<jupyter_code>#insert extreme or disruptive data handling here, can use binning, or some other method.<jupyter_output><empty_output><jupyter_text>[Insert appropriate preprocessing step here]<jupyter_code>#insert one-hot encoding here, Feature Selection and Extraction, PCA, binning, etc<jupyter_output><empty_output><jupyter_text>Model Training It's time to select and train our algorithm(s) to make models that can predict the test dataset.<jupyter_code>#insert all codes here, can use scikit-learn, can make own model<jupyter_output><empty_output><jupyter_text>Evaluation and Analysis From here onwards, we can only check how good our data is through cross-validation of our data with the training set. The test set is our final step and will be submitted to kaggle for scoring. [Insert model choosing and training here]<jupyter_code>#insert prediction and results here.<jupyter_output><empty_output><jupyter_text>Make notes in Markdown as to why a certain model is chosen, if necessary. Save Model<jupyter_code>#saving model is important, so we don't have to train a new model when sharing .ipynb file to others.
#insert saving model method here.<jupyter_output><empty_output><jupyter_text>Final Run<jupyter_code>#predict test set, save results in .csv, with header of : ID,Predicted
#put code here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13297424675980474, 'fraction_numerical': 0.06025921562026595, 'mean_word_length': 5.522502744237102, 'pattern_counts': {'":': 32, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '141768', 'license': 'MIT', 'n_tokens_mistral': 2223, 'n_tokens_neox': 1981, 'n_words': 586, 'path': 'Final Project_ Sistem Rekomendasi.ipynb', 'repo_name': 'kevinandrianliu/Final-Project-DSDM', 'size': '5941'}