<jupyter_start><jupyter_text>Defining functions for mapping and error<jupyter_code>def mapper(array):
    array = np.sort(array)
    int_map = np.arange(len(np.unique(array))).astype(int)
    dict_map = dict(zip(np.sort(np.unique(array)), int_map))
    tmp = pd.Series(array)
    res = tmp.map(lambda x: dict_map[x])
    
    inv_dict_map = {v: k for k, v in dict_map.items()}
    return res.values, inv_dict_map

def rmse(x, y):
    return np.sqrt((x - y)**2.0).sum()<jupyter_output><empty_output><jupyter_text>Grabbing Movie Lens data<jupyter_code>!wget http://files.grouplens.org/datasets/movielens/ml-1m.zip
!unzip ml-1m.zip<jupyter_output>--2015-06-25 10:25:08--  http://files.grouplens.org/datasets/movielens/ml-1m.zip
Resolving files.grouplens.org... 172.16.17.32
Connecting to files.grouplens.org|172.16.17.32|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5917392 (5.6M) [application/zip]
Saving to: 'ml-1m.zip'

ml-1m.zip           100%[=====================>]   5.64M   668KB/s   in 13s    

2015-06-25 10:25:22 (437 KB/s) - 'ml-1m.zip' saved [5917392/5917392]<jupyter_text>Parsing data and cleaning it up for NTFLib<jupyter_code>ratings = pd.read_table('ml-1m/ratings.dat', sep='::', names=['UserID', 'MovieID', 'Rating', 'Timestamp'])
ratings.Timestamp = ratings.Timestamp.map(lambda x: datetime.datetime.fromtimestamp(x).strftime('%Y-%m'))
# movies = pd.read_table('ml-1m/movies.dat', sep='::', names=['MovieID', 'Title', 'Genres'])
# users = pd.read_table('ml-1m/users.dat', sep='::', names=['UserID' ,'Gender', 'Age', 'Occupation::Zip-code'])
# Converting dates to integers
ratings['UserID'], inv_uid_dict = mapper(ratings['UserID'])
ratings['MovieID'], inv_mid_dict = mapper(ratings['MovieID'])
ratings['Timestamp'], inv_ts_dict = mapper(ratings['Timestamp'])
x_indices = ratings[['UserID', 'MovieID', 'Timestamp']].copy()
x_indices['UserID'] = x_indices['UserID'] - x_indices['UserID'].min() 
x_indices['MovieID'] = x_indices['MovieID'] - x_indices['MovieID'].min()
x_indices['Timestamp'] = x_indices['Timestamp'] - x_indices['Timestamp'].min() 

print x_indices.min()
x_indices = x_indices.values
x_vals = ratings['Rating'].values
print 'Number of unique movie IDs: {0}'.format(len(ratings['MovieID'].unique()))
print 'Max movie ID: {0}'.format(ratings['MovieID'].max())
indices_train, indices_test, val_train, val_test = train_test_split(
    x_indices, x_vals, test_size=0.40, random_state=42)

shape_uid = len(np.unique(x_indices[:,0]))
shape_mid = len(np.unique(x_indices[:,1]))
shape_ts = len(np.unique(x_indices[:,2]))
shape = [shape_uid, shape_mid, shape_ts]
shape
indices_train
# shape = [len(np.unique(ratings[x])) for x in ['UserID', 'MovieID', 'Timestamp']]
bnf = betantf.BetaNTF(shape, n_components=5, n_iters=10)
before = bnf.score(indices_train, val_train)
initial = bnf.impute(x_indices)
reconstructed = bnf.fit(indices_train, val_train)
after = bnf.score()
assert(after < before)
debug
prediction = bnf.impute(indices_test)
rmse(prediction, val_test) / float(prediction.shape[0])
!cat ml-1m/README<jupyter_output>SUMMARY
================================================================================

These files contain 1,000,209 anonymous ratings of approximately 3,900 movies 
made by 6,040 MovieLens users who joined MovieLens in 2000.

USAGE LICENSE
================================================================================

Neither the University of Minnesota nor any of the researchers
involved can guarantee the correctness of the data, its suitability
for any particular purpose, or the validity of results based on the
use of the data set.  The data set may be used for any research
purposes under the following conditions:

     * The user may not state or imply any endorsement from the
       University of Minnesota or the GroupLens Research Group.

     * The user must acknowledge the use of the data set in
       publications resulting from the use of the data set, and must
       send us an electronic or paper copy of those publications.

     * The user may no[...]
{'fraction_non_alphanumeric': 0.18460406962490808, 'fraction_numerical': 0.032606030889924004, 'mean_word_length': 5.306027820710974, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '23811', 'license': 'MIT', 'n_tokens_mistral': 1498, 'n_tokens_neox': 1373, 'n_words': 334, 'path': 'notebooks/.ipynb_checkpoints/ntf_movie_lens-checkpoint.ipynb', 'repo_name': 'stitchfix/NTFLib', 'size': '4083'}