<jupyter_start><jupyter_text>1. Data processingBefore we can begin analysing the data, we need to get it and "clean" it so that we can run computations on it.<jupyter_code>%matplotlib inline

import ast 
import csv
import numpy as np
from collections import Counter
import matplotlib
import matplotlib.pyplot as plt
# pretty plotting
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = [20, 5]
# First lets parse the data
reader = csv.reader(open("movie_recommendations.csv", "rb"), delimiter=",")
data = list(reader)
print data
# The first row has header info and the second row is empty, so we can ignore them.
# Note: the data is stored as strings, so we need to process it some more
text_data = np.array(data[2:])

movie_titles = [unicode(title, 'utf-8') for title in text_data[:,0]]
raw_movie_genres = text_data[:,1]
raw_omkar_ratings = text_data[:,2]
raw_imdb_ratings = text_data[:,3]
# -SOON->
# raw_meta_critic_ratings = result[:,4]
# raw_rotten_tomato_ratings = result[:,5]
# Now lets normalize these ratings so they are between 0 and 1
from __future__ import division # so that python will evaluate 3/10 as a floating pt operation instead of an integer op

def string_to_numpy(string_arr):
    tmp_list = []
    for string_val in string_arr:
        if string_val is 'N/A':
            tmp_list.append(0)
        else:
            tmp_list.append(eval(string_val))
    return np.asarray(tmp_list).astype("float")

omkar_ratings = string_to_numpy(raw_omkar_ratings)
imdb_ratings = string_to_numpy(raw_imdb_ratings)<jupyter_output><empty_output><jupyter_text>2. AnalysisLets look at the raw data first:<jupyter_code>assert len(imdb_ratings) == len(movie_titles)
# plt.xticks(range(len(imdb_ratings)), movie_titles, rotation=90) # <- too messy :(
# Remember, we scalled all scores to [0,1]!
plt.plot(imdb_ratings, alpha=0.5, label="IMDB rating")
plt.plot(omkar_ratings, alpha=1.0, label="Omkar's rating")
plt.legend()
plt.title('Plotting Omkar and IMDB ratings (scaled to [0,1])')
plt.show()<jupyter_output><empty_output><jupyter_text>_Phew!_ That's a pretty dense chart and on its own we can quickly surmise how closely related Omkar's ratings are w.r.t IMDB. For a single number statistic, let's look at [cross-correlation](https://en.wikipedia.org/wiki/Cross-correlation) between Omakar and IMDB:<jupyter_code>print "Overall IMDB corellation: ",np.corrcoef(omkar_ratings, imdb_ratings)[0,1]<jupyter_output>Overall IMDB corellation:  0.6404962063207162<jupyter_text>On its own, the correlation doesn't tell us much. Let's look at where the largest difference between Omkar and IMDB come up:<jupyter_code>def analyze_diff(diff_omkar_imdb, title):
    print 'Max difference: ', diff_omkar_imdb.max()
    print 'Min difference: ', diff_omkar_imdb.min()
    print 'Mean: ', diff_omkar_imdb.mean()
    print 'Std dev: ', diff_omkar_imdb.std()

    below_1_sigma = np.array(diff_omkar_imdb < (diff_omkar_imdb.mean() - diff_omkar_imdb.std()))
    above_1_sigma = np.array(diff_omkar_imdb > (diff_omkar_imdb.mean() + diff_omkar_imdb.std()))
    # everything that's not 1 sigma above/below the mean
    rest = np.logical_not(below_1_sigma) & np.logical_not(above_1_sigma)

    _x_axis = np.arange(len(imdb_ratings))

    plt.bar(_x_axis[above_1_sigma], diff_omkar_imdb[above_1_sigma], label='Above 1 $\sigma$')
    plt.bar(_x_axis[below_1_sigma], diff_omkar_imdb[below_1_sigma], label='Below 1 $\sigma$')
    plt.bar(_x_axis[rest], diff_omkar_imdb[rest], alpha=0.5, label='Within 2 $\sigma$')
    plt.legend()
    plt.title(title)
    
    high_positive_diff = []
    high_negative_diff = []

    for idx in range(len(movie_titles)):
        if above_1_sigma[idx]:
            high_positive_diff.append((movie_titles[idx], diff_omkar_imdb[idx]))
        if below_1_sigma[idx]:
            high_negative_diff.append((movie_titles[idx], diff_omkar_imdb[idx]))


    # Note: diff = Omkar - IMDB, so a positive score indicates Omkar rated a movie higher and vice versa
    print 'Movies that are above 1 sigma from the mean difference b/w Omkar and IMDB: (total: {})'.format(len(high_positive_diff))
    for movie_title, diff in high_positive_diff:
        print '\tMovie: {}, diff: {}'.format(movie_title.encode('utf-8'), diff)

    print 'Movies that are below 1 sigma from the mean difference b/w Omkar and IMDB: (total: {})'.format(len(high_negative_diff))
    for movie_title, diff in high_negative_diff:
        print '\tMovie: {}, diff: {}'.format(movie_title.encode('utf-8'), diff)
    
    return
analyze_diff(omkar_ratings - imdb_ratings, 'Difference b/w Omkar and IMDB (both of which were first scaled to [0,1])')<jupyter_output>Max difference:  0.30000000000000004
Min difference:  -0.16000000000000003
Mean:  0.041283185840707975
Std dev:  0.0667225093735769
Movies that are above 1 sigma from the mean difference b/w Omkar and IMDB: (total: 32)
	Movie: 5) Seven Psycopaths (2012), diff: 0.18
	Movie: 7) Con Air (1997), diff: 0.12
	Movie: 12) Silver Linings Playbook (2012), diff: 0.12
	Movie: 13) <NAME> (2013), diff: 0.15
	Movie: 21) <NAME> (2011), diff: 0.13
	Movie: 35) Stoker (2013), diff: 0.12
	Movie: 38) Pacific Rim (2013) , diff: 0.2
	Movie: 43) Only God Forgives (2013) , diff: 0.13
	Movie: 71) Speed Racer (2008) , diff: 0.3
	Movie: 80) Edge Of Tomorrow (2014), diff: 0.11
	Movie: 84) The Guest (2014), diff: 0.13
	Movie: 89) Boyhood (2014), diff: 0.11
	Movie: 90) <NAME> (2014), diff: 0.17
	Movie: 92) Birdman (2014), diff: 0.13
	Movie: 97) You're Next (2011), diff: 0.15
	Movie: 118) The DUFF (2015), diff: 0.15
	Movie: 126) Soul Plane (2004), diff: 0.26
	Movie: 136) Wet Hot American Summer (20[...]<jupyter_text>This is interesting: on average, it looks like Omkar rates movies ~4% higher than IMDB. With a standard deviation of ~6%, we see that Omkar tends to genrally be more generous with his ratings.Additionally, we can also look at the **absolute** difference b/w Omkar and IMDB in order to see which movies have very strong agreement b/w both datasets:<jupyter_code>analyze_diff(np.abs(omkar_ratings - imdb_ratings), 'Absolute difference b/w Omkar and IMDB (both of which were first scaled to [0,1])')<jupyter_output>Max difference:  0.30000000000000004
Min difference:  0.0
Mean:  0.061637168141592924
Std dev:  0.048549502507754805
Movies that are above 1 sigma from the mean difference b/w Omkar and IMDB: (total: 32)
	Movie: 5) Seven Psycopaths (2012), diff: 0.18
	Movie: 7) Con Air (1997), diff: 0.12
	Movie: 12) Silver Linings Playbook (2012), diff: 0.12
	Movie: 13) <NAME> (2013), diff: 0.15
	Movie: 21) Killer Joe (2011), diff: 0.13
	Movie: 35) Stoker (2013), diff: 0.12
	Movie: 38) Pacific Rim (2013) , diff: 0.2
	Movie: 43) Only God Forgives (2013) , diff: 0.13
	Movie: 47) Udaan (2010) , diff: 0.12
	Movie: 51) Videodrome (1983), diff: 0.13
	Movie: 71) Speed Racer (2008) , diff: 0.3
	Movie: 78) Blade Runner (1982), diff: 0.12
	Movie: 84) The Guest (2014), diff: 0.13
	Movie: 90) <NAME> (2014), diff: 0.17
	Movie: 92) Birdman (2014), diff: 0.13
	Movie: 97) You're Next (2011), diff: 0.15
	Movie: 118) The DUFF (2015), diff: 0.15
	Movie: 126) Soul Plane (2004), diff: 0.26
	Movie: 136) Wet H[...]<jupyter_text>Genre-based analysis<jupyter_code># Num unique genres
all_genres = []
for raw_genres in raw_movie_genres:
    genres = raw_genres.split('/')
    for genre in genres:
        word = genre.lower().strip()
        # spelling mistakes
        if word == 'crme':
            word = 'crime'
        elif word == 'myster':
            word = 'mystery'
        all_genres.append(word)

unique_genres = sorted(set(all_genres))
counts = Counter(all_genres)
print unique_genres
print counts
max_correlation = 0
max_corr_genre = 'N/A'

for genre in unique_genres:
    use = []
    for raw_genres in raw_movie_genres:
        use.append(genre in raw_genres.lower())
    if sum(use) < 3:
        print '> Genre "{}" has too few examples ({})'.format(genre, counts[genre])
        continue
    correlation = np.corrcoef(omkar_ratings[use], imdb_ratings[use])[0, 1]
    print 'Genre: {}, Num. data pts: {}, Correlation: {}'.format(genre, counts[genre], correlation)
    
    if correlation > max_correlation:
        max_correlation = correlation
        max_corr_genre = genre


print "Max. correlated genre: {}, ({})".format(max_corr_genre, max_correlation)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.121968787515006, 'fraction_numerical': 0.06398559423769508, 'mean_word_length': 4.3748387096774195, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 31, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '490443', 'license': 'MIT', 'n_tokens_mistral': 3432, 'n_tokens_neox': 2964, 'n_words': 892, 'path': 'analysis.ipynb', 'repo_name': 'shivaghose/omkar_movie_analysis', 'size': '8364'}