<jupyter_start><jupyter_text>Data SourcesIn this notebook, you will get some practice working with data sources that are associated with recommendation systems.  As a review, remember that the type of data you have available to you determines what recommendation systems are possible:* Collaborative Filtering    - requires user-item interaction data* Demographic Recommenders    - require user information    * Content Based Recommenders    - require item information    * Utility Based Recommenders    - no strong requirements, but perform better with as many features as possible    * Knowledge Based Recommenders    - require user provided information to filter items **Intro:** Read in and take a look at the data below. This data is available from [MovieTweetings](https://github.com/sidooms/MovieTweetings).  Following the link provides additional information about the data should you want to know more information.<jupyter_code>import numpy as np
import pandas as pd

movies_dat = pd.read_csv('../../data/movies.dat', sep='::', engine='python',
                         header=None, names=['movie_id', 'movie_title', 'movie_genre'])
users_dat = pd.read_csv('../../data/users.dat', sep='::', engine='python', \
                        header=None, names=['user_id', 'twitter_id'])
ratings_dat = pd.read_csv('../../data/ratings.dat', sep='::', engine='python', \
                          header=None, names=['user_id', 'movie_id','rating','time'])

#look at the datasets using this cell
#look at the datasets using this cell
#look at the datasets using this cell<jupyter_output><empty_output><jupyter_text>**Question 1:** Based on the data provided, which of the following recommender systems is absolutely not possible to build?<jupyter_code>import data_solution_part1 as sp

a = "Collaborative Filtering"
b = "Demographic Based Recommender"
c = "Content Based Recommender"
d = "Utility Based Recommender"
e = "Knowledge Based Recommender"

your_answer = #a

sp.answer_one(your_answer)<jupyter_output><empty_output><jupyter_text>**Question 2:** If we wanted to build a collaborative filtering recommender system, what dataframe(s) would we absolutely need?<jupyter_code>a = "movies_dat"
b = "users_dat"
c = "ratings_dat"
d = "movies_dat and ratings_dat"
e = "all three dataframes are needed"

your_answer = #a

sp.answer_two(your_answer)<jupyter_output><empty_output><jupyter_text>Exploring and understanding your dataset is super important!  Because of that, you need to do Exploratory Data Analysis (EDA) to understand your dataset!**Question 3:** How many unique users have rated movies?  How many unique movies are in the dataset?  What is the average rating provided in the dataset?  What is the minimum rating given in the dataset?  How many missing values are in `ratings_dat`? Fill in your answers to each below.<jupyter_code>unique_users = # your answer integer
unique_movies = # your answer integer
avg_rating = # your answer round to hundredths place
min_rating = # your answer round to integer
max_rating = # your answer round to integer
missing_vals = # your answer integer

your_answer = (unique_users, unique_movies, avg_rating, \
               min_rating, max_rating, missing_vals)

sp.answer_three(your_answer)<jupyter_output><empty_output><jupyter_text>**Question 4:** There are two movies that have an average rating of 10, while having more than 2 ratings - these are the two movies with the 'highest average, max number of ratings' combination.  What are these two movies?<jupyter_code>a = "A Woman Under the Influence"
b = "Amadeus"
c = "ratings_dat"
d = "movies_dat and ratings_dat"
e = "all three dataframes are needed"

your_answer = #(10_rating_4_count_movie, 10_rating_3_count_movie)

sp.answer_four(your_answer)
#use this cell for your work
#use this cell for your work
#use this cell for your work<jupyter_output><empty_output><jupyter_text>Use the space below to look at the distribution of ratings, and explore any other proporties you might be interested in understanding. Like which user gave the most ratings? Or which user gave the highest ratings on average combined with the most ratings?  We will dive into how to clean up some characteristics of the data, and then you will be ready to start building your own recommendation systems!<jupyter_code>#more cells for your work<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07952194897724661, 'fraction_numerical': 0.003217651114686279, 'mean_word_length': 4.937244201909959, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '372934', 'license': 'MIT', 'n_tokens_mistral': 1224, 'n_tokens_neox': 1187, 'n_words': 500, 'path': 'Part 2 - Data Sources/notebooks/data_sources_notebook1.ipynb', 'repo_name': 'tongni1975/rec-workshop', 'size': '4351'}