<jupyter_start><jupyter_text>Final project of <NAME> for the course Data Processing at the Universiteit van Amsterdam.My idea with this project was to visualize my Netflix watching habit, and to find out whether I watched more Netflix during the lockdowns and online university teaching or not.For this project I downloaded my activity log from my Netflix account and started to work with that. The dataset only contained 2 columns (Title (with the seasons and episode) and Date) therefore, at first I was expanding it with more information manually, assigning genres to the titles and another column stating whether that title is a series or a movie. Later on, I used scraped information from the iMDB to give a true picture of my viewing habits, to validate my manuallay added information.For this project I used those modules and libraries I studied during the course of Scientific Programming 1 and 2, and Data Processing. Throughout the project, I also coded my own tools and small library to read, manipulate, transform the datasets and dataframes, scrape the data from the internet, and visualize everything that can be seen here. Importing all the necessary module and my tools to run the notebook. The source code of the tools I wrote especially for this project can be found under the folder "myTools". As some of the methods used to handle dataframes and using them to visualize, a warning message was popping up after running a cell, so I decided to filter out those warnings in the beginning.<jupyter_code>import warnings
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from bokeh.plotting import figure
from bokeh.io import output_notebook, show
from pandas.core.common import SettingWithCopyWarning
from bokeh.palettes import Category20b_13, Category20b_14
from bokeh.models import ColumnDataSource, CategoricalColorMapper, HoverTool


from myTools import scrapeBreakingBadRatings, scrapeHomelandRatings, scrapeMovies, scrapeSeries
from myTools import watchingHabit, mostWatchedGenre, busiestDay, mostWatchedType, mostWatchedTypeUnique, heatMapBreakingBad, heatMapHomeland, watchingHabitMinutes, mostWatchedGenreMinutes, busiestDayMinutes, mostWatchedTypeMinutes
from myTools import readNetflixDataset, episodePerDay, titlePerGenre, titleType, titleTypeUnique, seriesData, moviesData, concatSeriesMovies, cleanDataFrame, mapRuntime, mapGenre, mapType, addSpentMinutes, minutesPerDay, minutesPerGenre, minutesPerTitleType

warnings.simplefilter(action="ignore", category=SettingWithCopyWarning) 
output_notebook()<jupyter_output><empty_output><jupyter_text>Reading the datasets and creating a copy of it, to use the original dataset with the original timeformat later on when needed.<jupyter_code>netflixRaw = readNetflixDataset('datasets/NetflixViewingHistory.xlsx')
netflix = netflixRaw.copy()
netflix['Date'] = pd.to_datetime(netflix['Date'])
display(netflix.head())<jupyter_output><empty_output><jupyter_text>Creating all the necessary dataframes that are needed to run the visualization tools. Here can be also seen, how I managed to map the scraped information to the dataset, and how the structure of the dataframe that is used for the visualizations looks like.<jupyter_code>seriesData = seriesData(netflix)
moviesData = moviesData(netflix)
concatedSeriesMovies = concatSeriesMovies(seriesData, moviesData)

# SM stands for Spent Minutes
netflixSM = cleanDataFrame(netflix)
netflixSM = mapRuntime(netflixSM, concatedSeriesMovies)
netflixSM = mapGenre(netflixSM, concatedSeriesMovies)
netflixSM = mapType(netflixSM, concatedSeriesMovies)
netflixSM = addSpentMinutes(netflixSM)

display(netflixSM.head())<jupyter_output><empty_output><jupyter_text>This is my watching habit visualizations where it can be seen how many titles I watched since my subscriptions and when, color coded by genres.<jupyter_code>watchingHabit(netflix)<jupyter_output><empty_output><jupyter_text>This visualizations also presents my watching habit, expect that the Y-axis is the number of minutes I spent on Netflix.<jupyter_code>watchingHabitMinutes(netflixSM)<jupyter_output><empty_output><jupyter_text>The following two bar graphs shows on which days I watched Netflix the most. First graph shows the number of titles, second shows the number of minutes spent.<jupyter_code>busiestDay(netflixRaw)
busiestDayMinutes(netflixSM)<jupyter_output><empty_output><jupyter_text>The following two bar graphs shows which genres I watched the most. First graph shows the number of titles within the given genre, second shows the number of minutes spent watching that genre.<jupyter_code>mostWatchedGenre(netflix)
mostWatchedGenreMinutes(netflixSM)<jupyter_output><empty_output><jupyter_text>The following two bar graphs shows how many series and movies I watched on Netflix. First graph shows the number of series and movies, second shows the number of minutes spent watching series and movies<jupyter_code>mostWatchedType(netflix)
mostWatchedTypeMinutes(netflixSM)<jupyter_output><empty_output><jupyter_text>This visualizations shows how many unique titles I watched on Netflix. Turns out, I watched more movies than series.<jupyter_code>mostWatchedTypeUnique(netflix)<jupyter_output><empty_output><jupyter_text>These heatmaps visualize the ratings of my two favorite series' episodes, Breaking Bad and Homeland. I decided to make these visualizations because they are close to my heart, and I was curious to see the distribution of the ratings. Safe to say, I am happy with these ratings, expect the final episode of Homeland's 4th season<jupyter_code>heatMapBreakingBad()
heatMapHomeland()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.04882363346895454, 'fraction_numerical': 0.0021227666725632407, 'mean_word_length': 7.100286532951289, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '516577', 'license': 'MIT', 'n_tokens_mistral': 1543, 'n_tokens_neox': 1483, 'n_words': 641, 'path': 'finalproject.ipynb', 'repo_name': 'tmsmolnar/sp-dp-uva', 'size': '5659'}