<jupyter_start><jupyter_text>Load the relevant paths and modules In the data directory, the following folders are included:- external: raw data from source- interim: cleaned data for intermediate use - processed: fully procesed and finished data<jupyter_code># set the auto-reload for the cells with %aimport
%load_ext autoreload
%autoreload 1
import os
import sys

# get source and data directories
src_dir = os.path.join(os.getcwd(), os.pardir, 'src')
data_dir = os.path.join(os.getcwd(), os.pardir, 'data')

# set the source data directory as the current path
src_data_dir = os.path.join(src_dir, 'data')
sys.path.append(src_data_dir)

interim_data_dir = os.path.join(data_dir, 'interim')
processed_data_dir = os.path.join(data_dir, 'processed')
# load the web scraping modules
%aimport feedback
import feedback as fb
%aimport clean
import clean as cl<jupyter_output><empty_output><jupyter_text>Clean the existing csv data files In clean.py, I pass the path to the data directory and the clean_imdb_data function reads the tmdb data file and does the following cleaning steps:- removes non-alphanumeric characters- makes lowercase- removes duplicate names- converts dates to year only- convert existing genre format to "genre1|genre2|genre3" format- remove entries with missing financial data- overwrites existing file with cleaned data<jupyter_code>#audit movie financial csv file
cl.clean_imdb_data(data_dir)<jupyter_output><empty_output><jupyter_text>Scrape movie data and put into separate csv filesI segment the movies this way in case I encounter some kind of error for a particular year, I only disregard that year so as to not lose all of the data from the other years.<jupyter_code>import time
import numpy as np

# get a list of movies for a particular year
years = ['2000', '2001', '2002', '2003', '2004', '2005', '2006', \
         '2007', '2008', '2009', '2010', '2011', '2012', '2013', \
         '2014', '2015', '2016']

for year in years:
    print('starting year: ' + str(year))
    movie_urls = []
    movie_data = []
    start_time = time.time()
    
    # get a list of movie urls for each desired year
    movie_urls = fb.get_movie_urls(year)
    
    # scrape the website for the movie data
    movie_data = fb.scrape(movie_urls)
    
    # add the interest score
    avg_rating = np.array([m['rating'] for m in movie_data]).mean()
    avg_num_votes = np.array([m['total_votes'] for m in movie_data]).mean()
    for movie in movie_data:
        movie['interest_score'] = ((avg_rating*avg_num_votes) + (movie['total_votes']*movie['rating']))/(avg_num_votes + movie['total_votes'])
    
    # create a csv for the given year
    fb.convert_to_csv(movie_data, interim_data_dir, str(year))
    
    end_time = time.time() - start_time
    print(end_time/60)
    print('finished year: ' + str(year))<jupyter_output>starting year: 2000
 getting movie urls... 
 done!
 scraping data from movie website...
 done!
 creating /Users/dmoton/projects/data_projects/movie-hype/notebooks/../data/interim/2000.csv...
 done!
0.050003015995
finished year: 2000
starting year: 2001
 getting movie urls... 
 done!
 scraping data from movie website...
 done!
 creating /Users/dmoton/projects/data_projects/movie-hype/notebooks/../data/interim/2001.csv...
 done!
0.0656933347384
finished year: 2001
starting year: 2002
 getting movie urls... 
 done!
 scraping data from movie website...
 done!
 creating /Users/dmoton/projects/data_projects/movie-hype/notebooks/../data/interim/2002.csv...
 done!
0.524883568287
finished year: 2002
starting year: 2003
 getting movie urls... 
 done!
 scraping data from movie website...
 done!
 creating /Users/dmoton/projects/data_projects/movie-hype/notebooks/../data/interim/2003.csv...
 done!
1.84589506785
finished year: 2003
starting year: 2004
 getting movie urls... 
 done!
 scraping data fr[...]<jupyter_text>Merge the csv year files into one csv<jupyter_code>import pandas as pd 

# join the interest data into a dataframe
mi = pd.DataFrame([])

for year in years:
    year_csv = os.path.join(interim_data_dir, year + '.csv')
    year_df = pd.read_csv(year_csv, skiprows=0)
    mi = mi.append(year_df)  
# add the interest score
ratings = list(mi['rating'])
num_votes = list(mi['total_votes'])

avg_rating = np.array(ratings).mean()
avg_num_votes = np.array(num_votes).mean()
mi['interest_score'] = ((avg_rating*avg_num_votes) + (mi['total_votes']*mi['rating']))/(avg_num_votes + mi['total_votes'])

output = os.path.join(processed_data_dir, 'movie_interest.csv')
mi.to_csv(output)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11345183234584157, 'fraction_numerical': 0.03928022822032039, 'mean_word_length': 4.4261904761904765, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '330010', 'license': 'MIT', 'n_tokens_mistral': 1646, 'n_tokens_neox': 1459, 'n_words': 453, 'path': 'notebooks/data-wrangling.ipynb', 'repo_name': 'culight/movie-hype', 'size': '4557'}