<jupyter_start><jupyter_text>IASIP: The Gang Codes Python<jupyter_code>%%HTML
<i> Pondy is the coolest!</i>
<img src="https://i.imgur.com/MyrwEFn.jpg" alt="hand drawn image of <NAME> at his sisters wedding."></img>
%%HTML
<i> "With a plastic bag for a helmet!!!" "</i>
<img src="https://i.imgur.com/Sthqe90.jpg" alt="hand drawn image of Frank at the Analyst's Office, ink on paper"></img>
%%HTML
<i> You gotta takem off son!!!</i>
<img src="https://i.imgur.com/WC4O9zu.jpg" alt="Z remembers some favorite jean shorts."></img>
%%HTML
<i> What the hell went on back there?!</i>
<img src="https://i.imgur.com/LjRBq6g.jpg" alt="<NAME>, C.E.O."></img>
%%!
cd ~/mtPy/py_iasip/
pwd
#cat iasip.sql
#determine if data file is valid. 
#use iasip[2,4,5,etc].csv and iasip.3.csv to validate updates before merging into iasip.sql file
import pandas as pd
import numpy as np
import os<jupyter_output><empty_output><jupyter_text>Use the following steps to determine where data is missing in the data .csv files. '''Once corrected, build a iasip.sql file.Create a dataframe to determine if merging the two update files will correct missingvalues before updatingFiles: iasip3.csv contains seasons 12 & 13 data, practice merging data frames hereFind which rows in these files are showing null (NaN) values. which data file is the most correct? Once the data files are corrected, use this same procedure to check if the update file is correct before exporting to the processed file directory.<jupyter_code>raw_data_path = os.path.join(os.path.pardir,'data','raw')
iasip_file_path = os.path.join(raw_data_path, 'iasip.csv')
iasip2_file_path = os.path.join(raw_data_path, 'iasip2.csv')
iasip3_file_path = os.path.join(raw_data_path, 'iasip3.csv')
iasip4_file_path = os.path.join(raw_data_path, 'iasip4.csv')
iasip5_file_path = os.path.join(raw_data_path, 'iasip5.csv')
iasip_df = pd.read_csv(iasip_file_path, index_col=['episode'])
iasip2_df = pd.read_csv(iasip2_file_path, index_col=['episode'])
iasip3_df = pd.read_csv(iasip3_file_path, index_col=['episode'])
iasip4_df = pd.read_csv(iasip4_file_path, index_col=['episode'])
iasip5_df = pd.read_csv(iasip5_file_path, index_col=['episode'])
iasip_df = pd.read_csv('data/raw/iasip.csv', index_col='episode')
iasip2_df = pd.read_csv('data/raw/iasip2.csv', index_col=['episode'])
iasip3_df = pd.read_csv('data/raw/iasip3.csv', index_col=['episode'])
iasip4_df = pd.read_csv('data/raw/iasip4.csv', index_col=['episode'])
iasip5_df = pd.read_csv('data/raw/iasip5.csv', index_col=['episode'])
type(iasip_df)
iasip2_df.index
iasip_df.info()
#this is missing the epidose column, 
iasip2_df.info()
iasip3_df.info()
iasip4_df.info()
iasip5_df.info()
iasip_df.columns
iasip_df[['e_title', 'season','day_of_week']]
iasip_df.e_title
iasip_df['season']
iasip_df[['season', 'e_title']]
df = pd.concat([iasip_df, update_df], axis=0, sort=True, ignore_index=True)               
#keywords can have null values, all other values should equal total entries
df.info()
#find which episodes are missing values in the e_time column
df[df.e_time.isnull()]
df.tail(21)
print(update_df.columns)
print(iasip_df.columns)
print(df.columns)
type(df)
season_7 = df.loc[71:83, ['season', 'e_title', 'location']]
print(season_7)
season_12 = df.loc[124:133, ['season', 'e_title', 'location']]
print(season_12)
df.iloc[71:83,1:2]
#should return season 7
df.loc[71:83, ['season', 'e_title']]
#this should show every day of the week plus three outliers
df.day_of_week.value_counts()
df.location.value_counts()
df[df.location.isnull()]
df.day_of_week.value_counts().plot(kind='bar', title='Epiode by Day of Week')
df.location.value_counts().plot(kind='pie', title='Epiode by Location')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1475586727812247, 'fraction_numerical': 0.019692473698408416, 'mean_word_length': 6.32806324110672, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '587904', 'license': 'MIT', 'n_tokens_mistral': 1442, 'n_tokens_neox': 1400, 'n_words': 340, 'path': 'notebooks/00_mt_iasip_notebook.ipynb', 'repo_name': 'worker-bee-micah/iasip_py', 'size': '3725'}