<jupyter_start><jupyter_text>Holiday Data Exploration and Cleaning<jupyter_code>import pandas as pd
import numpy as np
import datetime
import seaborn as sns
from matplotlib import rcParams
import matplotlib.pyplot as plt
# figure size in inches
rcParams['figure.figsize'] = 30,8
df_holidays = pd.read_excel ('../data/db_load_files/calendario.xls')

df_holidays.tail(50)
df_holidays.dtypes
df_holidays.columns<jupyter_output><empty_output><jupyter_text>Lets rename these ugly column names:<jupyter_code>df_holidays.rename(columns={'Dia': 'date','laborable / festivo / domingo festivo': 'day_type', 'Festividad': 'holiday_name', 'Tipo de Festivo':'holiday_type' }, inplace=True)
del df_holidays['Dia_semana']
df_holidays.sample(60)<jupyter_output><empty_output><jupyter_text>I only need the date range which coincides with my sales data..well merge these two tables later<jupyter_code>mask = (df_holidays['date'] > '2015-09-20') & (df_holidays['date'] <= '2021-05-11')
new_df_holidays = df_holidays.loc[mask]
new_df_holidays<jupyter_output><empty_output><jupyter_text>Do I have any missing dates?<jupyter_code>#List all my missing dates
pd.date_range(start = '2015-09-21', end = '2021-05-11' ).difference(new_df_holidays.date)
new_df_holidays = new_df_holidays.set_index("date")
new_df_holidays
new_df_holidays.index.is_all_dates
pd.infer_freq(new_df_holidays.index)<jupyter_output><empty_output><jupyter_text>How many unique holidays are there?<jupyter_code>new_df_holidays.day_type.unique()
new_df_holidays.holiday_type.unique()
new_df_holidays.holiday_name.unique()<jupyter_output><empty_output><jupyter_text>Save clean data to file:<jupyter_code>new_df_holidays.to_csv("../data/raw_data/holidays_clean.csv", index=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1182181610508281, 'fraction_numerical': 0.02227298686464877, 'mean_word_length': 9.068965517241379, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '428250', 'license': 'MIT', 'n_tokens_mistral': 687, 'n_tokens_neox': 681, 'n_words': 111, 'path': 'notebook/3-fix_holidays.ipynb', 'repo_name': 'FranLeston/CityPlayForecast', 'size': '1751'}