<jupyter_start><jupyter_text>YouTube Trending Project*  Data Cleaning and Curation Table of Contents:* 1.Exploratory Data Analysis* 2.Data Cleaning    * 2.1 Preprocessing Data        * 2.1.1 Drop Duplicate Rows        * 2.1.2 Drop Columns        * 2.1.3 Handling Missing Data    * 2.2 Post-Processed Data        * 2.2.1 Column Information        * 2.2.2 Exporting Curated Data    * 2.3 Export Cleaned Data w/ All Columns* 3.Modeling  2. Data Cleaning Loading Data and Libraries<jupyter_code>import helpers
import pandas as pd
import numpy as np
import datetime as dt

# Encoding and Data Split
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split

# Plotting Modules
import matplotlib.pyplot as plt

# Reading the stitched data
trend_data = helpers.load_df("Data/US_Data.csv")

df = trend_data.copy()
df.head()<jupyter_output><empty_output><jupyter_text>Checking shape of data frame and applying feature engineering<jupyter_code>np.shape(df)
helpers.featureEng(df)
# Must drop duplicate video observations collected over multiple days
df[df.video_id == 'tcYodQoapMg']<jupyter_output>/usr/local/lib/python3.7/site-packages/numpy/lib/histograms.py:822: RuntimeWarning: divide by zero encountered in double_scalars
  norm = n_equal_bins / _unsigned_subtract(last_edge, first_edge)
/usr/local/lib/python3.7/site-packages/numpy/lib/histograms.py:850: RuntimeWarning: invalid value encountered in multiply
  f_indices = _unsigned_subtract(tmp_a, first_edge) * norm<jupyter_text>2.1 Preprocessing Data*  2.1.1 Drop Duplicate Rows*  2.1.2 Drop Columns - Unnecessary or low correlation*  2.1.3 Handling Missing Data - Drop rows with missing values<jupyter_code>helpers.preprocess(df)<jupyter_output><empty_output><jupyter_text>2.2 Post-Processed Data*  2.2.1 Column Information*  2.2.2 Exporting Curated Data<jupyter_code>df.info()
df.to_csv("Data/Curated_US_Data.csv", index=False)<jupyter_output><empty_output><jupyter_text>2.3 Export Cleaned Data w/ All ColumnsExport data that has been cleaned of duplicates and missing values, but no dropped columns<jupyter_code>df = trend_data.copy()

helpers.featureEng(df)

df = df.drop_duplicates(subset=['video_id'], keep='last', ignore_index=True)

# Drop missing values
df.dropna()

df.to_csv("Data/FeatureEng_US_Data.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09167015487651738, 'fraction_numerical': 0.023440770196735034, 'mean_word_length': 5.373333333333333, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '162357', 'license': 'MIT', 'n_tokens_mistral': 825, 'n_tokens_neox': 794, 'n_words': 205, 'path': 'yt-cleaning.ipynb', 'repo_name': 'nickrwu/Youtube-Trending', 'size': '2389'}