<jupyter_start><jupyter_text>Handling Missing Data - fillna, interpolate, dropna<jupyter_code>import pandas as pd
df = pd.read_csv("weather_data.csv",parse_dates=['day'])
type(df.day[0])
df
df.set_index('day',inplace=True)
df<jupyter_output><empty_output><jupyter_text>fillna **Fill all NaN with one specific value**<jupyter_code>new_df = df.fillna(0)
new_df<jupyter_output><empty_output><jupyter_text>**Fill na using column names and dict**<jupyter_code>new_df = df.fillna({
        'temperature': 0,
        'windspeed': 0,
        'event': 'No Event'
    })
new_df<jupyter_output><empty_output><jupyter_text>**Use method to determine how to fill na values**<jupyter_code>new_df = df.fillna(method="ffill")
new_df
new_df = df.fillna(method="bfill")
new_df<jupyter_output><empty_output><jupyter_text>**Use of axis**<jupyter_code>new_df = df.fillna(method="bfill", axis="columns") # axis is either "index" or "columns"
new_df<jupyter_output><empty_output><jupyter_text>**limit parameter**<jupyter_code>new_df = df.fillna(method="ffill",limit=1)
new_df<jupyter_output><empty_output><jupyter_text>interpolate<jupyter_code>new_df = df.interpolate()
new_df
new_df = df.interpolate(method="time") 
new_df<jupyter_output><empty_output><jupyter_text>**Notice that in above temperature on 2017-01-04 is 29 instead of 30 (in plain linear interpolate)** **There are many other methods for interpolation such as quadratic, piecewise_polynomial, cubic etc. Just google "dataframe interpolate" to see complete documentation**  dropna<jupyter_code>new_df = df.dropna()
new_df
new_df = df.dropna(how='all')
new_df
new_df = df.dropna(thresh=1)
new_df<jupyter_output><empty_output><jupyter_text>Inserting Missing Dates<jupyter_code>dt = pd.date_range("01-01-2017","01-11-2017")
idx = pd.DatetimeIndex(dt)
df.reindex(idx)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1352973267866885, 'fraction_numerical': 0.018548827059465357, 'mean_word_length': 8.405128205128205, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '276507', 'license': 'MIT', 'n_tokens_mistral': 733, 'n_tokens_neox': 697, 'n_words': 122, 'path': "Program's_Contributed_By_Contributors/AI-Summer-Course/py-master/pandas/5_handling_missing_data_fillna_dropna_interpolate/handling_missing_data_fillna_dropna_interpolate.ipynb", 'repo_name': 'siddharthdeo99/Hacktoberfest2k21', 'size': '1833'}