<jupyter_start><jupyter_text>1. Ignoring Missing Data  1.1 Ignoring rows with missing columns<jupyter_code>dummy_df = pd.read_csv('dummy_data.csv', index_col=0)
dummy_df
removed_na_df = dummy_df.dropna()
removed_na_df<jupyter_output><empty_output><jupyter_text>Not a good idea since we have deleted 4 out of 9 rows (~44% data)  1.2 Removing rows with majority columns as missing<jupyter_code>dummy_majority_df = pd.read_csv('dummy_missing_majority.csv', index_col=0)
dummy_majority_df<jupyter_output><empty_output><jupyter_text>`thresh=x` attribute tells df.dropna() to keep rows with atleast 'x' Non-Null values<jupyter_code>dummy_majority_df.dropna(thresh=4)
dummy_majority_df.dropna(thresh=6)<jupyter_output><empty_output><jupyter_text>Dropping columns by percentage of missing values   Drop columns with say over 40% empty values **Note** : axis=0 is for rows and axis=1 is for columns<jupyter_code>dummy_majority_df.dropna(axis=1, thresh=int(0.4*len(dummy_majority_df)))<jupyter_output><empty_output><jupyter_text>`Marks` column is deleted since it had more than 40% empty values  Dropping rows with majority columns as null values Keep rows with atleast 60% of values filled<jupyter_code>dummy_majority_df.dropna(axis=0, thresh=int(0.6*len(dummy_majority_df.columns)))<jupyter_output><empty_output><jupyter_text>2. Imputing Values  2.1 Filling with generic values<jupyter_code>dummy_df = pd.read_csv('dummy_data.csv', index_col=0)
dummy_df
dummy_df.fillna(-1)<jupyter_output><empty_output><jupyter_text>**Note** changes will be permanently reflected only when inplace=True<jupyter_code># dummy_df.fillna(inplace=True)
dummy_df.bfill()  #Use the next non-null value to fill
dummy_df.ffill() #Use the previous non-null value to fill<jupyter_output><empty_output><jupyter_text>2.2 Filling with central tendencies<jupyter_code>dummy_df
mean_age = dummy_df['Age'].mean()
mean_height = dummy_df['Height(cm)'].mean()
map_dict = {'Age': mean_age, 'Height(cm)': mean_height}
map_dict
dummy_df.fillna(value=map_dict)<jupyter_output><empty_output><jupyter_text>2.3 Imputing values based on condition<jupyter_code>weight_df = pd.read_csv('dummy_age_weight.csv')
weight_df
weight_df["Weight(kg)"] = weight_df.groupby("Gender").transform(lambda x: x.fillna(x.mean()))
weight_df
weight_df.groupby("Gender")
for x in weight_df.groupby("Gender"):
    print("Printing Group")
    print(x)
    print("\n\n")<jupyter_output>Printing Group
('Female',     Gender  Weight(kg)
1   Female        55.0
3   Female        57.6
4   Female        60.0
6   Female        52.0
7   Female        53.0
11  Female        68.0)



Printing Group
('Male',    Gender  Weight(kg)
0    Male       70.00
2    Male       65.00
5    Male       73.75
8    Male       85.00
9    Male       75.00
10   Male       73.75)
{'fraction_non_alphanumeric': 0.10939186757826556, 'fraction_numerical': 0.03310543360921195, 'mean_word_length': 5.541176470588235, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '109233', 'license': 'MIT', 'n_tokens_mistral': 1062, 'n_tokens_neox': 1007, 'n_words': 223, 'path': 'Section 2/Dealing with missing Values.ipynb', 'repo_name': 'kamaleshreddy/Exploratory-Data-Analysis-with-Pandas-and-Python-3.x', 'size': '2779'}