<jupyter_start><jupyter_text>Deletion<jupyter_code>#Drop all rows that have any column value missing. This is case deletion
df.dropna()
df=pd.DataFrame(data, columns =['Name','Age','Sex','Goals', 'Assists', 'Value'])
df.head(n=7)
df.dropna(how='all') #When all the values of a sample (here row) is nan<jupyter_output><empty_output><jupyter_text>Mean mode Median imputation<jupyter_code>from sklearn.impute import SimpleImputer 
imp = SimpleImputer(missing_values=np.nan, strategy='mean')
X = np.array([[1,2],
             [np.nan,3],
             [7,6]])
imp.fit(X) 
print(imp.transform(X))      
imp.fit(df.to_numpy()[:,1].reshape(-1,1)) 
print(imp.transform(df.to_numpy()[:,1].reshape(-1,1)))<jupyter_output>[[21. ]
 [23. ]
 [20.6]
 [19. ]
 [25. ]
 [20.6]
 [15. ]]<jupyter_text>Univariate vs. Multivariate ImputationOne type of imputation algorithm is univariate, which imputes values in the i-th feature dimension using only non-missing values in that feature dimension (e.g. impute.SimpleImputer). By contrast, multivariate imputation algorithms use the entire set of available feature dimensions to estimate the missing values (e.g. impute.IterativeImputer).  Multivariate ImputationAt each step, a feature column is designated as output y and the other feature columns are treated as inputs X. A regressor is fit on (X, y) for known y. Then, the regressor is used to predict the missing values of y. This is done for each feature in an iterative fashion, and then is repeated for max_iter imputation rounds. The results of the final imputation round are returned.<jupyter_code>from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
X = np.array([[1, 2],
              [3, 6],
              [4, 8],
              [np.nan, 3],
              [7, np.nan]])

imp = IterativeImputer(max_iter=10, random_state=0)
imp.fit(X)  
IterativeImputer(add_indicator=False, estimator=None,
                 imputation_order='ascending', initial_strategy='mean',
                 max_iter=10, max_value=None, min_value=None,
                 missing_values=np.nan, n_nearest_features=None,
                 random_state=0, sample_posterior=False, tol=0.001,
                 verbose=0)

print(np.round(imp.transform(X)))<jupyter_output>[[ 1.  2.]
 [ 3.  6.]
 [ 4.  8.]
 [ 2.  3.]
 [ 7. 14.]]<jupyter_text>![autoimpune.png](attachment:autoimpune.png)<jupyter_code>#!pip install autoImpute<jupyter_output><empty_output><jupyter_text>![Screenshot%202019-11-26%20at%204.43.59%20PM.png](attachment:Screenshot%202019-11-26%20at%204.43.59%20PM.png) Two imputer classes* SingleImputer  - imputation methods, passing through the data once* MultipleImputer - imputation methods, passing through the data multiple times<jupyter_code>#Toy dataset
toy_df = pd.DataFrame({
    "age": np.random.choice(np.arange(20,80), 50),
    "gender": np.random.choice(["Male","Female"], 50),
    "employment": np.random.choice(["Unemployed","Employed", "Part Time", "Self-Employed"], 50),
    "salary": np.random.choice(np.arange(50_000, 1_000_000), 50),
    "weight": np.random.choice(np.arange(100, 300, 0.1), 50),
})
toy_df.head()
from autoimpute.imputations import SingleImputer
si = SingleImputer()
si.fit(toy_df)
si.statistics_<jupyter_output><empty_output><jupyter_text>series-imputersseries-imputers are simply workers behind the scenes that implement a given imputation model.<jupyter_code># get the series imputer for employment column
emp_series_imputer = si.statistics_["employment"]
print(emp_series_imputer)
print(emp_series_imputer.cat_imputer)
print(emp_series_imputer.num_imputer)
print(list(SingleImputer().strategies.keys()))
# string strategy broadcasts across all strategies
si_str = SingleImputer(strategy="mean")

# list strategy, where each item is a corresponding strategy
si_list = SingleImputer(strategy=["mean", "binary logistic", "median"])

# dictionary strategy, where we specify column and strategy together
# Note that with the dictionary, we can specify a SUBSET of columns to impute
si_dict = SingleImputer(strategy={"gender":"categorical", "salary": "pmm"})
# fitting with a single strategy
try:
    si_str.fit(toy_df)
except TypeError as te:
    print(f"{te.__class__.__name__}: {te}")
# fitting with a dictionaries of strategy
si_dict.fit(toy_df)<jupyter_output><empty_output><jupyter_text>img_kwgs argumentWhen imp_kwgs is None, all series-imputers for given strategies use their default arguments.img_kwgs argument control arguments to alter how the strategy ultimately works and performs.possible values* neighbors is the number of observations pmm will use to determine an imputation value.* If fill_value is set to random, pmm randomly selects one of the n neighbors as the imputation. Random is the default.* If the fill_value is set to mean, pmm takes the mean of the n neighbors and uses the mean as the imputation.<jupyter_code>si_dict_col = SingleImputer(
    strategy={"gender":"categorical", "salary": "pmm", "weight": "pmm"},
    imp_kwgs={"salary": {"neighbors": 10, "fill_value": "mean"}}
)
si_dict_col.fit(toy_df)<jupyter_output><empty_output><jupyter_text>Predictor argumentWhat all columns are used to build an imputation model for strategies that depend on predictors. We could also use a list or a dictionary to specify the predictors we want to use.<jupyter_code>preds_list = ["weight", "age", "salary"]
si_list_pred = SingleImputer(predictors=preds_list)
print(si_list_pred)

print()

preds_dict = {"weight": ["age", "gender"], "salary": ["education", "age"]}
si_dict_pred = SingleImputer(predictors=preds_dict)
print(si_dict_pred)
from autoimpute.imputations import MultipleImputer
#Toy dataset
toy_df = pd.DataFrame({
    "age": np.random.choice(np.arange(20,80), 50),
    "gender": np.random.choice(["Male","Female"], 50),
    "education": np.random.choice(["Graduate","masters"], 50),
    "employment": np.random.choice(["Unemployed","Employed", "Part Time", "Self-Employed"], 50),
    "salary": np.random.choice(np.arange(50_000, 1_000_000), 50),
    "weight": np.random.choice(np.arange(100, 300, 0.1), 50),
})
toy_df
toy_df.loc[toy_df.sample(frac=0.1).index, 'gender'] =np.nan
print(toy_df)
# imp = MultipleImputer(
#     n=2,
#     strategy={"salary": "pmm", "gender": "bayesian binary logistic", "age": "norm"},
#     predictors={"salary": "all", "gender": ["salary", "education", "weight"]},
#     imp_kwgs={"pmm": {"fill_value": "random"}},
#     visit="left-to-right",
#     return_list=True
# )

imp = MultipleImputer(
    n=1,
    strategy={"gender": "bayesian binary logistic"},
    predictors={"salary": "all", "gender": ["salary", "education", "weight"]},
    imp_kwgs={"pmm": {"fill_value": "random"}},
    visit="left-to-right",
    return_list=True
)


# Because we set return_list=True, imputations are done all at once, not evaluated lazily.
# This will return M*N, where M is the number of imputations and N is the size of original dataframe.
toy_df_full = imp.fit_transform(toy_df)
imps = pd.concat([toy_df_full[i][1]["gender"].to_frame() for i in range(len(toy_df_full))], axis=1)
conc = pd.concat([toy_df[['age','gender']], imps], axis=1)
#conc.columns = ["age",  "gender", "education","employment",  "salary",  "weight", "gender_1", "gender_2"]
conc.columns = ["age", "gender", "gender_1"]
print(conc)
imp.imputed_.keys()<jupyter_output><empty_output><jupyter_text>Exploring Missingness of the data<jupyter_code>from autoimpute.visuals import plot_md_percent, plot_md_locations
#Toy dataset
toy_df = pd.DataFrame({
    "age": np.random.choice(np.arange(20,80), 50),
    "gender": np.random.choice(["Male","Female"], 50),
    "education": np.random.choice(["Graduate","masters"], 50),
    "employment": np.random.choice(["Unemployed","Employed", "Part Time", "Self-Employed"], 50),
    "salary": np.random.choice(np.arange(50_000, 1_000_000), 50),
    "weight": np.random.choice(np.arange(100, 300, 0.1), 50),
})
toy_df.loc[toy_df.sample(frac=0.2).index, 'gender'] =np.nan
toy_df.loc[toy_df.sample(frac=0.2).index, 'salary'] =np.nan
toy_df.loc[toy_df.sample(frac=0.2).index, 'age'] =np.nan
print(toy_df)
plot_md_percent(toy_df)
plot_md_locations(toy_df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14442949737067384, 'fraction_numerical': 0.027393909746850922, 'mean_word_length': 5.379095163806552, 'pattern_counts': {'":': 39, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '22053', 'license': 'MIT', 'n_tokens_mistral': 3034, 'n_tokens_neox': 2817, 'n_words': 698, 'path': 'Chapter 02/Hello Missing.ipynb', 'repo_name': 'bpbpublications/Practical-Full-stack-Machine-Learning', 'size': '8177'}