<jupyter_start><jupyter_text>A quick exampleAmputation is the opposite of imputation: the generation of missing values in complete datasets. That is useful in an experimental setting where you want to evaluate the effect of missing values on the outcome of a model. :class:`~pyampute.ampute.MultivariateAmputation` is designed following scikit-learn's ``fit`` and ``transform`` paradigm, and can therefore seamless be integrated in a larger data processing pipeline.Here, we give a short demonstration. A more extensive example can be found in `this example`_. For people who are familiar with the implementation of multivariate amputation in R-function `ampute`_, `this blogpost`_ gives an overview of the similarities and differences with :class:`~pyampute.ampute.MultivariateAmputation`. Inspection of an incomplete dataset can be done with :class:`~pyampute.exploration.md_patterns.mdPatterns`.Note that the amputation methodology itself is proposed in `Generating missing values for simulation purposes`_ and in `The dance of the mechanisms`_.<jupyter_code># Author: <NAME> <https://rianneschouten.github.io/>
# Co-Author: <NAME> <https://davinaz.me/><jupyter_output><empty_output><jupyter_text>Transforming one dataset Multivariate amputation of one dataset can directly be performed with ``fit_transform``. Inspection of an incomplete dataset can be done with :class:`~pyampute.exploration.md_patterns.mdPatterns`. By default, :class:`~pyampute.ampute.MultivariateAmputation` generates 1 pattern with MAR missingness in 50% of the data rows for 50% of the variables.<jupyter_code>import numpy as np

from pyampute.ampute import MultivariateAmputation
from pyampute.exploration.md_patterns import mdPatterns

seed = 2022
rng = np.random.RandomState(seed)

m = 1000
n = 10
X_compl = np.random.randn(m,n)

ma = MultivariateAmputation(seed=seed)
X_incompl = ma.fit_transform(X_compl)

mdp = mdPatterns()
patterns = mdp.get_patterns(X_incompl)<jupyter_output><empty_output><jupyter_text>A separate fit and transform Integration in a larger pipeline requires separate ``fit`` and ``transform`` functionality.<jupyter_code>from sklearn.model_selection import train_test_split

X_compl_train, X_compl_test = train_test_split(X_compl, random_state=2022)
ma = MultivariateAmputation()
ma.fit(X_compl_train)
X_incompl_test = ma.transform(X_compl_test)<jupyter_output><empty_output><jupyter_text>Integration in a pipeline A short pipeline may look as follows.<jupyter_code>from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
import matplotlib.pyplot as plt

pipe = make_pipeline(MultivariateAmputation(), SimpleImputer())
pipe.fit(X_compl_train)

X_imp_test = pipe.transform(X_compl_test)<jupyter_output><empty_output><jupyter_text>By default, ``SimpleImputer`` imputes with the mean of the observed data. It is therefore like that we find the median in 50% of the rows (of the test set, which contains 25% of $m$) for 50% of the variables.<jupyter_code>medians = np.nanmedian(X_imp_test, axis=0)
print(np.sum(X_imp_test == medians[None,:], axis=0))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0827430779137154, 'fraction_numerical': 0.008692852543464262, 'mean_word_length': 6.966666666666667, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '563079', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 997, 'n_tokens_neox': 919, 'n_words': 307, 'path': 'docs/source/auto_examples/plot_easy_example.ipynb', 'repo_name': 'RianneSchouten/pyampute', 'size': '3126'}