<jupyter_start><jupyter_text>IntroThis notebook explores the missingness structure of Understanding Society data for wave 21 (2011-2012).To begin load data and count number of missing values by column. There are 5 missing types-1 Don't Know - An individual does not know the answer to the question (e.g. birth year)-2 Refusal - An individual refuses to answer a question (e.g. political opinion)-7 Proxy - An individual was unable to attend their interview. A shorter proxy interview was undertaken by mail/next of kin with some questions not answered.-8 Inapplicable - An observation does not have a logical answer by design. E.g. Anyone unemployed will not have an income variable.  For US NA also includes observations not carried forwards. Some variables are not recorded unless an individual changes state. They aren't technically missing. distinction between 'truly' NA values and tbose not carried forwards may be done by assigning missing by design values -10.-9 Missing (unintentionally) - an observation is truly missing. This can be for any number of data recording errors such as human error.<jupyter_code>import sys
import seaborn as sns
import numpy as np
sys.path.append('../minos/data_generation')
import US_utils
import US_missing_description as USmd

file_names = ["../data/corrected_US/2011_US_cohort.csv"]
data = US_utils.load_multiple_data(file_names)

table = USmd.missingness_table(data)
print(table)
USmd.missingness_bars(data, "depression_change", "ethnicity")
USmd.missingness_hist(data, "depression_change", "age")
print(sns.catplot("depression_change", "SF-12", data = data.replace(US_utils.missing_types, np.nan), kind = "box"))

USmd.missingness_bars(data, "depression", "ethnicity")
USmd.missingness_hist(data, "depression", "age")
print(sns.catplot("depression", "SF-12", data = data.replace(US_utils.missing_types, np.nan), kind = "box"))

USmd.missingness_bars(data, "SF-12", "ethnicity")
USmd.missingness_hist(data, "SF-12", "age")
print(sns.catplot("ethnicity", "SF-12", data = data.replace(US_utils.missing_types, np.nan), kind = "box"))
print(sns.catplot("region", "SF-12", data = data.replace(US_utils.missing_types, np.nan), kind = "box"))

print(data["ethnicity"].value_counts())
print(sns.catplot("education_state", "age", data = data.replace(US_utils.missing_types, np.nan), kind = "box"))<jupyter_output>/Users/robertclay/opt/anaconda3/envs/minos/lib/python3.7/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
{'fraction_non_alphanumeric': 0.10629629629629629, 'fraction_numerical': 0.015555555555555555, 'mean_word_length': 5.769423558897243, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '304124', 'license': 'MIT', 'n_tokens_mistral': 834, 'n_tokens_neox': 792, 'n_words': 278, 'path': 'notebooks/missing_exploration.ipynb', 'repo_name': 'Leeds-MRG/Minos', 'size': '2700'}