<jupyter_start><jupyter_text>Example Cleaning PipelineThis notebook demonstrates how to run the NCCID notebook using an dummy dataset.<jupyter_code>import pandas as pd
from nccid_cleaning import clean_data_df, patient_df_pipeline<jupyter_output><empty_output><jupyter_text>The example data contains 10 synthesized rows of NCCID clinical data and a subset of the possible columns. The columns have been chosen to be representative of the different types of information available, i.e., dates, categories, integers, floats, whilst also covering the known data quality issues within, i.e., typos in headings, incorrect formats, values embedded in strings, values outside of category ranges etc. Broadly speaking the NCCID clinical data can be split into 5 groups:- general: PatientID, SubmittingCentre, demographics- date: swab dates, scan dates, date of admission, intubation death etc.,- medical history: usually categorical e.g, presence of pre-existing lung conditions- admission metrics: usually numerical, e.g., heart rate on admission,- outcomes: usually categorial, e.g., test results, x-ray severity, death.The data is broken down into these 5 groups in the subsequent analysis of the cleaning pipeline.   Run the cleaning pipeline<jupyter_code># load example df
df = pd.read_csv("data/example.csv")
df.head()
# run clinical cleaning pipeline
df = clean_data_df(df, patient_df_pipeline)
df.head()
# just the new cleaned columns
df[[col for col in df.columns if col.islower()]].head()
def compare_dfs(columns, df):
    """Creates dataframe with equivalent columns side by side.

    Params
    -------
    columns: list
             list of original column names
    df: pd.DataFrame
        original data

    Returns
    -------
    comp_df: pd.DataFrame
            comparision dataframe with equivalent columns side by side.
    """

    comp_df = pd.concat(
        [
            pd.concat((df[col], df[col.lower().replace(" ", "_")]), axis=1)
            for col in columns
        ],
        axis=1,
    )
    return comp_df<jupyter_output><empty_output><jupyter_text>General ColumnsColumns like Pseudonym and Submitting centre are have already been preprocessed by the NCCID ingestion pipeline. As such subsequent cleaning is not applied to these columns. Mixed formats have been used in the demographic columns ethnicity and sex, where categories are submitting in multiple ways. For sex the mapping is simple, e.g., `0` to `F`, `1` to `M`. For ethnicity, various subgroups are aggregated into the broader set of ethnicity groupings: `The White, Black, Asian, Multiple, Other, Unknown`. Ages are rounded to the nearest whole number.<jupyter_code>gen_df = compare_dfs(["Ethnicity", "Age", "Sex"], df)
pd.concat((df[["Pseudonym", "SubmittingCentre"]], gen_df), axis=1)<jupyter_output><empty_output><jupyter_text>Dates The majority of date columns, including `Date of admission`, are submitted in US date format MM/DD/YYYY or some variant (e.g, M/D/YY). As such the cleaning pipeline assumes `month_first=True` for most date columns when coercing into pd.datetime. The exception is `SwabDate` which is submitted in UK format and therefore treated separately. Known errors in date entry include submissions of the string format `[Text] - YYYY-MM-DD`, for which the date is extracted using regex. Other errors such as non-date entries (row 7) are cast to `NaT`. In some cases, dates have been submitted in the wrong format (day first instead of month first or vice versa). The cleaning pipeline cannot correct for ambiguous cases, such as 05/06, and users should look to additional sources such as DICOM header dates to corroborate where possible.<jupyter_code># original and cleaned date columns
compare_dfs(["SwabDate", "Date of admission"], df)<jupyter_output><empty_output><jupyter_text>Medical history Preprocessing of the medical history fields involves serveral steps. For all categorical fields, values are mapped to their expected number of categories, as documented on the [HDRUK portal](https://web.www.healthdatagateway.org/dataset/31f0148b-f965-4136-ab39-6c5bbbf8c2d9). Any entries outside the set of expected categories are mapped to `NaN`. For Fields with an `Unknown` category, these are also mapped to `NaN`. In some cases, numerical categories are embedded within strings (row 4 below), for which values are extracted. This might throw away additonal information that could be useful, such as the type of disease in the example.<jupyter_code>compare_dfs(["PMH Lung disease"], df)<jupyter_output><empty_output><jupyter_text>For binarys field the cleaning pipeline converts entries into `bool`, mapping `Unknown` to `NaN`.The pipeline also merges columns that are known to be equivalent, but have been submitted with changes in their headers. An example of this is the 'PMH h1pertension' field which appears for some patients, caused by change at the submitting center. Another example is `PMH diabetes mellitus TYPE II` and `PMH diabetes mellitus type II` where the former was the name of the column in pervious versions of the data collection spreadsheet, and the latter is the the current name.<jupyter_code>df[["PMH hypertension", "PMH h1pertension", "pmh_hypertension"]]<jupyter_output><empty_output><jupyter_text>Admission metricsThese are exclusively expected to be numerical values (float or int), therefore, the preprocessing pipeline attempts to cast entries to numerical values using `pd.to_numerical(value, errors="coerce")`. This removes anomolous entires, such as blank spaces which have a high prevalence in the data set.Data is sometime embedded within strings of the format `[value] - YYYY-MM-DD`, e.g., Ferritin, row 3/4. For such cases, the cleaning pipeline is designed to extract the values from within the square brackets. In the case of blood pressure the pattern `[systolic/diastolic]` is assumed when extracting the information.<jupyter_code>compare_dfs(["Systolic BP", "Diastolic BP", "Ferritin"], df)<jupyter_output><empty_output><jupyter_text>The values for FiO2 are submitted different scales but different sites. The pipeline remaps these entries to the percentage sacle using the table provided on the FAQs page. In addition, some fields are clipped to a reasonable value range. This is currently implemented for `temperature_on_admission` (25- 45 celcius) and `fibrinogen__if_d-dimer_not_performed, urea_on_admission, o2_saturation` (all 0-100). This step can be avoided by removing `_clip_numeric_values` from the cleaning pipeline. Other fields in the clinical data, such as `Creatinine on admission` are also submitted using different units for different sites. The cleaning pipeline does not currently address this issue.<jupyter_code>compare_dfs(["FiO2", "Temperature on admission"], df)<jupyter_output><empty_output><jupyter_text>OutcomesThe outcome fields are largely categorical and therefore processed alongside the medical history fields. Therefore, categories corresponsing to "Unknown" are mapped to NaN and binary fields like `death` are mapped to `[True, False]`, with the exception of test results which are mapped to `[Positive, Negative]`. In addition the following known errors are corrected:- `RNA DETECTED (SARS-CoV-2)` is mapped to `Positive` for test results- Covid codes are extraced from entries with the pattern `CVCX`- the column `CXR severity 3` is renames `cxr_severity_2` to correctly reference a second chest x-ray.<jupyter_code>pd.concat(
    (
        compare_dfs(["1st RT-PCR result", "COVID CODE"], df),
        df[["CXR severity 3", "cxr_severity_2"]],
    ),
    axis=1,
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07478604344963792, 'fraction_numerical': 0.00750493745885451, 'mean_word_length': 5.038155802861685, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '59677', 'license': 'MIT', 'n_tokens_mistral': 2095, 'n_tokens_neox': 1977, 'n_words': 945, 'path': 'notebooks/example_cleaning_pipeline.ipynb', 'repo_name': 'deeuu/nccid-cleaning', 'size': '7595'}