<jupyter_start><jupyter_text>Notebook showing how to use the functions<jupyter_code>import warnings

warnings.filterwarnings("ignore")

import pandas as pd
import numpy as np
data = {'col_1': [3, 2, 1, 0], 'col_2': [4,5,6,np.nan], 'col_3': [4,5,6,np.nan]}
df = pd.DataFrame.from_dict(data)
print("Number of missing values col_2: ", df.col_2.isnull().sum())
print("Number of missing values col_3: ", df.col_3.isnull().sum())
# having the function in the notebook

# function to impute NaNs with mean value
def impute(series: pd.Series) -> pd.Series:
    mean = series.mean()
    return series.fillna(mean)
df["col_2"] = impute(df["col_2"])
print("Number of missing values col_2: ", df.col_2.isnull().sum())
print("Number of missing values col_3: ", df.col_3.isnull().sum())<jupyter_output>Number of missing values col_2:  0
Number of missing values col_3:  1<jupyter_text>After data cleaning stepMove functions to python scripts, and then also update the notebook to call the functions. This will greatly reduce the copy pasting.. especially when you have more notebooks in larger projects<jupyter_code># calling the function from the script
# as the notebook is in the same folder the import doesn't need scripts.data_preprocessing
from data_processing import impute_mean
df["col_3"] = impute_mean(df["col_3"])
print("Number of missing values col_2: ", df.col_2.isnull().sum())
print("Number of missing values col_3: ", df.col_3.isnull().sum())<jupyter_output>Number of missing values col_2:  0
Number of missing values col_3:  0
{'fraction_non_alphanumeric': 0.12034009156311315, 'fraction_numerical': 0.02419882275997384, 'mean_word_length': 5.6521739130434785, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '95288', 'license': 'MIT', 'n_tokens_mistral': 504, 'n_tokens_neox': 498, 'n_words': 168, 'path': 'optional/sample_notebook.ipynb', 'repo_name': 'nickeley/ds-eda-project', 'size': '1529'}