<jupyter_start><jupyter_text>Data PreprocessingThis notebooks loads and preprocesses the raw data for missing values imputation.  0 - Setup  0.1 - ImportsLoad the necessary dependencies.<jupyter_code>%%capture
from ydata.connectors import GCSConnector, LocalConnector
from ydata.utils.formats import read_json<jupyter_output><empty_output><jupyter_text>0.2 - Auxiliary FunctionsThe auxiliary functions are custom-designed utilities developed for this use case.<jupyter_code>from ydata.dataset import Dataset
from preprocess import preprocess_data<jupyter_output><empty_output><jupyter_text>1 - Load DataFirst extraction: 26 November 2020 - 15 July 2021Second extraction: 15 July 2021 - 29 September 2021<jupyter_code># Load the credentials
credentials = read_json('gcs_credentials.json')

# Create the connector for Google Cloud Storage
connector = GCSConnector('ydatasynthetic', gcs_credentials=credentials)

# Read the first extraction data 26 Nov 2020 to 15 Jul 2021
data = connector.read_file('gs://pipelines_artifacts/wind_measurements_pipeline/data/ethiopia_wind_lot_nan.csv')<jupyter_output><empty_output><jupyter_text>2 - Data ProcessingTrain: 26 November 2021 - 31 August 2021Holdout: 1 September 2021 - 29 September 2021Steps:  1. Select the relevant columns  2. Convert to Pandas DataFrame  3. Cast to the correct data types  4. Join the readings from both extractions into a single table  5. Split into train and holdout<jupyter_code>train, holdout = preprocess_data(data)<jupyter_output><empty_output><jupyter_text>3 - Store DataAfter data is preprocessed, store locally or to cloud storage.<jupyter_code>%%capture
local_connector = LocalConnector()

# Store the train
local_connector.write_file(data=train.reset_index(), path='train_allmeters.csv', index=True)

# Store the holdout
local_connector.write_file(data=holdout.reset_index(), path='holdout_allmeters.csv', index=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07544224765868887, 'fraction_numerical': 0.037460978147762745, 'mean_word_length': 7.079831932773109, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '597817', 'license': 'MIT', 'n_tokens_mistral': 640, 'n_tokens_neox': 564, 'n_words': 168, 'path': '5 - synthetic-data-applications/time-series/missing-values-imputation/1_Preprocess.ipynb', 'repo_name': 'ydataai/Blog', 'size': '1922'}