<jupyter_start><jupyter_text>01 Data Loading_[Estimated execution time: 1 min]_This tutorial will show different ways of loading data sets and how we can process different data types such as date and time.In MOGPTK there are two data structures that are important, the `Data` and `DataSet` classes. The `Data` class is the basic component that holds all data and related information for a single channel. For example, it contains the `X` and `Y` coordinates of our input data, which coordinates are for training and testing, the name of the channel, the labels for the X and Y axes, the latent function of our data (if they exist), data from predictions, data transformations (see [02 Data Preparation](https://github.com/GAMES-UChile/mogptk/blob/master/examples/02_Data_Preparation.ipynb)), and data formatters (discussed in this tutorial).The `DataSet` class is essentially an array of `Data` instances and thus represents a complete data set with multiple output channels. The separate `Data` instances can be obtained by indexing (e.g. `dataset[0]` returns the first data channel) and `DataSet` contains convenient functions over all channels.<jupyter_code>import mogptk
import pandas as pd<jupyter_output><empty_output><jupyter_text>Loading from CSV fileLoading CSV files can be done through the [`mogptk.LoadCSV`](https://games-uchile.github.io/mogptk/dataset.htmlmogptk.dataset.LoadCSV) function. We will use the airline passenger data set as an example and we'll inspect how the first lines of this file look like:<jupyter_code>with open('data/Airline_passenger.csv') as f:
    for i in range(5):
        print(f.readline(), end='')<jupyter_output>0.000000000000000000e+00 1.120000000000000000e+02
1.000000000000000000e+00 1.180000000000000000e+02
2.000000000000000000e+00 1.320000000000000000e+02
3.000000000000000000e+00 1.290000000000000000e+02
4.000000000000000000e+00 1.210000000000000000e+02<jupyter_text>We note that there are no column names in the first row, nor are columns separated by commas which usually is the case with CSV (comma-separated values) files. Our function is able to load this CSV file, but we have to explicitly pass `sep=' '` to tell that columns are separated by spaces. We also pass the `names=['time','passengers]` to set the column names explicitly as they cannot be extracted from the data. Note that `LoadCSV` is a wrapper around [pandas.read_csv](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html), so you can pass the same arguments.<jupyter_code>mogptk.LoadCSV('data/Airline_passenger.csv',
               names=['time','passengers'],
               sep=' ')<jupyter_output><empty_output><jupyter_text>Loading from DataFrameA more flexible way to load data is through pandas data frames directly because they provide functionality for loading CSV, Excel, JSON, SQL, and more data files. Furthermore, dataframes allow to filter and clean the data before handing them over to MOGPTK. We will be using the [`mogptk.LoadDataFrame`](https://games-uchile.github.io/mogptk/dataset.htmlmogptk.dataset.LoadDataFrame) to load the airline passenger data. First we load our data into a DataFrame using the `read_table` function:<jupyter_code>df = pd.read_table('data/Airline_passenger.csv',
                   names=['time','passengers'],
                   sep=' ')
df<jupyter_output><empty_output><jupyter_text>Given the DataFrame we can load this into a `DataSet` as follows. This function will by default load the first column as the `X` axis and the second column as the `Y` axis. Using the `dtypes` of the data frame for each column, it will automatically load convert `datetime` fields to numbers.<jupyter_code>data = mogptk.LoadDataFrame(df)
data<jupyter_output><empty_output><jupyter_text>Selecting columns and loading datetime valuesHere we will use the air quality data set which includes column names as well as date and time values. First we inspect the first five lines:<jupyter_code>with open('data/AirQualityUCI.csv') as f:
    for i in range(5):
        print(f.readline(), end='')<jupyter_output>Date;Time;CO(GT);PT08.S1(CO);NMHC(GT);C6H6(GT);PT08.S2(NMHC);NOx(GT);PT08.S3(NOx);NO2(GT);PT08.S4(NO2);PT08.S5(O3);T;RH;AH;;
10/03/2004;18.00.00;2.6;1360;150;11.9;1046;166;1056;113;1692;1268;13.6;48.9;0.7578;;
10/03/2004;19.00.00;2;1292;112;9.4;955;103;1174;92;1559;972;13.3;47.7;0.7255;;
10/03/2004;20.00.00;2.2;1402;88;9.0;939;131;1140;114;1555;1074;11.9;54.0;0.7502;;
10/03/2004;21.00.00;2.2;1376;80;9.2;948;172;1092;122;1584;1203;11.0;60.0;0.7867;;<jupyter_text>We note that that the separator between the columns is a semicolon `;`, so this will be our separator for `pandas.read_table`. There is no need to set the `names` parameter since all the column names are given in the first row of the data file.<jupyter_code>df = pd.read_table('data/AirQualityUCI.csv', sep=';')<jupyter_output><empty_output><jupyter_text>Data loading will automatically try and parse the `Date` column to see if it can be parsed as a datetime type. However we can also explicitly set the DataFrame column's dtype to `datetime64`. The toolkit will automatically recognize this and convert the date time values to numbers which is needed for training.<jupyter_code>df['Date'] = pd.to_datetime(df['Date'])
df['Time'] = pd.to_datetime(df['Time'], format='%H.%M.%S')<jupyter_output><empty_output><jupyter_text>Next we will load the data frame and set our input dimension to the `Date` column, and our output dimensions to the `CO(GT)` and `PT08.S1(CO)` columns. That means, we will have two channels `CO(GT)` and `PT08.S1(CO)` that share the same `X` coordinates.<jupyter_code>data = mogptk.LoadDataFrame(df, x_col='Date', y_col=['CO(GT)', 'PT08.S1(CO)'])
data<jupyter_output><empty_output><jupyter_text>DataSetsWe can expand data sets by appending another `DataSet` or `Data` instance, which will be added to the list of channels:<jupyter_code>data.append(mogptk.LoadDataFrame(df, x_col=['Date', 'Time'], y_col='NMHC(GT)'))
# How many channels do we have?
data.get_output_dims()
# And how many input dimensions do each of these channels have?
data.get_input_dims()
# What are the channel's names?
data.get_names()<jupyter_output><empty_output><jupyter_text>We can retrieve a single channel as follows:<jupyter_code>data[0]
data['CO(GT)']<jupyter_output><empty_output><jupyter_text>Using `get_train_data()` we get the `X` and `Y` training data. `get_data()` returns all data, and `get_test_data()` returns the test data.<jupyter_code>data.get_train_data()<jupyter_output><empty_output><jupyter_text>To see what data is used for training, we have to pass `transformed=True`. This returns the transformed data containing only numbers and with transformations applied to improve training results.<jupyter_code>data.get_train_data(transformed=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10510906163080076, 'fraction_numerical': 0.06982872200263504, 'mean_word_length': 5.901010101010101, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 10, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '366530', 'license': 'MIT', 'n_tokens_mistral': 2489, 'n_tokens_neox': 2145, 'n_words': 727, 'path': 'examples/01_Data_Loading.ipynb', 'repo_name': 'vishalbelsare/mogptk', 'size': '6831'}