<jupyter_start><jupyter_text>Exploratory Data Analysis, 2D and 3D plotting**Aims of the script:*** Load a device-specific dataframe (eg: one csv file containing the data acquired by one device)* Identify a part of the device-specific dataframe that you want to study: either a specific day or night (part 2) or several days (part 3)* Normalize the data using standardscaler* Plot the time series in 2D* Plot multidimensional data in 3D independently from time. Use K-means to identify multidimensional clusters**To run the script:**- To speed up the processing, this script was designed to work with a device-specific dataframe that should be located in the folder ../data/interim. The user enters the number of the device in a two-digit string format ('01', '02'... '51') and follows the instructions.- If the device-specific dataframes do not exist yet, it will be automatically generated. This script expects to find the following files in the folder ../Data/raw:'        * raw_data_part1.csv        * raw_data_part2.csv        * raw_data_part3.csv        * raw_data_part4.csv        * raw_data_part5.csv**Interpretation and conclusions:*** This code was written to be flexible to change device number and time frame.* Depending on the device, the data is variable among devices and between devices. There are trends going over several or highly instable signal difficult to analyse.> Questions:> Contact <NAME> at <EMAIL>  1 - Loading the relevant library<jupyter_code>import pandas as pd
import numpy as np
import time
from datetime import datetime
from pytz import timezone
import re
import matplotlib.pyplot as plt
from matplotlib.dates import DateFormatter
from mpl_toolkits.mplot3d import Axes3D
import seaborn as sns
sns.set(style="whitegrid")

# To play with the interactive plots, uncomment the following lines
# %matplotlib notebook

from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans<jupyter_output><empty_output><jupyter_text>!jupyter lab --version !conda list ipywidgets<jupyter_code>from functions import load_ds, load_raw_data, df_dev_generator, df_dev_formater, df_generator, day_night_csv<jupyter_output><empty_output><jupyter_text>2 - Plotting the data of a specific night or day    a. Load a specific dataset by inserting the number of the device as a string of 2 digits. Use the interactive plot to identify the date and time when the resident wakes up or goes to bed     b. Fill the information collected in the step a into the dataframe day_night_data. Run day_night_csv that will create csv files in the folder ../Data/interim    c. Load a specific day or night dataframe which has the following name structure:    name = str(device) + '_' + str(day_night) + '_' + str(days) + '_from_' + str(hour_formatted)    d. Data normalisation    e. 2D plotting of the time course     f. 3D plotting: representing the data of the different parameters (light, CO2, humidity and temperature) and look for clusters using K-mean clustering.  2a - Load the device-specific dataset<jupyter_code># Enter the number of the device as a string with two digits and execute the cell
device_nb = '01'

# Load the device-specific dataframe. Show the most important information
assert isinstance(device_nb, str) and len(device_nb)==2 and sum(d.isdigit() for d in device_nb)==2, 'WARNING: device_nb must be a string of 2-digits!'
assert int(device_nb)>=1 and int(device_nb)<=51, 'This device does not belong to the dataframe'
device, df_dev = load_ds(device_nb)

# Convert the variable device from a np.array to a string
regex = re.compile('[^A-Za-z0-9]')
device = regex.sub('', str(device))

# Show the full device-specific dataframe
print('\nShowing the dataframe. Use it to find the time window you would like to investigate:')
start = df_dev['ds'][0].strftime('%Y-%m-%d')
last = df_dev['ds'][df_dev.shape[0]-1].strftime('%Y-%m-%d')
df, predict_n, today_index, lookback_n = df_generator(df_dev, device, 'co2', start, last,  '1T', 0.016)
plt.show()<jupyter_output>Check report:
##############################################
['Device contained in the dataset: device01']
['Tenant using the device: tenant01']

Data types:
device                                object
tenant                                object
ds             datetime64[ns, Europe/Zurich]
light                                float64
temperature                          float64
humidity                             float64
co2                                  float64
dtype: object

Available data from the 2019-05-01 to the 2020-03-26.

Showing the dataframe. Use it to find the time window you would like to investigate:
Full dataset: 2019-05-01 to the 2020-03-26. Analysed data the 2019-05-01 to the 2020-03-26.<jupyter_text>**NOTE: *This figure is meant to be used in the interactive mode of jupyter notebook. It will be displayed small and with green horizontal lines on the Gitlab. To use the interactive mode, uncomment the line  %matplotlib notebook and run the notebook.***  2b - Generating day and night-specific dataframes This function generates dataframes for all parameters from the time they go to bed to the time they wake up.This function was for exploration but not further use in the analysis.    Parameters:- device: String indicating the device.- day_night_dataframe: Pandas dataframe containing the columns          day_gotobed(%Y-%m-%d), time_gotobed(%H:%M),          day_wakeup(%Y-%m-%d), time_wakeup(%H:%M)          Return:- save one csv for of the data of night- save one csv for of the data of day<jupyter_code># Example Device01, from 2019 05 02 to the 2019 05 12
# Indicate here the date and time when the resident goes to bed and wake up

day_night_data = [['2019-05-01', '20:00', '2019-05-02', '7:00'],
 ['2019-05-02', '20:00', '2019-05-03', '6:30'],
 ['2019-05-03', '20:30', '2019-05-04', '9:00'],
 ['2019-05-04', '19:00', '2019-05-05', '6:00'],
 ['2019-05-05', '19:00', '2019-05-06', '8:30'],
 ['2019-05-06', '19:00', '2019-05-07', '6:00'],
 ['2019-05-07', '20:00', '2019-05-08', '8:30'],
 ['2019-05-08', '18:00', '2019-05-09', '7:00'],
 ['2019-05-09', '18:00', '2019-05-10', '6:30'],
 ['2019-05-10', '18:00', '2019-05-11', '6:30'],
 ['2019-05-11', '18:00', '2019-05-12', '6:00'],
 ['2019-05-12', '18:00', '2019-05-13', '5:00']]

day_night_df = pd.DataFrame(day_night_data, columns = ['day_gotobed', 'time_gotobed',
                                                        'day_wakeup', 'time_wakeup'])
day_night_df.head(12)
day_night_csv(df_dev, device, day_night_df)<jupyter_output>csv generation completed in 2 seconds.<jupyter_text>2c - Load a specific day or night dataframe: Example of the day of the 2019-05-03 that started at 6:30AM<jupyter_code>day_night = 'Day' # String. Day or Night
days = '2019-05-03' # String.
hour_formatted = '6h30' 
folder = '../data/interim/'

name = str(device) + '_' + str(day_night) + '_' + str(days) + '_from_' + str(hour_formatted)

file_name = folder + name + '.csv'
file_name

df = pd.read_csv(file_name)
df = df.drop(columns=['Unnamed: 0'])
df.head(-10)<jupyter_output><empty_output><jupyter_text>2d - Data normalization<jupyter_code>data = df.iloc[:, 3:]
scaler = StandardScaler()
scaled_data = scaler.fit_transform(data)
scaled_data = pd.DataFrame(scaled_data, columns=data.columns)
scaled_data.head(2)
df_scaled = df.drop(columns=['light', 'temperature', 'humidity', 'co2'])
df_scaled = pd.concat([df_scaled, scaled_data], axis=1)
df_scaled.loc[:, 'ds'] = pd.to_datetime(df_scaled.loc[:, 'ds'])
# df_scaled.dtypes
df_scaled = df_scaled.set_index('ds')
df_scaled.head(2)<jupyter_output><empty_output><jupyter_text>2e. 2D plotting: distribution and time course<jupyter_code>scaled_data.hist(figsize=(7,4), linewidth=1)
plt.show()
# Correlation matrix
df_scaled.corr()<jupyter_output><empty_output><jupyter_text>There is one correlation between the parameters:- **co2 is correlated with the temperature***Note: This is variable among devices.*<jupyter_code>plt.close()
ax = df_scaled.plot(figsize=(7,4), linewidth=1, fontsize=10)
myFmt = DateFormatter("%d/%m %H:%M")
ax.xaxis.set_major_formatter(myFmt)
plt.gcf().autofmt_xdate()
plt.xlabel('Date and time', fontsize=8)
plt.ylabel('Normalised parameter value (StandardScaler)', fontsize=8)
plt.title(name, fontsize=14)
plt.legend(loc='upper right')
plt.show()
# Plotting of raw data

CO2 = df_scaled[['co2']]
CO2.plot(figsize=(7,4), linewidth=1, fontsize=10)
ax.xaxis.set_major_formatter(myFmt)
plt.gcf().autofmt_xdate()
plt.show()
# Identifying trend in Time Series

CO2.rolling(100).mean().plot(figsize=(7,4), linewidth=1, fontsize=10)
plt.xlabel('Time (arbitrary unit). Full day or night displayed', fontsize=10)
plt.gcf().autofmt_xdate()
plt.show()
# First-order differencing
# import matplotlib.dates as mdates
plt.close()
from matplotlib.dates import DateFormatter
CO2.diff().plot(figsize=(7,4), linewidth=1, fontsize=10)
plt.xlabel('Time (arbitrary unit). Full day or night displayed', fontsize=10)
plt.gcf().autofmt_xdate()
# First-order differencingplt.show()
# First-order differencing. Smoothed data
CO2.rolling(100).mean().diff().plot(figsize=(7,4), linewidth=1, fontsize=10)
plt.xlabel('Time (arbitrary unit). 1 day displayed', fontsize=10)
plt.gcf().autofmt_xdate()
plt.show()<jupyter_output><empty_output><jupyter_text>**Conclusion and limitation*** Using the smoothed data of the derivative might be helpful. It may give the possibility to identify transients changes in CO2 that can be translated into the resident activity (coming and leaving the room).* However, it would be more accurate to have more data than a specific day or night to calculate the normal resident activity. Automatically detecting day and night represents a challenge that I addressed using Prophet.  2f. 3D plotting to see multiparameter clusters, independently from time This code is from this [tutorial](/Users/guillaume/Documents/DS2020/guillaume-azarias/03-Machine-Learning/Day-8-Clustering/Exercice_1_GA.py) done at the Propulsion Academy.<jupyter_code># This is the original function

def plot_3D(data_3D, labels=None, **kwargs):
    fig_ = plt.figure(figsize=(7, 7))
    ax = Axes3D(fig_)
    ax.scatter(data_3D.iloc[:, 0], data_3D.iloc[:, 1], data_3D.iloc[:, 2],
               c=labels,
               s=20,
               alpha=0.15)
    
    label = list(data_3D.columns.values)
    ax.set_xlabel(label[0])
    ax.set_ylabel(label[1])
    ax.set_zlabel(label[2])
    return ax
    
def kmeans_clusters_3D(data_, n_clusters):
    km_ = KMeans(n_clusters)
    km_.fit(data_)

    plot_3D(data_, km_.labels_.astype(float))

    return
# For K-means clustering, I chose to replace NaN by 0.
# Further investigation of K-means clustering may need to consider better fillna() method
if scaled_data.isnull().sum().sum()>0:
    scaled_data = scaled_data.fillna(0)
    print('NaN removed.')
else:
    print('No NaN in the dataframe.')
n_cluster = 4

kmeans_clusters_3D(scaled_data[['temperature', 'light', 'co2']], n_cluster)
plt.title('Graphical 3D representation of physical parameters divided in ' + str(n_cluster) + ' clusters')
plt.show()
n_cluster = 4

kmeans_clusters_3D(scaled_data[['temperature', 'humidity', 'co2']], n_cluster)
plt.title('Graphical 3D representation of physical parameters divided in ' + str(n_cluster) + ' clusters')
plt.show()
n_cluster = 5

kmeans_clusters_3D(scaled_data[['temperature', 'humidity', 'co2']], n_cluster)
plt.title('Graphical 3D representation of physical parameters divided in ' + str(n_cluster) + ' clusters')
plt.show()<jupyter_output><empty_output><jupyter_text>3 - Plotting in 3D the dataframe of a week    a. Load a specific dataset by inserting the number of the device as a string of 2 digits. Use the interactive plot to identify the date and time when the resident wakes up or goes to bed    b. Decide for the start and end of the dataframe you want to plot    c. Load and plot the data specific to the start and end of the dataframe    d. Data normalisation    e. 2D plotting of the time course     f. 3D plotting: representing the data of the different parameters (light, CO2, humidity and temperature) and look for clusters using K-mean clustering.    *Example: dataframe from the 2020-01-07 to the 2020-01-14*  3a - Load the device-specific dataset<jupyter_code># Enter the number of the device as a string with two digits and execute the cell
device_nb = '01'

# Load the device-specific dataframe. Show the most important information
assert isinstance(device_nb, str) and len(device_nb)==2 and sum(d.isdigit() for d in device_nb)==2, 'WARNING: device_nb must be a string of 2-digits!'
assert int(device_nb)>=1 and int(device_nb)<=51, 'This device does not belong to the dataframe'
device, df_dev = load_ds(device_nb)

# Convert the variable device from a np.array to a string
regex = re.compile('[^A-Za-z0-9]')
device = regex.sub('', str(device))

# Show the full device-specific dataframe
print('\nShowing the dataframe. Use it to find the time window you would like to investigate:')
start = df_dev['ds'][0].strftime('%Y-%m-%d')
last = df_dev['ds'][df_dev.shape[0]-1].strftime('%Y-%m-%d')
df, predict_n, today_index, lookback_n = df_generator(df_dev, device, 'co2', start, last,  '1T', 0.016)
plt.show()<jupyter_output>Check report:
##############################################
['Device contained in the dataset: device01']
['Tenant using the device: tenant01']

Data types:
device                                object
tenant                                object
ds             datetime64[ns, Europe/Zurich]
light                                float64
temperature                          float64
humidity                             float64
co2                                  float64
dtype: object

Available data from the 2019-05-01 to the 2020-03-26.

Showing the dataframe. Use it to find the time window you would like to investigate:
Full dataset: 2019-05-01 to the 2020-03-26. Analysed data the 2019-05-01 to the 2020-03-26.<jupyter_text>3b. Decide for the start and end of the dataframe you want to plot<jupyter_code>start = '2020-01-07'
last = '2020-01-14'<jupyter_output><empty_output><jupyter_text>3c. Load and plot the data specific to the start and end of the dataframe<jupyter_code>data = pd.DataFrame(columns=['light', 'temperature', 'humidity', 'co2'])
df, predict_n, today_index, lookback_n = df_generator(df_dev, device, 'co2', start, last,  '1T', 0.016)
plt.show()
data['co2'] = df.loc[:, 'y']
df, predict_n, today_index, lookback_n = df_generator(df_dev, device, 'light', start, last,  '1T', 0.016)
plt.show()
data['light'] = df.loc[:, 'y']
df, predict_n, today_index, lookback_n = df_generator(df_dev, device, 'temperature', start, last,  '1T', 0.016)
plt.show()
data['temperature'] = df.loc[:, 'y']
df, predict_n, today_index, lookback_n = df_generator(df_dev, device, 'humidity', start, last,  '1T', 0.016)
plt.show()
data['humidity'] = df.loc[:, 'y']<jupyter_output>Full dataset: 2019-05-01 to the 2020-03-26. Analysed data the 2020-01-07 to the 2020-01-14.<jupyter_text>3d. Data normalisation<jupyter_code>data.head()
# Scaling
scaler = StandardScaler()
scaled_data = scaler.fit_transform(data)
scaled_data = pd.DataFrame(scaled_data, columns=data.columns)
scaled_data.head(2)<jupyter_output><empty_output><jupyter_text>3e. 3D plotting of clusters<jupyter_code>n_cluster = 5

kmeans_clusters_3D(scaled_data[['temperature', 'humidity', 'co2']], n_cluster)
plt.title('Graphical 3D representation of physical parameters divided in ' + str(n_cluster) + ' clusters')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12263178244619152, 'fraction_numerical': 0.04678437943033896, 'mean_word_length': 4.403551532033426, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 60, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '540819', 'license': 'MIT', 'n_tokens_mistral': 5499, 'n_tokens_neox': 4974, 'n_words': 1633, 'path': '1_exploration/Data_Exploration.ipynb', 'repo_name': 'guillaume-azarias/Human-Activity-Prediction', 'size': '15551'}