<jupyter_start><jupyter_text>Explorative Data AnalysisIn this exercise, you will learn how to handle datasets, to load, manipulate and visualize data in python. Furthermore, we will have a look at calculating features on data.  PandasA common package for data handling and analysis in python is `pandas`. The fundamental data structure introduced by this package is the data frame. A data frame is a table that can be handled like a table in a relational data base: We can select rows, columns or both.<jupyter_code>import numpy as np
import pandas as pd
from scipy.io import arff
import matplotlib.pyplot as plt

data = arff.loadarff('S08.arff')
df = pd.DataFrame(data[0])


#plt.plot(df["Sensor_T8_Acceleration_X"])
df<jupyter_output><empty_output><jupyter_text>Selecting rows and columnsUsing pandas, it is simple to select only certain rows or columns of a data frame. One option is to select columns by name:<jupyter_code>accx = df.loc[:,"Sensor_T8_Acceleration_X"]<jupyter_output><empty_output><jupyter_text>Here, `:` stands for "select all rows". If just a single column is returned, the result is of type `Series`, otherwise it is a data frame again. We can also access columns by index:<jupyter_code>acc = df.iloc[:,1:4]
acc<jupyter_output><empty_output><jupyter_text>Rows can be accessed in the same way. For example, following expression returns only the first 5 rows:<jupyter_code>df.iloc[0:5,:]<jupyter_output><empty_output><jupyter_text>We can combine both, lik this:<jupyter_code>df.loc[0:5,["Sensor_T8_Acceleration_X", "Sensor_T8_Acceleration_Y"]]<jupyter_output><empty_output><jupyter_text>Another useful option is to access rows or columns via boolean expression. For example, the following expression returns all rows, where the value of `Sensor_T8_Acceleration_X` is less than 0.7.<jupyter_code>df.loc[df.Sensor_T8_Acceleration_X < 0.7,:]<jupyter_output><empty_output><jupyter_text>Inserting new valuesInserting new values into a data frame is possible in the same way. The following expression sets all values of column `Sensor_T8_Acceleration_Y` that are smaller than 0 to 0.<jupyter_code>df.loc[df.Sensor_T8_Acceleration_Y < 0,"Sensor_T8_Acceleration_Y"] = 0
df<jupyter_output><empty_output><jupyter_text>ApplyOften, we want to apply a function to a complete row or column of the data. The function `apply` allows that. The following expression calculates the mean per column:<jupyter_code>df.iloc[:,1:31].apply(np.mean)<jupyter_output><empty_output><jupyter_text>Exercise 1Compute the distribution of classes in `df`. (`collections.Counter`)<jupyter_code>from collections import Counter

hist = Counter(df.loc[:,"class"].values)
hist<jupyter_output><empty_output><jupyter_text>Plot the distribution of classes as a bar plot.<jupyter_code>histdf = pd.DataFrame.from_dict(hist, orient='index')
histdf.plot(kind='bar')
print(histdf)<jupyter_output>0
b'open'   1806
b'take'   2458
b'put'    1695
b'close'  2201
b'walk'    513
b'other'   214
b'fill'   5754<jupyter_text>Plot multiple accelerometer axes (e.g. (z.B. "Sensor_T8_Acceleration_X","Sensor_T8_Acceleration_Y", "Sensor_T8_Acceleration_Z") as a line plot. The different axes should be drawn in different colors.<jupyter_code>dfs = df.loc[:,["Sensor_LeftLowerLeg_Acceleration_X", "Sensor_LeftLowerLeg_Acceleration_Y", "Sensor_LeftLowerLeg_Acceleration_Z"]]
#dfs = df.loc[:,["Sensor_RightForeArm_Acceleration_X", "Sensor_RightForeArm_Acceleration_Y", "Sensor_RightForeArm_Acceleration_Z"]]

plt.subplot(2,1,1)
plt.plot(dfs)
plt.subplot(2, 1, 2)
plt.plot(df.loc[:,"class"])<jupyter_output><empty_output><jupyter_text>Exercise 2Next, we want to calculate some features of the data. For sequential data, we will typically calculate features in a window-based fashion. That is, for rows 1 to n, we calculate some feature (mean, ...), then for rows n+1 to 2n and so on. The windows can also be overlapping. In the lecture, you learned about other features that are, for example, based on the frequency of the signal. Implement a function `feature` which calculates a feature of the data, given a window size, an overlap, a dataset, and a statistical feature function (mean, ...). Calculate the mean, median and variance of accelerometer data of the right leg with window sizes 128, 256 and 512. Use 50% overlap and plot the result.<jupyter_code>import matplotlib.pyplot as plt

def feature(wsize,overlap,data,ffunct):
    start = 0
    numwindows = int(len(data)/(wsize-overlap))
    result = np.zeros(numwindows)
    for i in range(numwindows):
        end = start + wsize-1
        windowed = data[start:end]
        result[i] = ffunct(windowed)
        start = start + wsize - overlap
    return(result)

meanX = feature(64,32,df.loc[:,"Sensor_T8_Acceleration_X"].values,np.var)
plt.plot(meanX)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09422555764019179, 'fraction_numerical': 0.01855326245570148, 'mean_word_length': 6.07669616519174, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '159069', 'license': 'MIT', 'n_tokens_mistral': 1619, 'n_tokens_neox': 1554, 'n_words': 498, 'path': '.ipynb_checkpoints/02-data-analysis-checkpoint.ipynb', 'repo_name': 'stefanluedtke/WS19AI2Exercises', 'size': '4797'}