<jupyter_start><jupyter_text>Data preprocessing concept work Development notebook for data processing concepts:1. Potential cleaning1. Peak detection1. Data splitting1. Visual check of every beat1. Saving in appropriate manner<jupyter_code>import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as mp
import peakutils as pu
from IPython.display import clear_output
import sys, os, glob
import scipy as sp
import scipy.interpolate as spi<jupyter_output><empty_output><jupyter_text>Read in csv file, single file for now. To be translated into batch mode down the line.<jupyter_code>fname = '20190822-092110.csv'
data = pd.read_csv('../01_raw/'+fname,header=None,names=['time','signal'])
data.shape<jupyter_output><empty_output><jupyter_text>Some data cleaning may be necessary from serial communication issues, etc. Check time stamp for contiuity and delete first lines (should be fixed on Arduino side with delay and flush?). Rollover will mess up this feature! Not used for now<jupyter_code>#if not data['time'].is_monotonic:
#    data['dt'] = data['time'].diff()
#    idx = data[data['dt'] < 0].index[0]
#    data = data[idx:]
#data.shape<jupyter_output><empty_output><jupyter_text>Plot overview of recorded data<jupyter_code>data.head(1000).plot("time","signal")<jupyter_output><empty_output><jupyter_text>Persisiting noise from interference may be filtered out fairly effciently with an rolling average approach. Not used for now.<jupyter_code>data['avg'] = data['signal'].rolling(4,center=True).mean()<jupyter_output><empty_output><jupyter_text>Show original and cleaned signal<jupyter_code>data.head(1000).plot("time","avg")<jupyter_output><empty_output><jupyter_text>Peak detecting on signal<jupyter_code>idx = pu.indexes(np.array(data['avg'].fillna(0)), thres=0.8, min_dist=150)
len(idx)<jupyter_output><empty_output><jupyter_text>Plotting<jupyter_code>mp.plot(data['time'],data['avg'],label='Raw')
mp.plot(data['time'].iloc[idx],data['avg'].iloc[idx],'rx')
#mp.plot([data['time'].iloc[idx[:-2]+120],data['time'].iloc[idx[:-2]+120]],[0,700],'b--')
#mp.plot([data['time'].iloc[idx[:-2]-60],data['time'].iloc[idx[:-2]-60]],[0,700],'g--')
#mp.legend()
mp.axis([data['time'].iloc[41500],data['time'].iloc[42500],0,700])
#mp.axis([160000,170000,0,700])<jupyter_output><empty_output><jupyter_text>Splitting according to determined peak<jupyter_code>matplotlib.use('Agg')
for i in range(1,2):
    data_int = pd.DataFrame(np.zeros((200,),dtype=int))
    #clear_output()
    #print(str(i)+' of '+str(len(idx)))
    mp.clf()
    sample = data.iloc[idx[i]:idx[i+1]]
    f = spi.interp1d(np.linspace(0,200,len(sample)), sample["avg"].values, kind='linear')
    out = f(np.linspace(0,199,200))
    outname = fname[0:15]+'-{0:04d}'.format(i)+'.csv'
    sample.to_csv('../03_samples/'+outname,header=False,index=False)
    mp.plot(sample["time"],sample["avg"])
    mp.savefig('../03_samples/'+outname[:-3]+'png')
f = spi.interp1d(np.linspace(0,200,len(sample)), sample["avg"].values, kind='linear')
out = pd.DataFrame(f(np.linspace(0,199,200)))
out<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17694543106231836, 'fraction_numerical': 0.04229899903132063, 'mean_word_length': 7.534435261707989, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '351486', 'license': 'MIT', 'n_tokens_mistral': 1209, 'n_tokens_neox': 1129, 'n_words': 202, 'path': '02_Pre/02_development/01_preprocessing.ipynb', 'repo_name': 'markusmeingast/Pulse-Monitor', 'size': '3097'}