<jupyter_start><jupyter_text>Essentials of Post-processing    Keywords:  ```Low-Pass Filter```, ```Python```, ```Plotting```  I. Introduction     1. Background  As important as signal acquisition, is making sense out of thedata once collected. This generally requires a set of preparatory steps thatinvolve taking the recorded data, loading it, and performing utilitary taskssuch as filtering, scaling, clipping, graphical representation, and several others.Although there are a number of ways to address these tasks, executing themprogramaticaly can have tremendous advantages over other approaches. Withthis set of experiments, we will briefly explore essential concepts related withsaid preparatory steps, usually known as post-processing.    2. Objectives* Get familiar with commonly used functions for loading, post-processing and plotting pre-recorded data using the Python programming language * Understand how to extract common metrics from pre-recorded data* Implement a code base to help automate post-processing tasks, in particular by performing operations for batches of data rather than for individual files   3. Materials * Anaconda Python 2.7* OpenSignals (r)evolution software (optional)* 1x BITalino (r)evolution Board BT (optional)  II. Experimental      1. Loading and Plotting Retrieving a file and providing a graphical representation of its content (eitherin part or in full) is generally an entry point to get a better grasp of the datawe’re dealing with. As you know by now, it is also a useful tool to support thereporting of tasks related with data analytics and interpretation. In this experiment we will implement code for loading and plotting data, while controlling some of the visualization parameters.    1. Using OpenSignals, record a file with Accelerometer (ACC) sensor data in ASCII format, consisting of a full revolution of the ACC sensor performed very slowly around an axis that is perpendicular to the one being recorded; a pre-recorded example can be downloaded from:     https://github.com/BITalinoWorld/revolution-sample-data/tree/master/ACC      2. Create a new (empty) Python script in the Spyder IDE    3. Write a script that loads the file, retrieves the ACC data column and graphically plots the amplitude versus time (in seconds)    4. Add code to include title and axis labels in the plot    5. Run your script and verify the behaviour; notice how, even though a biomechanical activity was recorded and the revolution has been done slowly, the signal still exhibits high frequency components that, in this particular case, should mostly correspond to noise    6. One way to remove these high frequency components is by filtering the signal; implement a simple n-point moving average (aka rolling mean or smoother) filter in your script; consider 10% of the sampling rate for the window size (i.e. n)    7. Modify the plotting part of your script to show an overlay of the raw and filtered data; format the plot to present the raw data as a black line with thickness of 3 points and the filtered data as a dashed red line    8. Run your script and verify the behaviour 2. Write a script that loads the file, retrieves the ACC data column and graphically plots the amplitude versus time (in seconds)<jupyter_code>import matplotlib.pyplot as plt
import numpy as np

ACCdata = np.genfromtxt(r'..\X.Example_Files\SampleACC.txt', delimiter=",", skip_header=3, usecols=[0,-1], names=["x", "y"])
plt.plot(ACCdata['x'], ACCdata['y'])<jupyter_output><empty_output><jupyter_text>Explore         Detailed information regarding the plot(...) function, made availableby the matplotlib module in Python, and its formatting options can befound at:https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html         2. Calibration As we have seen before, with adequate previous knowledge of certain specificationsof the acquisition system and/or sensors, it is possible to convert the digitalcodes produced by an Analog-to-Digital Converter (ADC) into their correct original physical units. Sometimes though, such information may not be readilyavailable or accurately known beforehand, in which cases we may be able toresort to empirical evidence to support the conversion process. This is generallyknown as calibration, and will be the focus of our next experiment using ACCdata as a case study.    1. In the case of the accelerometer we know that the device is sensitive to the gravity acceleration and that, in a full revolution, it crosses the ±1g acceleration values; these will correspond to the maximum and minimum values in the output signal, and can effectively be used as calibration coefficients to convert raw data to the correct physical units    2. Create a copy of the Python script developed in Section 8.4    3. Modify the code in such way that the plots show the amplitude values in g (their correct physical units); use the maximum and minimum values of the filtered ACC data as coefficients to normalize and offset the signal in such way that it is bound to ±1g    4. Run your script and verify the behaviour; instead of being shown as the adimensional range of values that are the ADC codes, the data should be ranging between ±1g    Explore         You can find additional information about the BITalino ACC sensor(used in this experiment) on the data sheet available at:http://bitalino.com/datasheets/REVOLUTION_ACC_Sensor_Datasheet.pdf      3. Electromyographic (EMG) Onset Detection Until now we have been working with individual files, however, in multiple practical applications it is very useful to repeatedly apply the same operations to a batch of files. In this experiment we will explore this concept, applied to areal-world use case of automating the characterization of EMG onsets in a set of recorded files.    1. If you don’t have a previous recording already, record a file with Electromyography (EMG) sensor data in ASCII format (using OpenSignals), by applying the electrodes to a muscle of your liking and performing a cycle of CONTRACTION-RELEASE-REST five times (as before, maintaining the contraction for two seconds and resting for two seconds, and trying to start with a low intensity contraction, gradually increasing the level in each repetition in such way that the last corresponds to your maximum voluntary contraction ability); a pre-recorded example of EMG data can be downloaded from: https://github.com/BITalinoWorld/revolution-sample-data/tree/master/EMG    2. Create a new (empty) Python script in the Spyder IDE        3. Implement a script that loads the recorded file, computes the EMG envelope using a low pass filter with 5Hz cutoff frequency (see Figure 8.1), and plots an overlap with both signals       4. Run your script, verify the behaviour, and determine a suitable threshold (by visual observation), that could be used to isolate the onsets based on the EMG envelope       5. Expand your script to threshold the EMG envelope, detect the transitions in the resulting signal, and depict vertical lines in the plot marking, with different colors, the instants at which an onset and a return to a resting condition have been detected       6. Run your script and verify the behaviour       7. Exchange your EMG recording with four other class mates and collect the EMG recordings from them (alternatively collect additional data from your group colleagues)        8. Gather all the recordings in a new folder, having the folder of your script as the root      9. Copy the walktree(...) function (Figure 8.2) to the beginning of your script (just bellow the import lines)    10. Expand your script to, instead of loading a pre-defined file, traverse your working directory using the walktree(...) function with a specification to select only files with TXT extension; you can do this by aggregating your existing plotting code under a for statement in the form:         for filename in walktree(’.’, r’^.TXT$’):          11. Run your script and verify the behaviour; you should have a single plot with all the data from all the files accumulated in it       12. Modify your code to create a new figure in each iteration of the repetition structure (using the figure() function)       13. Run your script and verify the behaviour; you should obtain one plot per individual file   3.1. Example - Low Pass filter implementation in Python  The following Python code exemplifies a recursive folder transverser.<jupyter_code>from scipy import signal

def lowpass(s,f,order=2,fs=1000.0):
    b, a = signal.butter(order, f/(fs/2))
    return signal.lfilter(b, a, s)

def walktree (top=None, spec=None):
    if top is None:
        top = os.getcwd()
    if spec is None:
        spec = r'.*?'
        
    prog = re.compile(spec)
    
    for root, _, files in os.walk(top):
        for name in files:
            if prog.search(name):
                fname = os.path.join(root, name)
                yield fname<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0472299944040291, 'fraction_numerical': 0.0071628427532176835, 'mean_word_length': 4.183294663573085, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '233517', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2160, 'n_tokens_neox': 2035, 'n_words': 1285, 'path': 'D.Feature_Extraction/D001 Essentials of Post-processing.ipynb', 'repo_name': 'anascacais/ScientISST-notebooks', 'size': '8935'}