<jupyter_start><jupyter_text>Ematools demoThis notebook illustrates usage of `ematools` for exploring and preprocessing NDI EMA data.<jupyter_code>import os
import numpy as np
from ematools import NDIData, coords2df, rotation_ref_creator  # pip install git+https://github.com/rsprouse/ematools
from phonlab.third_party.robustsmoothing import smoothn  # pip install git+https://github.com/rsprouse/phonlab
import matplotlib.pyplot as plt
%matplotlib inline

datadir = "../data"

def sensor_plot(coords, allsensors):
    '''Quick and dirty plotting for these acquisitions.'''
    colormap = [
        ('REF', 'b'), ('RMA', 'g'), ('LMA', 'orange'), ('JAW', 'r'),
        ('UL', 'y'), ('LL', 'y'), ('JW', 'r'), ('TT', 'k'), ('TB', 'k'), ('TD', 'k'),
        ('PL', 'b')
    ]
    for sensor, color in colormap:
        try:
            sidx = np.where(np.array(allsensors) == sensor)[0][0] # sensor index
            plt.plot(coords[:,sidx,0], coords[:,sidx,1], color=color)
        except IndexError:
            pass<jupyter_output><empty_output><jupyter_text>Explore a .tsv fileLoad data in a .tsv file into an `NDIData` object. If you don't know the sensor names and locations yet, the sensor columns will automatically be prefixed with `Sn_`, where `n` is an index starting with 0.<jupyter_code>dfile = os.path.join(datadir, 'human_test_with_6dref_000.tsv')
drec = NDIData(dfile)<jupyter_output><empty_output><jupyter_text>Access the dataframe with the `df` attribute.<jupyter_code>drec.df.head()<jupyter_output><empty_output><jupyter_text>Other attributes are available for inspection:<jupyter_code>print(drec.rate)           # sample rate
print(drec.num_sensors)    # number of sensors loaded
print(drec.sensors)        # names of sensors, in order as they appear in dataframe
print(drec.state_cols)     # names of <sensor>_state columns<jupyter_output>200.15558093305927
15
['S0', 'S1', 'S2', 'S3', 'S4', 'S5', 'S6', 'S7', 'S8', 'S9', 'S10', 'S11', 'S12', 'S13', 'S14']
['S0_state', 'S1_state', 'S2_state', 'S3_state', 'S4_state', 'S5_state', 'S6_state', 'S7_state', 'S8_state', 'S9_state', 'S10_state', 'S11_state', 'S12_state', 'S13_state', 'S14_state']<jupyter_text>Checking sensor statesThere are multiple ways to explore the data columns for goodness of the data.The `state_cols` attribute contains a list of the `_state` columns. These contain comments on sensor observations. You can access these columns and use the Pandas [`describe()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html) method to find the ones in which the most frequent observations are `OK` and there is a high proportion of `freq` compared to `count`.<jupyter_code>drec.df[drec.state_cols].describe()<jupyter_output><empty_output><jupyter_text>The sensors at indexes [3, 5, 8, 9, 12] appear to contain meaningful data.If you prefer you can look at data values directly to find `NaN` or other values that are likely to indicate good or bad data.<jupyter_code>print(drec.df.S0_x.head())
print(drec.df.S3_x.head())<jupyter_output>0   NaN
1   NaN
2   NaN
3   NaN
4   NaN
Name: S0_x, dtype: float64
0   -0.155
1   -0.575
2   -0.111
3   -0.365
4   -0.202
Name: S3_x, dtype: float64<jupyter_text>Load a mix of known and unknown sensor dataWhen you know the mapping of sensors to columns, create a dictionary that maps known sensor names to index values. (Or provide a list of sensor names, using `None` as a placeholder for unknown sensors.) Use this mapping when you load the .tsv file and include the `load_all` parameter to indicate that unknown sensor values should also be loaded. The unknown sensors are prefixed with `Sn` as before.<jupyter_code>sensordict = {'REF': 3, 'JAW': 5, 'LMA': 8, 'RMA': 9, 'OS': 12}
sensorlist = [None, None, None, 'REF', None, 'JAW', None, None, 'LMA', 'RMA', None, None, 'OS']
drec = NDIData(dfile, sensordict, load_all=True)
drec.df.head()<jupyter_output><empty_output><jupyter_text>Examining the `state_cols` shows the dataframe now holds meaningful values for the known sensors, and the unknown sensors (prefixed with `Sn`) do not appear to contain anything meaningful.<jupyter_code>drec.df[drec.state_cols].describe()<jupyter_output><empty_output><jupyter_text>Load only known sensor dataWhen you are satisifed with the mapping of known sensors you can skip the loading of other sensor columns. Omit the `load_all` parameter (or set it to `False`) to load only known sensors provided by your sensor map.<jupyter_code>drec = NDIData(dfile, sensordict)
drec.df[drec.state_cols].describe()<jupyter_output><empty_output><jupyter_text>Extract coordinatesRetrieve coordinate values with the `coords()` method. The values returned are a 3D ndarray in which the first axis is time in seconds, the second axis maps the sensors, and the third axis contains the `x`, `y`, `z` values.The `coords()` method returns a tuple containing:1. The coordinate  ndarray.1. The time labels for the first dimension of the returned values.1. The sensor labels for the second dimension of the returned values.By default values for all times and all sensors are returned.<jupyter_code>coords, tlabels, slabels = drec.coords()
assert(
    coords.shape == (
        len(tlabels),
        len(slabels),
        len(['x', 'y', 'z'])
    )
)
coords.shape<jupyter_output><empty_output><jupyter_text>Below is a plot of all the sensors for a single acquisition frame. Only the `x` (0) and `y` (1) dimensions are plotted.<jupyter_code>frame = 0
x = coords[frame,:,0]
y = coords[frame,:,1]
fig, ax = plt.subplots()
ax.scatter(x, y)
for n, txt in enumerate(slabels):
    ax.annotate(txt, (x[n], y[n]))<jupyter_output><empty_output><jupyter_text>You can retrieve a subset of data values by specifying a `start` and `end` time for retrieval. You can also specify a subset of sensors with the `sensors` parameter.<jupyter_code>coords, tlabels, slabels = drec.coords(start=0.02, end=0.04, sensors=['RMA', 'LMA'])
print(coords.shape)
print(tlabels)
print(slabels)
print(coords)<jupyter_output>(4, 2, 3)
4    0.024981
5    0.029977
6    0.034973
7    0.039969
Name: sec, dtype: float64
['RMA', 'LMA']
[[[-133.622   51.254  -56.036]
  [ -21.821  132.776  -71.672]]

 [[-133.708   51.66   -55.522]
  [ -22.046  133.54   -71.078]]

 [[-133.537   51.35   -56.04 ]
  [ -22.179  133.243  -72.095]]

 [[-133.535   51.71   -55.548]
  [ -21.904  133.362  -70.99 ]]]<jupyter_text>Rotate coordinates with a `RotationRef` objectA `RotationRef` object is a special kind of `NDIData` object that can be used to  transform a set of coordinates into a reference space. There are two subtypes, `WaxBiteplate3Point` and `WaxBiteplateReferenced`. Both subtypes may have coordinate values from a 6DOF sensor; they differ in that the `WaxBiteplateReferenced` subtype's coordinates are relative to the 6DOF global reference sensor whereas the `WaxBiteplate3Point`'s coordinates are not.If there is no 6DOF sensor in a recording there can be no global reference sensor. If there is a 6DOF sensor, then it can be selected as a 'Global Reference' in the acquisition software at the time of recording, and the output data will be either the `3Point` or `Referenced` subtype depending on your selection. `WaxBiteplate3Point`A `WaxBiteplate3Point` recording does not have a global reference and will contain sensors at these locations: nasion, origin, molar, right_mastoid, left_mastoid. By default the sensor names for these are 'REF', 'OS', 'MS', 'RMA', 'LMA', respectively, and these names should be included in the sensor mapping you provide when you instantiate the object. The nasion and two mastoid sensors will also be present in non-biteplate recordings, and the mapping between these fixed points is used to calculate rotation and translation on a frame-by-frame basis.The `rotation_ref_creator()` function chooses the right kind of `RotationRef` object automatically based on the sensors you declare in your sensor mapping. The mapping must contain ['REF', 'RMA', 'LMA'] sensors for `WaxBiteplate3Point` objects or none of those sensors for `WaxBiteplateReferenced` objects.<jupyter_code>bp_sensors = [None, None, None, 'REF', 'MS', 'JAW', None, None, 'LMA', 'RMA', 'OS']
bpfile = os.path.join(datadir, 'human_test_without_6dref_biteplate_002.tsv')
bprec = rotation_ref_creator(bpfile, bp_sensors)
bprec<jupyter_output><empty_output><jupyter_text>Get raw coordinatesThe first step in rotating coordinates is to get the raw coordinates from an acquisition.<jupyter_code>rawd, tlabels, slabels = drec.coords()
sensor_plot(rawd, slabels)<jupyter_output><empty_output><jupyter_text>Apply smoothingIf desired, apply a smoothing function to the coordinates.<jupyter_code>srawd = smoothn(rawd, axis=[0], isrobust=True, s=20.0)[0]
sensor_plot(srawd, slabels)<jupyter_output><empty_output><jupyter_text>Rotate the smoothed coordinatesThe `WaxBiteplate3Point` object's `transform()` method rotates and translates an ndarray of coordinates. The first parameter passes the ndarray of coordinates. The second parameter provides a list of sensor labels in the order they appear in the second dimension of the coordinate ndarray. This list must include the same labels used to identify the nasion, right mastoid, and left mastoid sensors in the `WaxBiteplate3Point` object. By default these are the fixed sensors 'REF', 'RMA', and 'LMA'.An idealized head location is determined for the fixed sensors in the biteplate recording, and then the same sensors in the input coordinate data are mapped frame-by-frame to the idealized locations. The rotation and translation required to perform this mapping is then applied to all the coordinate data to complete the `transform()`.<jupyter_code>rotd = bprec.transform(srawd, slabels)
sensor_plot(rotd, drec.sensors)<jupyter_output><empty_output><jupyter_text>Rotate coordinates with a `WaxBiteplateReferenced` objectA `WaxBiteplateReferenced` object is similar to the `WaxBiteplate3Point` object, but has a reference sensor that provides global positioning for all the other sensor values. Transforming 6DOF data is simpler than 5DOF because it is not necessary to calculate a mapping of fixed sensors on a frame-by-frame basis. The same rotation and translation required to put the biteplate recording into reference orientation can simply be applied to all coordinates in related recordings with identical reference sensor placement. A `Referenced` wax biteplate recording should contain sensors at these locations: origin, molar. By default the sensor names for these are 'OS', 'MS', respectively, and these names should be included in the sensor mapping you provide when you instantiate the object. The 'REF' (nasion) reference sensor should not be included in the sensor mapping as it is in a different coordinate space than the other sensors.<jupyter_code>bp_sensors_sn715 = {'OS': 11,'MS': 12}
refbpfile = os.path.join(datadir, 'SN715_BitePlate.tsv')
bpref = rotation_ref_creator(refbpfile, bp_sensors_sn715)
bpref<jupyter_output><empty_output><jupyter_text>Get raw coordinates<jupyter_code>data_sensors_sn715 = {'UL': 1, 'LL': 2, 'JW': 3, 'TT': 4, 'TB': 5, 'TD': 6, 'PL': 13}
reftsvfile = os.path.join(datadir, 'SN715_MOCHA_TIMIT_451.tsv')
refrec = NDIData(reftsvfile, data_sensors_sn715)
rawref, reftlabels, refslabels = refrec.coords()
sensor_plot(rawref, refslabels)<jupyter_output><empty_output><jupyter_text>Apply smoothing<jupyter_code>srawref = smoothn(rawref, axis=[0], isrobust=True, s=20.0)[0]
sensor_plot(srawref, refslabels)<jupyter_output><empty_output><jupyter_text>Rotate the smoothed coordinatesNote that `transform()` for a 6DOF rotation reference does not include a parameter for passing sensor labels. This is because it is not necessary to find and map fixed sensors to the rotation reference. All sensors are simply transformed using the same rotation and translation required to put the biteplate recording into the reference orientation.<jupyter_code>rotref = bpref.transform(srawref)
sensor_plot(rotref, refslabels)<jupyter_output><empty_output><jupyter_text>Convert coordinates to a dataframeThe `coords2df()` function flattens a 3D ndarray of coordinates into a dataframe table. The `sec` parameter provides the time column, and the `sensors` parameter is used to construct the `_` column names. Here we create a new dataframe from smoothed and rotated coordinates.<jupyter_code>df = coords2df(rotref, sec=reftlabels, sensors=refslabels)
df.head()
df.plot('TB_x', 'TB_y')  # Should match the previous plot of TB<jupyter_output><empty_output><jupyter_text>Save the smoothed and rotated data to a fileOnce the coordinates have been converted to a dataframe, use [`to_csv()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) to save the data to a file.<jupyter_code>df.to_csv('somefile.tsv', sep='\t', index=False)<jupyter_output><empty_output><jupyter_text>A simple preprocessing functionIt is useful to preprocess your raw EMA data by applying smoothing to the raw coordinates and transforming them into reference orientation. The following simple function does this preprocessing and saves to an output file that is much smaller than the raw data file.<jupyter_code>def smooth_rotate_and_save_file(tsv, sensors, rotation_ref, out):
    '''
    Load raw NDI data from .tsv file, applying smoothing, then transform
    coordinates into a reference orientation, and save results to
    an output .tsv file.
    
    Parameters
    ----------
    tsv : str
        The path to the input .tsv file to be processed.
        
    sensors : list or dict
        The mapping of sensor names to sensor index for the input .tsv file.
        
    rotation_ref : a `RotationRef` object, e.g. `WaxBiteplate[56]D`
        The object with the reference orientation and that implements the
        `transform()` method.
        
    out : str
        The output .tsv filename.
        
    Returns
    -------
    No return value. Errors thrown during the processing will raise to the
    caller.
    '''
    rec = NDIData(tsv, sensors)
    rawd, tlabels, slabels = rec.coords()
    srawd = smoothn(rawd, axis=[0], isrobust=True, s=20.0)[0]
    if rotation_ref.has_global_ref:
        rotd = rotation_ref.transform(srawd)
    else:
        rotd = rotation_ref.transform(srawd, slabels)
    df = coords2df(rotd, sec=tlabels, sensors=slabels)
    df.to_csv(out, sep='\t', index=False)<jupyter_output><empty_output><jupyter_text>Example usage:<jupyter_code># The biteplate recording that provides the reference orientation
bp_sensors = [None, None, None, 'REF', 'MS', 'JAW', None, None, 'LMA', 'RMA', 'OS']
bpfile = os.path.join(datadir, 'human_test_without_6dref_biteplate_002.tsv')
bprec = rotation_ref_creator(bpfile, bp_sensors)

# The .tsv file to preprocess
tsvname = os.path.join(datadir, 'human_test_with_6dref_000.tsv')
sensordict = {'REF': 3, 'JAW': 5, 'LMA': 8, 'RMA': 9, 'OS': 12}

# Preprocess and save
smooth_rotate_and_save_file(
    tsv=tsvname,
    sensors=sensordict,
    rotation_ref=bprec,
    out='rotatedfile.tsv'
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.100904014889657, 'fraction_numerical': 0.025325711247008775, 'mean_word_length': 5.047025723472669, 'pattern_counts': {'":': 0, '<': 95, '<?xml version=': 0, '>': 95, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '404158', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 5060, 'n_tokens_neox': 4729, 'n_words': 1705, 'path': 'doc/ematools_demo.ipynb', 'repo_name': 'rsprouse/ematools', 'size': '15044'}