<jupyter_start><jupyter_text>Identify the sensorsThe original reference does not state which of the employed sensors corresponds to which column in the published data set. Here we try to identify the sensors by their tuning.<jupyter_code>import sys
import os
#add path to the directory containing the plumy module to PYTHONPATH
toplevel_path = os.path.abspath(os.path.join(os.path.pardir, os.path.pardir,os.path.pardir))
sys.path.append(os.path.join(toplevel_path, 'plumy'))
import cPickle
import pandas as pd
%pylab inline
from plumy.utils import DataSelector
from plumy.utils import HDFDataSelector
from plumy.utils import ZipFileDataSelector
plt.rc('text', usetex=False)
mpl.rcParams['savefig.dpi'] = 100 # for print, go to 600
from __future__ import unicode_literals

from plumy.bouts import *
path = os.path.join(toplevel_path, 'HDFcache-Board5')
dsh = HDFDataSelector(path, drop_duplicates=True, resample=True, verbose=False)
# use 6V heater voltage and 1500 RPM fan speed, distance 1.18 m
voltage = 5
speed = 1
dist = 3
#load all gases, except out #6, CO_1000 because it has only 5 trials, and we have CO_4000 after all
gases = [1,2,3,4,5,7,8,9,10,11]
data = []
for gas in gases:
    data.append(dsh.select(gas,dist,voltage,speed))<jupyter_output><empty_output><jupyter_text>Sensor 1 is unusable due to artefactsSensor 1 has proven to be unusable due to strong artefacts in the recordings that could not be mitigated by post-processing. The next figure shows a representative example.<jupyter_code>#first gas, first trial, centre board, first sensor
signal = data[0][0].Board5.Sensor1.get_data()
xx = np.arange(0, len(signal)/100, 0.01)
f = figure(figsize(7,2.5))
gs = mpl.gridspec.GridSpec(1,2, bottom=.2)
ax = f.add_subplot(gs[0])
ax.plot(xx,signal,'k')
ax.set_frame_on(False)
ax.xaxis.set_ticks_position('bottom')
ax.yaxis.set_ticks_position('left')
ax.set_ylabel("amplitude")
ax.set_xlabel('time [s]')
ax.set_xlim(0, 260)
ylim = ax.get_ylim()
ax = f.add_subplot(gs[1])
ax.plot(xx[21200:21300],signal.values[21200:21300],'k')
ax.set_frame_on(False)
ax.xaxis.set_ticks_position('bottom')
ax.yaxis.set_ticks_position('left')
ax.set_ylim(ylim)
ax.set_yticklabels([])
ax.set_xlabel('time [s]')
f.text(0.03,0.9,"A)", fontweight='bold', fontsize=12)
f.text(0.515,0.9,"B)", fontweight='bold', fontsize=12)
f.savefig('Figures/Fig. S1 - Sensor 1 artefacts.png', dpi=600)<jupyter_output><empty_output><jupyter_text>Fig. S1: Atrefacts in the signal from sensor 1. The signal jumps between high and low values. A) A representative example of an entire measurement. B) One of the "jump events" in detail. The signal contains "jumps" that occur inadvertently (Fig. S1A). These jumps occur on virtually all recordings of sensor 1. These jumps have a complex structure (Fig. S1B). We were unable to discern the cause for these jumps. Due to their complex structure it's difficult to remove them by post-processing without introducing other artefacts that might affect the analysis. Hence, we decided to not use recordings from sensor 1 at all.   Sensor tuning<jupyter_code>sensornames = ["Sensor{}".format(i) for i in xrange(2,9)] #exclude sensor 1 due to artifacts
num_gases, num_trials, num_sensors = len(data), len(data[0]), len(sensornames)


#TODO: use data baseline
raw_amps = numpy.zeros((num_gases, num_trials, num_sensors), dtype=float)
boardname = "Board5"
for g in xrange(num_gases):
    for t in xrange(num_trials):
        for si,sn in enumerate(sensornames):
            signal = eval("data[g][t].{}.{}".format(boardname, sn)).get_data().values
            baseline = signal[:2000] # 20 s
            m = np.mean(baseline) 
#            if np.std(baseline)/m > 0.01:
#                print("{} {} baseline std/mean={:.1f}%".format(dsh.GasNames[g+1], sn, 100*np.std(baseline)/m))
            s = signal - m
            int_window = np.arange(2000,25000)
            raw_amps[g,t,si] = np.mean(s[int_window])
mean_amps = np.mean(raw_amps, axis=1) #mean over trials
f = figure(figsize=(5,6))
gs = mpl.gridspec.GridSpec(5,2)
for gi, g in enumerate(gases):
    ax = f.add_subplot(gs[gi])
    ax.bar(np.arange(0,7), mean_amps[gi])
    ax.set_title(dsh.GasNames[gases[gi]])<jupyter_output><empty_output><jupyter_text>Interesting. The patterns are highly skewed towards sensor 3having by far the strongest response. Need to normalise each sensor to max 1.<jupyter_code>mean_amps_norm = mean_amps/np.max(mean_amps, axis=0)[:,newaxis].T
f = figure(figsize=(5,8))
gs = mpl.gridspec.GridSpec(5,2)
for gi, g in enumerate(gases):
    ax = f.add_subplot(gs[gi])
    ax.bar(np.arange(0,7), mean_amps_norm[gi])
    ax.set_title(dsh.GasNames[gases[gi]])
f.tight_layout()<jupyter_output><empty_output><jupyter_text>These are the response patterns. Sensor 4 responds strongly to all gases except Ethylene. Now let's look at the tuning curves of the sensors.<jupyter_code>f = figure(figsize=(5,7))
short_gas_names = ["AcA", "Ace", "Amm", "Bnz", "But", "CO", "Eth", "Mta", "Mol", "Tol"]
gs = mpl.gridspec.GridSpec(4,2)
for si, s in enumerate(sensornames):
    ax = f.add_subplot(gs[si])
    ax.bar(np.arange(0,10), mean_amps_norm[:,si])
    ax.set_title(s)
f.tight_layout()
for i,g in enumerate(gases):
    print("{} : {}".format(i,dsh.GasNames[g]))<jupyter_output>0 : Acetaldehyde_500
1 : Acetone_2500
2 : Ammonia_10000
3 : Benzene_200
4 : Butanol_100
5 : CO_4000
6 : Ethylene_500
7 : Methane_1000
8 : Methanol_200
9 : Toluene_200<jupyter_text>Trying to match the sensors to the tuning curves from the data sheet is difficult, since there's little overlap between the odors used in the data and the gases shown in the data sheet. Fonollosa at al, 2015 used the same setup and report the following sensor sequence in the accompanying description of the data set (in the UCI repository): * Sensor 1: TGS2600 -.-  (left out here because of artifacts),  * Sensor 2: TGS2602 can't be, weak response to Toluene!  * Sensor 3: TGS2602 not identical to Sensor 2!,  * Sensor 4: TGS2620, no response to methane, but strong response to Toluene? Doesn't make sense.  * Sensor 5: TGS2612, should have methane response, but doesn't,  * Sensor 6: TGS2620, certainly quite different from sensor 4! * Sensor 7: TGS2611, should have methane response but doesn't. * Sensor 8: TGS2610, should have methane response but doesn't.This sequence clearly doesn't match. maybe the sensors were re-arranged in their sockets between both studies, or the sequence in which they were stored in the file differs from the data set that we're using here.New info:11 12 10 02 00 00 20 20 top left-right   12   02   00   20                ch1, ch3, ch5, ch7bottom left right 11  10   00   20                ch0, ch2, ch4, ch6                So, according to the new info, we'd have  * Sensor 1: TGS2611 -.-  (left out here because of artifacts),  * Sensor 2: TGS2612 - methane response is there, but largest response to CO. * Sensor 3: TGS2610 - no methane response? * Sensor 4: TGS2602 - good response to Toluene * Sensor 5: TGS2600 - similar enough to sensor 6, except for the smaller ammonia response.  * Sensor 6: TGS2600 - dito * Sensor 7: TGS2620 - sufficiently similar to sensor 8, CO response is present, but not methane! * Sensor 8: TGS2620 - dito  Assessing sensor similarity<jupyter_code>import mdp
wn = mdp.nodes.WhiteningNode(output_dim=5)
# need sensors*trials x gases
raw_amps_reshape = raw_amps.reshape((num_gases, num_trials*num_sensors)).T
wn.train(raw_amps_reshape)
amps_pca = []
for i in xrange(num_sensors):
    amps_pca.append(wn.execute(raw_amps_reshape[i*num_trials:(i+1)*num_trials,:]))
amps_pca = np.concatenate(amps_pca)
def make_nice_pca_axes(ax):
    ax.xaxis.set_ticks_position('none')
    ax.yaxis.set_ticks_position('none')
    ax.set_frame_on(False)
    xl = ax.get_xlim()
    yl = ax.get_ylim()
    ap = {"arrowstyle":"->", "linewidth":.8}
    ax.annotate("",xy=(xl[1]*.9, 0), xytext=(xl[0]*.95,0), arrowprops=ap, zorder=-1)
    ax.annotate("",xy=(0, yl[1]*.9), xytext=(0,yl[0]*.95), arrowprops=ap, zorder=-1)
    ax.text(0, yl[1]*.95,"PC 2", fontsize=9, va='center', ha='center')
    ax.text(xl[1]*.97,0,"PC 1", fontsize=9, va='center', ha='center')
    ax.set_xlim(xl[0], xl[1]*2.25)
    ax.set_xticks([])
    ax.set_yticks([])
f = figure()
ax = f.add_subplot(111)
colors = cm.rainbow(np.linspace(0,1,num_sensors))
for i in xrange(num_sensors):
    for j in xrange(num_trials):
        v = amps_pca[j*num_sensors + i,:]
        l = ax.plot(v[0], v[1], '.', color=colors[i])
    l[0].set_label(sensornames[i])
make_nice_pca_axes(ax)
legend(numpoints=1, frameon=False, loc='upper right', fontsize=8)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12713340299547196, 'fraction_numerical': 0.0443515615929409, 'mean_word_length': 4.72358803986711, 'pattern_counts': {'":': 2, '<': 24, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '382427', 'license': 'MIT', 'n_tokens_mistral': 3387, 'n_tokens_neox': 3066, 'n_words': 890, 'path': 'ipnotebooks/manuscript_figures/Fig. S1 - Sensor 1 artefacts, and Sensor tuning.ipynb', 'repo_name': 'sesutton93/exploiting_plume_structure', 'size': '8613'}