<jupyter_start><jupyter_text>UT2000 Data ProcessingFor the MADS framework paper, we need to present the data. This notebooks helps reorganize the data so that it should be clear for anyone that picks it up.<jupyter_code>import warnings
warnings.filterwarnings('ignore')
import os
import os.path
from os import path

from datetime import datetime, timedelta
import pytz

import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.dates as mdates

import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>ID CrossoverWe need the IDs from the multiply modalities to cross-reference the participants.<jupyter_code>ids_1000 = pd.read_csv('../data/raw/ut1000/admin/id_crossover.csv')
# limiting so we don't have repeats with ut2000
ids_1000 = ids_1000[ids_1000['record'] < 2000]
ids_1000.head()
ids_2000 = pd.read_csv('../data/raw/ut2000/admin/id_crossover.csv')
ids_2000.head()
# combining ut1000 and 2000 records
ids = ids_1000.append(ids_2000)<jupyter_output><empty_output><jupyter_text>Fitbit DataThe Fitbit data is combined into a single CSV that contains all participants for a certain datatype. I need to import each file, separate by beiwe ID, and then export again.<jupyter_code>data_dir = '../data/raw/ut2000/fitbit/'
export_dir = '../data/processed/MADS_framework_data/'
local = pytz.timezone ("America/Chicago")
# looping through all the files
for file in os.listdir(data_dir):
    if file[-1] == 'v': # checking if csv file
        fitbit_label = file.split('merged')[0][:-1]
        if fitbit_label == 'activitylogs':
            df = pd.read_csv(data_dir+file,parse_dates=[[1,2]])
            df.set_index('Id',inplace=True)
        else:
            df = pd.read_csv(data_dir+file,index_col=0)
        for col in df.columns:
            if col in ['Time','ActivityHour','ActivityMinute','Date_StartTime']:
                df['UTC time'] = pd.to_datetime(df[col],errors='coerce') - timedelta(hours=6)
                
        fitbit_ids = df.index.unique().values
        for fid in fitbit_ids:
            df_byid = df[df.index.values == fid]
            try:
                bid = ids[ids['record'] == fid]['beiwe'].values[0]
                df_byid['beiwe'] = bid
                df_byid['fitbit'] = fid
                if path.isdir(export_dir + bid + '/Fitbit/'):
                    df_byid.to_csv(export_dir+bid+'/Fitbit/'+fitbit_label+'.csv')
                else:
                    os.mkdir(f'../data/processed/MADS_framework_data/{bid}')
                    os.mkdir(f'../data/processed/MADS_framework_data/{bid}/Fitbit')
                    df_byid.to_csv(export_dir+bid+'/Fitbit/'+fitbit_label+'.csv')
            except Exception as inst:
                print(f'File {fitbit_label} - Beiwe ID {bid} - Exception {inst}')<jupyter_output>File heartrate_1min - Beiwe ID uuc9c1q8 - Exception index 0 is out of bounds for axis 0 with size 0
File heartrate_1min - Beiwe ID uuc9c1q8 - Exception index 0 is out of bounds for axis 0 with size 0
File heartrate_1min - Beiwe ID uuc9c1q8 - Exception index 0 is out of bounds for axis 0 with size 0
File heartrate_1min - Beiwe ID uuc9c1q8 - Exception index 0 is out of bounds for axis 0 with size 0
File heartrate_1min - Beiwe ID uuc9c1q8 - Exception index 0 is out of bounds for axis 0 with size 0
File heartrate_1min - Beiwe ID uuc9c1q8 - Exception index 0 is out of bounds for axis 0 with size 0
File heartrate_1min - Beiwe ID uuc9c1q8 - Exception index 0 is out of bounds for axis 0 with size 0
File heartrate_1min - Beiwe ID uuc9c1q8 - Exception index 0 is out of bounds for axis 0 with size 0
File heartrate_1min - Beiwe ID uuc9c1q8 - Exception index 0 is out of bounds for axis 0 with size 0
File heartrate_1min - Beiwe ID uuc9c1q8 - Exception index 0 is out of bounds for axis 0 with size 0
[...]<jupyter_text>Beacon DataThe beacon data might not as easy to coax into a format that works. The beacon data in the UT1000 file does not present any useful data so we can skip straight to the UT2000 data.<jupyter_code># headers for datafiles - pms, sgp, sht
headers = [['UTC time','pm1','pm2p5','pm10',
           'std1','std2p5','std10',
           'pc0p3','pc0p5','pc1','pc2p5','pc5','pc10'],
           ['UTC time','eco2','tvoc'],
           ['UTC time','rh','tc']
          ]
dirs = ['PM','TVOC','TRH']
data_dir = '../data/raw/ut2000/beacon'
export_dir = '../data/processed/MADS_framework_data'
for beacon in os.listdir(data_dir):
    if beacon in ['beacon-d3-00','beacon-d3-01','beacon-d3-02']:
        print(f'No processing data for beacon {beacon}')
    else:
        print(f'Reading for beacon {beacon}')
        for sensor, header, sensor_dir in zip(['pms5003','sgp30','sht31d'], headers, dirs):
            for file in os.listdir(f'{data_dir}/{beacon}/bevo/{sensor}/'):
                df = pd.read_csv(f'{data_dir}/{beacon}/bevo/{sensor}/{file}',names=header)
                beacon_no = int(beacon.split('-')[2])
                if sensor == 'pms5003':
                    df = df[['UTC time','pm1','pm2p5','pm10']]
                    df.columns = ['UTC time','pm10','pm2.5','pm1']
                try:
                    bid = ids_2000[ids_2000['beacon'] == beacon_no]
                    bid = bid['beiwe'].values[0]
                    df['beiwe'] = bid
                    df['beacon'] = beacon_no
                    df.set_index('UTC time',inplace=True)

                    if path.isdir(f'{export_dir}/{bid}/BEVO/{sensor_dir}/'):
                        df.to_csv(f'{export_dir}/{bid}/BEVO/{sensor_dir}/{file}')
                    elif not path.isdir(f'{export_dir}/{bid}'):
                        os.mkdir(f'{export_dir}/{bid}/')
                        os.mkdir(f'{export_dir}/{bid}/BEVO/')
                        os.mkdir(f'{export_dir}/{bid}/BEVO/{sensor_dir}/')
                        df.to_csv(f'{export_dir}/{bid}/BEVO/{sensor_dir}/{file}')
                    elif not path.isdir(f'{export_dir}/{bid}/BEVO/'):
                        os.mkdir(f'{export_dir}/{bid}/BEVO/')
                        os.mkdir(f'{export_dir}/{bid}/BEVO/{sensor_dir}/')
                        df.to_csv(f'{export_dir}/{bid}/BEVO/{sensor_dir}/{file}')
                    else:
                        os.mkdir(f'{export_dir}/{bid}/BEVO/{sensor_dir}/')
                        df.to_csv(f'{export_dir}/{bid}/BEVO/{sensor_dir}/{file}')
                except Exception as inst:
                    print(f'{inst} - Beacon {beacon_no} not deployed')<jupyter_output>Reading for beacon beacon-d3-06
No processing data for beacon beacon-d3-01
Reading for beacon beacon-d3-08
Reading for beacon beacon-d3-09
No processing data for beacon beacon-d3-00
Reading for beacon beacon-d3-07
Reading for beacon beacon-d3-12
No processing data for beacon beacon-d3-02
Reading for beacon beacon-d3-05
Reading for beacon beacon-d3-03
Reading for beacon beacon-d3-10
Reading for beacon beacon-d3-11
index 0 is out of bounds for axis 0 with size 0 - Beacon 11 not deployed
index 0 is out of bounds for axis 0 with size 0 - Beacon 11 not deployed
index 0 is out of bounds for axis 0 with size 0 - Beacon 11 not deployed
index 0 is out of bounds for axis 0 with size 0 - Beacon 11 not deployed
index 0 is out of bounds for axis 0 with size 0 - Beacon 11 not deployed
index 0 is out of bounds for axis 0 with size 0 - Beacon 11 not deployed
index 0 is out of bounds for axis 0 with size 0 - Beacon 11 not deployed
index 0 is out of bounds for axis 0 with size 0 - Beacon 11 not deployed[...]
{'fraction_non_alphanumeric': 0.11561079925153703, 'fraction_numerical': 0.039561614541566426, 'mean_word_length': 2.6627508565834557, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '145026', 'license': 'MIT', 'n_tokens_mistral': 2673, 'n_tokens_neox': 2387, 'n_words': 718, 'path': 'notebooks/archive/2.0.0-hef-ut2000-data_processing.ipynb', 'repo_name': 'intelligent-environments-lab/utx000', 'size': '7482'}