<jupyter_start><jupyter_text>Exploring the UTx000 DatasetFrom the first cohort in Spring 2020<jupyter_code>import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Bluetooth ProcessingI don't even know what to expect with this dataset.<jupyter_code>import os

import pandas as pd
import numpy as np

from datetime import datetime, timedelta

import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Test Data ImportWe want to import some data from participants so that we can understand it a little bit more.  One datafile<jupyter_code>raw = pd.read_csv('/Volumes/HEF_Dissertation_Research/utx000/bpeace1/beiwe/bluetooth/25rlmdr1/bluetooth/2020-01-24 23_00_00.csv')
raw<jupyter_output><empty_output><jupyter_text>One participant<jupyter_code>pt_dir = '/Volumes/HEF_Dissertation_Research/utx000/bpeace1/beiwe/bluetooth/25rlmdr1/bluetooth/'
pt_df = pd.DataFrame()
for file in os.listdir(pt_dir):
    temp = pd.read_csv(f'{pt_dir}{file}',parse_dates = [1],infer_datetime_format=True,index_col=1)
    pt_df = pt_df.append(temp)
    
pt_df.head()<jupyter_output><empty_output><jupyter_text>Visualizing to see gaps<jupyter_code>fig, ax = plt.subplots(figsize=(16,4))
ax.scatter(pt_df.index, pt_df['RSSI'],s=10,color='black')
plt.show()
plt.close()<jupyter_output><empty_output><jupyter_text>Downsampling analysis Can't really do any downsampling because the bluetooth addresses seem unique at each time step. So, the processing will simply be appending the data together into one giant file.  Processing Pipeline<jupyter_code>data_dir = '/Volumes/HEF_Dissertation_Research/utx000/bpeace1/beiwe/bluetooth/'
bt_df = pd.DataFrame()
for pt in os.listdir(data_dir)[:4]:
    if len(pt) == 8:
        print(f'\t\tWorking for participant {pt}')
        pt_df = pd.DataFrame()
        for file in os.listdir(f'{data_dir}{pt}/bluetooth/'):
            if file[-1] == 'v':
                temp = pd.read_csv(f'{data_dir}{pt}/bluetooth/{file}',parse_dates=[1],infer_datetime_format=True,engine='python')
                pt_df = pt_df.append(temp)
            
        pt_df['UTC time'] = pd.to_datetime(pt_df['UTC time'])
        pt_df['Time'] = pt_df['UTC time'] - timedelta(hours=6)
        pt_df.set_index('Time',inplace=True)
        pt_df['Beiwe'] = pt
        
        bt_df = bt_df.append(pt_df)
    
bt_df.head()
temp = pd.read_csv('../data/processed/bpeace1-gps.csv')
temp<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12150668286755771, 'fraction_numerical': 0.021061158363710003, 'mean_word_length': 4.951807228915663, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '647236', 'license': 'MIT', 'n_tokens_mistral': 939, 'n_tokens_neox': 887, 'n_words': 167, 'path': 'notebooks/archive/3.5.0-hef-beiwe-bluetooth_processing.ipynb', 'repo_name': 'intelligent-environments-lab/utx000', 'size': '2469'}