<jupyter_start><jupyter_text>In this notebook, we are looking for gaps in the proximity data collected by one or more badges<jupyter_code>from __future__ import absolute_import, division, print_function

import os
import sys
import datetime
import pytz
import dateutil

import pandas as pd
import numpy as np
from plotnine import *
from plotnine.data import mpg

#from ggplot import *
import gzip
src_dir = os.path.join(r'C:\Users\UOU\Desktop\GJ\openbadge-analysis-master')
sys.path.append(src_dir)
%matplotlib inline

# Import the data analysis tools
import openbadge_analysis as ob
import openbadge_analysis.core as core
# config
filename = '../data/hub_data/proximity_archive.txt.gz'
log_filename = '../data/hub_data/hub.log.gz'

eastern = pytz.timezone('US/Eastern')
utc  = pytz.timezone('UTC')<jupyter_output><empty_output><jupyter_text>Find badge first reset If we want to get a real sense of how many gaps we have, we need to discard data after the the badges stared to reset. We need to look at the actualy logfile to find that time. This isn't pretty, but it works.<jupyter_code># only load lines in which the hub reported a unsynchronized badge
with gzip.open(log_filename, 'r') as f:
    unsync_raw = [ line for line in f if 'unsync' in line]
import re
# use regex to get date and badge
unsync_split = [ re.match('(.*) - INFO - \[(.*)\]',line).group(1,2) for line in unsync_raw]
# load into a dataframe and convert the date
df_log = pd.DataFrame(unsync_split,columns=['datetime','badge_address'])

# Sort by date
df_log['datetime'] = pd.to_datetime(df_log['datetime'], utc=True)
df_log.sort_values('datetime')
df_log.index = pd.to_datetime(df_log.pop('datetime'), utc=True)

# This log is in EST, so let's make the datatime tiemzone aware
# This way we can use it to filter the data
df_log = df_log.tz_convert(eastern)
print(df_log.index.dtype)
# reseting index, now that we converted the timezone (i'm not sure how to do this more efficiently...)
df_log = df_log.reset_index()
print(df_log.datetime.dtype)<jupyter_output>datetime64[ns, US/Eastern]<jupyter_text>The first reset (for every badge) is the first time we set the clock, so it's not really a reset - it is the first time we sent the date to the badge<jupyter_code>df_log_grp = df_log.groupby('badge_address')
start_time = df_log_grp.nth(0)
print("Start time\n",start_time,"\n\n")

first_reset = df_log_grp.nth(1)
print("First reset\n",first_reset,"\n\n")

# change to a more convinient format
first_reset = first_reset.reset_index()
first_reset.rename(columns={'datetime': 'first_reset_datetime'}, inplace=True)
first_reset['first_reset_timestamp'] = first_reset.pop('first_reset_datetime').astype(np.int64)//1e9
print("First reset, restructured\n", first_reset,"\n\n")<jupyter_output>Start time
 Empty DataFrame
Columns: [datetime]
Index: [] 


First reset
 Empty DataFrame
Columns: [datetime]
Index: [] 


First reset, restructured
 Empty DataFrame
Columns: [badge_address, first_reset_timestamp]
Index: []<jupyter_text>Look for gaps<jupyter_code># Load proximity data
with gzip.open(filename, 'r') as f:
    proximity_chunks = core.load_proximity_chunks_as_json_objects(f, log_version='2.0')
len(proximity_chunks)
# load data into a data frame
df = pd.DataFrame(proximity_chunks,columns=['badge_address','timestamp'])

# Converting timestamp (epoch) to UTC datetime
df['datetime'] = pd.to_datetime(df['timestamp'],unit='s', utc=True)

# Setting datetime as index
df.sort_values('datetime')
df.index = pd.to_datetime(df.pop('datetime'), utc=True)
df = df.tz_convert(utc)
df = df.tz_convert(eastern)
df = df.reset_index() # make it a column again

print(df.head(10))
print(df.dtypes)<jupyter_output>datetime      badge_address     timestamp
0 2017-06-30 17:43:58-04:00  E0:B2:01:E3:09:8C  1.498859e+09
1 2017-06-30 17:44:01-04:00  D8:59:E3:35:BD:C4  1.498859e+09
2 2017-06-30 17:44:58-04:00  E0:B2:01:E3:09:8C  1.498859e+09
3 2017-06-30 17:45:01-04:00  D8:59:E3:35:BD:C4  1.498859e+09
4 2017-06-30 17:45:58-04:00  E0:B2:01:E3:09:8C  1.498859e+09
5 2017-06-30 17:46:01-04:00  D8:59:E3:35:BD:C4  1.498859e+09
6 2017-06-30 17:46:58-04:00  E0:B2:01:E3:09:8C  1.498859e+09
7 2017-06-30 17:47:01-04:00  D8:59:E3:35:BD:C4  1.498859e+09
8 2017-06-30 17:47:58-04:00  E0:B2:01:E3:09:8C  1.498859e+09
9 2017-06-30 17:48:01-04:00  D8:59:E3:35:BD:C4  1.498859e+09
datetime         datetime64[ns, US/Eastern]
badge_address                        object
timestamp                           float64
dtype: object<jupyter_text>Filter based on first reset time<jupyter_code>if (len(first_reset)) > 0:
    df_fil = pd.merge(df,first_reset,how='left',on='badge_address')
    df_fil = df_fil[df_fil['timestamp'] < df_fil['first_reset_timestamp']]
    print("Records before filtering: {}, after: {}".format(len(df),len(df_fil)))

    del df_fil['first_reset_timestamp']
else:
    df_fil = df<jupyter_output><empty_output><jupyter_text>Check for gaps in data calculations are per-badge, so we must create a multi index. Then, per-group calculations can be done with group by<jupyter_code># http://pandas.pydata.org/pandas-docs/stable/advanced.html
# set multi-index
df_fil.set_index(['badge_address', 'datetime'], inplace=True, drop=False)

# sort is important for shift to work properly
df_fil.sort_index(inplace=True)
print(df_fil.head())
# http://pandas.pydata.org/pandas-docs/stable/groupby.html
# group by badges, and shift ones so we can get the time of the previous row
grp_shift = df_fil.groupby(level=[0]).shift(1)
df_with_lag = df_fil.copy()
df_with_lag['timestamp_lag']= grp_shift['timestamp']
df_with_lag['time_diff']=df_with_lag['timestamp']-df_with_lag['timestamp_lag']
print("index dtype: ",df_with_lag.index.dtype)
print("Timediff type:", df_with_lag.time_diff.dtype)
print(df_with_lag.head())<jupyter_output>index dtype:  object
Timediff type: float64
                                                             datetime  \
badge_address     datetime                                              
D8:59:E3:35:BD:C4 2017-06-30 17:44:01-04:00 2017-06-30 17:44:01-04:00   
                  2017-06-30 17:45:01-04:00 2017-06-30 17:45:01-04:00   
                  2017-06-30 17:46:01-04:00 2017-06-30 17:46:01-04:00   
                  2017-06-30 17:47:01-04:00 2017-06-30 17:47:01-04:00   
                  2017-06-30 17:48:01-04:00 2017-06-30 17:48:01-04:00   

                                                 badge_address     timestamp  \
badge_address     datetime                                                     
D8:59:E3:35:BD:C4 2017-06-30 17:44:01-04:00  D8:59:E3:35:BD:C4  1.498859e+09   
                  2017-06-30 17:45:01-04:00  D8:59:E3:35:BD:C4  1.498859e+09   
                  2017-06-30 17:46:01-04:00  D8:59:E3:35:BD:C4  1.498859e+09   
                  2017-06-30 17:47:01-04:00 [...]<jupyter_text>Scans are performed every 60 seconds<jupyter_code>seconds_between_chunks = 60
print("seconds between chunks: {}".format(seconds_between_chunks))

# time between chunks might vary slightly
seconds_between_chunks = seconds_between_chunks + 2
print("seconds between chunks: {}".format(seconds_between_chunks))

gaps = df_with_lag[df_with_lag['time_diff']>seconds_between_chunks]
print(gaps[['time_diff','timestamp']].head(10))<jupyter_output>seconds between chunks: 60
seconds between chunks: 62
                                             time_diff     timestamp
badge_address     datetime                                          
D8:59:E3:35:BD:C4 2017-07-06 17:48:47-04:00      180.0  1.499378e+09
                  2017-07-06 18:16:47-04:00      120.0  1.499379e+09
                  2017-07-06 18:27:47-04:00      120.0  1.499380e+09
                  2017-07-06 18:29:47-04:00      120.0  1.499380e+09
                  2017-07-06 18:33:47-04:00      120.0  1.499380e+09
                  2017-07-06 18:41:47-04:00      120.0  1.499381e+09
                  2017-07-06 18:46:47-04:00      120.0  1.499381e+09
                  2017-07-06 18:56:47-04:00      120.0  1.499382e+09
                  2017-07-06 19:02:47-04:00      120.0  1.499382e+09
                  2017-07-06 19:09:47-04:00      300.0  1.499383e+09<jupyter_text>How many gaps are ther then? If you look below, you'll see that there are gaps in the data, even before the first reset. However, if you look at the hub_plot_audio_voltage notebook, you will see that these gaps appear only when the battery voltage is very low.<jupyter_code>print(len(df))
print(len(gaps))
print(len(gaps)/len(df))
gaps<jupyter_output><empty_output><jupyter_text>Misc analysis of the gaps<jupyter_code>gaps[1:2].timestamp.map('{:.3f}'.format)
gaps[8:9].timestamp.map('{:.3f}'.format)

(ggplot(gaps)
 +aes(x='time_diff', fill='badge_address')
 +geom_histogram(binwidth=1)
 +geom_density(alpha=0.25)
 +ggtitle("Gap (sec)")
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12551159618008187, 'fraction_numerical': 0.11709868121873579, 'mean_word_length': 3.6227009984235417, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '171765', 'license': 'MIT', 'n_tokens_mistral': 3859, 'n_tokens_neox': 3236, 'n_words': 711, 'path': 'openbadge_analysis_examples/notebooks/hub_check_gaps_in_proximity_data.ipynb', 'repo_name': 'Finale21/openbadge', 'size': '8796'}