<jupyter_start><jupyter_text>dbCamHDThis notebook is an attempt to create a metadata database using the pycamhd library.  Setup environment<jupyter_code>%matplotlib inline
import pycamhd as camhd
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Get a list of files from the serverThis operation can take many minutes (10-15), and sometimes fails when the server has some random issue. I don't recommend doing this very often, if at all. I have changed these cells over to raw now that the dbcamhd.json file exists so the cells will not run automatically. It should not be necessary to rerun these next few cells until CamHD comes back online and begins to generate new video files.<jupyter_code>%%time
file_list, file_sizes = camhd.get_file_list();<jupyter_output><empty_output><jupyter_text>Create a Pandas dataframe from these Lists<jupyter_code>dbcamhd = pd.DataFrame(
    {'filename': file_list,
     'filesize': file_sizes,
    })
dbcamhd.tail()<jupyter_output><empty_output><jupyter_text>Get some additional information about the filesThis cell takes a couple of hours to run in a single thread, which is why it is commented out. How much faster would it go using Dask Delayed and a bunch of Dask workers?<jupyter_code>%%time
timestamp = []
frame_count = []
moov = []

for i in dbcamhd.index:
    filename = dbcamhd['file'][i]
    try:
        moov_atom = camhd.get_moov_atom(filename)
        timestamp.append(camhd.get_timestamp(filename, moov_atom))
        frame_count.append(camhd.get_frame_count(filename, moov_atom))
        moov.append(True)
    except:
        timestamp.append(0)
        frame_count.append(0)
        moov.append(False)
    if (i % 100) == 0:
        print(i)<jupyter_output><empty_output><jupyter_text>Add these to the dbcamhd dataframe<jupyter_code>dbcamhd['moov'] = pd.Series(moov, index=dbcamhd.index)
dbcamhd['timestamp'] = pd.Series(timestamp, index=dbcamhd.index)
dbcamhd['frame_count'] = pd.Series(frame_count, index=dbcamhd.index)<jupyter_output><empty_output><jupyter_text>Save dataframe to JSON file<jupyter_code>dbcamhd.to_json('dbcamhd.json', orient="records", lines=True)<jupyter_output><empty_output><jupyter_text>Load dataframe from JSON file (do this instead of camhd.get_file_list())Here we load the dbcamhd JSON file that contains the results from the above cells.<jupyter_code>dbcamhd = pd.read_json('dbcamhd.json', orient="records", lines=True)
dbcamhd.tail()<jupyter_output><empty_output><jupyter_text>Plot histogram of MOV sizes<jupyter_code>import holoviews as hv
hv.extension('bokeh')
from bokeh.plotting import figure, show

frequencies, edges = np.histogram(dbcamhd['filesize']/1024/1024/1024, bins=np.linspace(0,20,100))

p = figure(title="MOV Size Distribution")
p.quad(top=frequencies, bottom=0, left=edges[:-1], right=edges[1:], fill_color="blue", line_color="black")
p.xaxis.axis_label = 'Filesize (GB)'
p.yaxis.axis_label = 'N'
show(p)<jupyter_output><empty_output><jupyter_text>Plot histogram of timestamp deltas<jupyter_code>deltas = dbcamhd.timestamp.loc[dbcamhd.timestamp>140000000].diff()
frequencies, edges = np.histogram(deltas, range=(0, 100000), bins=range(10000))

p = figure(title="MOV Size Distribution", x_range=(0, 1000))
p.quad(top=frequencies, bottom=0, left=edges[:-1], right=edges[1:], fill_color="blue", line_color="black")
p.xaxis.axis_label = 'Delta (s)'
p.yaxis.axis_label = 'N'
show(p)<jupyter_output><empty_output><jupyter_text>Plot subset of frame_counts<jupyter_code>p = figure(x_axis_type='datetime', y_range=(0, 30000))
min_t = 1400000000

dates = pd.to_datetime(dbcamhd['timestamp'][dbcamhd.timestamp>min_t],unit='s')
frame_count = dbcamhd['frame_count'][dbcamhd.timestamp>min_t]

p.circle(dates, frame_count, size=1)
p.xaxis.axis_label = 'Date'
p.yaxis.axis_label = 'Frame Count'
show(p)<jupyter_output><empty_output><jupyter_text>Add deployment numbers to databaseSee the [asset management](https://github.com/ooi-integration/asset-management/blob/master/deployment/RS03ASHS_Deploy.csv) page for deployment information.<jupyter_code>import pandas as pd
dbcamhd = pd.read_json('dbcamhd.json', orient="records", lines=True)
dbcamhd.filename[0]
dbcamhd.tail()
dt = pd.to_datetime(dbcamhd.timestamp, unit='s')
dbcamhd['deployment'] = dbcamhd.timestamp*0
dbcamhd.loc[dt < '2016-07-26 21:18:00', 'deployment'] = 2
dbcamhd.loc[dt >= '2016-07-26 21:18:00', 'deployment'] = 3
dbcamhd.loc[dt >= '2017-08-14 06:00:00', 'deployment'] = 4
dbcamhd.tail()
dbcamhd.to_json('dbcamhd.json', orient="records", lines=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12494519947391496, 'fraction_numerical': 0.02761946514686541, 'mean_word_length': 6.617696160267112, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 50, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '475807', 'license': 'MIT', 'n_tokens_mistral': 1690, 'n_tokens_neox': 1588, 'n_words': 360, 'path': 'tim/dbcamhd.ipynb', 'repo_name': 'EmilioTesin100/rte-camhd', 'size': '4562'}