<jupyter_start><jupyter_text>Define the frame numbers from each file to process<jupyter_code>blob_urls = list(dbcamhd.blob_url[(dbcamhd.deployment == 2) & (dbcamhd.frame_count > 5000) & (dbcamhd.frame_count < 30000)])
blob_urls.sort()
frame_numbers = [7200]
import pycamhd as camhd
import numpy as np
from dask import delayed
import dask.array as dsa
delayed_frame_list = []
for blob_url in blob_urls:
    delayed_moov_atom = delayed(camhd.get_moov_atom)(blob_url)         
    for frame_number in frame_numbers:
        delayed_frame = delayed(camhd.get_frame)(blob_url, frame_number, 'rgb24', delayed_moov_atom)
        delayed_frame_list.append(dsa.from_delayed(delayed_frame, (1080, 1920, 3), np.uint8))
delayed_frame_array = dsa.stack(delayed_frame_list)<jupyter_output><empty_output><jupyter_text>A dask array is in many ways like a numpy array, except in this case it holds a set of instructions for how to acquire each chunk of the array, which makes it easy to farm this array out to workers in the cloud using the [distributed](http://distributed.readthedocs.io/en/latest/) scheduler.  Show one of the images<jupyter_code>frame = delayed_frame_array[300].compute()
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
import matplotlib.patches as patches
plt.rc('figure', figsize=(7, 3.3))
fig, ax = plt.subplots()
im1 = ax.imshow(frame)
im1.set_cmap('gray')
plt.yticks(np.arange(0,1081,270))
plt.xticks(np.arange(0,1921,480));
rect = patches.Rectangle((1140,200),300,400,linewidth=1.5,edgecolor='w',    facecolor='none')
ax.add_patch(rect)
plt.show();<jupyter_output><empty_output><jupyter_text>3) Do science!  Show the filter that will be used to filter images in the frequency domainTo deal with variations in lighting and high-frequency noise, we filter each subimage using a Butterworth bandpass filter.<jupyter_code>def butterworth(d1, d2, n):
    x = np.arange(-1024/2+0.5,1024/2+1-0.5)
    xx, yy = np.meshgrid(x, x)
    d = np.sqrt(xx**2+yy**2)
    bff = (1 - (1./(1 + (d/d1)**(2*n))))*(1/(1 + (d/d2)**(2*n)))
    return bff
d1 = 20 # low cut wavenumber
d2 = 400 # high cut wavenumber
n = 4
bff = butterworth(d1, d2, n)
plt.rc('figure', figsize=(6, 6))
imgplot = plt.imshow(bff, cmap='gray')<jupyter_output><empty_output><jupyter_text>Define the floc proxy functionThe floc proxy is simply the number of pixels in each filtered subimage that have a value greater than 4000.<jupyter_code>def frame_filter(frame, d1, d2, n):
    if frame.ndim == 3 and frame.shape[0] == 1:
        I = np.squeeze(frame[0, 0:1024, 0:1024])
    else:
        I = frame[0:1024, 0:1024]
    bff = butterworth(d1, d2, n)
    I_fft = np.fft.fft2(I)
    I_fft_shift = np.fft.fftshift(I_fft)
    I_fft_shift_filt = I_fft_shift*bff # filter with the Butterworth filter
    I_fft_filt = np.fft.ifftshift(I_fft_shift_filt)
    I_filt = np.fft.ifft2(I_fft_filt)
    return I_filt
def calc_floc_proxy(frame, d1, d2, n):
    I_filt = frame_filter(frame, d1, d2, n)
    return np.array([(np.absolute(I_filt)>4000).sum()])<jupyter_output><empty_output><jupyter_text>Show example for one frame<jupyter_code>I_filt = frame_filter(frame, d1, d2, n)
plt.rc('figure', figsize=(6, 6))
imgplot = plt.imshow(np.absolute(I_filt)>4000, cmap='gray')
plt.title('floc_proxy value = %i' % (np.absolute(I_filt)>4000).sum());<jupyter_output><empty_output><jupyter_text>Assemble a new Dask array including our computation using map_blocks<jupyter_code>floc_proxy = dsa.map_blocks(calc_floc_proxy, delayed_frame_array, d1, d2, n, dtype='i8', drop_axis=[1,2])
floc_proxy<jupyter_output><empty_output><jupyter_text>Compute the floc_proxy (subset)<jupyter_code>%%time
results = floc_proxy[0:10].compute()<jupyter_output><empty_output><jupyter_text>Start a Dask clusterUse the new Dask extension!<jupyter_code>client
%%time
results = floc_proxy[0::10].compute()<jupyter_output><empty_output><jupyter_text>Calculate all the results<jupyter_code>print('Number of images: %i' % len(floc_proxy))
print('Size of dataset (GB): %i' % round(len(floc_proxy)*1080*1920*2/1024/1024/1024))
%%time
results = floc_proxy.compute()<jupyter_output><empty_output><jupyter_text>Get a timestamp for each frame<jupyter_code>import datetime, math
import matplotlib.dates as dates
frame_timestamp = []
for blob_url in blob_urls:
    timestamp = dbcamhd['timestamp'][dbcamhd.blob_url == blob_url].iloc[0]
    for frame_number in frame_numbers:
        timestamp = timestamp + frame_number/29.97
        dt = datetime.datetime.fromtimestamp(timestamp)
        frame_timestamp.append(dates.date2num(dt))<jupyter_output><empty_output><jupyter_text>Plot a two-dimensional multivariate histogram of the results<jupyter_code>plt.rc('font', size=11)
fig, ax = plt.subplots()
fig.set_size_inches(14, 6)
fig.frameon = False
hb1 = ax.hexbin(frame_timestamp[0::10], results, vmin=0.1, vmax=10, bins='log', linewidths=0.25,
  gridsize=(200, 4000), mincnt=1, cmap=plt.cm.BuPu)
fig.colorbar(hb1)
ax.set_ylim([0, 8000])
ax.set_xlim([frame_timestamp[0],frame_timestamp[-1]])
ax.yaxis.grid(True)
ax.xaxis.grid(True)
months = dates.MonthLocator(interval=6)  # every month
monthsFmt = dates.DateFormatter('%b %Y')
ax.xaxis.set_major_locator(months)
ax.xaxis.set_major_formatter(monthsFmt)
plt.ylabel('Floc Proxy Value');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13236678591602336, 'fraction_numerical': 0.04349463377894935, 'mean_word_length': 5.971128608923885, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '94608', 'license': 'MIT', 'n_tokens_mistral': 2157, 'n_tokens_neox': 2016, 'n_words': 425, 'path': 'show_image.ipynb', 'repo_name': 'tjcrone/agu2019', 'size': '5311'}