<jupyter_start><jupyter_text>This script generated is for the statistical analysis of the<jupyter_code>import numpy as np
import scipy as sp
from scipy import stats
from scipy.optimize import curve_fit
from scipy.optimize import minimize
from scipy.interpolate import *
import matplotlib.pyplot as plt
import pandas as pd
import datetime, math
import matplotlib.dates as dates
import holoviews as hv
from holoviews import opts
from holoviews import dim
hv.extension('bokeh')
results = pd.read_pickle('/home/jovyan/data/camhd_data/regions_results_for_dep_5.pickle')

results.head()
total_floc = [sum(i) for i in results['label_stats']]
results['total_floc'] = total_floc
x = results['timestamp'].values
y = results['total_floc'].values
date = results['datetime']
type(x[1])
results.head()
data = pd.DataFrame({'x': x, 'y': y, 'date':date})
data.head()
import hvplot.pandas
data.hvplot.scatter('x', 'y', datashade = True)<jupyter_output><empty_output><jupyter_text>Remove outliers<jupyter_code>ss = data.loc[data['y']<7000]
ss.describe()
xx = ss['x'].values
yy = ss['y'].values
datedate = ss['date']
len(ss)
np.shape(ss)<jupyter_output><empty_output><jupyter_text>Plot Subset without crazy high values<jupyter_code>def func (xx, a1, a2):
    return a1 + a2 * np.log10(xx)
popt, pcov = curve_fit(func, xx, yy)
popt
y2 = func(xx, popt[0], popt[1])
data2 = pd.DataFrame({'xx': xx, 'Floc Volume': yy, 'y2': y2, 'Date': datedate})
data2.head()
import hvplot.pandas
import hvplot.pandas
plot1= data2.hvplot.scatter('Date', 'Floc Volume', datashade = True)
plot2= data2.hvplot.line('Date', 'y2', color = 'red')
plot1*plot2
xs = data2['xx']
ys = data2['Floc Volume']
hex_with_values = hv.HexTiles((xs, ys, 0.01))
hex_line = data2.hvplot.line('xx', 'y2')
overlay = (hex_with_values*hex_line)

# hex_with_values.opts(opts.HexTiles(width=500, height=400, tools=['hover'], colorbar=True))
#overlay = hex_with_values*hex_line
overlay<jupyter_output><empty_output><jupyter_text>Does our model work? Test it with synthetic data<jupyter_code>xx = np.arange(1,100)
err = np.random.randn(len(xx))*.1
yy = -np.log10(xx)+err
plt.plot(xx,yy,'.')
def func (xx, a1, a2):
    return a1 + a2 * np.log10(xx)
popt, pcov = curve_fit(func, xx, yy)
y2 = func(xx, popt[0], popt[1])
plt.plot(xx,yy,'.')
plt.plot(xx,y2,'-')
We do not yet have a model yet...
Exploring this process
Straight line fit..... 
Obviously these are not the winners .... looking at ways to refine our model. Open to ideas to what might work best

What are your ideas that for what might control this process..... 
input output model. Constant... stokes settling output. 

Input constant for some number of days..... settling 

Alturnatively the model, the input is a decay function. 

Step function for inupt, stokes setteling as an output. 

box model.... inputs and outputs..... depends on particle size.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14221140472879, 'fraction_numerical': 0.019471488178025034, 'mean_word_length': 5.436241610738255, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '408081', 'license': 'MIT', 'n_tokens_mistral': 1093, 'n_tokens_neox': 1053, 'n_words': 296, 'path': 'notebooks/figure_scripts/floc_logmodel.ipynb', 'repo_name': 'EmilioTesin100/camhd_floc_model', 'size': '2876'}