<jupyter_start><jupyter_text>DataUp until now, we have a CSV that has a lot of information, but first and foremost, it has the manually curated start step and stop step signifiers. Let's go ahead and import this file.<jupyter_code>step_seconds = pd.read_csv('./data/step_seconds.csv')<jupyter_output><empty_output><jupyter_text>Let's do a little investigating. How does this look like?<jupyter_code>step_seconds.head()<jupyter_output><empty_output><jupyter_text>Notably, we will mainly use the `Start Time` and `End Time` columns in our initial analyses. The other columns will come into play when we export our data.  Cool! Now, let's move on to our video data. To start with, we'll be looking at mouse `1058`. This is a Wild Type mouse where we extracted roughly 50 steps. We have it's MitoPark counterpart to analyze right after this, but this should be a good test case, since there's more ambulation and tracking looks better.<jupyter_code>h5_file = step_seconds[step_seconds['Mouse'] == 1058]['h5'][18]
DLCscorer = "DLC_resnet101_bottom_mp_pawsMar24shuffle1_150000"
fps = 100
h5_file
h5_file = 'C:\\Users\\ameet\\Box Sync\\BevanLab\\MitoPark Labeled videos\\1058-bottomDLC_resnet101_bottom_mp_pawsMar24shuffle1_150000.h5'
Dataframe = pd.read_hdf(h5_file)
Dataframe[DLCscorer].head()<jupyter_output><empty_output><jupyter_text>Paw analysisLet's start by looking at each paw<jupyter_code>start_time = '00:09.550'
end_time = '00:09.730'

def convert_time_to_seconds (time: str) -> int:
    
    """
    Takes time in the form MM:SS:ms and converts to seconds
    """
    time = time.split(':')
    out = int(time[0]) * 60 + float(time[1])
    return out
start_sec = convert_time_to_seconds(start_time)
end_sec = convert_time_to_seconds(end_time)
start_sec
np.round(end_sec - start_sec, 2)<jupyter_output><empty_output><jupyter_text>Looks like this step was about .18 seconds in duration<jupyter_code>def get_snippet(data, bodypart, DLCScorer, start_sec, end_sec, fps) -> pd.DataFrame: 
    return data[DLCscorer][bodypart][int(start_sec * fps): int(end_sec * fps)]
back_left = get_snippet(Dataframe, 'back-left-paw', DLCscorer, start_sec, end_sec, fps)
back_right = get_snippet(Dataframe, 'back-right-paw', DLCscorer, start_sec, end_sec, fps)


front_left = get_snippet(Dataframe, 'front-left-paw', DLCscorer, start_sec, end_sec, fps)
front_right = get_snippet(Dataframe, 'front-right-paw', DLCscorer, start_sec, end_sec, fps)
plt.plot(back_left['x'], back_left['y'], color = 'red', label = 'back left')
plt.plot(back_right['x'], back_right['y'], color = 'blue', label = 'back right')
plt.plot(front_right['x'], front_right['y'], color = 'green', label = 'front right')
plt.plot(front_left['x'], front_left['y'], color = 'black', label = 'front left')
plt.title('Single Footstep')
plt.legend()
#plt.savefig('trajectory.pdf')
plt.xlabel('X (pixels)')
plt.ylabel('Y (pixels)')
plt.savefig('trajectory.pdf', format='pdf', dpi=1200)

plt.show()<jupyter_output><empty_output><jupyter_text>Ok, it's claer that our front left data is a little skewed. Let's try and figure out a way to fix this.<jupyter_code>def polynomial(x, a,b,c, d):
    return a + b*x + c*x*x + d * x * x * x


def fit_paw_to_polynomial(paw):
    timedata = np.linspace(0, paw['x'].size, paw['x'].size)
    xdata = paw['x']
    ydata = paw['y']
    popt_x, pcov = curve_fit(polynomial, timedata, xdata)
    popt_y, pcov = curve_fit(polynomial, timedata, ydata)
    return polynomial(time, popt_x[0], popt_x[1], popt_x[2]), polynomial(time, popt_y[0], popt_y[1], popt_y[2])
time = np.arange(front_left['x'].shape[0])
np.random.seed(1729)
noise = 0.2 * np.random.normal(size=front_left['x'].size)
plt.plot(time, front_left['x'] + noise, color = 'red', label = 'front left x')

plt.plot(time, front_left['y'] + noise, color = 'green', label = 'front left y')
plt.legend()

timedata = np.linspace(0, 18, front_left['x'].size)
xdata = np.copy(back_right['x'])
ydata = np.copy(back_right['y'])
popt_x, pcov_x = curve_fit(polynomial, timedata, xdata)
popt_y, pcov_y = curve_fit(polynomial, timedata, ydata)<jupyter_output><empty_output><jupyter_text>1) Fit polynomial 2) Get 2 * cdf(- abs(dist/stderr)) -> Two tailed test - tells you if you're on either tail of distribution of line3) Sample according to probabilities from 24) Repeat steps 1-3 until params of polynomial don't change up to some epsilon<jupyter_code>def flip(p):
    return True if random.random() < p else False

def sample_and_replace(p_values, data, polynomial, timedata, popt_x):
    """
    Samples with p-values and replaces with polynomial values in place. 
    """
    to_replace = np.array(list(filter(lambda p: np.random.random() < (1 - p[1]), enumerate(p_values))))
    to_replace_idx = np.array([int(i[0]) for i in to_replace])
    data[to_replace_idx] = polynomial(timedata, *popt_x)[to_replace_idx]
    
    
def get_pvals(popt, pcov, data, polynomial, timedata):
    """
    Gets p values for every data point 
    """

    perr = np.sqrt(np.diag(pcov))
    dist = data - polynomial(timedata, *popt)

    stderr = np.std(dist)
    p_values = 2 * stats.norm.cdf(-np.abs(dist/stderr), 0, 1)
    return p_values
def iterate_and_fit(data, polynomial):
    popt_x_old, pcov_x = curve_fit(polynomial, timedata, data)
    i = 0
    while True: 
        pvals = get_pvals(popt_x_old, pcov_x, data, polynomial, timedata)
        if np.all(pvals > .03):
            break
        sample_and_replace(pvals, data, polynomial, timedata, popt_x_old)
        #plt.plot(timedata, data, label = i)
        popt_x, pcov_x = curve_fit(polynomial, timedata, data)
        if np.allclose(popt_x_old, popt_x):
            break
        else: 
            i += 1
            popt_x_old = popt_x
    #plt.legend()
    #plt.title('Iterative Regressions')
    #plt.savefig('test.png')
    #plt.show()
plt.plot(back_left['x'], back_left['y'], color = 'red', label = 'back left')
plt.plot(back_right['x'], back_right['y'], color = 'blue', label = 'back right')
plt.plot(front_right['x'], front_right['y'], color = 'green', label = 'front right')
plt.plot(front_left['x'], front_left['y'], color = 'black', label = 'front left')
plt.title('Single Footstep')

back_left_x = np.copy(back_left['x'])
back_left_y = np.copy(back_left['y'])
back_right_x = np.copy(back_right['x'])
back_right_y = np.copy(back_right['y'])
front_left_x = np.copy(front_left['x'])
front_left_y = np.copy(front_left['y'])
front_right_x = np.copy(front_right['x'])
front_right_y = np.copy(front_right['y'])

iterate_and_fit(back_left_x, polynomial)
iterate_and_fit(back_left_y, polynomial)
iterate_and_fit(back_right_x, polynomial)
iterate_and_fit(back_right_y, polynomial)
iterate_and_fit(front_left_x, polynomial)
iterate_and_fit(front_left_y, polynomial)
iterate_and_fit(front_right_x, polynomial)
iterate_and_fit(front_right_y, polynomial)

plt.plot(back_left_x, back_left_y, 'r--', label = 'back left resampled')
plt.plot(back_right_x, back_right_y, 'b--', label = 'back right resampled')
plt.plot(front_right_x, front_right_y, 'g--', label = 'front right resampled')
plt.plot(front_left_x, front_left_y, 'k--', label = 'front left resampled')
plt.legend()
plt.xlabel('X (pixels)')
plt.ylabel('Y (pixels)')
plt.savefig('single_footstep.png', format='png', dpi=1200)
plt.show()<jupyter_output><empty_output><jupyter_text>MetricsGreat! Now we've figured out how to detect and smooth outliers. Let's move on to metrics. The first metric is the time a single step takes - this is simple, we actually don't even need to look at the data for this. The second is the distance each paw travels during a step This is also fairly simple - euclidean distance from each point to each consecutive point. The third is the maximum width of a stride.<jupyter_code>plt.plot(xdata, ydata)
plt.plot(xdata, ydata)
popt_x, pcov_x = curve_fit(polynomial, timedata, xdata)
plt.plot(timedata, polynomial(timedata, *popt_x), label = 'init')
for i in range(10):
    perr = np.sqrt(np.diag(pcov_x))
    dist = xdata - polynomial(timedata, *popt_x)
    
    stderr = dist/np.std(dist)
    try:
        popt_x, pcov_x = curve_fit(polynomial, timedata, xdata, sigma = stderr)
        plt.plot(timedata, polynomial(timedata, *popt_x), label = i)
    except Exception as e:
        print(stderr)
        
plt.legend()
plt.show()
plt.plot(time, front_left['x'], color = 'red', label = 'front left raw data')
plt.plot(timedata, xdata, 'r--', label = 'FITTED & sampled')
plt.legend()
plt.savefig('test.png')
popt_x, pcov_x = curve_fit(polynomial, timedata, xdata, sigma = stderr)

plt.plot(timedata, polynomial(timedata, *popt_x), 'b--')
plt.plot(timedata, polynomial(timedata, *popt_x_init), 'r--')
plt.plot(timedata, xdata)

plt.plot(timedata, polynomial(timedata, reg.intercept_ ,reg.coef_[0],reg.coef_[1], 0), color = 'red')
plt.plot(time, front_left['x'], color = 'blue', label = 'front left x', alpha = .5)

plt.plot(time, front_left['x'], color = 'red', label = 'front left x')
plt.plot(time, polynomial(time, popt_x[0], popt_x[1], popt_x[2]), 'r--', label = 'front left x fitted')
plt.plot(time, front_left['y'], color = 'blue', label = 'front left y', alpha = .5)
plt.plot(time, polynomial(time, popt_y[0], popt_y[1], popt_y[2]), 'b--', label = 'front left y fitted')
plt.legend()
plt.plot(back_left['x'], back_left['y'], color = 'red', label = 'back left')
fitted_x, fitted_y = fit_paw_to_polynomial(back_left)
plt.plot(fitted_x, fitted_y, 'r--', label = 'FITTED back left')
plt.plot(back_right['x'], back_right['y'], color = 'black', label = 'back right')
fitted_x, fitted_y = fit_paw_to_polynomial(back_right)
plt.plot(fitted_x, fitted_y, 'k--', label = 'FITTED back right')
plt.plot(front_right['x'], front_right['y'], color = 'green', label = 'front right')
fitted_x, fitted_y = fit_paw_to_polynomial(front_right)
plt.plot(fitted_x, fitted_y, 'g--', label = 'FITTED front right')
plt.plot(front_left['x'], front_left['y'], color = 'blue', label = 'front left')
plt.plot(polynomial(time, popt_x[0], popt_x[1], popt_x[2]), polynomial(time, popt_y[0], popt_y[1], popt_y[2]), 'b--', label = 'FITTED front left')
plt.title('Single Footstep')
plt.legend()
plt.savefig('fitpoly.png')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1495189475751031, 'fraction_numerical': 0.012958963282937365, 'mean_word_length': 5.265067650676507, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '157264', 'license': 'MIT', 'n_tokens_mistral': 3987, 'n_tokens_neox': 3768, 'n_words': 921, 'path': '.ipynb_checkpoints/gait-checkpoint.ipynb', 'repo_name': 'BevanLab/open_field_gait', 'size': '10186'}