<jupyter_start><jupyter_text>This code is deliberately ad-hoc. Instead of making things difficult for myself by writing libraries and modules, etc, I'm writing code as a I go and only afterwards if need be will refactor by modularizing everything. My focus is on scientific insight, not on maintaining a codebase and the extra effort is simply not worth the mental energy and time spent organizing everything. Additionally, letting loose with my coding practices is an important step in my development. I'm learning how to be a fearless experimenter by doing things against my nature. I've previously been really tight about modularizing and organizing my code and never writing ad-hoc scripts, instead crusting over my code with abstractions upon abstractions upon abstractions. No longer will this be! Sometimes you need to solve specific problems with specific code and rely on the power of the tools you work on and your own ability to craft new solutions with them at ease. So this explains the erratic nature of the code in my project. It's me fearlessly experimenting with science and math. I've come to appreciate how developments are made in science and math. Very intelligent and curious people perceive a void, a possibility, a need which leads them to asking questions as to how that void can be filled. They start rigorously testing solutions and looking at all the nuances of their solutions, refining and refining and testing and testing until they've arrived at something substancial and versatile. For instance, Principal Component Analysis is the subject of much literature - because it is so nuanced. There are so many factors that come into play and the technique didn't just appear. It needed to be developed, tested and refined and be critiqued and worked on by brilliant people. This leads me to my new goal to become a fearless experimenter. I need to get past that barrier of inexpertise and rigorously play around with the new mathematical, statistical, and scientific ideas I learn. That means playing around with PCA, with graph theory, etc. It means not being afraid to embrace complex new ideas by engaging with them at the deepest level possible.<jupyter_code>import sys
# Add the ptdraft folder path to the sys.path list
sys.path.append('../src/')

import data_config as dc
wormData = dc.kato.data()

# Importing the transform code
import transform as tf
import dimensions_kit as dk
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import gridspec
transformed = []
pca = []
predicted = []

for worm, data in wormData.iteritems():
    wd_pca, wd_transformed = tf.scikit_pca(data['deltaFOverF_deriv'], 3)  
    
    #Storing this for later use
    transformed.append(wd_transformed)
    pca.append(wd_pca)
    """
    Using our PC's we can increase the dimensionality of 
        our reduced data, from 3 to 107, thus getting a blurred
        dataset as predicted by PC's 1 and 2. It contains
        most of the information contained in the original
        data
    
    The operation is very simple - a transformation
        of 3 dimensional data to 107 dimensional data as the 
        dot product of the components and transformed data
    """ 
    predicted.append(np.dot(wd_pca.components_.T, wd_transformed.T))
    

# How principle components are correlated with eachother
# Correlated: <>0, Not: ~ 0
# Tells you waht is common among the principle components
len(predicted)
def plot_all_predicted(): 
    fig = plt.figure(figsize=(10,10))
    gs = gridspec.GridSpec(3,2)
    for i in range(0,len(predicted)+1):
        # So we can layout our graphs
        # nicely in two dimensions
        # we need to map n <- R^1 to n'<-R^2 
        x,y = dk.transform((3,2), i) 
        ax = fig.add_subplot(gs[x,y])
        ax.set_title('Deriv-PCA Predicted ' + str(i))
        ax.pcolor(predicted[i-1])
    
"""
Here we will look at how the various datasets
compare with eachother. 

This enables us to answer the question of whether the same neurons have
similar PC strengths in all datasets. 
"""

# Here we plot the difference between two datasets
# The variance is fairly uniform except for at firing points
# where the pariance peaks
pred = predicted[0]
orig = wormData[0]['deltaFOverF_deriv']
diff = orig.T-pred
diff.shape


# This transforms a 2d array into a set of points
pts = np.array([[x,y,diff[x,y]] for x in range(diff.shape[0]) for y in range(diff.shape[1]) ])
"""
This shows you the variance between the predicted data
and the original derivative data
"""

%matplotlib inline
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111)
# ax.plot(pts[:,0], pts[:,1],pts[:,2])


ax.pcolormesh(diff*1000)
fig.show()<jupyter_output>/Library/Python/2.7/site-packages/matplotlib/figure.py:397: UserWarning: matplotlib is currently using a non-GUI backend, so cannot show the figure
  "matplotlib is currently using a non-GUI backend, "<jupyter_text>This tells us that the variance is fairl uniform except for at critical points whereit scales by some large factor. This makes sense - the PC would scale large points to make them outliers. The next step suggested is:  Comparing Adjacent PCs Across Datasets. To do this, we need to line up all neurons shared in two datasets. We first need to ensure that doing this is worthwhile.<jupyter_code># So that we can compare every worm with every worm, 
# we use combinations. (Binomial Coefficient, etc.)
from itertools import combinations
groupings  = [ i for i in combinations(range(5), r=2)]

# Tells us which neurons are present in the nth dataset
def get_neurons_present():
    neuron_names = [   
        { k[0]: idx 
        for idx, k in enumerate(wormData[i]['NeuronIds'])
        if k != None and (len(k)>0) }
        # Forall worms
        for i in range(len(wormData)) ]
    return neuron_names

neurons_present = [ 
    filter(lambda x: x!=None,i[:][0]) 
    for i in wormData.T['NeuronIds'].as_matrix()]

neuron_names_present = [set(ns) for ns in neurons_present]

# I used this stackoverflow to get this way of finding
# the intersection of sets: http://bit.ly/1RiV6FE
all_shared_neurons = set.intersection(*neuron_names_present)

# similars[(i,j)] = neurons shared between ith and jth dataset

similars = {}
for i,j in groupings: 
    ns_i = neurons_present[i]
    ns_j = neurons_present[j]
    
    # This makes accessing things easier. 
    # You don't need to worry about the order. 
    similars[(i,j)] = similars[(j,i)] = set.intersection(set(ns_i), set(ns_j))<jupyter_output>20<jupyter_text>We've just set up a framework for comparing neuron sharing across our datasets. Now let's use it to compute some preliminary metrics.<jupyter_code>all_possible_shared = list(set.union(*neuron_names_present))

print "Total number neurons present in > 2 datasets: " + str(len(all_possible_shared))

# Percentage of comparable neurons of the whole dataset
print str(int(float(len(all_possible_shared))/302*100)) + "%"<jupyter_output>64
Total number neurons present in > 2 datasets: 64
21%<jupyter_text>Because there are so few shared neurons, I'm not sure what kind of insight we can get by comparing the principle components for each neuron.Though after stepping back for a moment, I realized we can still get some insight if the shared neurons play important roles. Additionally, just comparing PCs for neurons across two datasets can provide some insight. So instead of comparing across all datasets, we can only compare across $x\geq2$ datasets. If we take this route, how many comparable neurons are there? **65**What percentage of the whole dataset are they? **21**What if we don't need to compare neurons shared across the entire dataset, but only across two readings?  Neurons Shared by Paired DatasetsNow I'm going to do something very cool using matplotlib_venn. I'm going to visualize the intersection of our datasets. I'm sure we can get a better metric than just percentage neurons shared.<jupyter_code>plt.clf()
%matplotlib inline
import numpy as np
from matplotlib_venn import *

import dimensions_kit as dk

dimensions = dk.dimensions(len(groupings))
f, axes = plt.subplots(*dimensions,figsize=(10,10))

f.suptitle('Overlapping neurons collected in datasets j and i', fontsize=20)
n=0
for k,i in groupings:
    
    names_k = neuron_names_present[k]
    names_i = neuron_names_present[i]
    
    x,y = dk.transform(dimensions,n)
    
    axis = axes[x][y]
    v = venn2(subsets=(names_k, names_i), ax=axis)
    
    intersection = float(len(names_k.intersection(names_i)))
    ttl = len(names_k)+len(names_i)-intersection
    ratio = int(intersection/ttl*100)
    
    axis.set_title('Dataset ${0}$ vs ${1}$: {2}% Overlap'.format(k,i,ratio ))
    plt.setp(axis.get_xticklabels(), visible=False)
    plt.setp(axis.get_yticklabels(), visible=False)

    n+=1
    
while n < dimensions[0]*dimensions[1]:
    x,y = dk.transform(dimensions,n)
    axes[x][y].axis('off')
    n+=1<jupyter_output><empty_output><jupyter_text>Starting to Compare The Two DatasetsHere is a list of ideas to consider. A set of very simple yet concrete questions we can go and answer. Originally I was unsure of how to look at complex datasets such as these. Once I had my revelation related to mathematics I recognized progress is driven by asking larger questions and breaking them down into smaller questions. Essentially, I'm looking for relationships - ways of relating dynamics to structure or structural patterns. Before I get into any serious graph theory or dynamical systems, or dynamical graph theory, I'm seeking inspiration from empirical data. Thus I start the process of developing a higher order theory by making observations on reality itself. My eventual goal is to start playing around with the pure mathematical ideas that arise from this experimental fishing expedition. I'll play around with the mathematical models I develop to build dynamical graphs with a goal in mind. First, I need to understand the empirical data. So here are some smaller questions I need to answer. - How many shared neurons play a significant role in a significant PC?- Are these roles consistent across datasets?- If any inconsistencies: why?- Is classifying neurons consistent with their expected effect on global dynamics, i.e. muscle neurons, sensory neurons, etc. - Correlation between neuron degree and PC weight?> I also feel compelled to mention my hazy revelation relating to NKS - production rules, and predicting dynamics. Well - simple things do create very complex, seemingly random behavior, but can some properties of this behavior be quantified - as a discrete dynamical system? Just a thought. Here are the components for all the datasets.<jupyter_code>components = [np.array(pc.components_) for pc in pca]
components
all_shared_neurons<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06856032568467801, 'fraction_numerical': 0.008697261287934863, 'mean_word_length': 4.29333986287953, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '456814', 'license': 'MIT', 'n_tokens_mistral': 3077, 'n_tokens_neox': 2901, 'n_words': 1400, 'path': 'laboratory/PCA Next Steps.ipynb', 'repo_name': 'openworm/neuronal-analysis', 'size': '10808'}