<jupyter_start><jupyter_text>Introduction<jupyter_code>import sys
#makin pyrfold visible to the system
sys.path.append('../../')
import pyrfold<jupyter_output><empty_output><jupyter_text>GoalThe goal of this notebook is to introduce you to the TimeCourseStructure class that has been developed for pyrfold. This class is especially good at processing the pickled dictionaries that are currently (140617) produced on hyak.  What's new* compressed dictionaray structures    * These dictionary stuctures take up less memory and are far easier to work with (computationally speaking) relative to the old run dictionaries (these dictionaries have every individual run for a given experiment)* pandas data structures for handling structure data over time  Loading in dataThe data structure can accept two types of data - the new compressed data structures and the old uncompressed time structures. Let's initalize the structure both ways so you can see what we're twerking with.<jupyter_code>#First we can load in the example data
import cPickle as pickle
with open('exampledata/example_timecourse.p', 'rb') as temppick:
    notcompressedtimedata = pickle.load(temppick)
with open('exampledata/compresseddict.p', 'rb') as temppick:
    compressedtimedata = pickle.load(temppick)<jupyter_output><empty_output><jupyter_text>Let's look at these structures now:**Non-compressed data**<jupyter_code>print "These are the individual runs of the this data"
print notcompressedtimedata.keys()
print 'This is the information contained in a single run'
print notcompressedtimedata[1].keys()<jupyter_output>This is the information contained in a single run
['sequence', 'energy', 'total bases', 'time', 'dotbracket', 'type']<jupyter_text>**compressed data**<jupyter_code>print 'This is the compressed data structure'
print compressedtimedata.keys()
print 'Dotbracket conatins:'
print 
print compressedtimedata['dotbracket'].keys()[:20]
print 
print 'All of the unique time points of a given experiment'
print 'energy is a dictionary of all of the unique structures (keys) that are seen with given free energy (value)'
print 
print compressedtimedata['energy'].keys()[:20]<jupyter_output>energy is a dictionary of all of the unique structures (keys) that are seen with given free energy (value)

['........(((((....)))))', '..((((....)))).....................(((((((....)))))))', '..(((...(((((....)))))))).....(((....)))(((((((((((((......))))))))))))).....', '.....', '.(((((.((((((....))).)))............)))))', '((((((....)))....(((................)))))).((((((((((......))))))))))', '..((((....))))......................', '..((((....))))....................', '...(((..(((((....)))))..............)))(((....)))...', '..(((...(((((....))))))))......(((((((.......)))))))(((....))).', '((((....(((((....)))))..........((((((((((....)))))))))).))))', '..(((...(((((....))))).........)))(((........)))', '..(((...(((((....))))).........))).', '...(((..(((((....)))))..............)))(((....))).......', '((((((..(((((....)))))..............)))))).....((((((......))))))', '..(((...(((((....))))))))........', '((((((..(((((....)))))..........((((((((((....))))))))))....))))))..........[...]<jupyter_text>You should explore these data structures until you undersand what's going on.<jupyter_code>#Import the object which does all of the heavey lifting
from pyrfold.analyze.timecourse import TimeCourseStructure<jupyter_output><empty_output><jupyter_text>Passing the data to the TimeCourseStructure objectNow this is a very important thing - There are two ways to initialize the object. 1. With the compressed dictionary (a much faster way to initilaize the data)2. With the uncomrpessed dictionary (which will actually be converted to a compressed dictionary before any work is done)    * **NOTE:** YOU HAVE TO INITIALIZE THE DICTIONARY WITH THE COMMAND .init_from_dictofruns()<jupyter_code>#initialize the object with the non-compressed dictionary
timeobj = TimeCourseStructure.init_from_dictofruns(dictionaryofruns=notcompressedtimedata)<jupyter_output><empty_output><jupyter_text>Now that we have the object initialized we can extract data from it. We do this through the call:    timeboj.generate_data(structurewindow = [], timewindow = [], cutoff = 0.0)    Structure window is a list containing the start and stop indexes (1 indexed) of the sequence you want look at. The time window tells the object when in time to evaluate these things. If you leave the time window blank it will look at all points in time. Cutoff will remove all structures that observed that do not appear at a MAXIMIM frequency which is above the cutoff. By default this cutoff value is set to 0.0, so all structures that appear will be considered.<jupyter_code>#Now we can see that that timeobj structure generated a compressed data structure:
timeobj.dictionary.keys()<jupyter_output><empty_output><jupyter_text>So what can we do?Now that we have the data loaded in the structure - what can we do? First - we can find the index of specific parts (assuming we know the sequence)Let's assume that the following sequence is a part that we are interested in:    AAGAGACGAAGGCUUCGGCCAAGUCGAAACGGAAACGUCGGAUAGUCGCCCGUCCCG<jupyter_code>#Now we can find the position index of this part

timeobj.find_sequence_index('AAGAGACGAAGGCUUCGGCCAAGUCGAAACGGAAACGUCGGAUAGUCGCCCGUCCCG')
#now we can generate data using this index:

timeobj.generate_data(structurewindow = [35,91])
#Now this data has been put into a pandas data frame:

timeobj.structuredataframe
# You will notice that this data is starting from time 0 and our part doesn't appear 
# for a while after this time point.
# To get around this I have added the option of "firstexposure" 
# this truncates the data frame to avoid time points when the requested structure 
# doesn't exist

timeobj.generate_data(structurewindow = [35,91],firstexposure=True)<jupyter_output><empty_output><jupyter_text>Now you can do anything that would normally be able to do with a pandas data frame. Here are some commands that I think are essential:<jupyter_code>print "look at all of the structures that are seen"
timeobj.structuredataframe.columns.tolist()
print "look at a specific structure"
timeobj.structuredataframe['....(((...(((....)))..)))...(((....))).))))))..))))))))).']
#these structures are very easy to plot using matplotlib 
%matplotlib inline
import matplotlib.pylab as plt
timelist = timeobj.structuredataframe.index.tolist()
structurelist = timeobj.structuredataframe['....(((...(((....)))..)))...(((....))).))))))..))))))))).']
plt.plot(timelist, structurelist)<jupyter_output>/usr/local/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')<jupyter_text>BUT PANDAS CAN DO A LOT OF THIS STUFF FOR YOUPandas is a very feature rich package for python. It contains many built in plotting features for data stored in its 'DataFrame' objects<jupyter_code>timeobj.structuredataframe.plot(legend=False)
timeobj.structuredataframe.plot(y = ['....(((...(((....)))..)))...(((....))).))))))..))))))))).'])
timeobj.structuredataframe.plot(y = ['....(((...(((....)))..)))...(((....))).))))))..))))))))).', '....(((...(((....)))..)))...(((....))).))))))..)))))))))('])
timeobj.structuredataframe.plot(subplots=True, figsize=(20,15), sharey=True);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.21269756228234662, 'fraction_numerical': 0.004821859094562015, 'mean_word_length': 4.142561983471074, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '337974', 'license': 'Apache-2.0', 'n_tokens_mistral': 2193, 'n_tokens_neox': 2089, 'n_words': 735, 'path': 'docs/tutorials/03_Analyzing_kinefold_simulation_data.ipynb', 'repo_name': 'carothersresearch/pyrfold', 'size': '7466'}