<jupyter_start><jupyter_text>Pandas: Python Data Analysis LibraryPandas is a toolkit built around the 'tables' package to help manipulate, read, write, and analyze large datasets<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np

from pylab import *<jupyter_output><empty_output><jupyter_text>The Series StructureSeries is meant to help organize time or other stream data, and can handle both regularly and irregularly sampled data. A nice feature people use is its built-in support for timestamps, which can be read in and used directly from a datafile without the need to translate. Here's an example of a time-dependent trend with lots of noise added<jupyter_code>s = pd.Series(np.random.randn(1000)+np.sin(np.arange(0,2,2e-3)*2*pi), index=pd.date_range('1/1/2000', periods=1000))
s.plot()
print s.describe()<jupyter_output>count    1000.000000
mean       -0.011977
std         1.255895
min        -3.352429
25%        -0.906763
50%         0.026917
75%         0.842357
max         4.917346
dtype: float64<jupyter_text>We can use the series object to do quick time-domain analyses, similar to what we can do with matplotlib but handled internally and self-consistently. The windowing (rolling) function are particularly useful.<jupyter_code>s.plot(alpha=0.25,label='Data')
r = s.rolling(50).mean()
v= s.rolling(50).std()
ma = s.rolling(10).max()
mi = s.rolling(10).min()
r.plot(label='Mean')
v.plot(label='Std')
ma.plot(label='Max')
mi.plot(label='Min')
legend(ncol=2)
show()<jupyter_output><empty_output><jupyter_text>Indexing and restricting the range can also be handled easily for a series, the same way one would do it for a numpy array<jupyter_code>sy=s['2000-01':'2001-06']
sy.plot()
show()

print sy[0],sy[400],sy['2001-04-01']<jupyter_output><empty_output><jupyter_text>Irregular SamplingHere's an example of scatter points to which we can add. We have 6 points originally, and add two more, then sort the index so that they're plotted sequentially.Note that for a series object, no two points can have the same index; if you want true scatter data, we'll have to use a different structure that we'll see shortly.<jupyter_code>sIrr = pd.Series([1,2,3,4,5,6],index=[0,10,11,12,42,60])
print sIrr.index

sIrr[1]=0
sIrr[-5]=20
print sIrr.index

sIrrSorted=sIrr.sort_index()
print sIrrSorted.index
print sIrrSorted

sIrrSorted.plot()
show()<jupyter_output>Int64Index([0, 10, 11, 12, 42, 60], dtype='int64')
Int64Index([0, 10, 11, 12, 42, 60, 1, -5], dtype='int64')
Int64Index([-5, 0, 1, 10, 11, 12, 42, 60], dtype='int64')
-5     20
 0      1
 1      0
 10     2
 11     3
 12     4
 42     5
 60     6
dtype: int64<jupyter_text>The DataFrame StructureWhat if we have multiple time-series data? How about an array with simple indices, where maybe we have uncorrelated x and y values? We need a dataframe, which is analogous to a numpy recarray.In this example, I have two series objects with different underlying signals that I call 'A' and 'B'.<jupyter_code>sA = pd.Series(np.random.randn(1000)+np.sin(np.arange(0,2,2e-3)*0.5*2*pi), index=pd.date_range('1/1/2010', periods=1000))
sB = pd.Series(np.random.randn(1000)+np.sin(np.arange(0,2,2e-3)*100*2*pi), index=pd.date_range('1/1/2010', periods=1000))

d = pd.DataFrame({'A':sA,'B':sB})
d.head()<jupyter_output><empty_output><jupyter_text>Note that the index here is the same as the index for the Series objects. They happened to be the same; we'll see shortly what happens when they're not the same. Below, we can index normally, and we see one way to access a column. We can also use dataframes to plot without resorting to for loops!<jupyter_code>print d['20100109':'20100110']
print d.A['20100109']

d.plot(alpha=0.5)
show()<jupyter_output>A         B
2010-01-09 -0.380351  0.069661
2010-01-10 -0.065276 -1.255418
-0.38035138904064836<jupyter_text>Suppose we have data from two different instruments with different time ranges. One gives us just A and B, and one includes an additional failsage flag. This new dataframe might look like the following<jupyter_code>sANew = pd.Series(np.random.randn(3000)+np.sin(np.arange(0,6,2e-3)*0.5*2*pi), index=pd.date_range('1/1/2015', periods=3000))
sBNew = pd.Series(np.random.randn(3000)+np.sin(np.arange(0,6,2e-3)*50*2*pi), index=pd.date_range('1/1/2015', periods=3000))
sCNew = pd.Series(np.random.randn(3000) > 0.5, index=pd.date_range('1/1/2015', periods=3000))

dNew = pd.DataFrame({'A':sANew,'B':sBNew,'C':sCNew})

dNew.head()<jupyter_output><empty_output><jupyter_text>Here we concatenate the new dataframes to make a new dataframe object; we see we have a new column with 'NaN' where C didn't previously exist. This also retained the original timestamps!<jupyter_code>dTot=pd.concat([d,dNew],sort=False)
dTot.head()<jupyter_output><empty_output><jupyter_text>We can use the 'dropna' function to get back only the rows or columns without the nans. Here we do it for the rows, next we do it for the columns (to get rid of C, which is redundant). We also see that we can get the trues and falses individually using the groupby command<jupyter_code>dValid=dTot.dropna(axis=0)
g=dValid.groupby('C')
print len(dTot),len(dValid),len(g.get_group(1)),len(g.get_group(0))
dValid.head()
dClean=dTot.dropna(axis=1)
dClean.head()<jupyter_output><empty_output><jupyter_text>If we don't drop columns with Nan, the dataframe will just ignore those columns when it needs values for all rows (e.g for plotting). Here is an example of a scatter plot, where you can see the time gap.<jupyter_code>dTot.plot(marker='o',linestyle='',markersize=2,alpha=0.5)
show()<jupyter_output><empty_output><jupyter_text>Using Dataframes to Construct New VariablesHere I'm going to make some fake data (it looks like some delta pulses in time-stream data) which might be gamma ray signals or dark matter events. The backgrounds are not coincident between these two channels, for whatever reason, and the foreground is mono-energetic and shared equally between channels. I will end up with a dataframe containing 1000 traces, readout for both A and B, and the truth information about whether the event is a good event or pileup.<jupyter_code>#create fake data
def signal(x,tau1,tau2,x0=500):
    dx=x-x0
    y=exp(-dx/tau1)/tau1-exp(-dx/tau2)/tau2
    y[dx<0]=0.0
    return y
 
def noisySignal(x,tau1,tau2,rms,x0=500):
    noise=np.random.normal(size=len(x),scale=rms)
    return signal(x,tau1,tau2,x0=x0)*2.0+noise
 
def noisyPileup(x,tau1,tau2,rms,nEvents,bad=False):
   
    noise=np.random.normal(size=len(x),scale=rms)
    x0=np.random.rand()*max(x)
    sig=signal(x,tau1,tau2,x0=x0)
   
    for i in range (1,nEvents):
        if(bad):
            x0+=50
        else:
            x0=np.random.rand()*max(x)
        amp=np.random.rand()*10
        sig+=signal(x,tau1,tau2,x0=x0)*amp
       
    sig+=noise
 
    return sig

cA=list()
cB=list()
truth=list()

x=np.arange(0,2000)

for i in range(0,1000):
    
    bad=np.random.rand()>0.1

    if(bad):
        chanA=np.random.rand()>0.5
        nEvents=(6-np.ceil(5*np.sqrt(np.random.uniform()))).astype(np.int)
        
        pileup=noisyPileup(x,1000,10,0.01,nEvents)
        noise=np.random.normal(size=len(x),scale=0.01)
        
        if(chanA):
            cA.append(pd.Series(pileup,index=x))
            cB.append(pd.Series(noise,index=x))
        else:
            cA.append(pd.Series(noise,index=x))
            cB.append(pd.Series(pileup,index=x))
    else:
        cA.append(pd.Series(noisySignal(x,1000,10,0.01),index=x))
        cB.append(pd.Series(noisySignal(x,1000,10,0.01),index=x))
    truth.append(not bad)
    
data=pd.DataFrame({'A':cA,'B':cB,'GoodEvent':truth})
data.head()<jupyter_output><empty_output><jupyter_text>Here I'm plotting a selection of single events in one channel; these are random, so I have no idea whether they are good or not.<jupyter_code>data['A'][0:9].apply(plot,alpha=0.5)
show()<jupyter_output><empty_output><jupyter_text>I can also plot a single event for both channels equally easily<jupyter_code>data['A'][0].plot()
data['B'][0].plot()<jupyter_output><empty_output><jupyter_text>Here I use the 'goodEvent' variable to select good versus bad events<jupyter_code>data.head()

g=data.groupby('GoodEvent')
for i in range(0,len(g)):
    for col in ['A','B']:
        g.get_group(i)[col].apply(plot,alpha=0.1)
        show()
for chan in ['A','B']:
    data['Max_'+chan]=data[chan].map(np.max)
    data['Min_'+chan]=data[chan].map(np.min)
    data['Range_'+chan]=data['Max_'+chan]-data['Min_'+chan]

hist(data['Range_A'],bins=100)

data.head()
scatter(data['Range_A'],data['Range_B'])
dataGood=data[(data['Range_A']>0.1) & (data['Range_B']>0.1)]

scatter(data['Range_A'],data['Range_B'])
scatter(dataGood['Range_A'],dataGood['Range_B'],color='red')

dataGood.head()<jupyter_output><empty_output><jupyter_text>MultiIndexThe above example is great, but having to combine strings and remember the naming scheme is a pain and not very good practice. Multiindices (indexing by pairs or tuples) solve this problem. Note that the Panel class was mean to do this, but that is now deprecated in favor or multiindexFor example, we can build an array with a multi-index to include column categories with variables grouped by channel<jupyter_code>data=pd.DataFrame({('A','trace'):cA,('B','trace'):cB,('Event','GoodEvent'):truth})
for chan in ['A','B']:
    data[chan,'Max']=data[chan]['trace'].map(np.max)
    data[chan,'Min']=data[chan]['trace'].map(np.min)
    data[chan,'Range']=data[chan,'Max']-data[chan,'Min']

data.head()<jupyter_output><empty_output><jupyter_text>This is great, but now the multi-index is disorganized. We can easily organize any index (on any level) using the sortindex and sortlevel functions<jupyter_code>data=data.sort_index(axis=1)
data.head()<jupyter_output><empty_output><jupyter_text>Saving dataPandas has many helpful writers that quickly write any structure to many formats of data (CSV, HDF5, etc.) (see full list at: http://pandas.pydata.org/pandas-docs/version/0.20/io.html)  Reading in dataPandas also has many helpful readers that quickly read in many formats of data (CSV, HDF5, etc.) into a DataFrame (see above link). If you create a 'data.txt' file you can try to run these cells to play around with importing data<jupyter_code>data = pd.read_csv('data.txt',delimiter=' ',skiprows=0,header=None)
data.head(5)
data = pd.read_csv('data.txt',delimiter=' ',skiprows=0
                   ,names=[160,135,126,117,109,100,90,82,73,63,54,46,-160,-150,-140,-130,-120,-110,-100,-90])
data.head(5)
data.sort_index(inplace=True,axis=1)
data=data[175:]
data.describe()
data.hist(figsize=(12,10),bins=100,log=True)
tight_layout()
show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13732097725358045, 'fraction_numerical': 0.05597678554713095, 'mean_word_length': 5.019154929577465, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 75, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '28598', 'license': 'MIT', 'n_tokens_mistral': 4164, 'n_tokens_neox': 3784, 'n_words': 1012, 'path': 'Pandas.ipynb', 'repo_name': 'fnal-cpc/bootcamp', 'size': '10683'}