<jupyter_start><jupyter_text>Pandas 是一个Python数据分析库，安装完 Anaconda 会安装好 Pandas  1 基础数据结构Pandas 包含了两种主要的数据结构+ Series+ DataFrameSeries 用来存储一维数据，而DataFrame则存储复杂的数据  Series表示一维数据，并且内部结构简单，由两个关联的数组组成，主数组用来存放数据，每个主数组有一个与之相关的标签。index | value--- | --- 0 | 12 1 | -4 2 | 22 3 | 0<jupyter_code>import numpy as np
import pandas as pd
a = pd.Series([12,-4,22,0])
a
# 讲标签替换成有意义的值
a = pd.Series([12,-4,22,0],index=['a','b','c','d'])
print a
print a.values
print a.index<jupyter_output>[12 -4 22  0]
Index([u'a', u'b', u'c', u'd'], dtype='object')<jupyter_text>选取数据<jupyter_code>print a[2]
print a['b']
print a[0:2]
print a[['b','d']]<jupyter_output>22
-4
a    12
b    -4
dtype: int64
b   -4
d    0
dtype: int64<jupyter_text>数据赋值<jupyter_code>a[2]=10
a['d']=-5<jupyter_output><empty_output><jupyter_text>从numpy对象中创建Series<jupyter_code>arr =np.array([1,2,3,4])
s = pd.Series(arr)
s
s1 = pd.Series(s)
s1[0]=-10
print s<jupyter_output>0   -10
1     2
2     3
3     4
dtype: int64<jupyter_text>筛选数据<jupyter_code>s[s>2]<jupyter_output><empty_output><jupyter_text>数学运算<jupyter_code>s/2
np.log(s)<jupyter_output><empty_output><jupyter_text>NaN字段中若为空或者不符合要求的数字定义的是，放回NaN（Not a Number)<jupyter_code>import numpy as np
import pandas as pd
s2 = pd.Series([5,-3,np.NaN,14])
s2
s2.isnull()
s2.notnull()
# 当做筛选条件
s2[s2.notnull()]
s2[s2.isnull()]<jupyter_output><empty_output><jupyter_text>字典使用<jupyter_code>mydic = {'red':200,'blue':1000,'yellow':500,'orange':1000}
myseries = pd.Series(mydic)
myseries
colors = ['red','yellow','orange','blue','green']
mySeries = pd.Series(mydic, index=colors)
mySeries<jupyter_output><empty_output><jupyter_text>Series 对象之前运算<jupyter_code>mydict2 = {'red':400,'yellow':1000,'black':700}
mySeries2 = pd.Series(mydict2)
mySeries + mySeries2<jupyter_output><empty_output><jupyter_text>DataFrame DataFrame 列表式跟Excel比较类似，其设计初衷将Series的使用场景扩展至多维index | color | object | price--- | --- | --- | ---0 | blue | ball | 1.21 | green | pen | 1.02 | yellow | pencil | 0.6DataFrame 对象则有所不同，它有两个索引数组1. 与行相关，与Series的索引组类似2. 一系列标签，每个标签与列数据关联<jupyter_code>import numpy as np
import pandas as pd
data = {'color':['blue','green','yellow'],'object':['ball','pen','pencil'],'price':[1.2,1.0,0.6]}
frame = pd.DataFrame(data)
frame<jupyter_output><empty_output><jupyter_text>当然也可以选择你感兴趣的内容<jupyter_code>frame2 = pd.DataFrame(data,columns=['object','price'])
frame2<jupyter_output><empty_output><jupyter_text>修改index值<jupyter_code>frame2 = pd.DataFrame(data,index=['one','two','three'])
frame2<jupyter_output><empty_output><jupyter_text>其他创建DataFrame的方式<jupyter_code>frame3 = pd.DataFrame(np.arange(16).reshape((4,4)),
                     index=['red','blue','yellow','white'],
                     columns=['ball','pen','pencil','paper'])
frame3<jupyter_output><empty_output><jupyter_text>选择元素<jupyter_code>frame.columns
frame.index
frame.values
frame['price']
frame.ix[2]
# 选择选择元素
frame.ix[[0,2]]
# 索引值选择类似切片
frame[0:1]
frame[0:3]
# 选择其中的元素
frame['object'][2]<jupyter_output><empty_output><jupyter_text>赋值<jupyter_code>frame.index.name='id'
frame.columns.name='item'
frame
# 增加一个新列
frame['new']=12 # 默认
frame
frame['new'] = [3.0,1.3,2.2]
frame
# 通过Series其他方式更新
ser = pd.Series(np.arange(3))
frame['new']=ser
frame<jupyter_output><empty_output><jupyter_text>元素的所属关系<jupyter_code>frame.isin([1.0,'pen'])
frame[frame.isin([1.0,'pen'])]<jupyter_output><empty_output><jupyter_text>删除某一列<jupyter_code>del frame['new']
frame<jupyter_output><empty_output><jupyter_text>筛选元素<jupyter_code>frame[frame<3]<jupyter_output><empty_output><jupyter_text>有嵌套的字典生成DataFrame对象pandas 将会将外部的key解释为列名称，将内部的key解释为索引的标签<jupyter_code>nestdict = {'red':{2012:22,2013:33},'white':{2011:13,2012:22,2013:16},'blue':{2011:17,2012:27,2013:18}}
frame2 = pd.DataFrame(nestdict)
frame2<jupyter_output><empty_output><jupyter_text>DataFrame转置<jupyter_code>frame2.T<jupyter_output><empty_output><jupyter_text>index对象<jupyter_code>ser = pd.Series([5,0,3,8,4],index=['red','blue','yellow','white','green'])
ser.index<jupyter_output><empty_output><jupyter_text>index 对象方法<jupyter_code>ser.idxmin()
ser.idxmax()<jupyter_output><empty_output><jupyter_text>含有重复标签的index<jupyter_code>serd = pd.Series(range(6),index=['white','white','blue','green','green','yellow'])
serd
serd['white']
serd.index.is_unique<jupyter_output><empty_output><jupyter_text>索引对象其他的作用+ 更换索引+ 删除+ 对齐  更换索引<jupyter_code>ser = pd.Series([2,5,7,4],index=['one','two','three','four'])
ser
ser.reindex(['three','four','five','one'])<jupyter_output><empty_output><jupyter_text>上述reindex函数删除了'two'标签，增加了'five'标签，并且该值为NaN自动填充标签<jupyter_code>ser3 = pd.Series([1,5,6,3],index=[0,3,5,6])
ser3<jupyter_output><empty_output><jupyter_text>上述索引列并不完整，而是缺失了1，2和4，常见的需求为插值，得到一个完整的序列，reindex函数的method选项的值进行给定<jupyter_code>ser3.reindex(range(6),method='ffill')<jupyter_output><empty_output><jupyter_text>在插值过程中，所以缺失的索引所对应的值是比其小的索引值，如果想要后面的值，修改method的选项<jupyter_code>ser3.reindex(range(6),method='bfill')<jupyter_output><empty_output><jupyter_text>删除操作<jupyter_code>ser = pd.Series(np.arange(4.),index=['red','blue','yellow','white'])
ser
ser.drop('yellow')<jupyter_output><empty_output><jupyter_text>DataFrame删除操作<jupyter_code>frame = pd.DataFrame(np.arange(16).reshape((4,4)),
                    index=['red','blue','yellow','white'],
                    columns=['ball','pen','pencil','paper'])
frame
# 删除行
frame.drop(['blue','yellow'])<jupyter_output><empty_output><jupyter_text>删除列操作，指定列名，也要指定axis的值<jupyter_code>frame.drop(['pen','pencil'],axis=1)<jupyter_output><empty_output><jupyter_text>算术和数据对齐<jupyter_code>s1 = pd.Series([3,2,5,1],['white','yellow','green','blue'])
s2 = pd.Series([1,4,7,2,1],['white','yellow','black','blue','brown'])
s1+s2
frame1 = pd.DataFrame(np.arange(16).reshape((4,4)),
                     index=['red','blue','yellow','white'],
                     columns=['ball','pen','pencil','paper'])
frame2 = pd.DataFrame(np.arange(12).reshape((4,3)),
                     index=['blue','green','white','yellow'],
                     columns=['mug','pen','ball'])
frame1+frame2<jupyter_output><empty_output><jupyter_text>数据结构之间的运算  算术运算+ add()+ sub()+ div()+ mul()<jupyter_code>frame1.add(frame2)
frame1.sub(frame2)
frame1.div(frame2)
frame1.mul(frame2)<jupyter_output><empty_output><jupyter_text>DataFrame与Series之间运算<jupyter_code>frame = pd.DataFrame(np.arange(16).reshape((4,4)),
                    index=['red','blue','yellow','white'],
                    columns=['ball','pen','pencil','paper'])
frame
ser = pd.Series(np.arange(4),index=['ball','pen','pencil','paper'])
ser
frame - ser
ser['mug']=9
frame - ser<jupyter_output><empty_output><jupyter_text>函数应用与映射  操作元素的函数Pandas与numpy 一样，有大量关于元素操作的通用函数（universal function）<jupyter_code>frame
np.sqrt(frame)<jupyter_output><empty_output><jupyter_text>按行和列执行操作函数<jupyter_code>f = lambda x:x.max()-x.min()
frame.apply(f)
# 对行进行操作
frame.apply(f, axis=1)<jupyter_output><empty_output><jupyter_text>apply 函数不一定返回标量，也可以返回一个向量<jupyter_code>def f(x):
    return pd.Series([x.min(),x.max()],index=['min','max'])
frame.apply(f)
frame.apply(f,axis=1)<jupyter_output><empty_output><jupyter_text>统计函数<jupyter_code>frame.sum()
frame.mean()
frame.describe()<jupyter_output><empty_output><jupyter_text>排序  Series series对象排序只有索引这一列<jupyter_code>ser =pd.Series([5,0,3,8,4],index=['red','blue','yellow','white','green'])
ser
ser.sort_index()<jupyter_output><empty_output><jupyter_text>DataFrame排序 DataFrame 可以对两个轴任意一条进行排序，如果用索引列进行排序，直接使用sort_index()函数，如果对列进行排序，指定axis=1<jupyter_code>frame
frame.sort_index()
frame.sort_index(axis=1)<jupyter_output><empty_output><jupyter_text>对其中的元素进行排序<jupyter_code>ser.sort_values()
frame.sort_values(by='pen')
#基于多列排序
frame.sort_values(by=['pen','pencil'])<jupyter_output><empty_output><jupyter_text>ranking 操作与排序操作相关，该操作为序列的每一个元素安排一个位置，从1开始<jupyter_code>ser.rank()
ser.rank(method='first')
# 降序排列
ser.rank(ascending=False)<jupyter_output><empty_output><jupyter_text>相关性和协方差 correlation和covariance是两个重要的统计量，pandas计算这两个量使用corr()和cov().+ 标准差$$D(X)=E([X - E(X)]^2)$$+ 协方差$$COV(X,Y)=E([X-E(X)][Y-E(Y)])$$ + 相关系数$$\frac{COV(X,Y)}{\sqrt(D(X)) \times \sqrt(D(Y))}$$<jupyter_code>seq1 = pd.Series([3,4,3,4,5,4,3,2],index=['2006','2007','2008','2009','2010','2011','2012','2013'])
seq2 = pd.Series([1,2,3,4,4,3,2,1],index=['2006','2007','2008','2009','2010','2011','2012','2013'])
print seq1.corr(seq2)
print seq1.cov(seq2)
frame2 = pd.DataFrame([[1,4,3,6],[4,5,6,1],[3,3,1,5],[4,1,6,4]],
                     index=['red','blue','yellow','white'],
                     columns=['ball','pen','pencil','paper'])
frame2
frame2.corr()
frame2.cov()
frame2.corrwith(ser)
frame2.corrwith(frame)<jupyter_output><empty_output><jupyter_text>NaN数据<jupyter_code>ser = pd.Series([0,1,2,np.NaN,9],index=['red','blue','yellow','white','green'])
ser
ser['green']=None
ser<jupyter_output><empty_output><jupyter_text>过滤NaN数据<jupyter_code>ser.dropna()
ser[ser.notnull()]
frame3 = pd.DataFrame([[6,np.nan,6],[np.nan,np.nan,np.nan],[2,np.nan,5]],
                     index=['blue','green','red'],
                     columns=['ball','mug','pen'])
frame3<jupyter_output><empty_output><jupyter_text>在DataFrame中使用dropna()函数，一旦这一行或者列存在NaN，则将其整行或者整列全部删除<jupyter_code>frame3.dropna()
# 改进
frame3.dropna(how='all')<jupyter_output><empty_output><jupyter_text>为NaN赋值<jupyter_code>frame3.fillna(0)
frame3.fillna({'ball':1,'mug':0,'pen':99})<jupyter_output><empty_output><jupyter_text>等级索引和分级等级索引（hierarchical indexing）是pandas的一个重要的功能，单条轴也可以有多级索引，可以像操作两维结构那样处理多维数据<jupyter_code>mser = pd.Series(np.random.rand(8),
                index=[['white','white','white','blue','blue','red','red','red'],
                      ['up','down','right','up','down','up','down','left']])
mser
mser.index
mser['white']
mser[:,'up']
mser['white','up']<jupyter_output><empty_output><jupyter_text>使用unstack()函数将其转换成DataFrame,其中第二列的索引转换成列<jupyter_code>mser.unstack()<jupyter_output><empty_output><jupyter_text>逆操作，将DataFrame转换成Series对象，使用stack()函数<jupyter_code>frame.stack()<jupyter_output><empty_output><jupyter_text>对于DataFrame来讲，可以对其DataFrame对象的行和列分别进行定义等级索引<jupyter_code>mframe = pd.DataFrame(np.random.randn(16).reshape((4,4)),
                     index=[['white','white','red','red'],['up','down','up','down']],
                     columns=[['pen','pen','paper','paper'],[1,2,1,2]])
mframe<jupyter_output><empty_output><jupyter_text>调整顺序和为层级排序<jupyter_code># 指定index和columns的名称
mframe.columns.names=['object','id']
mframe.index.names=['colors','status']
mframe
mframe.swaplevel('colors','status')
mframe.sortlevel('colors')<jupyter_output><empty_output><jupyter_text>按层进行统计数据<jupyter_code>mframe.sum(level='colors')<jupyter_output><empty_output><jupyter_text>如果相对某一列进行统计分析<jupyter_code>mframe.sum(level='id',axis=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.20108593896274107, 'fraction_numerical': 0.04025463396367721, 'mean_word_length': 9.463271302644467, 'pattern_counts': {'":': 0, '<': 211, '<?xml version=': 0, '>': 211, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '53', 'dir': 'jupyter-structured-clean-dedup', 'id': '620385', 'license': 'MIT', 'n_tokens_mistral': 5173, 'n_tokens_neox': 5186, 'n_words': 340, 'path': 'Data_Analytics_in_Action/pandas.ipynb', 'repo_name': 'gaufung/Data_Analytics_Learning_Note', 'size': '10682'}