<jupyter_start><jupyter_text>Pandas - 数据处理  一、简介Pandas是一个强大的分析结构化数据的工具集；它的使用基础是Numpy（提供高性能的矩阵运算）；用于数据挖掘和数据分析，同时也提供数据清洗功能。- Series：一维数组，与Numpy中的一维array类似。二者与Python基本的数据结构List也很相近。Series如今能保存不同种数据类型，字符串、boolean值、数字等都能保存在Series中。- Time- Series：以时间为索引的Series。- DataFrame：二维的表格型数据结构。很多功能与R中的data.frame类似。可以将DataFrame理解为Series的容器。- Panel ：三维的数组，可以理解为DataFrame的容器。- Panel4D：是像Panel一样的4维数据容器。- PanelND：拥有factory集合，可以创建像Panel4D一样N维命名容器的模块。  二、基本使用 (1) 生成序列<jupyter_code>import pandas as pd
import numpy as np
a1 = pd.Series([1,2,3,np.nan,44,1])
print(a1)<jupyter_output>0     1.0
1     2.0
2     3.0
3     NaN
4    44.0
5     1.0
dtype: float64<jupyter_text>(2) 生成日期<jupyter_code># Signature: pd.date_range(start=None, end=None, periods=None, freq='D',
# tz=None, normalize=False, name=None, closed=None, **kwargs)
datas = pd.date_range('20190101',periods=6)
print(datas)<jupyter_output>DatetimeIndex(['2019-01-01', '2019-01-02', '2019-01-03', '2019-01-04',
               '2019-01-05', '2019-01-06'],
              dtype='datetime64[ns]', freq='D')<jupyter_text>(3)生成数据<jupyter_code># 定义行列
df = pd.DataFrame(np.random.randn(6,4),index=datas,columns=['a','b','c','d'])
print(df)
# 不定义行列
df1 = pd.DataFrame(np.arange(12).reshape(3,4))
print(df1)
print(df1.describe()) #计算<jupyter_output>0  1   2   3
0  0  1   2   3
1  4  5   6   7
2  8  9  10  11
         0    1     2     3
count  3.0  3.0   3.0   3.0
mean   4.0  5.0   6.0   7.0
std    4.0  4.0   4.0   4.0
min    0.0  1.0   2.0   3.0
25%    2.0  3.0   4.0   5.0
50%    4.0  5.0   6.0   7.0
75%    6.0  7.0   8.0   9.0
max    8.0  9.0  10.0  11.0<jupyter_text>(4) 基本操作<jupyter_code># 用字典定义
df2 = pd.DataFrame({'A':'aaa','B':pd.Categorical([1,2])}) #缺失的会复制
print(df2)
print(df2.dtypes)
print(df2.index) # 打印行索引
print(df2.columns)# 打印列索引
print(df2.values) # 打印值
print(df2.T) # 转制
df = pd.DataFrame(np.random.randn(6,4),index=datas,columns=['a','b','c','d'])
df.sort_index(axis=1,ascending=False)
df.sort_index(axis=0,ascending=False)
# 根据某一参数排序
df.sort_values(by='c')<jupyter_output><empty_output><jupyter_text>三、进阶使用 (1) 切片取值<jupyter_code>df = pd.DataFrame(np.random.randn(6,4),index=datas,columns=['a','b','c','d'])
df['a']
# 直接索引
df[0:3]
# 通过下标查找(类似于EXCEL中的行列号)
df.iloc[0:3]
df.iloc[0:3,1:2]
df.iloc[0:1,:]
# 通过标签查找(类似于EXCEL中的标签行列)
df.loc['20190101',:]<jupyter_output><empty_output><jupyter_text>(2)筛选<jupyter_code># 按条件筛选
df[df['a']>0.01]<jupyter_output><empty_output><jupyter_text>（3）修改<jupyter_code>df = pd.DataFrame(np.random.randn(6,4),index=datas,columns=['a','b','c','d'])
df.iloc[2,2] = 5.5555
df
df[df['a']<0] = 1
df
df.a[df['a']>0.5] = 5
df<jupyter_output><empty_output><jupyter_text>(4) 处理丢失数据<jupyter_code>df = pd.DataFrame(np.random.randn(6,4),index=datas,columns=['a','b','c','d'])
df.iloc[0,1] = np.nan
df.iloc[1,2] = np.nan
df
df.dropna(axis=0,how='all') # all
df.dropna(axis=0,how='any') # all
df = pd.DataFrame(np.random.randn(6,4),index=datas,columns=['a','b','c','d'])
df.iloc[0,1] = np.nan
df.iloc[1,2] = np.nan
df.fillna(value=1000)
df = pd.DataFrame(np.random.randn(6,4),index=datas,columns=['a','b','c','d'])
df.iloc[0,1] = np.nan
df.iloc[1,2] = np.nan
df.isnull()<jupyter_output><empty_output><jupyter_text>(5) 导入导出数据 pandas可以读取csv、excel、hdf、sql、json、msgpack、html、gbq、stata、clipboard、pickle。读取的格式为read_xxx,写的格式为to_xxx<jupyter_code># 读取CSV
import pandas as pd
data = pd.read_csv('lessons/csvtest1.csv')
print(data)
# 存储为任意格式
data.to_pickle('lessons/csvtest1.pickle')<jupyter_output><empty_output><jupyter_text>(6)合并数据<jupyter_code>import pandas as pd
import numpy as np
df0 = pd.DataFrame(np.ones((3,4))*0, columns=['a','b','c','d'])
df1 = pd.DataFrame(np.ones((3,4))*1, columns=['a','b','c','d'])
df2 = pd.DataFrame(np.ones((3,4))*2, columns=['a','b','c','d'])
print(df0)
print(df1)
print(df2)
res = pd.concat([df0,df1,df2], axis=0)
print(res)
res = pd.concat([df0,df1,df2], axis=1)
print(res)
res = pd.concat([df0,df1,df2], axis=0, ignore_index=True)
print(res)
# join,['inner','outer']
import pandas as pd
import numpy as np
df0 = pd.DataFrame(np.ones((3,4))*0, columns=['a','b','c','d'],index=[1,2,3])
df1 = pd.DataFrame(np.ones((3,4))*1, columns=['b','c','d','e'],index=[2,3,4])
print(df0)
print(df1)
res = pd.concat([df0,df1],join='outer',sort=False)
print(res)
res = pd.concat([df0,df1],join='inner',ignore_index=True)
print(res)
# res = pd.merge()   merge<jupyter_output><empty_output><jupyter_text>(7) plot画图<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
data  = pd.Series(np.random.randn(1000),index=np.arange(1000))
data = data.cumsum()
data.plot()
plt.show()
data = pd.DataFrame(np.random.randn(1000,4),index=np.arange(1000),columns=list('ABCD'))
print(data.head())
data = data.cumsum()
data.plot()
plt.show()<jupyter_output>A         B         C         D
0 -0.823284  0.731889  1.395714  0.427060
1 -0.966378 -0.241757 -0.943584 -1.063513
2  0.164956 -0.484679  0.683918 -0.418225
3 -1.294836 -0.520445  0.139156  0.928332
4  1.394723 -0.191527  0.661297  0.743933
{'fraction_non_alphanumeric': 0.20627623425944433, 'fraction_numerical': 0.09814111533080151, 'mean_word_length': 5.6103038309114925, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '500126', 'license': 'MIT', 'n_tokens_mistral': 2734, 'n_tokens_neox': 2627, 'n_words': 325, 'path': '99 Cache/.ipynb_checkpoints/lesson1-2 Pandas-checkpoint.ipynb', 'repo_name': 'Leon-Hou/PythonLearningNotebook', 'size': '5003'}