<jupyter_start><jupyter_text>用值列表生成 Series 时，Pandas 默认自动生成整数索引：<jupyter_code>s = pd.Series([1, 3, 5, np.nan, 6, 8])
s<jupyter_output><empty_output><jupyter_text>用含日期时间索引与标签的 NumPy 数组生成 DataFrame：<jupyter_code>dates = pd.date_range('20130101', periods=6)
dates
df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
df<jupyter_output><empty_output><jupyter_text>用 Series 字典对象生成 DataFrame：<jupyter_code>df2 = pd.DataFrame({'A': 1.,
                    'B': pd.Timestamp('20130102'),
                    'C': pd.Series(1, index=list(range(4)), dtype='float32'),
                    'D': np.array([3] * 4, dtype='int32'),
                    'E': pd.Categorical(["test", "train", "test", "train"]),
                    'F': 'foo'})
df2<jupyter_output><empty_output><jupyter_text>DataFrame 的列有不同数据类型<jupyter_code>df2.dtypes<jupyter_output><empty_output><jupyter_text>IPython支持 tab 键自动补全列名与公共属性。下面是部分可自动补全的属性： - `df2.A`- `df2.abs`- `df2.add`- `df2.add_prefix`- `df2.add_suffix`- `df2.align`- `df2.all`- `df2.any`- `df2.append`- `df2.apply`- `df2.applymap`- `df2.D`- `df2.bool`- `df2.boxplot`- `df2.C`- `df2.clip`- `df2.clip_lower`- `df2.clip_upper`- `df2.columns`- `df2.combine`- `df2.combine_first`- `df2.compound`- `df2.consolidate`列 A、B、C、D 和 E 都可以自动补全；为简洁起见，此处只显示了部分属性。  查看数据<jupyter_code># 查看 DataFrame 头部数据
df.head()
# 查看 DataFrame 尾部数据
df.tail(3)
# 显示索引
df.index
# 显示列名
df.columns<jupyter_output><empty_output><jupyter_text>[DataFrame.to_numpy()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_numpy.htmlpandas.DataFrame.to_numpy)输出底层数据的 NumPy 对象。注意，[DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.htmlpandas.DataFrame)的列由多种数据类型组成时，该操作耗费系统资源较大，这也是 Pandas 和 NumPy 的本质区别：NumPy 数组只有一种数据类型，DataFrame 每列的数据类型各不相同。调用`DataFrame.to_numpy()`时，Pandas 查找支持 DataFrame 里所有数据类型的 NumPy 数据类型。还有一种数据类型是 object，可以把 DataFrame 列里的值强制转换为 Python 对象。下面的 df 这个 DataFrame 里的值都是浮点数，DataFrame.to_numpy() 的操作会很快，而且不复制数据。以上已被弃用，使用`df.values`。<jupyter_code>df.values
df2.values<jupyter_output><empty_output><jupyter_text>`df.describe()`快速查看数据的统计摘要<jupyter_code>df.describe()
# 转制数据
df.T
# 按轴排序
df.sort_index(axis=1, ascending=False)
# 按值排序
df.sort_values(by='B')<jupyter_output><empty_output><jupyter_text>选择> **提醒**选择、设置标准 Python / Numpy 的表达式已经非常直观，交互也很方便，但对于生产代码，我们还是推荐优化过的 Pandas 数据访问方法：`.at`、`.iat`、`.loc` 和 `.iloc`。详见[索引与选择数据](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.htmlindexing)、[多层索引与高级索引](https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.htmladvanced)文档。  获取数据选择单列，产生 Series，与 `df.A` 等效：<jupyter_code>df['A']<jupyter_output><empty_output><jupyter_text>用 [ ] 切片行：<jupyter_code>df[0:3]<jupyter_output><empty_output><jupyter_text>[按标签选择](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.htmlindexing-label)用标签提取一行数据：<jupyter_code>df.loc[dates[0]]
# 用标签选择多列数据
df.loc[:, ['A', 'B']]
# 用标签切片，包含行与列结束点
df.loc['20130102':'20130104', ['A', 'B']]
# 返回对象降维
df.loc['20130102', ['A', 'B']]
# 提取标量值
df.loc[dates[0], 'A']
# 快速访问标量，与 loc 等效
df.at[dates[0], 'A']<jupyter_output><empty_output><jupyter_text>[按位置选择](http://pandas.pydata.org/Pandas-docs/stable/indexing.htmlindexing-integer)用整数位置选择：<jupyter_code>df.iloc[3]
# 类似 NumPy / Python，用整数切片：
df.iloc[3:5, 0:2]
# 类似 NumPy / Python，用整数列表按位置切片：
df.iloc[[1, 2, 4], [0, 2]]
# 显式整行切片
df.iloc[1:3, :]
# 显式整列切片
df.iloc[:, 1:3]
# 显式提取值
df.iloc[1, 1]
# 快速访问标量 等效与 iloc
df.iat[1, 1]<jupyter_output><empty_output><jupyter_text>布尔索引用单列的值选择数据：<jupyter_code>df[df.A > 0]
# 选择 DataFrame 里满足条件的值
df[df > 0]
# 用 isin() 筛选
df2 = df.copy()
df2['E'] = ['one', 'one', 'two', 'three', 'four', 'three']
df2
df2[df2['E'].isin(['two', 'four'])]<jupyter_output><empty_output><jupyter_text>赋值用索引自动对齐新增列的数据：<jupyter_code>s1 = pd.Series([1, 2, 3, 4, 5, 6], index=pd.date_range('20130102', periods=6))
s1
df['F'] = s1
# 按标签赋值
df.at[dates[0], 'A'] = 0

# 按位置赋值
df.iat[0, 1] = 0

# 按 NumPy 数组赋值
df.loc[:, 'D'] = np.array([5] * len(df))

df
# 用 where 条件赋值
df2 = df.copy()
df2[df2 > 0] = -df2
df2<jupyter_output><empty_output><jupyter_text>缺失值Pandas 主要用 np.nan 表示缺失数据。 计算时，默认不包含空值。详见[缺失数据](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.htmlmissing-data)。重建索引（reindex）可以更改、添加、删除指定轴的索引，并返回数据副本，即不更改原数据。<jupyter_code>df1 = df.reindex(index=dates[0:4], columns=list(df.columns) + ['E'])
df1.loc[dates[0]:dates[1], 'E'] = 1
df1
# 删除所有含缺失值的行
df1.dropna(how='any')
# 填充缺失值
df1.fillna(value=5)
# 提取 nan 值的布尔掩码
pd.isna(df1)<jupyter_output><empty_output><jupyter_text>运算[二进制操作](https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.htmlbasics-binop) 统计一般情况下，运算时排除缺失值。描述性统计：<jupyter_code>df.mean()
# 在另一个轴(即，行)上执行同样的操作
df.mean(1)
# 不同维度对象运算时，要先对齐。 此外，Pandas 自动沿指定维度广播
s = pd.Series([1, 3, 5, np.nan, 6, 8], index=dates).shift(2)
s
df.sub(s, axis='index')<jupyter_output><empty_output><jupyter_text>Apply 函数Apply 函数处理数据：<jupyter_code>df.apply(np.cumsum)
df.apply(lambda x: x.max() - x.min())<jupyter_output><empty_output><jupyter_text>[直方图](https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.htmlbasics-discretization)<jupyter_code>s = pd.Series(np.random.randint(0, 7, size=10))
s
s.value_counts()<jupyter_output><empty_output><jupyter_text>字符串方法Series 的 str 属性包含一组字符串处理功能，如下列代码所示。注意，str 的模式匹配默认使用[正则表达式](https://docs.python.org/3/library/re.html)。详见[矢量字符串方法](https://pandas.pydata.org/pandas-docs/stable/user_guide/text.htmltext-string-methods)。<jupyter_code>s = pd.Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])
s.str.lower()<jupyter_output><empty_output><jupyter_text>[合并（Merge）](https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.htmlmerging) 结合（Concat）Pandas 提供了多种将 Series、DataFrame 对象组合在一起的功能，用索引与关联代数功能的多种设置逻辑可执行连接（join）与合并（merge）操作。`concat()` 用于连接 Pandas 对象：<jupyter_code>df = pd.DataFrame(np.random.randn(10, 4))
df
# 分解为多组
pieces = [df[:3], df[3:7], df[7:]]
pd.concat(pieces)<jupyter_output><empty_output><jupyter_text>连接（join）SQL 风格的合并。 详见[数据库风格连接](https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.htmlmerging-join)<jupyter_code>left = pd.DataFrame({'key': ['foo', 'foo'], 'lval': [1, 2]})
right = pd.DataFrame({'key': ['foo', 'foo'], 'rval': [4, 5]})
left
right
pd.merge(left, right, on='key')
left = pd.DataFrame({'key': ['foo', 'bar'], 'lval': [1, 2]})
right = pd.DataFrame({'key': ['foo', 'bar'], 'rval': [4, 5]})
left
right
pd.merge(left, right, on='key')<jupyter_output><empty_output><jupyter_text>[追加（Append）](https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.htmlmerging-concatenation)为 DataFrame 追加行。<jupyter_code>df = pd.DataFrame(np.random.randn(8, 4), columns=['A', 'B', 'C', 'D'])
df
s = df.iloc[3]
df.append(s, ignore_index=True)<jupyter_output><empty_output><jupyter_text>[分组（Grouping）](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.htmlgroupby)“group by” 指的是涵盖下列一项或多项步骤的处理流程：- 分割：按条件把数据分割成多组；- 应用：为每组单独应用函数；- 组合：将处理结果组合成一个数据结构。<jupyter_code>df = pd.DataFrame({'A': ['foo', 'bar', 'foo', 'bar',
                         'foo', 'bar', 'foo', 'foo'],
                   'B': ['one', 'one', 'two', 'three',
                         'two', 'two', 'one', 'three'],
                   'C': np.random.randn(8),
                   'D': np.random.randn(8)})
df
# 先分组，再用 sum()函数计算每组的汇总数据
df.groupby('A').sum()<jupyter_output><empty_output><jupyter_text>多列分组后，生成多层索引，也可以应用 sum 函数：<jupyter_code>df.groupby(['A', 'B']).sum()<jupyter_output><empty_output><jupyter_text>重塑（Reshaping）详见[多层索引](https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.htmladvanced-hierarchical)与[重塑](https://pandas.pydata.org/pandas-docs/stable/user_guide/reshaping.htmlreshaping-stacking) 堆叠（Stack）<jupyter_code>tuples = list(zip(*[['bar', 'bar', 'baz', 'baz',
                     'foo', 'foo', 'qux', 'qux'],
                    ['one', 'two', 'one', 'two',
                     'one', 'two', 'one', 'two']]))
index = pd.MultiIndex.from_tuples(tuples, names=['first', 'second'])
df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=['A', 'B'])
df2 = df[:4]
df2<jupyter_output><empty_output><jupyter_text>[`stack()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.stack.htmlpandas.DataFrame.stack)方法把 DataFrame 列压缩至一层：<jupyter_code>stacked = df2.stack()
stacked<jupyter_output><empty_output><jupyter_text>压缩后的 DataFrame 或 Series 具有多层索引， `stack()` 的逆操作是 [`unstack()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.unstack.htmlpandas.DataFrame.unstack)，默认为拆叠最后一层：<jupyter_code>stacked.unstack()
stacked.unstack(1)
stacked.unstack(0)<jupyter_output><empty_output><jupyter_text>[数据透视表（Pivot Tables）](https://pandas.pydata.org/pandas-docs/stable/user_guide/reshaping.htmlreshaping-pivot)<jupyter_code>df = pd.DataFrame({'A': ['one', 'one', 'two', 'three'] * 3,
                   'B': ['A', 'B', 'C'] * 4,
                   'C': ['foo', 'foo', 'foo', 'bar', 'bar', 'bar'] * 2,
                   'D': np.random.randn(12),
                   'E': np.random.randn(12)})
df<jupyter_output><empty_output><jupyter_text>用上述数据生成数据透视表非常简单：<jupyter_code>pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'])<jupyter_output><empty_output><jupyter_text>[时间序列(TimeSeries)](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.htmltimeseries)Pandas 为频率转换时重采样提供了虽然简单易用，但强大高效的功能，如，将秒级的数据转换为 5 分钟为频率的数据。这种操作常见于财务应用程序，但又不仅限于此。<jupyter_code>rng = pd.date_range('1/1/2012', periods=100, freq='S')
ts = pd.Series(np.random.randint(0, 500, len(rng)), index=rng)
ts.resample('5Min').sum()
# 时区表示
rng = pd.date_range('3/6/2012 00:00', periods=5, freq='D')
ts = pd.Series(np.random.randn(len(rng)), rng)
ts
ts_utc = ts.tz_localize('UTC')
ts_utc
# 转换成其它时区
ts_utc.tz_convert('US/Eastern')
# 转换时间段
rng = pd.date_range('1/1/2012', periods=5, freq='M')
ts = pd.Series(np.random.randn(len(rng)), index=rng)
ts
ps = ts.to_period()
ps
ps.to_timestamp()<jupyter_output><empty_output><jupyter_text>Pandas 函数可以很方便地转换时间段与时间戳。下例把以 11 月为结束年份的季度频率转换为下一季度月末上午 9 点：<jupyter_code>prng = pd.period_range('1990Q1', '2000Q4', freq='Q-NOV')
ts = pd.Series(np.random.randn(len(prng)), prng)
ts.index = (prng.asfreq('M', 'e') + 1).asfreq('H', 's') + 9
ts.head()<jupyter_output><empty_output><jupyter_text>[类别型（Categoricals）](https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.htmlcategorical)Pandas 的 DataFrame 里可以包含类别数据。[API 文档](https://pandas.pydata.org/pandas-docs/stable/reference/arrays.htmlapi-arrays-categorical)<jupyter_code>df = pd.DataFrame({"id": [1, 2, 3, 4, 5, 6],
                   "raw_grade": ['a', 'b', 'b', 'a', 'a', 'e']})
# 将 grade 的原生数据转换为类别型数据
df["grade"] = df["raw_grade"].astype("category")
df["grade"]<jupyter_output><empty_output><jupyter_text>用有含义的名字重命名不同类型，调用 `Series.cat.categories`。<jupyter_code>df["grade"].cat.categories = ["very good", "good", "very bad"]<jupyter_output><empty_output><jupyter_text>重新排序各类别，并添加缺失类，`Series.cat` 的方法默认返回新 Series。<jupyter_code>df["grade"] = df["grade"].cat.set_categories(["very bad", "bad", "medium",
                                              "good", "very good"])
df["grade"]<jupyter_output><empty_output><jupyter_text>注意，这里是按生成类别时的顺序排序，不是按词汇排序：<jupyter_code>df.sort_values(by="grade")<jupyter_output><empty_output><jupyter_text>按类列分组（groupby）时，即便某类别为空，也会显示：<jupyter_code>df.groupby("grade").size()<jupyter_output><empty_output><jupyter_text>[可视化](https://pandas.pydata.org/pandas-docs/stable/user_guide/visualization.htmlvisualization)<jupyter_code>ts = pd.Series(np.random.randn(1000),
               index=pd.date_range('1/1/2000', periods=1000))
ts = ts.cumsum()
ts.plot()<jupyter_output><empty_output><jupyter_text>DataFrame 的 `plot()` 方法可以快速绘制所有带标签的列：<jupyter_code>df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,
                  columns=['A', 'B', 'C', 'D'])
df = df.cumsum()
import matplotlib.pyplot as plt
plt.figure()
df.plot()
plt.legend(loc='best')<jupyter_output><empty_output><jupyter_text>数据输入 / 输出 [CSV](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.htmlio-store-in-csv)<jupyter_code># 写入 CSV 文件
df.to_csv('foo.csv')

# 读取 CSV 文件数据
pd.read_csv('foo.csv')<jupyter_output><empty_output><jupyter_text>[HDF5](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.htmlio-hdf5)写入 HDF5 Store：<jupyter_code>df.to_hdf('foo.h5', 'df')<jupyter_output><empty_output><jupyter_text>读取 HDF5 Store：<jupyter_code>pd.read_hdf('foo.h5', 'df')<jupyter_output><empty_output><jupyter_text>[Excel](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.htmlio-excel)写入 Excel 文件：<jupyter_code>df.to_excel('foo.xlsx', sheet_name='Sheet1')<jupyter_output><empty_output><jupyter_text>读取 Excel 文件：<jupyter_code>pd.read_excel('foo.xlsx', 'Sheet1', index_col=None, na_values=['NA'])<jupyter_output><empty_output><jupyter_text>各种坑（Gotchas）执行某些操作，将触发异常，如：<jupyter_code>>>> if pd.Series([False, True, False]):
...    print("I was true")

Traceback
    ...
ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.19778103089606286, 'fraction_numerical': 0.02203559596270899, 'mean_word_length': 6.320924985899605, 'pattern_counts': {'":': 2, '<': 173, '<?xml version=': 0, '>': 180, 'https://': 26, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '43', 'dir': 'jupyter-structured-clean-dedup', 'id': '224456', 'license': 'MIT', 'n_tokens_mistral': 6615, 'n_tokens_neox': 6760, 'n_words': 640, 'path': 'Datatype/pandas_basic.ipynb', 'repo_name': 'EvaGalois/Deep_Learning', 'size': '12979'}