<jupyter_start><jupyter_text>In-Memory Reading and WritingTo assist with serialization and deserialization of in-memory data, we have file interfaces that can read and write to Arrow Buffers.These have similar semantics to Python’s built-in io.BytesIO.<jupyter_code>writer = pa.BufferOutputStream()
writer.write(b'hello, friends')
# 14

buf = writer.getvalue()
print(buf)
# <pyarrow.lib.Buffer at 0x2b9df4d9d180>
print(buf.size)
# 14

reader = pa.BufferReader(buf)
reader.seek(7)
reader.read(7)
# b'friends'<jupyter_output><pyarrow.lib.Buffer object at 0x114442688>
14<jupyter_text>Data Types and In-Memory Data ModelApache Arrow defines columnar array data structures by composing type metadata with memory buffers, like the ones explained in the documentation on Memory and IO. These data structures are exposed in Python through a series of interrelated classes:* Type Metadata: Instances of pyarrow.DataType, which describe a logical array type* Schemas: Instances of pyarrow.Schema, which describe a named collection of types. These can be thought of as the column types in a table-like object.* Arrays: Instances of pyarrow.Array, which are atomic, contiguous columnar data structures composed from Arrow Buffer objects* Record Batches: Instances of pyarrow.RecordBatch, which are a collection of Array objects with a particular Schema* Tables: Instances of pyarrow.Table, a logical table data structure in which each column consists of one or more pyarrow.Array objects of the same type.<jupyter_code>t1 = pa.int32()
print(t1)
f0 = pa.field('int32_field', t1)
print(f0.name)
print(f0.type)
t6 = pa.list_(t1)
print(t6)
t1 = pa.int32()
t2 = pa.string()
t3 = pa.binary()
t4 = pa.binary(10)
t5 = pa.timestamp('ms')

fields = [
    pa.field('s0', t1),
    pa.field('s1', t2),
    pa.field('s2', t4),
    pa.field('s3', t6),
]

t7 = pa.struct(fields)
print(t7)
my_schema = pa.schema([('field0', t1),
                       ('field1', t2),
                       ('field2', t4),
                       ('field3', t6)])

my_schema
data = [
    pa.array([1, 2, 3, 4]),
    pa.array(['foo', 'bar', 'baz', None]),
    pa.array([True, None, False, True])
]
batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2'])
print(batch.num_columns)
print(batch.num_rows)
print(batch.schema)
print(batch[1])
batch2 = batch.slice(1, 3)
print(batch2[1])<jupyter_output>3
4
f0: int64
f1: string
f2: bool
[
  "foo",
  "bar",
  "baz",
  null
]
[
  "bar",
  "baz",
  null
]<jupyter_text>表PyArrow Table类型不是Apache Arrow规范的一部分，而是一个帮助将多个记录批次和数组片段作为单个逻辑数据集进行争论的工具。作为一个相关的例子，我们可能在套接字流中接收多个小记录批次，然后需要将它们连接成连续的内存以便在NumPy或pandas中使用。Table对象使这种方法更有效，而无需额外的内存复制。考虑到我们上面创建的记录批次，我们可以使用以下方法创建一个包含批次的一个或多个副本的表Table.from_batches：<jupyter_code>batches = [batch] * 5
table = pa.Table.from_batches(batches)
print(table)
print(table.num_rows)
print(table.num_columns)
# 表的列是实例Column，它是一个或多个相同类型的数组的容器。
c = table[0]
c
print(c.data)
print(c.data.num_chunks)
print(c.data.chunk(0))
# 可以将这些对象转换为连续的NumPy数组，以便在pandas中使用：
c.to_pandas()
# pyarrow.concat_tables如果模式相同，也可以将多个表连接在一起以形成单个表 ：
tables = [table] * 2
table_all = pa.concat_tables(tables)
table_all.num_rows
c = table_all[0]
c.data.num_chunks
df_new = table_all.to_pandas()
df_new[:5]<jupyter_output><empty_output><jupyter_text>这类似于Table.from_batches，但使用表作为输入而不是记录批次。记录批次可以制作成表格，但不是相反，所以如果您的数据已经是表格形式，那么请使用 pyarrow.concat_tables。  Arrow定义了两种用于序列化记录批次的二进制格式：- 流格式：用于发送任意长度的记录批次序列。必须从头到尾处理格式，并且不支持随机访问- 文件或随机访问格式：用于序列化固定数量的记录批次。支持随机访问，因此在与内存映射一起使用时非常有用<jupyter_code>import pyarrow as pa

data = [
    pa.array([1, 2, 3, 4]),
    pa.array(['foo', 'bar', 'baz', None]),
    pa.array([True, None, False, True])
]

batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2'])
batch.num_rows
# 现在，我们可以开始编写包含一些批次的流。为此我们使用RecordBatchStreamWriter，
# 它可以写入可写 NativeFile对象或可写Python对象：
# 这里我们使用了内存中的Arrow缓冲流，但这可能是一个套接字或其他一些IO接收器。
sink = pa.BufferOutputStream()
writer = pa.RecordBatchStreamWriter(sink, batch.schema)

for i in range(5):
    writer.write_batch(batch)

writer.close()
buf = sink.getvalue()
print(buf.size)
print(type(buf))
reader = pa.ipc.open_stream(buf)
print(reader.schema)

batches = [b for b in reader]
len(batches)
# 检查返回的批次是否与原始输入相同：
batches[0].equals(batch)
# 在RecordBatchFileWriter具有相同的API RecordBatchStreamWriter：
sink = pa.BufferOutputStream()
writer = pa.RecordBatchFileWriter(sink, batch.schema)

for i in range(10):
    writer.write_batch(batch)

writer.close()

buf = sink.getvalue()
buf.size<jupyter_output><empty_output><jupyter_text>RecordBatchFileReader和 之间的区别在于RecordBatchStreamReader输入源必须具有seek随机访问的 方法。流读取器仅需要读取操作。我们也可以使用该pyarrow.ipc.open_file方法打开一个文件：<jupyter_code>reader = pa.ipc.open_file(buf)
reader.num_record_batches
# 10
b = reader.get_batch(3)
# 流和文件阅读器类有一个特殊的read_pandas方法来简化读取多个记录批次并将它们转换为单个DataFrame输出
df = pa.ipc.open_file(buf).read_pandas()
df[:5]<jupyter_output><empty_output><jupyter_text>任意对象序列化在pyarrow我们能够序列化和反序列化多种Python对象。虽然不是pickle模块的完全替代品，但这些功能可以明显更快，特别是在处理NumPy阵列的集合时。<jupyter_code>import numpy as np

data = {
    i: np.random.randn(500, 500)
    for i in range(100)
}
buf = pa.serialize(data).to_buffer()
print(type(buf))
print(buf.size)<jupyter_output><class 'pyarrow.lib.Buffer'>
200028864<jupyter_text>pyarrow.serialize创建一个中间对象，可以将其转换为缓冲区（to_buffer方法）或直接写入输出流。pyarrow.deserialize 将类缓冲区对象转换回原始Python对象：处理NumPy数组时，pyarrow.deserialize可能会明显快于pickle因为生成的数组是对输入缓冲区的零拷贝引用。阵列越大，性能节省越多。<jupyter_code>restored_data = pa.deserialize(buf)
restored_data[0]
%timeit restored_data = pa.deserialize(buf)
import pickle
pickled = pickle.dumps(data)
%timeit unpickled_data = pickle.loads(pickled)
import pyarrow as pa

# 考虑一个有两个成员的类，其中一个是NumPy数组：
class MyData:
    def __init__(self, name, data):
        self.name = name
        self.data = data

# 我们编写函数来将它转换为具有更简单类型的字典：
def _serialize_MyData(val):
    return {'name': val.name, 'data': val.data}

def _deserialize_MyData(data):
    return MyData(data['name'], data['data'])

# 那么，我们必须在注册这些功能SerializationContext，这样 MyData可以确认：
context=pa.SerializationContext()
context.register_type(MyData, 'MyData',
                      custom_serializer=_serialize_MyData,
                      custom_deserializer=_deserialize_MyData)

val=MyData("hello", "world")
# 最后，我们使用此上下文作为附加参数pyarrow.serialize：
buf = pa.serialize(val, context=context).to_buffer()
restored_val = pa.deserialize(buf, context=context)
# 该SerializationContext还具有方便的方法serialize和 deserialize，所以这些都是等效的声明：
buf = context.serialize(val).to_buffer()
restored_val = context.deserialize(buf)
print(restored_val, restored_val.data)<jupyter_output><__main__.MyData object at 0x10f9f47b8> world<jupyter_text>基于组件的序列化对于序列化包含一定数量的NumPy数组，箭头缓冲区或其他数据类型的Python对象，可能需要传输其序列化表示而不必使用该to_buffer方法生成中间副本 。为了激发这一点，假设我们有一个NumPy数组列表：..该调用pa.serialize(data)不会复制每个NumPy数组中的内存。然后，可以将此序列化表示分解为包含一系列pyarrow.Buffer对象的字典，该对象包含每个数组的元数据以及对数组内部内存的引用。为此，请使用以下to_components方法：<jupyter_code>import numpy as np
data = [np.random.randn(10, 10) for i in range(5)]

serialized = pa.serialize(data)
components = serialized.to_components()
memoryview(components['data'][0])
# memoryview可以转换回箭Buffer与 pyarrow.py_buffer：
mv = memoryview(components['data'][0])
buf = pa.py_buffer(mv)

# 使用以下方法从基于组件的表示重建对象 deserialize_components：
restored_data = pa.deserialize_components(components)
restored_data[0]<jupyter_output><empty_output><jupyter_text>序列化pandas对象默认序列化方面进行了优化像熊猫的对象处理DataFrame和Series。结合上面的基于组件的序列化，这可以实现不包含任何Python对象的pandas DataFrame对象的零拷贝传输：<jupyter_code>import pandas as pd

df = pd.DataFrame({'a': [1, 2, 3, 4, 5]})
context = pa.default_serialization_context()

serialized_df = context.serialize(df)
df_components = serialized_df.to_components()
original_df = context.deserialize_components(df_components)
original_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11616688396349413, 'fraction_numerical': 0.019035202086049544, 'mean_word_length': 6.678678678678678, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '621772', 'license': 'Apache-2.0', 'n_tokens_mistral': 3502, 'n_tokens_neox': 3605, 'n_words': 516, 'path': 'notebook/procs-pyarrow.ipynb', 'repo_name': 'samlet/stack', 'size': '7670'}