<commit_before>fastavro
========

The Python [`avro`][avro_pypi] package distributed by 
[Apache][apache_avro] is **incredibly slow**. When dealing with very 
large datasets, the performance of [`avro`][avro_pypi] is unacceptable,
more or less unfit for any production use.

**`fastavro`** is less feature complete than [`avro`][avro_pypi], 
however performance reading and writing Avro files is *significantly*
faster.

If you compile the optional [Cython][cython] C-extension modules, then 
**`fastavro`** is even faster, almost on par with the native Java
implementation of Avro.

The focus of this fork ([e-heller/fastavro]) is to improve the 
Cython implementation of `fastavro` to achieve performance that is
even faster. So far I've managed some success.

I hope these improvements will eventually be pulled into the upstream 
repo, but I'm not very optimistic that will happen anytime soon.

This **`fastavro`** fork supports these Python versions:

* Python 2.6
* Python 2.7
* Python 3.4
* Python 3.5


[e-heller/fastavro]: https://github.com/e-heller/fastavro
[avro_pypi]: https://pypi.python.org/pypi/avro
[apache_avro]: http://avro.apache.org
[Cython]: http://cython.org


Basic Usage
===========

Reading from Avro format
------------------------

```python
import fastavro

with open('some_file.avro', 'rb') as fo:

    # Create a `reader` object to iterate over the records:
    reader = fastavro.Reader(fo)

    # Obtain the file's Schema if you need it:
    schema = reader.schema

    # Iteratively read the records:
    for record in reader:
        process_record(record)
        
    # Read the records in one shot:
    records = list(reader)
```


Writing to Avro format
----------------------

```python
import fastavro

# Define an Avro Schema as a dict:
schema = {
    'doc': 'A weather reading.',
    'name': 'Weather',
    'namespace': 'test',<commit_msg>Fix code example in README<commit_after>fastavro
========

The Python [`avro`][avro_pypi] package distributed by 
[Apache][apache_avro] is **incredibly slow**. When dealing with very 
large datasets, the performance of [`avro`][avro_pypi] is unacceptable,
more or less unfit for any production use.

**`fastavro`** is less feature complete than [`avro`][avro_pypi], 
however performance reading and writing Avro files is *significantly*
faster.

If you compile the optional [Cython][cython] C-extension modules, then 
**`fastavro`** is even faster, almost on par with the native Java
implementation of Avro.

The focus of this fork ([e-heller/fastavro]) is to improve the 
Cython implementation of `fastavro` to achieve performance that is
even faster. So far I've managed some success.

I hope these improvements will eventually be pulled into the upstream 
repo, but I'm not very optimistic that will happen anytime soon.

This **`fastavro`** fork supports these Python versions:

* Python 2.6
* Python 2.7
* Python 3.4
* Python 3.5


[e-heller/fastavro]: https://github.com/e-heller/fastavro
[avro_pypi]: https://pypi.python.org/pypi/avro
[apache_avro]: http://avro.apache.org
[Cython]: http://cython.org


Basic Usage
===========

Reading from Avro format
------------------------

```python
import fastavro

with open('some_file.avro', 'rb') as input:
    # Create a `Reader` object
    reader = fastavro.Reader(input)

    # Obtain the writer's schema if required
    schema = reader.schema

    # Iteratively read the records:
    for record in reader:
        process_record(record)
        
    # Read the records in one shot:
    records = list(reader)
```


Writing to Avro format
----------------------

```python
import fastavro

# Define an Avro Schema as a dict:
schema = {
    'doc': 'A weather reading.',
    'name': 'Weather',
    'namespace': 'test',
{'fraction_non_alphanumeric': 0.13383097840575847, 'fraction_numerical': 0.004265529192215409, 'mean_word_length': 3.603680981595092, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2573245', 'n_tokens_mistral': 1203, 'n_tokens_neox': 1128, 'n_words': 432}