<commit_before># ScalaFunctional
[![TravisCI](https://travis-ci.org/EntilZha/ScalaFunctional.svg?branch=master)](https://travis-ci.org/EntilZha/ScalaFunctional)
[![Coverage by codecov.io](https://codecov.io/github/EntilZha/ScalaFunctional/coverage.svg?branch=master)](https://codecov.io/github/EntilZha/ScalaFunctional?branch=master)
[![ReadTheDocs](https://readthedocs.org/projects/scalafunctional/badge/?version=latest)](http://scalafunctional.readthedocs.org/en/)
[![Latest Version](https://badge.fury.io/py/scalafunctional.svg)](https://pypi.python.org/pypi/scalafunctional/)
[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/EntilZha/ScalaFunctional?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)

## Introduction
`ScalaFunctional` is a Python package that makes working with data easy. It takes inspiration from
several sources that include Scala collections, Apache Spark RDDs, Microsoft LINQ and more generally
functional programming. It also offers native reading and writing of data formats such as text, csv,
and json files. Support for SQLite3, other databases, and compressed files is planned for the next
release.

The combination of these ideas makes `ScalaFunctional` a great choice
for declarative transformation, creating pipelines, and data analysis.

[Original blog post for ScalaFunctional](http://entilzha.github.io/blog/2015/03/14/functional-programming-collections-python/)

## Installation
`ScalaFunctional` is available on [pypi](https://pypi.python.org/pypi/ScalaFunctional) and can be
installed by running:
```bash
# Install from command line
$ pip install scalafunctional
```

Then in python run: `from functional import seq`

## Examples
`ScalaFunctional` is useful for many tasks, and can natively open several common file types. Here
are a few examples of what you can do.

### Simple Example
```python
from functional import seq

seq(1, 2, 3, 4)\
    .map(lambda x: x * 2)\
    .filter(lambda x: x > 4)\
    .reduce(lambda x, y: x + y)
# 14
```

### Streams, Transformations and Actions
`ScalaFunctional` has three types of functions:

1. Streams: read data for use by the collections API.
2. Transformations: transform data from streams with functions such as `map`, `flat_map`, and
`filter`
3. Actions: These cause a series of transformations to evaluate to a concrete value. `to_list`,
`reduce`, and `to_dict` are examples of actions.

In the expression `seq(1, 2, 3).map(lambda x: x * 2).reduce(lambda x, y: x + y)`, `seq` is the
stream, `map` is the transformation, and `reduce` is the action.

### Filtering a list of account transactions
```python
from functional import seq
from collections import namedtuple

Transaction = namedtuple('Transaction', 'reason amount')
transactions = [
    Transaction('github', 7),
    Transaction('food', 10),
    Transaction('coffee', 5),
    Transaction('digitalocean', 5),
    Transaction('food', 5),
    Transaction('riotgames', 25),
    Transaction('food', 10),
    Transaction('amazon', 200),
    Transaction('paycheck', -1000)
]

# Using the Scala/Spark inspired APIs
food_cost = seq(transactions)\
    .filter(lambda x: x.reason == 'food')\
    .map(lambda x: x.amount).sum()

# Using the LINQ inspired APIs
food_cost = seq(transactions)\
    .where(lambda x: x.reason == 'food')\
    .select(lambda x: x.amount).sum()

# Using ScalaFunctional with fn
from fn import _
food_cost = seq(transactions).filter(_.reason == 'food').map(_.amount).sum()
```

### Word Count and Joins
The account transactions example could be done easily in pure python using list comprehensions. To
show some of the things `ScalaFunctional` excels at, take a look at a couple of word count examples.

```python
words = 'I dont want to believe I want to know'.split(' ')
seq(words).map(lambda word: (word, 1)).reduce_by_key(lambda x, y: x + y)
# [('dont', 1), ('I', 2), ('to', 2), ('know', 1), ('want', 2), ('believe', 1)]
```

In the next example we have chat logs formatted in jsonl which contain messages and metadata. A
typical json file will have one valid json on each line of a file. Below are a few lines out of
`examples/chat_logs.jsonl`.

```json
{"message":"hello anyone there?","date":"10/09","user":"bob"}
{"message":"need some help with a program","date":"10/09","user":"bob"}
{"message":"sure thing. What do you need help with?","date":"10/09","user":"dave"}
```

```python
from operator import add
import re
messages = seq.jsonl('examples/chat_lots.jsonl')

# Split words on space and normalize before doing word count
def extract_words(message):
    return re.sub('[^0-9a-z ]+', '', message.lower()).split(' ')


word_counts = messages\
    .map(lambda log: extract_words(log['message']))\
    .flatten().map(lambda word: (word, 1))\
    .reduce_by_key(add).order_by(lambda x: x[1])

```

Next, lets continue that example but introduce a json database of users from `examples/users.json`.
In the previous example we showed how `ScalaFunctional` can do word counts, in the next example lets
show how `ScalaFunctional` can join different data sources.

```python
# First read the json file<commit_msg>Add jsonl link, fix typo, and make clearer<commit_after># ScalaFunctional
[![TravisCI](https://travis-ci.org/EntilZha/ScalaFunctional.svg?branch=master)](https://travis-ci.org/EntilZha/ScalaFunctional)
[![Coverage by codecov.io](https://codecov.io/github/EntilZha/ScalaFunctional/coverage.svg?branch=master)](https://codecov.io/github/EntilZha/ScalaFunctional?branch=master)
[![ReadTheDocs](https://readthedocs.org/projects/scalafunctional/badge/?version=latest)](http://scalafunctional.readthedocs.org/en/)
[![Latest Version](https://badge.fury.io/py/scalafunctional.svg)](https://pypi.python.org/pypi/scalafunctional/)
[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/EntilZha/ScalaFunctional?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)

## Introduction
`ScalaFunctional` is a Python package that makes working with data easy. It takes inspiration from
several sources that include Scala collections, Apache Spark RDDs, Microsoft LINQ and more generally
functional programming. It also offers native reading and writing of data formats such as text, csv,
and json files. Support for SQLite3, other databases, and compressed files is planned for the next
release.

The combination of these ideas makes `ScalaFunctional` a great choice
for declarative transformation, creating pipelines, and data analysis.

[Original blog post for ScalaFunctional](http://entilzha.github.io/blog/2015/03/14/functional-programming-collections-python/)

## Installation
`ScalaFunctional` is available on [pypi](https://pypi.python.org/pypi/ScalaFunctional) and can be
installed by running:
```bash
# Install from command line
$ pip install scalafunctional
```

Then in python run: `from functional import seq`

## Examples
`ScalaFunctional` is useful for many tasks, and can natively open several common file types. Here
are a few examples of what you can do.

### Simple Example
```python
from functional import seq

seq(1, 2, 3, 4)\
    .map(lambda x: x * 2)\
    .filter(lambda x: x > 4)\
    .reduce(lambda x, y: x + y)
# 14
```

### Streams, Transformations and Actions
`ScalaFunctional` has three types of functions:

1. Streams: read data for use by the collections API.
2. Transformations: transform data from streams with functions such as `map`, `flat_map`, and
`filter`
3. Actions: These cause a series of transformations to evaluate to a concrete value. `to_list`,
`reduce`, and `to_dict` are examples of actions.

In the expression `seq(1, 2, 3).map(lambda x: x * 2).reduce(lambda x, y: x + y)`, `seq` is the
stream, `map` is the transformation, and `reduce` is the action.

### Filtering a list of account transactions
```python
from functional import seq
from collections import namedtuple

Transaction = namedtuple('Transaction', 'reason amount')
transactions = [
    Transaction('github', 7),
    Transaction('food', 10),
    Transaction('coffee', 5),
    Transaction('digitalocean', 5),
    Transaction('food', 5),
    Transaction('riotgames', 25),
    Transaction('food', 10),
    Transaction('amazon', 200),
    Transaction('paycheck', -1000)
]

# Using the Scala/Spark inspired APIs
food_cost = seq(transactions)\
    .filter(lambda x: x.reason == 'food')\
    .map(lambda x: x.amount).sum()

# Using the LINQ inspired APIs
food_cost = seq(transactions)\
    .where(lambda x: x.reason == 'food')\
    .select(lambda x: x.amount).sum()

# Using ScalaFunctional with fn
from fn import _
food_cost = seq(transactions).filter(_.reason == 'food').map(_.amount).sum()
```

### Word Count and Joins
The account transactions example could be done easily in pure python using list comprehensions. To
show some of the things `ScalaFunctional` excels at, take a look at a couple of word count examples.

```python
words = 'I dont want to believe I want to know'.split(' ')
seq(words).map(lambda word: (word, 1)).reduce_by_key(lambda x, y: x + y)
# [('dont', 1), ('I', 2), ('to', 2), ('know', 1), ('want', 2), ('believe', 1)]
```

In the next example we have chat logs formatted in [json lines (jsonl)](http://jsonlines.org/) which
contain messages and metadata. A typical jsonl file will have one valid json on each line of a file.
Below are a few lines out of `examples/chat_logs.jsonl`.

```json
{"message":"hello anyone there?","date":"10/09","user":"bob"}
{"message":"need some help with a program","date":"10/09","user":"bob"}
{"message":"sure thing. What do you need help with?","date":"10/09","user":"dave"}
```

```python
from operator import add
import re
messages = seq.jsonl('examples/chat_lots.jsonl')

# Split words on space and normalize before doing word count
def extract_words(message):
    return re.sub('[^0-9a-z ]+', '', message.lower()).split(' ')


word_counts = messages\
    .map(lambda log: extract_words(log['message']))\
    .flatten().map(lambda word: (word, 1))\
    .reduce_by_key(add).order_by(lambda x: x[1])

```

Next, lets continue that example but introduce a json database of users from `examples/users.json`.
In the previous example we showed how `ScalaFunctional` can do word counts, in the next example lets
show how `ScalaFunctional` can join different data sources.

```python
# First read the json file
{'fraction_non_alphanumeric': 0.13475384316014788, 'fraction_numerical': 0.01284296555750146, 'mean_word_length': 4.976162790697674, 'pattern_counts': {'":': 18, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 20, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3557755', 'n_tokens_mistral': 3326, 'n_tokens_neox': 3092, 'n_words': 1084}