<commit_before>Scio
====

[![Build Status](https://travis-ci.org/spotify/scio.svg?branch=master)](https://travis-ci.org/spotify/scio)
[![codecov.io](https://codecov.io/github/spotify/scio/coverage.svg?branch=master)](https://codecov.io/github/spotify/scio?branch=master)
[![GitHub license](https://img.shields.io/github/license/spotify/scio.svg)]()
[![Maven Central](https://img.shields.io/maven-central/v/com.spotify/scio-core_2.11.svg)](https://maven-badges.herokuapp.com/maven-central/com.spotify/scio-core_2.11)

> Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]

> Verb: I can, know, understand, have knowledge.

Scio is a Scala API for [Google Cloud Dataflow](https://github.com/GoogleCloudPlatform/DataflowJavaSDK) inspired by [Spark](http://spark.apache.org/) and [Scalding](https://github.com/twitter/scalding). See the [current API documentation](http://spotify.github.io/scio/) for more information.

# Features

- Scala API close to that of Spark and Scalding core APIs
- Fully managed service<sup>*</sup>
- Unified batch and streaming programming model<sup>*</sup>
- Integration with Google Cloud products: Cloud Storage, BigQuery, Pub/Sub, Datastore, Bigtable<sup>*</sup>
- HDFS source/sink
- Interactive mode with Scio REPL
- Type safe BigQuery
- Integration with [Algebird](https://github.com/twitter/algebird) and [Breeze](https://github.com/scalanlp/breeze)
- Pipeline orchestration with [Scala Futures](http://docs.scala-lang.org/overviews/core/futures.html)
- Distributed cache

<sup>*</sup> provided by Google Cloud Dataflow

# Quick Start

The ubiquitous word count example can be run directly with SBT in local mode, using `README.md` as input.

```bash
sbt "project scio-examples" "run-main com.spotify.scio.examples.WordCount --input=README.md --output=wc"
cat wc/part-00000-of-00001.txt
```

# Documentation

- [Scio Wiki](https://github.com/spotify/scio/wiki) - wiki page
- [ScalaDocs](http://spotify.github.com/scio) - current API documentation
- [Scio REPL](https://github.com/spotify/scio/wiki/Scio-REPL) - tutorial for the interactive Scio REPL
- [Scio, Spark and Scalding](https://github.com/spotify/scio/wiki/Scio,-Scalding-and-Spark) - comparison of these frameworks
- [Type safe BigQuery](https://github.com/spotify/scio/wiki/Type-Safe-BigQuery) - tutorial for the type safe BigQuery API

# Artifacts

Scio includes the following artifacts:
<commit_msg>add HDFS link to README.md<commit_after>Scio
====

[![Build Status](https://travis-ci.org/spotify/scio.svg?branch=master)](https://travis-ci.org/spotify/scio)
[![codecov.io](https://codecov.io/github/spotify/scio/coverage.svg?branch=master)](https://codecov.io/github/spotify/scio?branch=master)
[![GitHub license](https://img.shields.io/github/license/spotify/scio.svg)]()
[![Maven Central](https://img.shields.io/maven-central/v/com.spotify/scio-core_2.11.svg)](https://maven-badges.herokuapp.com/maven-central/com.spotify/scio-core_2.11)

> Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]

> Verb: I can, know, understand, have knowledge.

Scio is a Scala API for [Google Cloud Dataflow](https://github.com/GoogleCloudPlatform/DataflowJavaSDK) inspired by [Spark](http://spark.apache.org/) and [Scalding](https://github.com/twitter/scalding). See the [current API documentation](http://spotify.github.io/scio/) for more information.

# Features

- Scala API close to that of Spark and Scalding core APIs
- Fully managed service<sup>*</sup>
- Unified batch and streaming programming model<sup>*</sup>
- Integration with Google Cloud products: Cloud Storage, BigQuery, Pub/Sub, Datastore, Bigtable<sup>*</sup>
- HDFS source/sink
- Interactive mode with Scio REPL
- Type safe BigQuery
- Integration with [Algebird](https://github.com/twitter/algebird) and [Breeze](https://github.com/scalanlp/breeze)
- Pipeline orchestration with [Scala Futures](http://docs.scala-lang.org/overviews/core/futures.html)
- Distributed cache

<sup>*</sup> provided by Google Cloud Dataflow

# Quick Start

The ubiquitous word count example can be run directly with SBT in local mode, using `README.md` as input.

```bash
sbt "project scio-examples" "run-main com.spotify.scio.examples.WordCount --input=README.md --output=wc"
cat wc/part-00000-of-00001.txt
```

# Documentation

- [Scio Wiki](https://github.com/spotify/scio/wiki) - wiki page
- [ScalaDocs](http://spotify.github.com/scio) - current API documentation
- [Scio REPL](https://github.com/spotify/scio/wiki/Scio-REPL) - tutorial for the interactive Scio REPL
- [Scio, Spark and Scalding](https://github.com/spotify/scio/wiki/Scio,-Scalding-and-Spark) - comparison of these frameworks
- [Type safe BigQuery](https://github.com/spotify/scio/wiki/Type-Safe-BigQuery) - tutorial for the type safe BigQuery API
- [HDFS](https://github.com/spotify/scio/wiki/HDFS) - using Scio with HDFS files

# Artifacts

Scio includes the following artifacts:

{'fraction_non_alphanumeric': 0.1541769041769042, 'fraction_numerical': 0.006552006552006552, 'mean_word_length': 4.822407628128724, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 23, 'https://': 31, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7402453', 'n_tokens_mistral': 1762, 'n_tokens_neox': 1679, 'n_words': 358}