<commit_before># Frameless

[![Travis Badge](https://travis-ci.org/typelevel/frameless.svg?branch=master)](https://travis-ci.org/typelevel/frameless)
[![Codecov Badge](https://codecov.io/gh/typelevel/frameless/branch/master/graph/badge.svg)](https://codecov.io/gh/typelevel/frameless)
[![Maven Badge](https://img.shields.io/maven-central/v/org.typelevel/frameless-dataset_2.11.svg)](https://maven-badges.herokuapp.com/maven-central/org.typelevel/frameless-dataset_2.11)
[![Gitter Badge](https://badges.gitter.im/typelevel/frameless.svg)](https://gitter.im/typelevel/frameless)

Frameless is a proof-of-concept library for working with [Spark](http://spark.apache.org/) using more expressive types.
It consists of the following modules:

* `dataset` for more strongly typed `Dataset`s  (supports Spark 2.0.x)
* `cats` for using Spark with [cats](https://github.com/typelevel/cats) (supports Cats 0.9.x)


The Frameless project and contributors support the
[Typelevel](http://typelevel.org/) [Code of Conduct](http://typelevel.org/conduct.html) and want all its
associated channels (e.g. GitHub, Gitter) to be a safe and friendly environment for contributing and learning.

## Documentation

* [TypedDataset: Feature Overview](http://typelevel.github.io/frameless/docs/book/FeatureOverview.html)
* [Comparing TypedDatasets with Spark's Datasets](http://typelevel.github.io/frameless/docs/book/TypedDatasetVsSparkDataset.html)
* [Typed Encoders in Frameless](http://typelevel.github.io/frameless/docs/book/TypedEncoder.html)
* [Injection: Creating Custom Encoders](http://typelevel.github.io/frameless/docs/book/Injection.html)
* [Using Cats with RDDs](http://typelevel.github.io/frameless/docs/book/Cats.html)
* [Proof of Concept: TypedDataFrame](http://typelevel.github.io/frameless/docs/book/TypedDataFrame.html)

## Why?

Benefits of using `TypedDataset` compared to the standard Spark `Dataset` API:

* Typesafe columns referencing and expressions
* Customizable, typesafe encoders
* Typesafe casting and projections
* Enhanced type signature for some built-in functions

## Quick Start
Frameless is compiled against Scala 2.11.x.

Note that while Frameless is still getting off the ground, it is very possible that breaking changes will be
made for at least the next few versions.

To use Frameless in your project add the following in your `build.sbt` file as needed:
<commit_msg>Add Job to the README index<commit_after># Frameless

[![Travis Badge](https://travis-ci.org/typelevel/frameless.svg?branch=master)](https://travis-ci.org/typelevel/frameless)
[![Codecov Badge](https://codecov.io/gh/typelevel/frameless/branch/master/graph/badge.svg)](https://codecov.io/gh/typelevel/frameless)
[![Maven Badge](https://img.shields.io/maven-central/v/org.typelevel/frameless-dataset_2.11.svg)](https://maven-badges.herokuapp.com/maven-central/org.typelevel/frameless-dataset_2.11)
[![Gitter Badge](https://badges.gitter.im/typelevel/frameless.svg)](https://gitter.im/typelevel/frameless)

Frameless is a proof-of-concept library for working with [Spark](http://spark.apache.org/) using more expressive types.
It consists of the following modules:

* `dataset` for more strongly typed `Dataset`s  (supports Spark 2.0.x)
* `cats` for using Spark with [cats](https://github.com/typelevel/cats) (supports Cats 0.9.x)


The Frameless project and contributors support the
[Typelevel](http://typelevel.org/) [Code of Conduct](http://typelevel.org/conduct.html) and want all its
associated channels (e.g. GitHub, Gitter) to be a safe and friendly environment for contributing and learning.

## Documentation

* [TypedDataset: Feature Overview](http://typelevel.github.io/frameless/docs/book/FeatureOverview.html)
* [Comparing TypedDatasets with Spark's Datasets](http://typelevel.github.io/frameless/docs/book/TypedDatasetVsSparkDataset.html)
* [Typed Encoders in Frameless](http://typelevel.github.io/frameless/docs/book/TypedEncoder.html)
* [Injection: Creating Custom Encoders](http://typelevel.github.io/frameless/docs/book/Injection.html)
* [Job\[A\]](http://typelevel.github.io/frameless/docs/book/Job.htmlé)
* [Using Cats with RDDs](http://typelevel.github.io/frameless/docs/book/Cats.html)
* [Proof of Concept: TypedDataFrame](http://typelevel.github.io/frameless/docs/book/TypedDataFrame.html)

## Why?

Benefits of using `TypedDataset` compared to the standard Spark `Dataset` API:

* Typesafe columns referencing and expressions
* Customizable, typesafe encoders
* Typesafe casting and projections
* Enhanced type signature for some built-in functions

## Quick Start
Frameless is compiled against Scala 2.11.x.

Note that while Frameless is still getting off the ground, it is very possible that breaking changes will be
made for at least the next few versions.

To use Frameless in your project add the following in your `build.sbt` file as needed:

{'fraction_non_alphanumeric': 0.12878631774160312, 'fraction_numerical': 0.00535751081805069, 'mean_word_length': 5.41215323645971, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 18, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1132766', 'n_tokens_mistral': 1619, 'n_tokens_neox': 1567, 'n_words': 374}