<commit_before># Succinct-Spark

[Spark](http://spark.apache.org/) and 
[Spark SQL](http://spark.apache.org/docs/latest/sql-programming-guide.html) 
interfaces for [Succinct](http://succinct.cs.berkeley.edu/). 
This library facilitates compressing RDDs in Spark and DataFrames in Spark SQL
and enables queries directly on the compressed representation.

## Requirements

This library requires Spark 1.4+.

## Dependency Information

### Apache Maven

To build your application with Succinct-Spark, you can link against this library
using Maven by adding the following dependency information to your pom.xml file:

```xml
<dependency>
    <groupId>amplab</groupId>
    <artifactId>succinct-spark</artifactId>
    <version>0.1.3</version>
</dependency>
```

### SBT

Add the dependency to your SBT project by adding the following to `build.sbt` 
(see the [Spark Packages listing](http://spark-packages.org/package/amplab/succinct)
for spark-submit and Maven instructions):

```
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"
libraryDependencies += "amplab" % "succinct" % "0.1.3"
```

The succinct-spark jar file can also be added to a Spark shell using the 
`--jars` command line option. For example, to include it when starting the 
spark shell:

```
$ bin/spark-shell --jars succinct-0.1.3.jar
```

## Usage

The Succinct-Spark library exposes three APIs: 
* A SuccinctRDD API that views an RDD as an unstructured "flat-file" and enables queries on its compressed representation.
* A SuccinctKVRDD API that provides a key-value abstraction for the data, and supports search and random-access over the _values_.
* DataFrame API that integrates with the Spark SQL interface via Data Sources, and supports SQL queries on compressed structured data.

**Note: The Spark SQL interface is experimental, and only efficient for selected
SQL operators. We aim to make the Spark SQL integration more efficient in
future releases.**

### SuccinctRDD API

We expose a `SuccinctRDD` that extends `RDD[Array[Byte]]`. Since each record is
represented as an array of bytes, `SuccinctRDD` can be used to encode a 
collection of any type of records by providing a serializer/deserializer for
the record type. 

`SuccinctRDD` can be used as follows:

```scala
import edu.berkeley.cs.succinct.SuccinctRDD

// Read text data from file; sc is the SparkContext
val textRDD = sc.textFile("README.md")

// Convert the textRDD to a SuccinctRDD after serializing each record into an
// array of bytes. Persist the RDD in memory to perform in-memory queries.
val succinctTextRDD = SuccinctRDD(textRDD.map(_.getBytes)).cache

// Count the number of records containing "Succinct" in the data
val succinctCount = succinctTextRDD.count("Succinct")

// Fetch all records that contain the string "Succinct"
val succinctRecords = succinctTextRDD.search("Succinct").collect
```

More examples of the SuccinctRDD API can be found [here](src/main/scala/edu/berkeley/cs/succinct/examples/WikiSearch.scala).

#### Input Constraints

We don't support non-ASCII characters in the input for now, since the
algorithms depend on using certain non-ASCII characters as internal symbols.

#### Construction Time

Another constraint to consider is the construction time for Succinct
data-structures. As for any block compression scheme, Succinct requires
non-trivial amount of time to compress an input dataset. It is strongly
advised that the SuccinctRDD be cached in memory (using RDD.cache()) 
and persisted on disk after construcion completes, to be able to re-use 
the constructed data-structures without trigerring re-construction:

```scala
import edu.berkeley.cs.succinct.SuccinctRDD
import org.apache.spark.storage.StorageLevel

// Construct the succinct RDD as before, and save it as follows
succinctTextRDD.save("/README.md.succinct")

// Load into memory again as follows; sc is the SparkContext
val loadedSuccinctRDD = SuccinctRDD(sc, "/README.md.succinct", StorageLevel.MEMORY_ONLY)
```

### SuccinctKVRDD API

The `SuccinctKVRDD` implements the `RDD[(K, Array[Byte]]` interface, where key
can be of the specified (_ordered_) type while the value is a serialized array of 
bytes.  

`SuccinctKVRDD` can be used as follows:

```scala
import edu.berkeley.cs.succinct.kv.SuccinctKVRDD

// Create a set of key value pairs; sc is the SparkContext
val kvRDD = sc.textFile("README.md").zipWithIndex.map(t => (t._2, t._1.getBytes))

// Convert the kvRDD to a SuccinctKVRDD.
val succinctKVRDD = SuccinctKVRDD(kvRDD).cache

// Fetch keys corresponding to values containing the string "Succinct"
val keys = succinctKVRDD.search("Succinct")

// Fetch the value corresponding to key 0
val value = succinctKVRDD.get(0)
```

More examples of the SuccinctKVRDD API can be found [here](src/main/scala/edu/berkeley/cs/succinct/examples/KVSearch.scala). 

### DataFrame API<commit_msg>Updated README with more descriptive examples<commit_after># Succinct-Spark

[Spark](http://spark.apache.org/) and 
[Spark SQL](http://spark.apache.org/docs/latest/sql-programming-guide.html) 
interfaces for [Succinct](http://succinct.cs.berkeley.edu/). 
This library facilitates compressing RDDs in Spark and DataFrames in Spark SQL
and enables queries directly on the compressed representation.

## Requirements

This library requires Spark 1.4+.

## Dependency Information

### Apache Maven

To build your application with Succinct-Spark, you can link against this library
using Maven by adding the following dependency information to your pom.xml file:

```xml
<dependency>
    <groupId>amplab</groupId>
    <artifactId>succinct-spark</artifactId>
    <version>0.1.3</version>
</dependency>
```

### SBT

Add the dependency to your SBT project by adding the following to `build.sbt` 
(see the [Spark Packages listing](http://spark-packages.org/package/amplab/succinct)
for spark-submit and Maven instructions):

```
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"
libraryDependencies += "amplab" % "succinct" % "0.1.3"
```

The succinct-spark jar file can also be added to a Spark shell using the 
`--jars` command line option. For example, to include it when starting the 
spark shell:

```
$ bin/spark-shell --jars succinct-0.1.3.jar
```

## Usage

The Succinct-Spark library exposes three APIs: 
* A SuccinctRDD API that views an RDD as an unstructured "flat-file" and enables queries on its compressed representation.
* A SuccinctKVRDD API that provides a key-value abstraction for the data, and supports search and random-access over the _values_.
* DataFrame API that integrates with the Spark SQL interface via Data Sources, and supports SQL queries on compressed structured data.

**Note: The Spark SQL interface is experimental, and only efficient for selected
SQL operators. We aim to make the Spark SQL integration more efficient in
future releases.**

### SuccinctRDD API

We expose a `SuccinctRDD` that extends `RDD[Array[Byte]]`. Since each record is
represented as an array of bytes, `SuccinctRDD` can be used to encode a 
collection of any type of records by providing a serializer/deserializer for
the record type. 

`SuccinctRDD` can be used as follows:

```scala
import edu.berkeley.cs.succinct.SuccinctRDD

// Read text data from file; sc is the SparkContext
val wikiData = ctx.textFile("/path/to/data").map(_.getBytes)

// Convert the wikiRDD to a SuccinctRDD after serializing each record into an
// array of bytes. Persist the RDD in memory to perform in-memory queries.
val wikiSuccinctData = SuccinctRDD(wikiData).persist()

// Count the number of occurrences of "Berkeley" in text
val berkeleyOccCount = wikiSuccinctData.countOffsets("Berkeley")
println("# of times Berkeley appears in text = " + berkeleyOccCount)

// Count the number of occurrences of "Berkeley" in text
val stanfordOccCount = wikiSuccinctData.countOffsets("Berkeley")
println("# of times Berkeley appears in text = " + stanfordOccCount)

// Find all offsets of occurrences of "Berkeley" in text
val searchOffsets = wikiSuccinctData.searchOffsets("Berkeley")
println("First 10 locations in the RDD where Berkeley occurs: ")
searchOffsets.take(10).foreach(println)

// Find all occurrences of the regular expression "(berkeley|stanford)\.edu"
val regexOccurrences = wikiSuccinctData.regexSearchOffsets("(stanford|berkeley)\.edu").collect()
println("# of matches for the regular expression (stanford|berkeley)\.edu = " + regexOffsets.count)

// Fetch all records that contain the string "Succinct"
val succinctRecords = succinctTextRDD.search("Succinct").records.toStringRDD

// Perform a regex search to find all records containing a particular entry
val regexResults = wikiSuccinctData.regexSearch("(stanford|berkeley)\.edu")
      .map(new String(_))
println("# of records containing the regular expression (stanford|berkeley)\.edu = " + regexResults.count)
```

#### Input Constraints

We don't support non-ASCII characters in the input for now, since the
algorithms depend on using certain non-ASCII characters as internal symbols.

#### Construction Time

Another constraint to consider is the construction time for Succinct
data-structures. As for any block compression scheme, Succinct requires
non-trivial amount of time to compress an input dataset. It is strongly
advised that the SuccinctRDD be cached in memory (using RDD.cache()) 
and persisted on disk after construcion completes, to be able to re-use 
the constructed data-structures without trigerring re-construction:

```scala
import edu.berkeley.cs.succinct.SuccinctRDD
import org.apache.spark.storage.StorageLevel

// Construct the succinct RDD as before, and save it as follows
succinctTextRDD.save("/path/to/data")

// Load into memory again as follows; sc is the SparkContext
val loadedSuccinctRDD = SuccinctRDD(sc, "/path/to/data", StorageLevel.MEMORY_ONLY)
```

### SuccinctKVRDD API

The `SuccinctKVRDD` implements the `RDD[(K, Array[Byte]]` interface, where key
can be of the specified (_ordered_) type while the value is a serialized array of 
bytes.  

`SuccinctKVRDD` can be used as follows:

```scala
import edu.berkeley.cs.succinct.kv.SuccinctKVRDD

val wikiData = ctx.textFile(dataPath, partitions).map(_.getBytes)
val wikiKVData = wikiData.zipWithIndex().map(t => (t._2, t._1)).repartition(partitions)

val succinctKV = SuccinctKVRDD(wikiKVData).persist()

// Get a particular value
val value = succinctKV.get(0)
println("Value corresponding to key 0 = " + new String(value))

// Random access into a value
val valueData = succinctKV.extract(0, 1, 3)
println("Value data for key 0 at offset 1 and length 3 = " + new String(valueData))

// Search on values
val keys = succinctKV.search(searchQuery)
println("First 10 keys matching the search query:")
keys.take(10).foreach(println)

// Regex search on values
val regexKeys = succinctKV.regexSearch(regexQuery)
println("First 10 keys matching the regex query:")
regexKeys.take(10).foreach(println)
``` 

### DataFrame API
{'fraction_non_alphanumeric': 0.09275997443622752, 'fraction_numerical': 0.004382360996987127, 'mean_word_length': 4.85462319615179, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '755942', 'n_tokens_mistral': 3434, 'n_tokens_neox': 3206, 'n_words': 1228}