<commit_before>Sparkster
=======

A tool for managing Spark clusters in R.

## Installation

Sparkster is not yet on CRAN; it is currently available from GitHub. Make sure you have the devtools package installed, and then run:

```R
devtools::install_github("rstudio/sparkster")
```

## Example

To start a new 1-master 1-slave Spark cluster in EC2 run the following code:

```R
sparkster::start_ec2(access_key_id = "<KEY>",
                     secret_access_key = "<KEY>",
                     pem_file = "sparkster.pem")
```

The `access_key_id`, `secret_access_key` and `pem_file` need to be retrieved from the AWS console.<commit_msg>update readme from splyr and sparkster<commit_after>Spark
=======

A set of tools to provision, connect and interface to Apache Spark from within the
R language and echosystem. This package supports connecting to local and remote
Apache Spark clusters and provides support for R packages like dplyr and DBI.

## Installation

Spark is not yet on CRAN; it is currently available from GitHub. Install the devtools package followed by:

```R
devtools::install_github("rstudio/spark")
```

## Examples

### Local

This example starts a new Spark local instance and performs basic table operations:

```R
library(spark)
library(dplyr)
library(nycflights13)

db <- src_spark()
copy_to(db, flights, "flights")
src_tbls(db)

tbl(db, "flights") %>% collect

```

### Remote

To start a new 1-master 1-slave Spark cluster in EC2 run the following code:

```R
library(spark)
start_ec2(access_key_id = "<KEY>",
          secret_access_key = "1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
          pem_file = "sparkster.pem")
```

The `access_key_id`, `secret_access_key` and `pem_file` need to be retrieved from the AWS console.
{'fraction_non_alphanumeric': 0.1004566210045662, 'fraction_numerical': 0.017123287671232876, 'mean_word_length': 4.444099378881988, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3122384', 'n_tokens_mistral': 596, 'n_tokens_neox': 556, 'n_words': 200}