<commit_before>**ETL Code for Loading Data Into a Redshift-based Data Warehouse**

```
  _    _                       _       _____          _     _     _  __ _     ______ _______ _
 | |  | |                     ( )     |  __ \        | |   | |   (_)/ _| |   |  ____|__   __| |
 | |__| | __ _ _ __ _ __ _   _|/ ___  | |__) |___  __| |___| |__  _| |_| |_  | |__     | |  | |
 |  __  |/ _` | '__| '__| | | | / __| |  _  // _ \/ _` / __| '_ \| |  _| __| |  __|    | |  | |
 | |  | | (_| | |  | |  | |_| | \__ \ | | \ \  __/ (_| \__ \ | | | | | | |_  | |____   | |  | |____
 |_|  |_|\__,_|_|  |_|   \__, | |___/ |_|  \_\___|\__,_|___/_| |_|_|_|  \__| |______|  |_|  |______|
                          __/ |
                         |___/
```

This README outlines how to get started.
You are probably looking for the [Wiki](https://github.com/harrystech/harrys-redshift-etl/wiki) pages,
which include a lot more information about the ETL and what it does (and why it does what it does).
And if something appears amiss, check out the [issues page](https://github.com/harrystech/harrys-redshift-etl/issues).

# Running the ETL

## Pre-requisites

What else you need in order to use this ETL depends on where you'd like run it and whether you anticipate
making changes to the ETL code.

### Spark

Install Spark if you'd like to be able to test jobs on your local machine.
On a Mac, simply use `brew install apache-spark` for an easy [Homebrew](http://brew.sh/) installation.

Our ETL code base includes everything to run the ETL in a Spark cluster on AWS
using [Amazon EMR](https://aws.amazon.com/elasticmapreduce/) so that local testing may be less important to you.
Running it locally offers shorter development cycles while running it in AWS means less network activity (going in
and out of your VPC).

### JAR files

Download the JAR files for the following software before deployment into an AWS Spark Cluster in order
to be able to connect to PostgreSQL databases and write CSV files for a Dataframe:

| Software | Version | JAR file  |
|---|---|---|
| [PostgreSQL JDBC](https://jdbc.postgresql.org/) driver | 9.4 | [postgresql-9.4.1208.jar](https://jdbc.postgresql.org/download/postgresql-9.4.1208.jar) |
| [Spark-CSV package](https://spark-packages.org/) | 1.4 | [spark-csv_2.10-1.4.0.jar](http://repo1.maven.org/maven2/com/databricks/spark-csv_2.10/1.4.0/spark-csv_2.10-1.4.0.jar) |<commit_msg>Reference news from readme<commit_after>**ETL Code for Loading Data Into a Redshift-based Data Warehouse**

```
  _    _                       _       _____          _     _     _  __ _     ______ _______ _
 | |  | |                     ( )     |  __ \        | |   | |   (_)/ _| |   |  ____|__   __| |
 | |__| | __ _ _ __ _ __ _   _|/ ___  | |__) |___  __| |___| |__  _| |_| |_  | |__     | |  | |
 |  __  |/ _` | '__| '__| | | | / __| |  _  // _ \/ _` / __| '_ \| |  _| __| |  __|    | |  | |
 | |  | | (_| | |  | |  | |_| | \__ \ | | \ \  __/ (_| \__ \ | | | | | | |_  | |____   | |  | |____
 |_|  |_|\__,_|_|  |_|   \__, | |___/ |_|  \_\___|\__,_|___/_| |_|_|_|  \__| |______|  |_|  |______|
                          __/ |
                         |___/
```

This README outlines how to get started.

You are probably looking for the [Wiki](https://github.com/harrystech/harrys-redshift-etl/wiki) pages,
which include a lot more information about the ETL and what it does (and why it does what it does).
And if something appears amiss, check out the [issues page](https://github.com/harrystech/harrys-redshift-etl/issues).

And if you're looking for changes, check out the [NEWS](./NEWS.md).

# Running the ETL

## Pre-requisites

What else you need in order to use this ETL depends on where you'd like run it and whether you anticipate
making changes to the ETL code.

### Spark

Install Spark if you'd like to be able to test jobs on your local machine.
On a Mac, simply use `brew install apache-spark` for an easy [Homebrew](http://brew.sh/) installation.

Our ETL code base includes everything to run the ETL in a Spark cluster on AWS
using [Amazon EMR](https://aws.amazon.com/elasticmapreduce/) so that local testing may be less important to you.
Running it locally offers shorter development cycles while running it in AWS means less network activity (going in
and out of your VPC).

### JAR files

Download the JAR files for the following software before deployment into an AWS Spark Cluster in order
to be able to connect to PostgreSQL databases and write CSV files for a Dataframe:

| Software | Version | JAR file  |
|---|---|---|
| [PostgreSQL JDBC](https://jdbc.postgresql.org/) driver | 9.4 | [postgresql-9.4.1208.jar](https://jdbc.postgresql.org/download/postgresql-9.4.1208.jar) |
| [Spark-CSV package](https://spark-packages.org/) | 1.4 | [spark-csv_2.10-1.4.0.jar](http://repo1.maven.org/maven2/com/databricks/spark-csv_2.10/1.4.0/spark-csv_2.10-1.4.0.jar) |
{'fraction_non_alphanumeric': 0.14435103547262662, 'fraction_numerical': 0.014763174082427721, 'mean_word_length': 2.425561797752809, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '30582', 'n_tokens_mistral': 1764, 'n_tokens_neox': 1658, 'n_words': 441}