<commit_before># Cassandra Schema Evolution with Spark

Best practices in data modeling for [Cassandra](http://cassandra.apache.org/) often lead to a "table per query" and new or different queries needed to be supported the schema of an existing system will need to evolve. This is short discussion about how you might evolve a time-series schema using the [Apache Spark™](http://spark.apache.org/) with the DataStax [Spark Cassandra Connector](https://github.com/datastax/spark-cassandra-connector)

## Problem Statement *(pick one to focus on)*

### Option 1

We have an application taking sensor readings and storing them in Cassandra. We've only been storing data from 2 sensor arrays to evaluate the performance of our data model. Using `nodetool cfhistograms` showed what several partitions have too many values.

We want to reduce the values per partition by using a difference partition key.

### Option 2

We have an application taking only 1 sensor reading and storing them in Cassandra. We are going to add new sensors to the network that will provide 3 readings per sensor.

We know that will increase the number of values per partition so we'd like to modify the table's partition key to reduce the possible values per partition.

## Steps

1. Analyze possible columns to add to partition key, thus reducing values
2. Create a table with that new partition key
3. Change driver to write to both old & new table
4. **Bulk move data from old table into the new table** (run as background, batch job)
5. Wait for execution of bulk move to complete
6. Change driver to write to *only* new table
7. Truncate old table, drop old table
8. Run `nodetool cleanup` on cluster

### Schema - Old & New / Source & Target<commit_msg>Edited the introduction for clarity.<commit_after># Cassandra Schema Evolution with Spark

Best practices in data modeling for [Cassandra](http://cassandra.apache.org/) often lead to a "table per query." When new or different queries need to be supported, the schema of an existing system will have to *evolve*. This is short discussion about how you might alter a schema using the [Apache Spark™](http://spark.apache.org/) with the DataStax [Spark Cassandra Connector](https://github.com/datastax/spark-cassandra-connector).

## Problem Statement *(pick one to focus on)*

### Option 1

We have an application taking sensor readings and storing them in Cassandra. We've only been storing data from 2 sensor arrays to evaluate the performance of our data model. Using `nodetool cfhistograms` showed what several partitions have too many values.

We want to reduce the values per partition by using a difference partition key.

### Option 2

We have an application taking only 1 sensor reading and storing them in Cassandra. We are going to add new sensors to the network that will provide 3 readings per sensor.

We know that will increase the number of values per partition so we'd like to modify the table's partition key to reduce the possible values per partition.

## Steps

1. Analyze possible columns to add to partition key, thus reducing values
2. Create a table with that new partition key
3. Change driver to write to both old & new table
4. **Bulk move data from old table into the new table** (run as background, batch job)
5. Wait for execution of bulk move to complete
6. Change driver to write to *only* new table
7. Truncate old table, drop old table
8. Run `nodetool cleanup` on cluster

### Schema - Old & New / Source & Target
{'fraction_non_alphanumeric': 0.05664174813110983, 'fraction_numerical': 0.007475560667050029, 'mean_word_length': 4.319571865443425, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1330610', 'n_tokens_mistral': 886, 'n_tokens_neox': 837, 'n_words': 538}