<commit_before># Apache Spark-HBase Connector

### Why
Spark needs a connector library to access HBase.
  - **[Apache HBase Connectors](https://github.com/apache/hbase-connectors)** provides a Spark-HBase connector
   - [Hortonworks Spark-HBase connector](https://github.com/hortonworks-spark/shc) is another connector, it has been quite popular
    over the years, it appears to be no more supported nor updated.

### Configuration
  - You need the HBase client configuration file `hbase-site.xml` 
    - Copy it to `SPARK_CONF_DIR` (default is $SPARK_HOME/conf`, note: HBASE_CONF_DIR does not work)
  - See also the note below on server-side jar configuration to use SQL filter pushdown

### Spark 2.x and the Apache Spark HBase connector 
  - Deploy the connector using packages from Maven Central. Example:
  ```
  bin/spark-shell --master yarn --num-executors 1 --executor-cores 2 \
    --packages org.apache.hbase:hbase-server:2.2.4,org.apache.hbase:hbase-common:2.2.4,org.apache.hbase:hbase-zookeeper:2.2.4,org.apache.hbase.connectors.spark:hbase-spark:1.0.0
  ```
  
### Spark 3.x and the Apache Spark HBase connector
- You need to compile the connector with scala 2.12 (Spark 3.x does not support scala 2.11)
- An extra PR is needed: 
  - See HBase Connectors - [HBASE-25326](https://issues.apache.org/jira/browse/HBASE-25326)
  - See also HBase-connectors fork and experimental [branch for Spark 3 compatibility](https://github.com/LucaCanali/hbase-connectors/tree/compileWithSpark3) 

 - Build as in this example (customize HBase, Spark and Hadoop versions, as needed):  
    ```
    mvn -Dspark.version=3.0.1 -Dscala.version=2.12.10 -Dscala.binary.version=2.12 -Dhbase.version=2.2.4 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.0 -Dhadoop.version=3.2.0 -DskipTests clean package
    ```
  - Deploy using Spark 3.0, as in this example:
  ```
  # Customize the JAR path, for convenience there are versions uploaded to a web server
  JAR1=http://cern.ch/canali/res/hbase-spark-1.0.1_spark-3.0.1-cern1_4.jar
  JAR2=http://cern.ch/canali/res/hbase-spark-protocol-shaded-1.0.1_spark-3.0.1-cern1_4.jar
  bin/spark-shell --master yarn --num-executors 1 --executor-cores 2 \
  --jars $JAR1,$JAR2 --packages org.apache.hbase:hbase-shaded-mapreduce:2.2.4<commit_msg>Update Spark-HBase connector example.<commit_after># Apache Spark-HBase Connector

### Why
Spark needs a connector library to access HBase.
  - **[Apache HBase Connectors](https://github.com/apache/hbase-connectors)** provides a Spark-HBase connector
   - [Hortonworks Spark-HBase connector](https://github.com/hortonworks-spark/shc) is another connector, it has been quite popular
    over the years, it appears to be no more supported nor updated.

### Configuration
  - You need the HBase client configuration file `hbase-site.xml` 
    - Copy it to `SPARK_CONF_DIR` (default is $SPARK_HOME/conf`, note: HBASE_CONF_DIR does not work)
  - See also the note below on server-side jar configuration to use SQL filter pushdown

### Spark 2.x and the Apache Spark HBase connector 
  - Deploy the connector using packages from Maven Central. Example:
  ```
  bin/spark-shell --master yarn --num-executors 1 --executor-cores 2 \
    --packages org.apache.hbase:hbase-server:2.2.4,org.apache.hbase:hbase-common:2.2.4,org.apache.hbase:hbase-zookeeper:2.2.4,org.apache.hbase.connectors.spark:hbase-spark:1.0.0
  ```
  
### Spark 3.x and the Apache Spark HBase connector
- You need to compile the connector with scala 2.12 (Spark 3.x does not support scala 2.11)
- An extra PR is needed: 
  - See HBase Connectors - [HBASE-25326](https://issues.apache.org/jira/browse/HBASE-25326)
  - See also HBase-connectors fork and experimental [branch for Spark 3 compatibility](https://github.com/LucaCanali/hbase-connectors/tree/compileWithSpark3) 

 - Build as in this example (customize HBase, Spark and Hadoop versions, as needed):  
    ```
    mvn -Dspark.version=3.0.1 -Dscala.version=2.12.10 -Dscala.binary.version=2.12 -Dhbase.version=2.2.4 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.0 -Dhadoop.version=3.2.0 -DskipTests clean package
    ```
  - Deploy using Spark 3.0, as in this example:
  ```
  # Customize the JAR path, for convenience there are versions uploaded to a web server
  JAR1=https://cern.ch/canali/res/hbase-spark-1.0.1_spark-3.0.1-cern1_4.jar
  JAR2=https://cern.ch/canali/res/hbase-spark-protocol-shaded-1.0.1_spark-3.0.1-cern1_4.jar
  bin/spark-shell --master yarn --num-executors 1 --executor-cores 2 \
  --jars $JAR1,$JAR2 --packages org.apache.hbase:hbase-shaded-mapreduce:2.2.4
{'fraction_non_alphanumeric': 0.12439772229522558, 'fraction_numerical': 0.03679369250985545, 'mean_word_length': 3.727743271221532, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5813786', 'n_tokens_mistral': 1703, 'n_tokens_neox': 1556, 'n_words': 416}