<commit_before>Cluster Setup
-------------

The Taxi360 demo requires a CDH cluster with the following services installed and running:
* Kudu
* Impala-Kudu
* Solr (ensure sufficient number of servers)
* YARN
* Spark on YARN
* HDFS
* Kafka
* Hue

Build the Demo and Upload to Cluster
------------------------------------

Build the demo code
-------------------
    mvn clean package

Upload Artifacts to the Cluster
-------------------------------

The application JAR built in the previous step and required data files should be uploaded to a gateway node on the cluster. This should be a host configured as a client to the Hadoop cluster:

* target/Taxi360
* data/yellow_tripdata_2009-01.10000.csv
* solr/taxi360/schema.xml

Set Up Kafka
------------

Log into the gateway node and create the Kafka topic used for the example. Note that you may need to adjust the --replication-factor parameter based on the number of Kafka brokers in the cluster:

    kafka-topics --zookeeper ZKHOST:ZKPORT --partition 2 --replication-factor 2 --create --topic taxi-trip-input

Where ZKHOST:ZKPORT is the host and port number for a ZooKeeper server

You can test the successful creation of the topic with the following:

    kafka-topics --zookeeper ZKHOST:ZKPORT --list

Set up Solr
-----------

Log into the gateway node and execute the following steps:<commit_msg>updating README<commit_after>Cluster Setup
-------------

The Taxi360 demo requires a CDH cluster with the following services installed and running:
* Kudu
* Impala-Kudu
* Solr (ensure sufficient number of servers)
* YARN
* Spark on YARN
* HDFS
* Kafka
* Hue

Build the Demo and Upload to Cluster
------------------------------------

Build the demo code
-------------------
    mvn clean package

Upload Artifacts to the Cluster
-------------------------------

The application JAR built in the previous step and required data files should be uploaded to a gateway node on the cluster. This should be a host configured as a client to the Hadoop cluster:

* target/Taxi360
* data/yellow_tripdata_2009-01.10000.csv
* solr/taxi360/schema.xml

Set Up Kafka
------------

Log into the gateway node and create the Kafka topic used for the example. Note that you may need to adjust the --replication-factor parameter based on the number of Kafka brokers in the cluster:

    kafka-topics --zookeeper ZKHOST:ZKPORT --partition 2 --replication-factor 2 --create --topic taxi-trip-input

Where ZKHOST:ZKPORT is the host and port number for a ZooKeeper server in the cluster.

You can test the successful creation of the topic with the following:

    kafka-topics --zookeeper ZKHOST:ZKPORT --list

Set up Solr
-----------

Log into the gateway node and execute the following steps:
{'fraction_non_alphanumeric': 0.13169479090242112, 'fraction_numerical': 0.016140865737344093, 'mean_word_length': 2.6263297872340425, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4240172', 'n_tokens_mistral': 796, 'n_tokens_neox': 723, 'n_words': 355}