<commit_before># Kite SDK Examples

The Kite Examples project provides examples of how to use the Kite SDK.

Each example is a standalone Maven module with associated documentation.

## Basic Examples

* `dataset` is the closest to a HelloWorld example of Kite. It shows how to create datasets and perform streaming writes and reads over them.
* `dataset-hbase` shows how to store entities in HBase using the `RandomAccessDataset` API.
* `dataset-staging` shows how to use two datasets to manage Parquet-formatted data
* `logging` is an example of logging events from a command-line programs to Hadoop via Flume, using log4j as the logging API.
* `logging-webapp` is like `logging`, but the logging source is a webapp.

## Advanced Examples

* `demo` is a full end-to-end example of a webapp that logs events using Flume and performs session analysis using Crunch and Hive.

## Getting Started

The easiest way to run the examples is on the
[Cloudera QuickStart VM](https://ccp.cloudera.com/display/SUPPORT/Cloudera+QuickStart+VM),
which has all the necessary Hadoop services pre-installed, configured, and
running locally. See the notes below for any initial setup steps you should take.

The current examples run on version 4.4.0 of the QuickStart VM.

Checkout the latest [branch](https://github.com/kite-sdk/kite-examples/branches) of this repository in the VM:

```bash
git clone git://github.com/kite-sdk/kite-examples.git
cd kite-examples
```

If you are using a prepared Kite VM, the `git clone` command is already done for you.

Then choose the example you want to try and refer to the README in the relevant subdirectory.

### Setting up the QuickStart VM

There are two ways to run the examples with the QuickStart VM:

1. Logged in to the VM guest (username and password are both `<PASSWORD>`).
2. From your host computer.

The advantage of the first approach is that you don't need to install anything extra on
your host computer, such as Java or Maven, so there are no extra set up steps.

The second approach is preferable when you want to use tools from your own development
environment (browser, IDE, command line). However, there are a few extra steps you
need to take to configure the QuickStart VM, listed below.

* __Enable port forwarding__ For VirtualBox, open the Settings dialog for the VM,
select the Network tab, and click the Port Forwarding button. Map the following ports -
in each case the host port and the guest port should be the same.
    * 7180 (Cloudera Manager web UI)
    * 8020, 50010, 50020, 50070, 50075 (HDFS NameNode and DataNode)
    * 8021 (MapReduce JobTracker)
    * 8888 (Hue web UI)
    * 9083 (Hive/HCatalog metastore)
    * 41415 (Flume agent)
    * 11000 (Oozie server)
    * 21050 (Impala JDBC port)
* __Bind daemons to the wildcard address__ Daemons that are accessed from the host need
to listen on all network interfaces. In [Cloudera Manager]
(http://localhost:7180/cmf/services/status) for each of the services listed below,
select the service, click "View and Edit" under the Configuration tab then
search for "wildcard", check the box, then save changes.<commit_msg>Adding bash instructions for port forwarding<commit_after># Kite SDK Examples

The Kite Examples project provides examples of how to use the Kite SDK.

Each example is a standalone Maven module with associated documentation.

## Basic Examples

* `dataset` is the closest to a HelloWorld example of Kite. It shows how to create datasets and perform streaming writes and reads over them.
* `dataset-hbase` shows how to store entities in HBase using the `RandomAccessDataset` API.
* `dataset-staging` shows how to use two datasets to manage Parquet-formatted data
* `logging` is an example of logging events from a command-line programs to Hadoop via Flume, using log4j as the logging API.
* `logging-webapp` is like `logging`, but the logging source is a webapp.

## Advanced Examples

* `demo` is a full end-to-end example of a webapp that logs events using Flume and performs session analysis using Crunch and Hive.

## Getting Started

The easiest way to run the examples is on the
[Cloudera QuickStart VM](https://ccp.cloudera.com/display/SUPPORT/Cloudera+QuickStart+VM),
which has all the necessary Hadoop services pre-installed, configured, and
running locally. See the notes below for any initial setup steps you should take.

The current examples run on version 4.4.0 of the QuickStart VM.

Checkout the latest [branch](https://github.com/kite-sdk/kite-examples/branches) of this repository in the VM:

```bash
git clone git://github.com/kite-sdk/kite-examples.git
cd kite-examples
```

If you are using a prepared Kite VM, the `git clone` command is already done for you.

Then choose the example you want to try and refer to the README in the relevant subdirectory.

### Setting up the QuickStart VM

There are two ways to run the examples with the QuickStart VM:

1. Logged in to the VM guest (username and password are both `<PASSWORD>`).
2. From your host computer.

The advantage of the first approach is that you don't need to install anything extra on
your host computer, such as Java or Maven, so there are no extra set up steps.

The second approach is preferable when you want to use tools from your own development
environment (browser, IDE, command line). However, there are a few extra steps you
need to take to configure the QuickStart VM, listed below.

* __Enable port forwarding__ For VirtualBox, open the Settings dialog for the VM,
select the Network tab, and click the Port Forwarding button. Map the following ports -
in each case the host port and the guest port should be the same. Also, your VM should
not be running when you are making these changes.
    * 7180 (Cloudera Manager web UI)
    * 8020, 50010, 50020, 50070, 50075 (HDFS NameNode and DataNode)
    * 8021 (MapReduce JobTracker)
    * 8888 (Hue web UI)
    * 9083 (Hive/HCatalog metastore)
    * 41415 (Flume agent)
    * 11000 (Oozie server)
    * 21050 (Impala JDBC port)

If you have `VBoxManage` installed on your host machine, you can do this via command line
as well. In bash, this would look something like:
```bash
# Set VM_NAME to the name of your VM as it appears in VirtualBox
VM_NAME="QuickStart VM"
PORTS="7180 8020 50010 50020 50070 50075 8021 8888 9083 41415 11000 21050"
for port in $PORTS; do
  VBoxManage modifyvm "$VM_NAME" --natpf1 "Rule $port,tcp,,$port,,$port"
done
```
* __Bind daemons to the wildcard address__ Daemons that are accessed from the host need
to listen on all network interfaces. In [Cloudera Manager]
(http://localhost:7180/cmf/services/status) for each of the services listed below,
select the service, click "View and Edit" under the Configuration tab then
search for "wildcard", check the box, then save changes.
{'fraction_non_alphanumeric': 0.06200177147918512, 'fraction_numerical': 0.02745792736935341, 'mean_word_length': 4.239752513534416, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1929045', 'n_tokens_mistral': 2047, 'n_tokens_neox': 1863, 'n_words': 1008}