<commit_before># CDK End-to-End Demo

This module provides an example of logging application events from a webapp to Hadoop
via Flume (using log4j as the logging API), extracting session data from the events using
Crunch, running the Crunch job periodically using Oozie, and analyzing the
session data with SQL using Impala or Hive.

## Pre-requisites

*   __Enable Flume user impersonation__ Flume needs to be able to impersonate the owner
 of the dataset it is writing to. (This is like Unix `sudo`, see
[Configuring Flume's Security Properties](http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/cdh4sg_topic_4_2.html)
for further information.) In Cloudera Manager, for the [HDFS service](http://localhost:7180/cmf/services/status),
click "View and Edit" under the Configuration tab then
search for "Cluster-wide Configuration Safety Valve for core-site.xml"
and add the following XML snippet, then save changes.

```
<property>
  <name>hadoop.proxyuser.flume.groups</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.flume.hosts</name>
  <value>*</value>
</property>
```
*   __Install the CDK event serializer module__ This is necessary
since Flume 1.3.0 does not come with a HDFS sink that can write Avro data files.
Note that the HDFS sink in Flume 1.4.0 can write Avro data files so this step is not
needed for that version of Flume or later.

```bash
sudo wget https://repository.cloudera.com/artifactory/libs-release-local/com/cloudera/cdk/cdk-flume-avro-event-serializer/0.4.0/cdk-flume-avro-event-serializer-0.4.0.jar \
  -P /usr/lib/flume-ng/lib/
# or if wget is not available:
( cd /usr/lib/flume-ng/lib/ ; sudo curl -O https://repository.cloudera.com/artifactory/libs-release-local/com/cloudera/cdk/cdk-flume-avro-event-serializer/0.4.0/cdk-flume-avro-event-serializer-0.4.0.jar ; )
```
*   __Start a Flume agent__ You can do this via Cloudera Manager by
selecting "View and Edit" under the Flume service Configuration tab, then clicking on the
"Agent (Default)" category, and pasting the contents of the `flume.properties` file in
this project into the text area for the "Configuration File" property.

    If you are running this example from you machine and not from a QuickStart VM login,
then make sure you change the value of the `proxyUser` setting in the agent
configuration to the user that you are logged in as. Save changes,
then start the Flume agent.
*   __Set up Hive Oozie sharelib__ Add the HCatalog Core JAR to the Hive Oozie
sharelib, by logging in to the VM and running:

```bash
sudo -u oozie hadoop fs -put \
  /usr/lib/hcatalog/share/hcatalog/hcatalog-core-0.5.0-cdh4.3.0.jar \
  /user/oozie/share/lib/hive
```
*   __Ensure Oozie is running__ From Cloudera Manager, start the Oozie service.
*   __(Optional) Upgrade Impala__ To use Impala in this example you need to be running
Impala 1.1 or later. If the
version of QuickStart VM you are running is earlier than Impala 1.1, you can upgrade it
using [these instructions](http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_upgrading.html)
Make sure you follow the instructions under the heading "To upgrade Impala in a
Cloudera Managed environment, using parcels". You should also upgrade CDH at the same
time.

## Building

To build the project, type

```bash
mvn install
```

This creates the following artifacts:

* a JAR file containing the compiled Avro specific schema `session.avsc` (in `demo-core`)
* a WAR file for the webapp that logs application events (in `demo-logging-webapp`)
* a JAR file for running the Crunch job to transform events into sessions (in
`demo-crunch`)
* an Oozie application for running the Crunch job on a periodic basis (in `demo-oozie`)
* a WAR file for the webapp that displays reports generated by Impala (in
`demo-reports-webapp`)

## Running

### Create the datasets

First we need to create the datasets: one called `events` for the raw events,
and `sessions` for the derived sessions.

We store the raw events metadata in HDFS so Flume can find the schema (it would be nice
if we could store it using HCatalog, so we may lift this restriction in the future).
The sessions dataset metadata is stored using HCatalog, which will allow us to query it
via Hive.

```bash
mvn cdk:create-dataset \
  -Dcdk.rootDirectory=/tmp/data \
  -Dcdk.datasetName=events \
  -Dcdk.avroSchemaFile=standard_event.avsc \
  -Dcdk.hcatalog=false \
  -Dcdk.partitionExpression='[year("timestamp", "year"), month("timestamp", "month"), day("timestamp", "day"), hour("timestamp", "hour"), minute("timestamp", "minute")]'

mvn cdk:create-dataset \
  -Dcdk.rootDirectory=/tmp/data \
  -Dcdk.datasetName=sessions \
  -Dcdk.avroSchemaFile=demo-core/src/main/avro/session.avsc
```

A few comments about these commands. The schema for the `events` dataset,
`standard_event.avsc`, is loaded from the classpath (it's in the CDK JAR),
whereas the schema for `sessions` is a local file.

The `--partition-expression` argument is used to specify how the data is partitioned.
Here we partition by time fields, using JEXL to specify the field partitioners.

Note that you can drop the datasets if you created them on a previous attempt with:

```bash
mvn cdk:drop-dataset -Dcdk.rootDirectory=/tmp/data -Dcdk.datasetName=events -Dcdk.hcatalog=false
mvn cdk:drop-dataset -Dcdk.rootDirectory=/tmp/data -Dcdk.datasetName=sessions
```

You can check that the data directories were created, using Hue (login as `cloudera` if
 you are logged in to the VM, or as your host login if you are running from your
 machine): [`/tmp/data/events`](http://localhost:8888/filebrowser/#/tmp/data/events),
 [`/tmp/data/sessions`](http://localhost:8888/filebrowser/#/tmp/data/sessions).

### Create events

Next we can run the webapps. They can be used in a Java EE 6 servlet
container; for this example we'll start an embedded Tomcat instance using Maven:

```bash
mvn tomcat7:run
```

Navigate to [http://localhost:8080/demo-logging-webapp/](http://localhost:8080/demo-logging-webapp/),
which presents you with a very simple web page for sending messages.

The message events are sent to the Flume agent
over IPC, and the agent writes the events to the HDFS file sink.

Rather than creating lots of events manually, it's easier to simulate two users with
a script as follows:

```bash
./bin/simulate-activity.sh 1 10 > /dev/null &
./bin/simulate-activity.sh 2 10 > /dev/null &
```

### Generate the derived sessions

Wait about 30 seconds for Flume to flush the events to the
[filesystem](http://localhost:8888/filebrowser/#/tmp/data/events),
then run the Crunch job to generate derived session data from the events:

```bash
(cd demo-crunch; mvn cdk:run-tool)
```

The `cdk:run-tool` Maven goal executes the `run` method of the `Tool`,
in this case `CreateSessions`, which launches a Crunch job on the cluster.

The `Tool` class to run, as well as the cluster settings, are found from the configuration
of the `cdk-maven-plugin`.

When it's complete you should see a file in [`/tmp/data/sessions`]
(http://localhost:8888/filebrowser/#/tmp/data/sessions).

### Run session analysis

The `sessions` dataset is now populated with data, and there are a variety of ways
to explore the results.

One way is to use the `demo-reports-webapp` running at
[http://localhost:8080/demo-reports-webapp/](http://localhost:8080/demo-reports-webapp/),
which uses JDBC to run Impala queries for a few pre-defined reports. (Note this only
work with Impala 1.1 or later, see instructions above.)

Another way is to run ad hoc SQL queries using the Hue interfaces to
[Impala](http://localhost:8888/impala/) or [Hive](http://localhost:8888/beeswax/).
Here are some queries to try out:
<commit_msg>Added information about refreshing Impala's cached metadata information to the demo readme.<commit_after># CDK End-to-End Demo

This module provides an example of logging application events from a webapp to Hadoop
via Flume (using log4j as the logging API), extracting session data from the events using
Crunch, running the Crunch job periodically using Oozie, and analyzing the
session data with SQL using Impala or Hive.

## Pre-requisites

*   __Enable Flume user impersonation__ Flume needs to be able to impersonate the owner
 of the dataset it is writing to. (This is like Unix `sudo`, see
[Configuring Flume's Security Properties](http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/cdh4sg_topic_4_2.html)
for further information.) In Cloudera Manager, for the [HDFS service](http://localhost:7180/cmf/services/status),
click "View and Edit" under the Configuration tab then
search for "Cluster-wide Configuration Safety Valve for core-site.xml"
and add the following XML snippet, then save changes.

```
<property>
  <name>hadoop.proxyuser.flume.groups</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.flume.hosts</name>
  <value>*</value>
</property>
```
*   __Install the CDK event serializer module__ This is necessary
since Flume 1.3.0 does not come with a HDFS sink that can write Avro data files.
Note that the HDFS sink in Flume 1.4.0 can write Avro data files so this step is not
needed for that version of Flume or later.

```bash
sudo wget https://repository.cloudera.com/artifactory/libs-release-local/com/cloudera/cdk/cdk-flume-avro-event-serializer/0.4.0/cdk-flume-avro-event-serializer-0.4.0.jar \
  -P /usr/lib/flume-ng/lib/
# or if wget is not available:
( cd /usr/lib/flume-ng/lib/ ; sudo curl -O https://repository.cloudera.com/artifactory/libs-release-local/com/cloudera/cdk/cdk-flume-avro-event-serializer/0.4.0/cdk-flume-avro-event-serializer-0.4.0.jar ; )
```
*   __Start a Flume agent__ You can do this via Cloudera Manager by
selecting "View and Edit" under the Flume service Configuration tab, then clicking on the
"Agent (Default)" category, and pasting the contents of the `flume.properties` file in
this project into the text area for the "Configuration File" property.

    If you are running this example from you machine and not from a QuickStart VM login,
then make sure you change the value of the `proxyUser` setting in the agent
configuration to the user that you are logged in as. Save changes,
then start the Flume agent.
*   __Set up Hive Oozie sharelib__ Add the HCatalog Core JAR to the Hive Oozie
sharelib, by logging in to the VM and running:

```bash
sudo -u oozie hadoop fs -put \
  /usr/lib/hcatalog/share/hcatalog/hcatalog-core-0.5.0-cdh4.3.0.jar \
  /user/oozie/share/lib/hive
```
*   __Ensure Oozie is running__ From Cloudera Manager, start the Oozie service.
*   __(Optional) Upgrade Impala__ To use Impala in this example you need to be running
Impala 1.1 or later. If the
version of QuickStart VM you are running is earlier than Impala 1.1, you can upgrade it
using [these instructions](http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_upgrading.html)
Make sure you follow the instructions under the heading "To upgrade Impala in a
Cloudera Managed environment, using parcels". You should also upgrade CDH at the same
time.

## Building

To build the project, type

```bash
mvn install
```

This creates the following artifacts:

* a JAR file containing the compiled Avro specific schema `session.avsc` (in `demo-core`)
* a WAR file for the webapp that logs application events (in `demo-logging-webapp`)
* a JAR file for running the Crunch job to transform events into sessions (in
`demo-crunch`)
* an Oozie application for running the Crunch job on a periodic basis (in `demo-oozie`)
* a WAR file for the webapp that displays reports generated by Impala (in
`demo-reports-webapp`)

## Running

### Create the datasets

First we need to create the datasets: one called `events` for the raw events,
and `sessions` for the derived sessions.

We store the raw events metadata in HDFS so Flume can find the schema (it would be nice
if we could store it using HCatalog, so we may lift this restriction in the future).
The sessions dataset metadata is stored using HCatalog, which will allow us to query it
via Hive.

```bash
mvn cdk:create-dataset \
  -Dcdk.rootDirectory=/tmp/data \
  -Dcdk.datasetName=events \
  -Dcdk.avroSchemaFile=standard_event.avsc \
  -Dcdk.hcatalog=false \
  -Dcdk.partitionExpression='[year("timestamp", "year"), month("timestamp", "month"), day("timestamp", "day"), hour("timestamp", "hour"), minute("timestamp", "minute")]'

mvn cdk:create-dataset \
  -Dcdk.rootDirectory=/tmp/data \
  -Dcdk.datasetName=sessions \
  -Dcdk.avroSchemaFile=demo-core/src/main/avro/session.avsc
```

A few comments about these commands. The schema for the `events` dataset,
`standard_event.avsc`, is loaded from the classpath (it's in the CDK JAR),
whereas the schema for `sessions` is a local file.

The `--partition-expression` argument is used to specify how the data is partitioned.
Here we partition by time fields, using JEXL to specify the field partitioners.

Note that you can drop the datasets if you created them on a previous attempt with:

```bash
mvn cdk:drop-dataset -Dcdk.rootDirectory=/tmp/data -Dcdk.datasetName=events -Dcdk.hcatalog=false
mvn cdk:drop-dataset -Dcdk.rootDirectory=/tmp/data -Dcdk.datasetName=sessions
```

You can check that the data directories were created, using Hue (login as `cloudera` if
 you are logged in to the VM, or as your host login if you are running from your
 machine): [`/tmp/data/events`](http://localhost:8888/filebrowser/#/tmp/data/events),
 [`/tmp/data/sessions`](http://localhost:8888/filebrowser/#/tmp/data/sessions).

### Create events

Next we can run the webapps. They can be used in a Java EE 6 servlet
container; for this example we'll start an embedded Tomcat instance using Maven:

```bash
mvn tomcat7:run
```

Navigate to [http://localhost:8080/demo-logging-webapp/](http://localhost:8080/demo-logging-webapp/),
which presents you with a very simple web page for sending messages.

The message events are sent to the Flume agent
over IPC, and the agent writes the events to the HDFS file sink.

Rather than creating lots of events manually, it's easier to simulate two users with
a script as follows:

```bash
./bin/simulate-activity.sh 1 10 > /dev/null &
./bin/simulate-activity.sh 2 10 > /dev/null &
```

### Generate the derived sessions

Wait about 30 seconds for Flume to flush the events to the
[filesystem](http://localhost:8888/filebrowser/#/tmp/data/events),
then run the Crunch job to generate derived session data from the events:

```bash
(cd demo-crunch; mvn cdk:run-tool)
```

The `cdk:run-tool` Maven goal executes the `run` method of the `Tool`,
in this case `CreateSessions`, which launches a Crunch job on the cluster.

The `Tool` class to run, as well as the cluster settings, are found from the configuration
of the `cdk-maven-plugin`.

When it's complete you should see a file in [`/tmp/data/sessions`]
(http://localhost:8888/filebrowser/#/tmp/data/sessions).

### Run session analysis

The `sessions` dataset is now populated with data, and there are a variety of ways
to explore the results.

One way is to use the `demo-reports-webapp` running at
[http://localhost:8080/demo-reports-webapp/](http://localhost:8080/demo-reports-webapp/),
which uses JDBC to run Impala queries for a few pre-defined reports. (Note this only
work with Impala 1.1 or later, see instructions above.) Impala caches dataset
metadata on startup and, as a result, you may receive errors about the `sessions`
table not existing. To force Impala to reload its metadata cache, issue the command
`INVALIDATE METADATA` from the Impala shell or the Impala Hue application before
running any other SQL queries.

Another way is to run ad hoc SQL queries using the Hue interfaces to
[Impala](http://localhost:8888/impala/) or [Hive](http://localhost:8888/beeswax/).
Here are some queries to try out:

{'fraction_non_alphanumeric': 0.09518723297310883, 'fraction_numerical': 0.01130937421462679, 'mean_word_length': 4.382820426107542, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 31, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1013828', 'n_tokens_mistral': 5096, 'n_tokens_neox': 4863, 'n_words': 1987}