<commit_before># CDK End-to-End Demo

This module provides an example of logging application events from a webapp to Hadoop
via Flume (using log4j as the logging API), extracting session data from the events using
Crunch, running the Crunch job periodically using Oozie, and analyzing the
session data with SQL using Hive.

## Pre-requisites

Before trying this example, you need to have installed the CDK event serializer module in
Flume (this is explained in the `logging` example).

Next, start a Flume agent on the QuickStart VM. You can do this via Cloudera Manager by
selecting "View and Edit" under the Flume service Configuration tab, then clicking on the
"Agent (Default)" category, and pasting the contents of the `flume.properties` file in
this project into the text area for the "Configuration File" property.

If you are running this example from you machine and not from a QuickStart VM login,
then make sure you change the value of the `proxyUser` setting in the agent
configuration to the user that you are logged in as. Save changes,
then start the Flume agent.

For Oozie you need to have Oozie's sharelib installed (which is taken care of already in
the QuickStart VM) and the Oozie service must be running - so start it using Cloudera
Manager.

Finally add the HCatalog Core JAR to the Hive Oozie sharelib,
by logging in to the VM and running:

```bash
sudo -u oozie hadoop fs -put \
  /usr/lib/hcatalog/share/hcatalog/hcatalog-core-0.5.0-cdh4.3.0.jar \
  /user/oozie/share/lib/hive
```

## Building

To build the project, type

```bash
mvn package
```

This creates the following artifacts:

* a JAR file containing the compiled Avro specific schema `session.avsc` (in `demo-core`)
* a WAR file for the webapp that logs application events (in `demo-webapp`)<commit_msg>Need to install demo artifacts to local mvn repo since it's a multimodule project.<commit_after># CDK End-to-End Demo

This module provides an example of logging application events from a webapp to Hadoop
via Flume (using log4j as the logging API), extracting session data from the events using
Crunch, running the Crunch job periodically using Oozie, and analyzing the
session data with SQL using Hive.

## Pre-requisites

Before trying this example, you need to have installed the CDK event serializer module in
Flume (this is explained in the `logging` example).

Next, start a Flume agent on the QuickStart VM. You can do this via Cloudera Manager by
selecting "View and Edit" under the Flume service Configuration tab, then clicking on the
"Agent (Default)" category, and pasting the contents of the `flume.properties` file in
this project into the text area for the "Configuration File" property.

If you are running this example from you machine and not from a QuickStart VM login,
then make sure you change the value of the `proxyUser` setting in the agent
configuration to the user that you are logged in as. Save changes,
then start the Flume agent.

For Oozie you need to have Oozie's sharelib installed (which is taken care of already in
the QuickStart VM) and the Oozie service must be running - so start it using Cloudera
Manager.

Finally add the HCatalog Core JAR to the Hive Oozie sharelib,
by logging in to the VM and running:

```bash
sudo -u oozie hadoop fs -put \
  /usr/lib/hcatalog/share/hcatalog/hcatalog-core-0.5.0-cdh4.3.0.jar \
  /user/oozie/share/lib/hive
```

## Building

To build the project, type

```bash
mvn install
```

This creates the following artifacts:

* a JAR file containing the compiled Avro specific schema `session.avsc` (in `demo-core`)
* a WAR file for the webapp that logs application events (in `demo-webapp`)
{'fraction_non_alphanumeric': 0.05755001370238422, 'fraction_numerical': 0.0038366675801589476, 'mean_word_length': 4.312954876273653, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3571196', 'n_tokens_mistral': 1050, 'n_tokens_neox': 1015, 'n_words': 558}