<commit_before># marklogic-kinesis-connector
Connector from AWS Kinesis streams to MarkLogic database

**marklogic-kinesis-connector** acts as a pipeline between an [Amazon Kinesis] stream and a [MarkLogic] database. Data can be sent from Kinesis directly to MarkLogic as XML, JSON, or free/unstructured text documents where it will be automatically indexed and available for use. The connector is written in Java and can run in AWS or locally to pull data from Kinesis.

## Requirements

 + **Java JDK 1.7**: This connector has been built with Java version 1.7.
 + **Ant**: A build.xml script is provided to build the connector with Ant.
 + **AWS Kinesis Account**: An Amazon AWS Kinesis account to use as a source of data.
 + **MarkLogic database**: The host, port, username,and password a MarkLogic database either in AWS or on premise.

## Overview

Incoming records from one (or many) shards of an AWS Kinesis Stream will be written to MarkLogic, optionally into a collection and using a specified URI base path.  By specifying the type of data coming through the Kinesis stream (XML, JSON, TEXT) MarkLogic will ingest and automatically index them and make them available for immediate use in your applications.

## Configuration

The MarkLogicConnector.properties file should be modified for the Kinesis Stream(**kinesisInputStream**, **regionName**), destination MarkLogic database/cluster details(**marklogic.host**, **marklogic.port**, **marklogic.user**), App Name (**appName**)

## Building the Connector

To download required dependencies and build the project execute **ant build**
The build directory should be created with the following structure:

build
├── marklogic-kinesis.jar
├── config/
│   ├── MarkLogic.properties
│   ├── log4j.properties
├── lib/ (dependecy jars)

## Running the Connector

Set up your AWS credentials in ~/.aws/credentials

Modify MarkLogicConnector.properties with the appropriate Kinesis stream details: **kinesisInputStream** and **regionName** and set the type of data which will be coming through the Kinesis stream in **kinesisInputStream.documenttype**
For the MarkLogic database set the **marklogic.host** **marklogic.port** **marklogic.user** **marklogic.user.password**

Run the connector with **java -jar marklogic-kinesis.jar**

## Related

[Amazon Kinesis](http://aws.amazon.com/kinesis/)

[Ant](http://ant.apache.org/)

[AWS Kinesis Account](http://aws.amazon.com/account/)

[Kinesis Client Library](https://github.com/awslabs/amazon-kinesis-client/)<commit_msg>Details<commit_after># marklogic-kinesis-connector
Connector from AWS Kinesis streams to MarkLogic database

**marklogic-kinesis-connector** acts as a pipeline between an Amazon Kinesis stream and a MarkLogic database. Data can be sent from Kinesis directly to MarkLogic as XML, JSON, or free/unstructured text documents where it will be automatically indexed and available for use in applications for analysis and processing. The connector uses the MarkLogic Java Client and can run in AWS or locally to pull data from Kinesis.

## Requirements

 + **Java JDK 1.7**: This connector has been built with Java version 1.7.
 + **Ant**: A build.xml script is provided to build the connector with Ant.
 + **AWS Kinesis Account**: An Amazon AWS Kinesis account to use as a source of data.
 + **MarkLogic database**: The host, port, username,and password a MarkLogic database either in AWS or on premise.

## Overview

Incoming records from one (or many) shards of an AWS Kinesis Stream will be written to MarkLogic, optionally into a collection and using a specified URI base path.  By specifying the type of data coming through the Kinesis stream (XML, JSON, TEXT) MarkLogic will ingest and automatically index them and make them available for immediate use in your applications.

## Configuration

The MarkLogicConnector.properties file should be modified for the Kinesis Stream(**kinesisInputStream**, **regionName**), destination MarkLogic database/cluster details(**marklogic.host**, **marklogic.port**, **marklogic.user**), App Name (**appName**)

## Building the Connector

To download required dependencies and build the project execute **ant build**

When this completes the **build** directory should have been created containing **marklogic-kinesis.jar** and the following two sub-directories:

**config** directory containing **MarkLogic.properties** and **log4j.properties**

**lib** directory containing dependency jars

## Running the Connector

Set up your AWS credentials in ~/.aws/credentials

Modify **MarkLogicConnector.properties** with the appropriate Kinesis stream details: **kinesisInputStream** and **regionName** as well as setting the type of data which will be coming through the Kinesis stream in **kinesisInputStream.documenttype**

For MarkLogic set the **marklogic.host**, **marklogic.port**, **marklogic.user** and **marklogic.user.password**

Ensure the user has been created in MarkLogic and has rest-writer privileges to write to the MarkLogic REST api

Run the connector with **java -jar marklogic-kinesis.jar**

## Related

[Amazon Kinesis](http://aws.amazon.com/kinesis/)

[Ant](http://ant.apache.org/)

[AWS Kinesis Account](http://aws.amazon.com/account/)

[Kinesis Client Library](https://github.com/awslabs/amazon-kinesis-client/)
{'fraction_non_alphanumeric': 0.09175179720015135, 'fraction_numerical': 0.0018917896329928112, 'mean_word_length': 5.198124267291911, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1577969', 'n_tokens_mistral': 1451, 'n_tokens_neox': 1396, 'n_words': 645}