<commit_before>[![Build Status](https://travis-ci.org/TWCable/grabbit.svg?branch=master)](https://travis-ci.org/TWCable/grabbit)

[ ![Download](https://api.bintray.com/packages/twcable/aem/Grabbit/images/download.svg) ](https://bintray.com/twcable/aem/Grabbit/_latestVersion)

# Purpose #

The purpose of Grabbit is to provide a fast and reliable way of copying content from one Sling (specifically Adobe CQ/AEM) instance to another.

Existing solutions have been tried and found insufficient for very large data sets (GB-TB), especially over a network. CQ's .zip packages are extremely space inefficient, causing a lot of extra I/O. [`vlt rcp`](http://jackrabbit.apache.org/filevault/usage.html) and <NAME>'s [`recap`](http://adamcin.net/net.adamcin.recap/) use essentially the same mechanism: WebDAV using XML, doing an HTTP handshake for every node and many sets of properties, which means that any latency whatsoever on the network hurts performance enormously.
 
Grabbit creates a stream of data using [Google's Protocol Buffers](https://developers.google.com/protocol-buffers/) aka "ProtoBuf". Protocol Buffers are an extremely efficient (in terms of CPU, memory and wire size) binary protocol that includes compression. 

Moreover, by doing a continuous stream, we avoid the latency issues. Depending on the size and nature of the data, as well as network latency, we have so far seen speed improvements ranging from 2 to 10 times that of Recap/vlt. 

*__"Grabbit"__ obviously refers to this "grabbing" content from one CQ instance and copying it to another. However it also refers to "Jackrabbit," the reference JCR implementation that the content is being copied to and from.*
 
# Runtime Dependencies

* AEM/CQ v5.6.1
* To run Grabbit in your AEM/CQ instance, **you need to install a Fragment Bundle once per instance. It can be found [here](https://bintray.com/artifact/download/twcable/aem/dependencies/Sun-Misc-Fragment-Bundle-1.0.0.zip)**

# Building #

## Full Clean Build & Install ##

`gradlew clean build install refreshAllBundles` 

## Pre-requisites ##

### Installing Protocol Buffers Compiler ###

#### For Windows ####

Install the binary from https://code.google.com/p/protobuf/downloads/detail?name=protoc-2.4.1-win32.zip&can=2&q= and then set the `/path/to/protoc/parent` on your PATH variable.

#### For Macs ####

```

brew tap homebrew/versions

brew install protobuf241

brew link --force --overwrite protobuf241

```

_For both Windows and Mac : To verify that installation was successful, `protoc --version` should display `2.4.1`_

# Running Grabbit #

[This] (grabbit.sh) shell script can be used to initiate grabbit jobs for a given Grabbit configuration and to check the status of those jobs.

1. Create a configuration file with the path(s) you wish to copy (see "Config Format" section below, you'll also set the server side URL and credentials in this step)
2. Execute the grabbit.sh script (which comes with the codebase, by default lives in the root of the project)
3. Enter the full address for the client instance including port (details for the server instance should be in the config fil you create in step 1)
4. Enter Username for the client instance
5. Enter the Password for the client instance
6. Enter the path to the Grabbit config you created in step. Screenshot of how it will look like : ![alt text](assets/GrabbitConfig.jpg)
7. Once the Grabbit content sync is kicked off, you will get a confirmation of the kicked off jobs like : ![alt text](assets/GrabbitComplete.jpg)

<commit_msg>Fix broken Protocol Buffers download link in README<commit_after>[![Build Status](https://travis-ci.org/TWCable/grabbit.svg?branch=master)](https://travis-ci.org/TWCable/grabbit)

[ ![Download](https://api.bintray.com/packages/twcable/aem/Grabbit/images/download.svg) ](https://bintray.com/twcable/aem/Grabbit/_latestVersion)

# Purpose #

The purpose of Grabbit is to provide a fast and reliable way of copying content from one Sling (specifically Adobe CQ/AEM) instance to another.

Existing solutions have been tried and found insufficient for very large data sets (GB-TB), especially over a network. CQ's .zip packages are extremely space inefficient, causing a lot of extra I/O. [`vlt rcp`](http://jackrabbit.apache.org/filevault/usage.html) and Mark Adamcin's [`recap`](http://adamcin.net/net.adamcin.recap/) use essentially the same mechanism: WebDAV using XML, doing an HTTP handshake for every node and many sets of properties, which means that any latency whatsoever on the network hurts performance enormously.
 
Grabbit creates a stream of data using [Google's Protocol Buffers](https://developers.google.com/protocol-buffers/) aka "ProtoBuf". Protocol Buffers are an extremely efficient (in terms of CPU, memory and wire size) binary protocol that includes compression. 

Moreover, by doing a continuous stream, we avoid the latency issues. Depending on the size and nature of the data, as well as network latency, we have so far seen speed improvements ranging from 2 to 10 times that of Recap/vlt. 

*__"Grabbit"__ obviously refers to this "grabbing" content from one CQ instance and copying it to another. However it also refers to "Jackrabbit," the reference JCR implementation that the content is being copied to and from.*
 
# Runtime Dependencies

* AEM/CQ v5.6.1
* To run Grabbit in your AEM/CQ instance, **you need to install a Fragment Bundle once per instance. It can be found [here](https://bintray.com/artifact/download/twcable/aem/dependencies/Sun-Misc-Fragment-Bundle-1.0.0.zip)**

# Building #

## Full Clean Build & Install ##

`gradlew clean build install refreshAllBundles` 

## Pre-requisites ##

### Installing Protocol Buffers Compiler ###

#### For Windows ####

Install the binary from https://github.com/google/protobuf/releases/download/v2.4.1/protoc-2.4.1-win32.zip and then set the `/path/to/protoc/parent` on your PATH variable.

#### For Macs ####

```

brew tap homebrew/versions

brew install protobuf241

brew link --force --overwrite protobuf241

```

_For both Windows and Mac : To verify that installation was successful, `protoc --version` should display `2.4.1`_

# Running Grabbit #

[This] (grabbit.sh) shell script can be used to initiate grabbit jobs for a given Grabbit configuration and to check the status of those jobs.

1. Create a configuration file with the path(s) you wish to copy (see "Config Format" section below, you'll also set the server side URL and credentials in this step)
2. Execute the grabbit.sh script (which comes with the codebase, by default lives in the root of the project)
3. Enter the full address for the client instance including port (details for the server instance should be in the config fil you create in step 1)
4. Enter Username for the client instance
5. Enter the Password for the client instance
6. Enter the path to the Grabbit config you created in step. Screenshot of how it will look like : ![alt text](assets/GrabbitConfig.jpg)
7. Once the Grabbit content sync is kicked off, you will get a confirmation of the kicked off jobs like : ![alt text](assets/GrabbitComplete.jpg)


{'fraction_non_alphanumeric': 0.08698707138842046, 'fraction_numerical': 0.00927487352445194, 'mean_word_length': 4.525621118012422, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2289912', 'n_tokens_mistral': 2080, 'n_tokens_neox': 1953, 'n_words': 906}