<commit_before>CoAnSys/importers
=============

## About
CoAnSys/importers is a project containing methods to easily load [BWMeta document](../examples/bwmeta) packages into HBase.

It provides two way to do so - (a) with RichTSV file or (b) via REST.

## Prerequirements
This package depends on (Hadoop)[http://hadoop.apache.org/] from [Cloudera distribution](https://ccp.cloudera.com/display/SUPPORT/CDH+Downloads), version CDH 4.0.1 and appropriate [HBase](hbase.apache.org) version. 
As CDH4 depends on [Protocol Buffers](http://code.google.com/p/protobuf/) 2.4.1 so is CoAnSys/importers.

## Quick Start

### Instalation
```
# repository clone
cd ~  
git clone REPOSITORY_ADDRESS/CoAnSys.git  
cd CoAnSys/importers  
# package instalation  
## if you are not providing hadoop libraries, please modify pom.xml  
## by removing texts "<scope>provided</scope>"
mvn install
mvn assembly:single
```

### Example data acquisition
```
# get
```
[example data](../examples/otworz-ksiazke-tylkobwmeta-20110316-00000.zip)

```
cd ~
mkdir HBASE_IMPORT
cd HBASE_IMPORT
wget 'LOCALIZATION_FROM_THE_LINK'
```

### HBase output table creation
```
echo "create 'testProto','m','c'" | hbase shell
```

### import
#### via REST
```
cd ~/CoAnSys/importers/target
# the following command assign "TESTCOLLECTION" collection name 
# to all articles from  BWMeta zip files from the folder
# ~/HBASE_IMPORT/ and send them
# to HBase table "testProto" localized on localhost.
# Port 8080 is the default port for such a communication.
# Please specify the port adequate for your hbase configuration
java -jar importers-1.0-SNAPSHOT-jar-with-dependencies.jar ~/HBASE_IMPORT/ TESTCOLLECTION localhost 8080 testProto
```

#### via RichTSV file (TBD)
<commit_msg>replace example file for import<commit_after>CoAnSys/importers
=============

## About
CoAnSys/importers is a project containing methods to easily load [BWMeta document](../examples/bwmeta) packages into HBase.

It provides two way to do so - (a) with RichTSV file or (b) via REST.

## Prerequirements
This package depends on [Hadoop](http://hadoop.apache.org/) from [Cloudera distribution](https://ccp.cloudera.com/display/SUPPORT/CDH+Downloads), version CDH 4.0.1 and appropriate [HBase](hbase.apache.org) version. 
As CDH4 depends on [Protocol Buffers](http://code.google.com/p/protobuf/) 2.4.1 so is CoAnSys/importers.

## Quick Start

### Instalation
```
# repository clone
cd ~  
git clone REPOSITORY_ADDRESS/CoAnSys.git  
cd CoAnSys/importers  
# package instalation  
## if you are not providing hadoop libraries, please modify pom.xml  
## by removing texts "<scope>provided</scope>"
mvn install
mvn assembly:single
```

### Example data acquisition
```
# get
```
[example data](../examples/GROTOAP-10.zip)

```
cd ~
mkdir HBASE_IMPORT
cd HBASE_IMPORT
wget 'LOCALIZATION_FROM_THE_LINK'
```

### HBase output table creation
```
echo "create 'testProto','m','c'" | hbase shell
```

### import
#### via REST
```
cd ~/CoAnSys/importers/target
# the following command assign "TESTCOLLECTION" collection name 
# to all articles from  BWMeta zip files from the folder
# ~/HBASE_IMPORT/ and send them
# to HBase table "testProto" localized on localhost.
# Port 8080 is the default port for such a communication.
# Please specify the port adequate for your hbase configuration
java -cp importers-1.0-SNAPSHOT-jar-with-dependencies.jar pl.edu.icm.coansys.importers.io.writers.hbaserest.HBaseRestWriter_Bwmeta ~/HBASE_IMPORT/ TESTCOLLECTION localhost 8080 testProto
```

#### via RichTSV file (TBD)

{'fraction_non_alphanumeric': 0.12475247524752475, 'fraction_numerical': 0.013861386138613862, 'mean_word_length': 4.787234042553192, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7348654', 'n_tokens_mistral': 1208, 'n_tokens_neox': 1129, 'n_words': 353}