<commit_before># RNAcentral data import pipeline

## About

This is the main pipeline that is used internally for loading the data into the RNAcentral database.
[More information](http://www.ebi.ac.uk/seqdb/confluence/display/RNAC/RNAcentral+data+import+pipeline)

## Installation

### Perl dependencies

-   BioPerl
-   DBI
-   DBD::Oracle
-   Log4perl

### Installation

```
git clone https://github.com/RNAcentral/rnacentral-import-pipeline.git
cd rnacentral-import-pipeline
# initialize submodules
git submodule init
git submodule update
# add sensitive connection details to config/hive_params
cp config/hive_params_template config/hive_params
```

## Running in Hive mode

	source scripts/hive_pipeline.sh

## Running in non-Hive mode


	perl scripts/rnac_loader.pl <options>

or

	source scripts/rnac_loader_wrapper.sh

## License

See [LICENSE](https://github.com/RNAcentral/rnacentral-import-pipeline/blob/master/LICENSE) for more information.
<commit_msg>Add note on running luigi tests<commit_after># RNAcentral data import pipeline

## About

This is the main pipeline that is used internally for loading the data into the RNAcentral database.
[More information](http://www.ebi.ac.uk/seqdb/confluence/display/RNAC/RNAcentral+data+import+pipeline)

## Installation

### Perl dependencies

-   BioPerl
-   DBI
-   DBD::Oracle
-   Log4perl

### Installation

```
git clone https://github.com/RNAcentral/rnacentral-import-pipeline.git
cd rnacentral-import-pipeline
# initialize submodules
git submodule init
git submodule update
# add sensitive connection details to config/hive_params
cp config/hive_params_template config/hive_params
```

## Running in Hive mode

	source scripts/hive_pipeline.sh

## Running in non-Hive mode


	perl scripts/rnac_loader.pl <options>

or

	source scripts/rnac_loader_wrapper.sh

## Testing

The tests can be run using [py.test](http://pytest.org). For example, running
Ensembl importing tests can be done with:

```sh
py.test luigi/tests/ensembl_test.py
```

## License

See [LICENSE](https://github.com/RNAcentral/rnacentral-import-pipeline/blob/master/LICENSE) for more information.

{'fraction_non_alphanumeric': 0.09774436090225563, 'fraction_numerical': 0.0009398496240601503, 'mean_word_length': 4.738544474393531, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2513141', 'n_tokens_mistral': 701, 'n_tokens_neox': 663, 'n_words': 180}