<commit_before>* dlt tools 

** Introduction 
Basically contains 2 Clis 
- eodn_listner  : Listen for new scenes from host using websocket and download when a new file is uploaded
- eodn_download : Download the existing scenes specified in the list


** Arguments 
It contains the following arguments:
- -s , --scenes  : comma separated list of scenes
- -H , --host    : host to get scenes from Eg: http://dev.crest.iu.edu:8888/exnodes
- -r , --regex   : Pass a regex to fetch by ~/exnodes?metadata.scene=reg=/<Argumnet>~
		   Download everything that matches the regex

** Using path to download folders 

Currently folder like downloads with wildcards are only supported in the downloader and can be used in the following way , where -p indicates that it is a path and -F indicates the filter

#+begin_SRC bash
eodn_download -p Landsat/LC8/006/040/2016/ -F "*.png"

# Recurrsively download all the files under the folder (No way to download only the immediate files below the folder exist in this way)
eodn_download -p Landsat/LC8/006/ -F "*.png"

#+end_SRC


** General testing procedure [ Vagrant ]

Testing requires installation of lors-tools and python. Currently dlt-tools supported systems is restricted by the support for lors-tools and hence the following are supported :
- Centos 6,7 - 32 or 64 bit
- RHEL 6,7

All the various environments are present as vagrant file with an init script to install dlt-lors and dlt-tools from the yum repo. Additionaly the home folder has been shared as /e , so this can be used to directly install and try dlt-tools.

Run src_bash[:exports code]{vagrant up && vagrant ssh} on these images and use the shell to test it out.


<commit_msg>fix org code<commit_after>* dlt tools 

** Introduction 
Basically contains 2 Clis 
- eodn_listner  : Listen for new scenes from host using websocket and download when a new file is uploaded
- eodn_download : Download the existing scenes specified in the list


** Arguments 
It contains the following arguments:
- -s , --scenes  : comma separated list of scenes
- -H , --host    : host to get scenes from Eg: http://dev.crest.iu.edu:8888/exnodes
- -r , --regex   : Pass a regex to fetch by ~/exnodes?metadata.scene=reg=/<Argumnet>~
		   Download everything that matches the regex

** Using path to download folders 

Currently folder like downloads with wildcards are only supported in the downloader and can be used in the following way , where -p indicates that it is a path and -F indicates the filter

#+begin_SRC bash
eodn_download -p Landsat/LC8/006/040/2016/ -F "*.png"

# Recurrsively download all the files under the folder (No way to download only the immediate files below the folder exist in this way)
eodn_download -p Landsat/LC8/006/ -F "*.png"

#+end_SRC


** General testing procedure [ Vagrant ]

Testing requires installation of lors-tools and python. Currently dlt-tools supported systems is restricted by the support for lors-tools and hence the following are supported :
- Centos 6,7 - 32 or 64 bit
- RHEL 6,7

All the various environments are present as vagrant file with an init script to install dlt-lors and dlt-tools from the yum repo. Additionaly the home folder has been shared as /e , so this can be used to directly install and try dlt-tools.

Run ~vagrant up && vagrant ssh~ on these images and use the shell to test it out.



{'fraction_non_alphanumeric': 0.06552962298025135, 'fraction_numerical': 0.016756433273488927, 'mean_word_length': 3.735127478753541, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7000286', 'n_tokens_mistral': 1008, 'n_tokens_neox': 931, 'n_words': 481}