<commit_before># SftpExtractor

## 1) Installation

Add this line to your application's Gemfile:

```ruby
gem 'sftp_extractor'
```

### Development and Testing

To test the `sftp-extractor` you'll need to have a SFTP server running your machine. To do that on Mac OS X follow [this instructions](http://www.gooze.eu/howto/using-openssh-with-smartcards/openssh-server-on-mac-os-x).

Where it says 'Allow access for:' by default is `Administrators` which I changed for my username.

Test your port 22 is now opened:

```bash
$ nc -zv localhost 22

# => Connection to localhost port 22 [tcp/ssh] succeeded!
```

or simply

```bash
$ sftp nunocosta@127.0.0.1
Password:
Connected to 127.0.0.1.
sftp> dir
Applications  Desktop       Documents     Downloads     Google Drive  Library       Movies        Music         Pictures      Public
```

## 2) Usage

```bash
source /srv/rails-env/env.sh && cd /srv/sftp-extractor && time ruby bin/sftp_extractor.rb -c conf/campaign_cosy.yml -e $rails_env
```

### Development:

```bash
ruby bin/sftp_extractor.rb -c conf/template.yml -e development
```

## 3) Settings

__TO BE TRANSLATED__

| *chave* | *descrição* | *opcional?* |
| ------------- |-------------|:-----:|
| credentials[:server] | IP / hostname do servidor SFTP | |
| credentials[:user] | username | |
| credentials[:pwd] | password | |
| credentials[:timeout] | parametro aceite pela classe Net::SSH (funciona apenas no CONNECT) | |
| root | pasta relativa à home, onde estão os ficheiros que queremos extrair | |
| folder[:in] | lista de patterns que filtram os ficheiros que se pretendem extrair | |
| folder[:local_out] | pasta relativa onde vão ser gravados os ficheiros depois de extraidos  | |
| folder[:on_success] | depois de extrair os ficheiros move-os para esta pasta no SFTP | |
| folder[:on_success][:move_to] | pasta para onde serão movidos os ficheiros que foram copiados | X |
| folder[:on_success][:cleanup] | período ao fim do qual remove os ficheiros da pasta definida em folder[:on_success][:move_to]. Exemplo: '6 hours' | X |
| folder[:extraction_mode] | Opções: *'last'*: extrai o ficheiro mais recente com o PATTERN;  *'full'*: extrai todos os ficheiros existentes com o PATTERN) | X |
| retry_on_error[:max_times] | quantas vezes espera 'period' até que todos os ficheiros dos patterns tenham sido extraidos | X |
| retry_on_error[:period] | se pelo menos um dos ficheiros não tiver sido bem extraido, espera esta quantidade de tempo | X |


```yaml
# conf/campaign_cosy.yml

production:
  credentials:
    server: xx.xxx.xxx.xxx
    user: xXxXxXx
    pwd: <PASSWORD>
    timeout: 20
  root: "."
  folder:
    in: { patterns: ["*customers.csv"] }
    local_out: incoming/campaign_cosy
    on_success:
      move_to: PROCESSED
      cleanup: 6 hours
    extraction_mode: full
  logger:
    path: /var/log/sftp-extractor/campaign_cosy.log
    level: info
  retry_on_error:
    mail_config:
      to:
        - <EMAIL>
        - <EMAIL>
      subject: "Extraction from SFTP failed (CUSTOMERS LIST)"
      body: "Execution Log:"
```

## Contributing

1. [Fork it]( https://github.com/nunommc/sftp_extractor/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request<commit_msg>Updating documentation<commit_after># SftpExtractor

## 1) Installation

Add this line to your application's Gemfile:

```ruby
gem 'sftp_extractor'
```

### Before you start...

To test the `sftp-extractor` you'll need to have a SFTP server running your machine. To do that on Mac OS X follow [this instructions](http://www.gooze.eu/howto/using-openssh-with-smartcards/openssh-server-on-mac-os-x).

Where it says 'Allow access for:' by default is `Administrators` which I changed for my username.

Test your port 22 is now opened:
```bash
$ nc -zv localhost 22
# => Connection to localhost port 22 [tcp/ssh] succeeded!
```

or simply,
```bash
$ sftp nunommc@127.0.0.1
Password:
Connected to 127.0.0.1.
sftp> dir
Applications  Desktop       Documents     Downloads     Google Drive  Library       Movies        Music         Pictures      Public
```

## 2) Usage

```bash
source /srv/rails-env/env.sh && cd /srv/sftp-extractor && time ruby bin/sftp_extractor.rb -c conf/campaign.yml -e $rails_env
```

### Development:

```bash
ruby bin/sftp_extractor.rb -c conf/template.yml -e development
```

## 3) Settings

__TO BE TRANSLATED__


| *key*       |             | *description* | *optional* |
| ----------- | ----------- | ------------- | :--------: |
| **credentials** | :server     | SFTP server hostname/IP | |
|               | :user       | username | |
|               | :pwd        | password | |
|               | :timeout    | parameter accepted by Net::SSH constructor (applied only during the CONNECT) | |
|  **root**  | relative path to the default user's home, where are stored the files you want to extract | |
| **folder** | :in | list of patterns to filter the files you want to extract | |
|            | :local_out | relative path where the extracted files are should be saved  | |
|            | :on_success[:move_to] | after successfully extracting the files they'll be moved into this folder on the SFTP server | X |
|            | :on_success[:cleanup] | period of time after which files are going to be deleted from `folder[:on_success][:move_to]`. Example: `6 hours` | X |
|            | :extraction_mode | Options:<BR/> - **last**: extracts the most recent file matching the PATTERN;<BR/> - **full**: extracts every file matching the PATTERN | X |
| **retry_on_error** | :period | if at least one of the files wasn't successfully extracted, waits this amount of time | X |
|                    | :max_times | no. of times is going to wait  `period` time to retry the missing patterns | X |


```yaml
# conf/campaign.yml
production:
  credentials:
    server: xx.xxx.xxx.xxx
    user: xXxXxXx
    pwd: <PASSWORD>
    timeout: 20
  root: "."
  folder:
    in: { patterns: ["*customers.csv"] }
    local_out: incoming/campaign
    on_success:
      move_to: PROCESSED
      cleanup: 6 hours
    extraction_mode: full
  logger:
    path: /var/log/sftp-extractor/campaign.log
    level: info
  retry_on_error:
    mail_config:
      to:
        - <EMAIL>
        - <EMAIL>
      subject: "Extraction from SFTP failed (CUSTOMERS LIST)"
      body: "Execution Log:"
```

## Contributing

1. [Fork it]( https://github.com/nunommc/sftp_extractor/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request
{'fraction_non_alphanumeric': 0.12661421997921923, 'fraction_numerical': 0.00890604126465786, 'mean_word_length': 3.1085365853658535, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 15, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7260247', 'n_tokens_mistral': 2305, 'n_tokens_neox': 2155, 'n_words': 730}