<commit_before># Paatos-Scraper
Scraper to scrape decision data from web pages and PDF documents in Finland

## Usage

### Options

    Paatos-Scraper

      Scraper to scrape decision data from web pages and PDF documents in Finland. 

    Options

      --source source                The source where to retrieve the date. Supported sources are: oulu 
      --output-zip output zip file   The target zip-file where retrieved data will be stored            
      --organization-id              Organization id in source system format                            
      --max-events                   Limit number of events                                             
      --help                         Print this usage guide. 

### Installing dependencies

Before running the application you must install dependecies.

- `npm install`

### Example

Extracts latest event from Oulu's Kaupunginhallitus decision data

- `node app.js --source oulu --organization-id 690 --max-events=1 --output-zip=/tmp/output.zip`

## Running tests

Before running tests make sure that you have installed dependencies by running following command:

- `npm install`<commit_msg>Updated usage in README<commit_after># Paatos-Scraper
Scraper to scrape decision data from web pages and PDF documents in Finland

## Usage

### Options

    Paatos-Scraper

      Scraper to scrape decision data from web pages and PDF documents in Finland. 

    Options

      --source source                The source where to retrieve the date. Supported sources are: oulu,vantaa     
      --print-organizations          Print organizations and exit without extracting data                          
      --output-zip output zip file   The target zip-file where retrieved data will be stored                       
      --organization-id              Organization id in source system format                                       
      --max-events                   Limit number of events                                                        
      --pdf-download-interval        Interval between downloading PDF-files in milliseconds. Defaults to 100ms     
      --html-download-interval       Interval between downloading HTML-pages in milliseconds. Defaults to 10ms     
      --events-after                 Extract only events after specified date. Date should be formatted in         
                                     following format: YYYY-MM-DD                                                  
      --error-log                    Path to error log file. By default errors are written into console.           
      --help                         Print this usage guide.

### Installing dependencies

Before running the application you must install dependecies.

- `npm install`

### Example

Extracts latest event from Oulu's Kaupunginhallitus decision data

- `node app.js --source oulu --organization-id 690 --max-events=1 --output-zip=/tmp/output.zip`

## Running tests

Before running tests make sure that you have installed dependencies by running following command:

- `npm install`
{'fraction_non_alphanumeric': 0.05206638464041653, 'fraction_numerical': 0.004230393752033843, 'mean_word_length': 1.7348754448398576, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4949507', 'n_tokens_mistral': 721, 'n_tokens_neox': 651, 'n_words': 302}