<commit_before>INSTALL INSTRUCTIONS
=

Installing CrawText V5 
==
OSX and Debian Distributions
=== 

Previous recommandations 
====
1. Activate your new virtual env (no mandatory)
2. Make sure you have git installed
``` apt-get install git ```

Install V5 
=====
1. Clone the repo
``` git clone https://github.com/cortext/crawtext ```

2. Moove into crawtext directory
``` cd crawtext  ```

2. switch to V5 branch

``` git checkout V5 ```

3. Run the installer for your OS:

* debian-based distrib
``` chmod+x  deb-install.sh```
``` ./deb-install.sh```

* MAC OS-X distrib
``` chmod+x  mac-install.sh```
``` ./mac-install.sh```

Windows distribution 
===
Previous recommandations 
=====
As Crawtext is not compatible with MS-DOS version
you will have to install a Virtual Machine  and Docker

1. Install a VM
Install VirtualBox VM following the step on [https://www.virtualbox.org/]


2.Install Docker

Your machine must be running Windows 7.1, 8/8.1 or newer to run Docker. Windows 10 is not currently supported
Follow the step on [https://docs.docker.com/installation/windows/]

Install V5 
=====

1. Create the environnement for CrawText
```docker build -t crawtext ./install```

2. Activate the mongoengine in background

```sudo docker run  --name mongo-srv -v /home/c24b/projets/docker/data:/data/db -P -d mongo:latest```


Start a crawl 
===

For the moment you only have two options
* Create a small python script into the current dir
    nano new_project.py
    ```
    from crawtext import Crawtext
    params = {"key":"","query":"(mottanai) OR (motanai)", "lang": "fr", "repeat": True, "max_depth":2}
    c = Crawtext("mottanai")
    c.start(params)
    ```
    you can see also the [example.py] file
    a complete list of options will be soon available
    
* Use a YAML file and config.py
    YAML file + config.py
<commit_msg>Following INSTALL instruct tests<commit_after>INSTALL INSTRUCTIONS
==

Installing CrawText V5 
====
OSX and Debian Distributions
====== 

Previous recommandations 
========
1. Activate your new virtual env (no mandatory)
2. Make sure you have git installed
``` apt-get install git ```

Install V5 
==========
1. Clone the repo
``` git clone https://github.com/cortext/crawtext ```

2. Moove into crawtext directory
``` cd crawtext  ```

2. switch to V5 branch

``` git checkout V5 ```

3. Run the installer for your OS:

* debian-based distrib
``` chmod+x  deb-install.sh```
``` ./deb-install.sh```

* MAC OS-X distrib
``` chmod+x  mac-install.sh```
``` ./mac-install.sh```

Windows distribution 
======
Previous recommandations 
==========
As Crawtext is not compatible with MS-DOS version
you will have to install a Virtual Machine  and Docker

1. Install a VM
Install VirtualBox VM following the step on [https://www.virtualbox.org/]


2.Install Docker

Your machine must be running Windows 7.1, 8/8.1 or newer to run Docker. Windows 10 is not currently supported
Follow the step on [https://docs.docker.com/installation/windows/]

Install V5 
==========

1. Create the environnement for CrawText
```docker build -t crawtext ./install```

2. Activate the mongoengine in background

```sudo docker run  --name mongo-srv -v /home/c24b/projets/docker/data:/data/db -P -d mongo:latest```


Start a crawl 
======

For the moment you only have two options
* Create a small python script into the current dir
    nano new_project.py
    ```
    from crawtext import Crawtext
    params == {"key":"","query":"(mottanai) OR (motanai)", "lang": "fr", "repeat": True, "max_depth":2}
    c == Crawtext("mottanai")
    c.start(params)
    ```
    you can see also the [example.py] file
    a complete list of options will be soon available
    
* Use a YAML file and config.py
    YAML file + config.py

{'fraction_non_alphanumeric': 0.13149437600428496, 'fraction_numerical': 0.01339046598821639, 'mean_word_length': 3.6455223880597014, 'pattern_counts': {'":': 10, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2577932', 'n_tokens_mistral': 1258, 'n_tokens_neox': 1167, 'n_words': 449}