<commit_before># WebWeeder

A command-line tool for crawling and scraping information from websites for discourse analysis.

There are two main parts to the application:

- `webcrawl` - Crawls a list of domains, downloads the raw HTML for each matching page and stores it alongside some metadata in JSON format
- `webweed` - Extracts the required information from the raw HTML files and presents it in a usable format
- `config.py` - User-editable configuration file which contains configuration defaults and also defines how each domain should be crawled

This is a work in progress.

Both Windows and Linux are supported.
Python 2 is not supported.

## Usage

These instructions are for systems running Windows.
Adapt the instructions as needed to suit a Linux system.

### Install

1. Install the latest version of Python 3 from [here](https://python.org/downloads/) (enable the "Add Python 3.x to PATH" option when prompted)
1. Install the latest version of Git from [here](https://git-scm.com/downloads)
1. Create a folder named `WebWeeder` where you wish to install the application (just on your desktop is fine)
1. Open the folder and `Shift+RightClick`->`Open Command Window Here`
1. Enter these commands (press enter and wait for it to complete after each one):
    1. `python3 -m venv ./venv`
    1. `venv/bin/pip install -e git+https://github.com/jakemarsden/WebWeeder.git ./`
    1. Ensure it works: `webcrawl --help`

### Update

1. Open the `WebWeeder` folder and `Shift+RightClick`->`Open Command Window Here`
1. Enter these commands (press enter and wait for it to complete after each one):
    1. `venv/bin/pip install -e git+https://github.com/jakemarsden/WebWeeder.git ./`
    1. Ensure it still works: `webcrawl --help`

### Uninstall

1. Remove the `WebWeeder` folder you created during installation
1. Optionally uninstall Python 3
1. Optionally uninstall Git
<commit_msg>Removed erroneous install instructions from README<commit_after># WebWeeder

A command-line tool for crawling and scraping information from websites for discourse analysis.

There are two main parts to the application:

- `webcrawl` - Crawls a list of domains, downloads the raw HTML for each matching page and stores it alongside some metadata in JSON format
- `webweed` - Extracts the required information from the raw HTML files and presents it in a usable format
- `config.py` - User-editable configuration file which contains configuration defaults and also defines how each domain should be crawled

This is a work in progress.

Both Windows and Linux are supported.
Python 2 is not supported.

{'fraction_non_alphanumeric': 0.06784888203546646, 'fraction_numerical': 0.008095605242868157, 'mean_word_length': 4.138613861386139, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1479632', 'n_tokens_mistral': 711, 'n_tokens_neox': 676, 'n_words': 363}