<commit_before># NUS Scraper

This folder contains the scraper which produces our [v2 API data][api-v2] from internal NUS APIs.

**Note:** as this project gets its data from internal NUS APIs, you won't be able to contribute to this project without having the API domain and auth keys. Sadly, at the request of the university, we won't be able to release this information to non-core developers. If you have any ideas to improve the scraper, please [create an issue](https://github.com/nusmodifications/nusmods/issues/new/choose) instead.

## Getting Started

Node 10 or above is required. We use Node 10 in production.

Use `yarn` to install dependencies, then set up `env.json` with all the necessary keys and API base URL, then run the test script to check the setup is okay.

```
yarn

cp env.example.json
vim env.example.json

yarn dev help
yarn dev test | yarn bunyan
```

### Setting up ElasticSearch

We use ElasticSearch for our module search page. For local development it is not necessary to set this up because the scraper will automatically fall back to storing all data on the file system. To set up the ElasticSearch config, simple specify the `elasticConfig` key in `env.json` with the necessary configuration options that will be passed into the ElasticSearch client.

## Yarn Commands

### For production

- `build` - use Babel to compile the code for production
- `scrape` - run the scraper. See CLI commands below for a list of commands. Remember to use `NODE_ENV=production` in production.
- `docs` - copy the Swagger docs into the data folder

### For development

- `scrape` - run the scraper (see below for CLI commands). Note that the scraper has to be compiled through the `build` command first
- `dev` - compile and run the scraper. This is the same as running `build` and `scrape`
- `build` - compile the scraper using the TypeScript compiler
- `bunyan` - pipe output from the scraper through this so they can be read on the CLI. This is an alias of `bunyan -L -o short --color` - use local timestamp, short output format and color formatting. Run `yarn bunyan --help` to see all options.
- `test` - run all unit and integration tests
  - `test:watch` - run tests in watch mode, which runs only when code is changed
- `lint` - run both linter and type checker
  - `lint:code` - lint the code through ESLint

<commit_msg>Clean README for scrapers/nus-v2 (#2211)<commit_after># NUS Scraper

This folder contains the scraper which produces our [v2 API data][api-v2] from internal NUS APIs.

**Note:** as this project gets its data from internal NUS APIs, you won't be able to contribute to this project without having the API domain and auth keys. Sadly, at the request of the university, we won't be able to release this information to non-core developers. If you have any ideas to improve the scraper, please [create an issue](https://github.com/nusmodifications/nusmods/issues/new/choose) instead.

## Getting Started

Node 10 or above is required. We use Node 10 in production.

Use `yarn` to install dependencies, then set up `env.json` with all the necessary keys and API base URL, then run the test script to check the setup is okay.

```sh
yarn

cp env.example.json env.json
vim env.json  # add your appKey, studentKey and the baseUrl here

yarn dev help
yarn dev test | yarn bunyan
```

### Setting up ElasticSearch

We use ElasticSearch for our module search page. For local development it is not necessary to set this up because the scraper will automatically fall back to storing all data on the file system. To set up the ElasticSearch config, simple specify the `elasticConfig` key in `env.json` with the necessary configuration options that will be passed into the ElasticSearch client.

## Yarn Commands

### For production

- `build` - use Babel to compile the code for production
- `scrape` - run the scraper. See CLI commands below for a list of commands. Remember to use `NODE_ENV=production` in production.
- `docs` - copy the Swagger docs into the data folder

### For development

- `scrape` - run the scraper (see below for CLI commands). Note that the scraper has to be compiled through the `build` command first
- `dev` - compile and run the scraper. This is the same as running `build` and `scrape`
- `build` - compile the scraper using the TypeScript compiler
- `bunyan` - pipe output from the scraper through this so they can be read on the CLI. This is an alias of `bunyan -L -o short --color` - use local timestamp, short output format and color formatting. Run `yarn bunyan --help` to see all options.
- `test` - run all unit and integration tests
  - `test:watch` - run tests in watch mode, which runs only when code is changed
- `lint` - run both linter and type checker
  - `lint:code` - lint the code through ESLint


{'fraction_non_alphanumeric': 0.06802009208874006, 'fraction_numerical': 0.0035579740477187107, 'mean_word_length': 3.93698347107438, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5077762', 'n_tokens_mistral': 1341, 'n_tokens_neox': 1283, 'n_words': 736}