<commit_before># Task grader
This simple tool manages every step of grading a contest task, from the generation of test data to the grading of a solution output.

## Installing

Install dependencies: on Debian/stable:

    apt install build-essential sharutils python2.7 fp-compiler gcj-4.9 cgroup-tools

you also need to have a binary called `gcj` (not provided by Debian/stable):

    ln -s /usr/bin/gcj-4.9 /usr/bin/gcj

Execute `install.sh` in the taskgrader directory to install. It will help you install everything. If needed, modify `config.py` to suit your needs.

## Executing
The taskgrader itself can be executed with

    python taskgrader.py

It will wait for an input JSON on its standard input, then proceed to evaluation and then output the result JSON on standard output. Read `schema_input.json` and `schema_output.json` for a description of the expected formats.

You can also use the grading tool, executed with

    python grade.py [option]... FILE...

Full usage instructions are given by `grade.py -h`.

## How does it work?

Here's a description of the evaluation process, managed by the function `evaluation(evaluationParams)`.

* `evaluationParams` is the input JSON
* A build folder is created for the evaluation
* The `defaultParams.json` file from the task is read and its variables added
* A `dictWithVars` is created to add the variables, and returns the JSON data as if variables were replaced by their values
* `generators` are compiled
* `generations` describe how `generators` are to be executed in order to generate all the test files
* Some `extraTests` are "manually" added into the tests pool
* The `sanitizer` and the `checker` are compiled
* The `solutions` are compiled
* For each `execution`, we check whether the test file passes the `sanitizer` test, then we execute the solution with the test file as standard input and save the output, and we use the `checker` to grade the solution
* We return the evaluation report

## Evaluation components

The evaluation is made against a task which has multiple components.

### Generators

The `generators` are generating the testing environment. They are executed, optionally with various parameters, to generate files, which can be:

* test files: inputs for the solution, and if necessary, expected results
* libraries, for the compilation and execution of solutions

Some of these files can be passed directly in the evaluation JSON, without the need of a generator.

### Sanitizer

The `sanitizer` checks whether a test input is valid. It expects the test input on its stdin, and its exit code indicates the validity of the data.

### Checker

The `checker` checks whether the output of a solution corresponds to the expected result. It expects three arguments on the command line:

* `test.solout` the solution output
* `test.in` the reference input
* `test.out` the reference output

All checkers are passed these three arguments, whether they use it or not. The checker outputs the grading of the solution; its exit code can indicate an error while checking (invalid arguments, missing files, ...).

## Grading a solution<commit_msg>Better documentation on filterTests<commit_after># Task grader
This simple tool manages every step of grading a contest task, from the generation of test data to the grading of a solution output.

## Installing

Install dependencies: on Debian/stable:

    apt install build-essential sharutils python2.7 fp-compiler gcj-4.9 cgroup-tools

you also need to have a binary called `gcj` (not provided by Debian/stable):

    ln -s /usr/bin/gcj-4.9 /usr/bin/gcj

Execute `install.sh` in the taskgrader directory to install. It will help you install everything. If needed, modify `config.py` to suit your needs.

## Executing
The taskgrader itself can be executed with

    python taskgrader.py

It will wait for an input JSON on its standard input, then proceed to evaluation and then output the result JSON on standard output. Read `schema_input.json` and `schema_output.json` for a description of the expected formats.

You can also use the grading tool, executed with

    python grade.py [option]... FILE...

Full usage instructions are given by `grade.py -h`.

## How does it work?

Here's a description of the evaluation process, managed by the function `evaluation(evaluationParams)`.

* `evaluationParams` is the input JSON
* A build folder is created for the evaluation
* The `defaultParams.json` file from the task is read and its variables added
* A `dictWithVars` is created to add the variables, and returns the JSON data as if variables were replaced by their values
* `generators` are compiled
* `generations` describe how `generators` are to be executed in order to generate all the test files and optional libraries
* `extraTests` are added into the tests pool
* The `sanitizer` and the `checker` are compiled
* The `solutions` are compiled
* All `executions` are done for the solutions
* The full evaluation report is returned on standard output

### Executions

Each execution is the grading of one solution against multiple test files. For each `execution`:
* Test files corresponding to `filterTests` are selected, then for each test file:
* It passes first the `sanitizer` test
* Then the solution is executed, with the test file as standard input and the output saved
* Finally the `checker` grades the solution according to its output on that particular test file

`filterTests` is a list of globs (as `"test*.in"` or `"mytest.in"`) selecting test files to use among all the test files generated by the generators, and the `extraTests` given. One can specify directly test files into this array to use only specific ones.

## Evaluation components

The evaluation is made against a task which has multiple components.

### Generators

The `generators` are generating the testing environment. They are executed, optionally with various parameters, to generate files, which can be:

* test files: inputs for the solution, and if necessary, expected results
* libraries, for the compilation and execution of solutions

Some of these files can be passed directly in the evaluation JSON, without the need of a generator.

### Sanitizer

The `sanitizer` checks whether a test input is valid. It expects the test input on its stdin, and its exit code indicates the validity of the data.

### Checker

The `checker` checks whether the output of a solution corresponds to the expected result. It expects three arguments on the command line:

* `test.solout` the solution output
* `test.in` the reference input
* `test.out` the reference output

All checkers are passed these three arguments, whether they use it or not. The checker outputs the grading of the solution; its exit code can indicate an error while checking (invalid arguments, missing files, ...).

## Grading a solution
{'fraction_non_alphanumeric': 0.0584958217270195, 'fraction_numerical': 0.0017592728338953233, 'mean_word_length': 4.405705229793978, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1577283', 'n_tokens_mistral': 1726, 'n_tokens_neox': 1686, 'n_words': 1003}