<commit_before>RedPen Server
=============

The RedPen server delivers the majority of RedPen's functionality via a simple HTTP REST API.

Starting the RedPen server
--------------------------

Please refer to the :doc:`command` page for details on how to start the RedPen server.

RedPen Server API
-----------------

Configuration
~~~~~~~~~~~~~

**/rest//config/redpens**

Return the configuration for available, preconfigured redpens.

GET Parameters:

- *lang=xx* restricts the returned configurations to those that match the specified language. By default, all configurations are returned.

The JSON response is as follows:

.. code-block:: json

    {
      "version": "1.1.2",
      "documentParsers": ["PLAIN", "MARKDOWN", "WIKI"],
      "redpens": {
         "en": {
          "lang": "en",
          "tokenizer": "cc.redpen.tokenizer.WhiteSpaceTokenizer",
          "validators": {
             "CommaNumber": { "languages": [], "properties": {} },
             "Contraction": { "languages": ["en"], "properties": {} },
             "DoubledWord": { "languages": [], "properties": {} },
             "EndOfSentence": { "languages": ["en"], "properties": {} },
             "InvalidExpression": { "languages": [], "properties": {} },
             "InvalidSymbol": { "languages": [], "properties": {} },
             "InvalidWord": { "languages": ["en"], "properties": {} },
             "ParagraphNumber": { "languages": [], "properties": {} },
             "Quotation": { "languages": ["en"], "properties": {} },
             "SectionLength": { "languages": [], "properties": {"max_char_num": "2000"} },
             "SentenceLength": { "languages": [], "properties": {"max_len": "200"} },
             "SpaceBetweenAlphabeticalWord": { "languages": [], "properties": {} },
             "Spelling": { "languages": [], "properties": {} },
             "StartWithCapitalLetter": { "languages": ["en"], "properties": {} },
             "SuccessiveWord": { "languages": [], "properties": {} },
             "SymbolWithSpace": { "languages": [], "properties": {} },
             "WordNumber": { "languages": [], "properties": {} }
          }
        },
        "ja": {
          "lang": "ja",
          "tokenizer": "cc.redpen.tokenizer.JapaneseTokenizer",
          "validators": {
             "CommaNumber": { "languages": [], "properties": {} },
             "DoubledWord": { "languages": [], "properties": {} },
             "HankakuKana": { "languages": ["ja"], "properties": {} },
             "InvalidSymbol": { "languages": [], "properties": {} },
             "KatakanaEndHyphen": { "languages": ["ja"], "properties": {} },
             "KatakanaSpellCheck": { "languages": ["ja"], "properties": {} },
             "ParagraphNumber": { "languages": [], "properties": {} },
             "SectionLength": { "languages": [], "properties": {"max_num": "1500"} },
             "SentenceLength": { "languages": [], "properties": {"max_len": "100"} },
             "SpaceBetweenAlphabeticalWord": { "languages": [], "properties": {} },
             "SuccessiveWord": { "languages": [], "properties": {} }
          }
        }
      }
    }


- The *version* property indicates the version of RedPen.
- The *documentParsers* array contains all supported document parsers
- The *redpens* object shows the available pre-configured redpens and how they are configured. Within each object:

   - *lang* specifies the language the redpen is designed for
   - *tokenizer* specifies the tokenizer class used by the redpen
   - *validators* shows which validators are configured within the redpen. This object is in a format suitable for the *document/validate/json* request below. For each validator:

       - The *languages* array indicates which languages for which the validator is suitable. An empty array indicates all languages.
       - The *properties* object specifies the currently configured properties for this validator, as described in :doc:`validator`


Document Validation
~~~~~~~~~~~~~~~~~~~

**/document/validate**

This POST request validates a document and returns the errors as JSON.

POST Parameters:

- *document* contains the text of the document RedPen is to validate
- *documentParser* specifies which parser should be used to parse the document. Valid options are:
    - PLAIN
    - MARKDOWN
    - WIKI
- *lang* specifies the language used to tokenize the document. Currently, values of **ja** (Japanese) and **en** (English/Whitespace) are supported.

**/document/validate/json**

This POST request processes a redpen validation request, specified in JSON, and returns redpen errors in a supported RedPen format.

Request format:

.. code-block:: json

    {
      "document": "Theyre is a blak rownd borl.",
      "lang": "en",
      "format": "json2",
      "documentParser": "PLAIN",
      "config": {
        "CommaNumber": {},
        "Contraction": {},
        "DoubledWord": {},
        "EndOfSentence": {},
        "InvalidExpression": {},
        "InvalidSymbol": {},
        "InvalidWord": {},
        "ParagraphNumber": {},
        "Quotation": {},
        "SectionLength": {
          "properties": {
            "max_char_num": "2000"
          }
        },
        "SentenceLength": {
          "properties": {
            "max_len": "200"
          }
        },
        "SpaceBetweenAlphabeticalWord": {},
        "Spelling": {},
        "StartWithCapitalLetter": {},
        "SuccessiveWord": {},
        "SymbolWithSpace": {},
        "WordNumber": {}
      }
    }


- The *document* property specifies the text of the document to validate
- The *lang* property indicates the language of the document. It determines how the document will be tokenized by RedPen.
- The *documentParser* property should contain the name of a valid RedPen documentparser (ie: PLAIN, MARKDOWN or WIKI)
- The *format* property determines the format for the results. It can be one of json, json2, plain, plain2 or xml.
- The *config* object specifies the validator configuration for the request. It consists of a series of objects that are named after a RedPen validator. If the object is present, the validator will be configured. Within this named object, a *properties* object can be used to set the name and values of any property used by the validator, as described in :doc:`validator`

Response (json2 format):

.. code-block:: json

    {
      "errors": [
        {
          "sentence": "Theyre is a blak rownd borl.",
          "position": {
            "start": {
              "offset": 0,
              "line": 1
            },
            "end": {
              "offset": 27,
              "line": 1
            }
          },
          "errors": [
            {
              "subsentence": {
                "offset": 0,
                "length": 6
              },<commit_msg>Changes to JSON validation request - Changed to reflect new format - Added symbol table override to the example<commit_after>RedPen Server
=============

The RedPen server delivers the majority of RedPen's functionality via a simple HTTP REST API.

Starting the RedPen server
--------------------------

Please refer to the :doc:`command` page for details on how to start the RedPen server.

RedPen Server API
-----------------

Configuration
~~~~~~~~~~~~~

**/rest//config/redpens**

Return the configuration for available, preconfigured redpens.

GET Parameters:

- *lang=xx* restricts the returned configurations to those that match the specified language. By default, all configurations are returned.

The JSON response is as follows:

.. code-block:: json

    {
      "version": "1.1.2",
      "documentParsers": ["PLAIN", "MARKDOWN", "WIKI"],
      "redpens": {
         "en": {
          "lang": "en",
          "tokenizer": "cc.redpen.tokenizer.WhiteSpaceTokenizer",
          "validators": {
             "CommaNumber": { "languages": [], "properties": {} },
             "Contraction": { "languages": ["en"], "properties": {} },
             "DoubledWord": { "languages": [], "properties": {} },
             "EndOfSentence": { "languages": ["en"], "properties": {} },
             "InvalidExpression": { "languages": [], "properties": {} },
             "InvalidSymbol": { "languages": [], "properties": {} },
             "InvalidWord": { "languages": ["en"], "properties": {} },
             "ParagraphNumber": { "languages": [], "properties": {} },
             "Quotation": { "languages": ["en"], "properties": {} },
             "SectionLength": { "languages": [], "properties": {"max_char_num": "2000"} },
             "SentenceLength": { "languages": [], "properties": {"max_len": "200"} },
             "SpaceBetweenAlphabeticalWord": { "languages": [], "properties": {} },
             "Spelling": { "languages": [], "properties": {} },
             "StartWithCapitalLetter": { "languages": ["en"], "properties": {} },
             "SuccessiveWord": { "languages": [], "properties": {} },
             "SymbolWithSpace": { "languages": [], "properties": {} },
             "WordNumber": { "languages": [], "properties": {} }
          }
        },
        "ja": {
          "lang": "ja",
          "tokenizer": "cc.redpen.tokenizer.JapaneseTokenizer",
          "validators": {
             "CommaNumber": { "languages": [], "properties": {} },
             "DoubledWord": { "languages": [], "properties": {} },
             "HankakuKana": { "languages": ["ja"], "properties": {} },
             "InvalidSymbol": { "languages": [], "properties": {} },
             "KatakanaEndHyphen": { "languages": ["ja"], "properties": {} },
             "KatakanaSpellCheck": { "languages": ["ja"], "properties": {} },
             "ParagraphNumber": { "languages": [], "properties": {} },
             "SectionLength": { "languages": [], "properties": {"max_num": "1500"} },
             "SentenceLength": { "languages": [], "properties": {"max_len": "100"} },
             "SpaceBetweenAlphabeticalWord": { "languages": [], "properties": {} },
             "SuccessiveWord": { "languages": [], "properties": {} }
          }
        }
      }
    }


- The *version* property indicates the version of RedPen.
- The *documentParsers* array contains all supported document parsers
- The *redpens* object shows the available pre-configured redpens and how they are configured. Within each object:

   - *lang* specifies the language the redpen is designed for
   - *tokenizer* specifies the tokenizer class used by the redpen
   - *validators* shows which validators are configured within the redpen. This object is in a format suitable for the *document/validate/json* request below. For each validator:

       - The *languages* array indicates which languages for which the validator is suitable. An empty array indicates all languages.
       - The *properties* object specifies the currently configured properties for this validator, as described in :doc:`validator`


Document Validation
~~~~~~~~~~~~~~~~~~~

**/document/validate**

This POST request validates a document and returns the errors as JSON.

POST Parameters:

- *document* contains the text of the document RedPen is to validate
- *documentParser* specifies which parser should be used to parse the document. Valid options are:
    - PLAIN
    - MARKDOWN
    - WIKI
- *lang* specifies the language used to tokenize the document. Currently, values of **ja** (Japanese) and **en** (English/Whitespace) are supported.

**/document/validate/json**

This POST request processes a redpen validation request, specified in JSON, and returns redpen errors in a supported RedPen format.

Request format:

.. code-block:: json

    {
      "document": "Theyre is a blak rownd borl.",
      "format": "json2",
      "documentParser": "PLAIN",
      "config": {
        "lang": "en",
        "validators": {
          "CommaNumber": {},
          "Contraction": {},
          "DoubledWord": {},
          "EndOfSentence": {},
          "InvalidExpression": {},
          "InvalidSymbol": {},
          "InvalidWord": {},
          "ParagraphNumber": {},
          "Quotation": {},
          "SectionLength": {
            "properties": {
              "max_char_num": "2000"
            }
          },
          "SentenceLength": {
            "properties": {
              "max_len": "200"
            }
          },
          "SpaceBetweenAlphabeticalWord": {},
          "Spelling": {},
          "StartWithCapitalLetter": {},
          "SuccessiveWord": {},
          "SymbolWithSpace": {},
          "WordNumber": {}
        },
        "symbols": {
          "AMPERSAND": {
            "after_space": false,
            "before_space": true,
            "invalid_chars": "＆",
            "value": "&"
          },
          "ASTERISK": {
            "after_space": true,
            "before_space": true,
            "invalid_chars": "＊",
            "value": "*"
          }
        }
      }
    }


- The *document* property specifies the text of the document to validate
- The *documentParser* property should contain the name of a valid RedPen documentparser (ie: PLAIN, MARKDOWN or WIKI)
- The *format* property determines the format for the results. It can be one of json, json2, plain, plain2 or xml.
- The *config* object specifies the validator configuration for the request. This consists of:
    - A *config* object, consisting of a series of objects that are named after a RedPen validator. If the object is present, the validator will be configured. Within this named object, a *properties* object can be used to set the name and values of any property used by the validator, as described in :doc:`validator`
    - The *lang* property indicates the language of the document. It determines how the document will be tokenized by RedPen.
    - A *symbols* object containing overridden symbols, as described in :doc:`configuration`. Each entry must be a validate symbol name, and can contain the following elements:
        - *value* specifies the Symbol's value
        - *invalid_chars* is a string of invalid alternatives for this Symbol
        - *before_space* and *after_space* specify if a space is required before or after the Symbol.

Response (json2 format):

.. code-block:: json

    {
      "errors": [
        {
          "sentence": "Theyre is a blak rownd borl.",
          "position": {
            "start": {
              "offset": 0,
              "line": 1
            },
            "end": {
              "offset": 27,
              "line": 1
            }
          },
          "errors": [
            {
              "subsentence": {
                "offset": 0,
                "length": 6
              },
{'fraction_non_alphanumeric': 0.15262114687991252, 'fraction_numerical': 0.004784361971157132, 'mean_word_length': 2.417893015650549, 'pattern_counts': {'":': 288, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5902376', 'n_tokens_mistral': 3955, 'n_tokens_neox': 3818, 'n_words': 1262}