<commit_before>Lexical Analysis
================

Encoding
--------

All Nim source files are in the UTF-8 encoding (or its ASCII subset). Other
encodings are not supported. Any of the standard platform line termination
sequences can be used - the Unix form using ASCII LF (linefeed), the Windows
form using the ASCII sequence CR LF (return followed by linefeed), or the old
Macintosh form using the ASCII CR (return) character. All of these forms can be
used equally, regardless of platform.


Indentation
-----------

Nim's standard grammar describes an `indentation sensitive`:idx: language.
This means that all the control structures are recognized by indentation.
Indentation consists only of spaces; tabulators are not allowed.

The indentation handling is implemented as follows: The lexer annotates the
following token with the preceding number of spaces; indentation is not
a separate token. This trick allows parsing of Nim with only 1 token of
lookahead.

The parser uses a stack of indentation levels: the stack consists of integers
counting the spaces. The indentation information is queried at strategic
places in the parser but ignored otherwise: The pseudo terminal ``IND{>}``
denotes an indentation that consists of more spaces than the entry at the top
of the stack; ``IND{=}`` an indentation that has the same number of spaces. ``DED``
is another pseudo terminal that describes the *action* of popping a value
from the stack, ``IND{>}`` then implies to push onto the stack.

With this notation we can now easily define the core of the grammar: A block of
statements (simplified example)::

  ifStmt = 'if' expr ':' stmt
           (IND{=} 'elif' expr ':' stmt)*
           (IND{=} 'else' ':' stmt)?

  simpleStmt = ifStmt / ...

  stmt = IND{>} stmt ^+ IND{=} DED  # list of statements
       / simpleStmt                 # or a simple statement



Comments
--------

Comments start anywhere outside a string or character literal with the
hash character ``#``.
Comments consist of a concatenation of `comment pieces`:idx:. A comment piece
starts with ``#`` and runs until the end of the line. The end of line characters
belong to the piece. If the next line only consists of a comment piece with
no other tokens between it and the preceding one, it does not start a new
comment:


.. code-block:: nim
  i = 0     # This is a single comment over multiple lines.
    # The scanner merges these two pieces.
    # The comment continues here.


`Documentation comments`:idx: are comments that start with two ``##``.
Documentation comments are tokens; they are only allowed at certain places in
the input file as they belong to the syntax tree!


Multiline comments
------------------

Starting with version 0.13.0 of the language Nim supports multiline comments.
They look like:

.. code-block:: nim
  #[Comment here.
  Multiple lines
  are not a problem.]#

Multiline comments support nesting:

.. code-block:: nim
  #[  #[ Multiline comment in already
     commented out code. ]#
  proc p[T](x: T) = discard
  ]#

Multiline documentation comments also exist and support nesting too:

.. code-block:: nim
  proc foo =
    ##[Long documentation comment
    here.
    ]##


Identifiers & Keywords
----------------------

Identifiers in Nim can be any string of letters, digits
and underscores, beginning with a letter. Two immediate following
underscores ``__`` are not allowed::

  letter ::= 'A'..'Z' | 'a'..'z' | '\x80'..'\xff'
  digit ::= '0'..'9'
  IDENTIFIER ::= letter ( ['_'] (letter | digit) )*

Currently any Unicode character with an ordinal value > 127 (non ASCII) is
classified as a ``letter`` and may thus be part of an identifier but later
versions of the language may assign some Unicode characters to belong to the
operator characters instead.

The following keywords are reserved and cannot be used as identifiers:

.. code-block:: nim
   :file: ../keywords.txt

Some keywords are unused; they are reserved for future developments of the
language.


Identifier equality
-------------------

Two identifiers are considered equal if the following algorithm returns true:

.. code-block:: nim
  proc sameIdentifier(a, b: string): bool =
    a[0] == b[0] and
      a.replace(re"_|–", "").toLower == b.replace(re"_|–", "").toLower

That means only the first letters are compared in a case sensitive manner. Other
letters are compared case insensitively and underscores are ignored.

This rather unorthodox way to do identifier comparisons is called
`partial case insensitivity`:idx: and has some advantages over the conventional
case sensitivity:

It allows programmers to mostly use their own preferred
spelling style, be it humpStyle, snake_style or dash–style and libraries written
by different programmers cannot use incompatible conventions.
A Nim-aware editor or IDE can show the identifiers as preferred.
Another advantage is that it frees the programmer from remembering
the exact spelling of an identifier. The exception with respect to the first
letter allows common code like ``var foo: Foo`` to be parsed unambiguously.

Historically, Nim was a fully `style-insensitive`:idx: language. This meant that
it was not case-sensitive and underscores were ignored and there was no even a
distinction between ``foo`` and ``Foo``.


String literals
---------------
<commit_msg>Replaced en-dash with hyphen for identifier equality (#6272)<commit_after>Lexical Analysis
================

Encoding
--------

All Nim source files are in the UTF-8 encoding (or its ASCII subset). Other
encodings are not supported. Any of the standard platform line termination
sequences can be used - the Unix form using ASCII LF (linefeed), the Windows
form using the ASCII sequence CR LF (return followed by linefeed), or the old
Macintosh form using the ASCII CR (return) character. All of these forms can be
used equally, regardless of platform.


Indentation
-----------

Nim's standard grammar describes an `indentation sensitive`:idx: language.
This means that all the control structures are recognized by indentation.
Indentation consists only of spaces; tabulators are not allowed.

The indentation handling is implemented as follows: The lexer annotates the
following token with the preceding number of spaces; indentation is not
a separate token. This trick allows parsing of Nim with only 1 token of
lookahead.

The parser uses a stack of indentation levels: the stack consists of integers
counting the spaces. The indentation information is queried at strategic
places in the parser but ignored otherwise: The pseudo terminal ``IND{>}``
denotes an indentation that consists of more spaces than the entry at the top
of the stack; ``IND{=}`` an indentation that has the same number of spaces. ``DED``
is another pseudo terminal that describes the *action* of popping a value
from the stack, ``IND{>}`` then implies to push onto the stack.

With this notation we can now easily define the core of the grammar: A block of
statements (simplified example)::

  ifStmt = 'if' expr ':' stmt
           (IND{=} 'elif' expr ':' stmt)*
           (IND{=} 'else' ':' stmt)?

  simpleStmt = ifStmt / ...

  stmt = IND{>} stmt ^+ IND{=} DED  # list of statements
       / simpleStmt                 # or a simple statement



Comments
--------

Comments start anywhere outside a string or character literal with the
hash character ``#``.
Comments consist of a concatenation of `comment pieces`:idx:. A comment piece
starts with ``#`` and runs until the end of the line. The end of line characters
belong to the piece. If the next line only consists of a comment piece with
no other tokens between it and the preceding one, it does not start a new
comment:


.. code-block:: nim
  i = 0     # This is a single comment over multiple lines.
    # The scanner merges these two pieces.
    # The comment continues here.


`Documentation comments`:idx: are comments that start with two ``##``.
Documentation comments are tokens; they are only allowed at certain places in
the input file as they belong to the syntax tree!


Multiline comments
------------------

Starting with version 0.13.0 of the language Nim supports multiline comments.
They look like:

.. code-block:: nim
  #[Comment here.
  Multiple lines
  are not a problem.]#

Multiline comments support nesting:

.. code-block:: nim
  #[  #[ Multiline comment in already
     commented out code. ]#
  proc p[T](x: T) = discard
  ]#

Multiline documentation comments also exist and support nesting too:

.. code-block:: nim
  proc foo =
    ##[Long documentation comment
    here.
    ]##


Identifiers & Keywords
----------------------

Identifiers in Nim can be any string of letters, digits
and underscores, beginning with a letter. Two immediate following
underscores ``__`` are not allowed::

  letter ::= 'A'..'Z' | 'a'..'z' | '\x80'..'\xff'
  digit ::= '0'..'9'
  IDENTIFIER ::= letter ( ['_'] (letter | digit) )*

Currently any Unicode character with an ordinal value > 127 (non ASCII) is
classified as a ``letter`` and may thus be part of an identifier but later
versions of the language may assign some Unicode characters to belong to the
operator characters instead.

The following keywords are reserved and cannot be used as identifiers:

.. code-block:: nim
   :file: ../keywords.txt

Some keywords are unused; they are reserved for future developments of the
language.


Identifier equality
-------------------

Two identifiers are considered equal if the following algorithm returns true:

.. code-block:: nim
  proc sameIdentifier(a, b: string): bool =
    a[0] == b[0] and
      a.replace("_", "").toLower == b.replace("_", "").toLower

That means only the first letters are compared in a case sensitive manner. Other
letters are compared case insensitively and underscores are ignored.

This rather unorthodox way to do identifier comparisons is called
`partial case insensitivity`:idx: and has some advantages over the conventional
case sensitivity:

It allows programmers to mostly use their own preferred
spelling style, be it humpStyle, snake_style or dash–style and libraries written
by different programmers cannot use incompatible conventions.
A Nim-aware editor or IDE can show the identifiers as preferred.
Another advantage is that it frees the programmer from remembering
the exact spelling of an identifier. The exception with respect to the first
letter allows common code like ``var foo: Foo`` to be parsed unambiguously.

Historically, Nim was a fully `style-insensitive`:idx: language. This meant that
it was not case-sensitive and underscores were ignored and there was no even a
distinction between ``foo`` and ``Foo``.


String literals
---------------

{'fraction_non_alphanumeric': 0.09144681250586799, 'fraction_numerical': 0.003379964322598817, 'mean_word_length': 3.6576300830782684, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5285642', 'n_tokens_mistral': 2899, 'n_tokens_neox': 2661, 'n_words': 1479}