<commit_before>---
title: Lexical Syntax
layout: default
chapter: 1
---

# Lexical Syntax

Scala programs are written using the Unicode Basic Multilingual Plane
(_BMP_) character set; Unicode supplementary characters are not
presently supported.  This chapter defines the two modes of Scala's
lexical syntax, the Scala mode and the _XML mode_. If not
otherwise mentioned, the following descriptions of Scala tokens refer
to _Scala mode_, and literal characters ‘c’ refer to the ASCII fragment
`\u0000` – `\u007F`.

In Scala mode, _Unicode escapes_ are replaced by the corresponding
Unicode character with the given hexadecimal code.

```ebnf
UnicodeEscape ::= ‘\’ ‘u’ {‘u’} hexDigit hexDigit hexDigit hexDigit
hexDigit      ::= ‘0’ | … | ‘9’ | ‘A’ | … | ‘F’ | ‘a’ | … | ‘f’
```

<!--
TODO SI-4583: UnicodeEscape used to allow additional backslashes,
and there is something in the code `evenSlashPrefix` that alludes to it,
but I can't make it work nor can I imagine how this would make sense,
so I removed it for now.
-->

To construct tokens, characters are distinguished according to the following
classes (Unicode general category given in parentheses):

1. Whitespace characters. `\u0020 | \u0009 | \u000D | \u000A`.
1. Letters, which include lower case letters (`Ll`), upper case letters (`Lu`),
   titlecase letters (`Lt`), other letters (`Lo`), letter numerals (`Nl`) and the
   two characters `\u0024 ‘$’` and `\u005F ‘_’`, which both count as upper case
   letters.
1. Digits `‘0’ | … | ‘9’`.
1. Parentheses `‘(’ | ‘)’ | ‘[’ | ‘]’ | ‘{’ | ‘}’ `.
1. Delimiter characters ``‘`’ | ‘'’ | ‘"’ | ‘.’ | ‘;’ | ‘,’ ``.
1. Operator characters. These consist of all printable ASCII characters
   `\u0020` - `\u007F` which are in none of the sets above, mathematical
   symbols (`Sm`) and other symbols (`So`).

## Identifiers

```ebnf
op       ::=  opchar {opchar}
varid    ::=  lower idrest
plainid  ::=  upper idrest
           |  varid
           |  op
id       ::=  plainid
           |  ‘`’ stringLiteral ‘`’
idrest   ::=  {letter | digit} [‘_’ op]
```

There are three ways to form an identifier. First, an identifier can
start with a letter which can be followed by an arbitrary sequence of
letters and digits. This may be followed by underscore ‘_’
characters and another string composed of either letters and digits or
of operator characters.  Second, an identifier can start with an operator
character followed by an arbitrary sequence of operator characters.
The preceding two forms are called _plain_ identifiers.  Finally,
an identifier may also be formed by an arbitrary string between
back-quotes (host systems may impose some restrictions on which
strings are legal for identifiers).  The identifier then is composed
of all characters excluding the backquotes themselves.

As usual, a longest match rule applies. For instance, the string

```scala
big_bob++=`def`
```

decomposes into the three identifiers `big_bob`, `++=`, and
`def`. The rules for pattern matching further distinguish between
_variable identifiers_, which start with a lower case letter, and<commit_msg>Quote `_` properly in lexical syntax.<commit_after>---
title: Lexical Syntax
layout: default
chapter: 1
---

# Lexical Syntax

Scala programs are written using the Unicode Basic Multilingual Plane
(_BMP_) character set; Unicode supplementary characters are not
presently supported.  This chapter defines the two modes of Scala's
lexical syntax, the Scala mode and the _XML mode_. If not
otherwise mentioned, the following descriptions of Scala tokens refer
to _Scala mode_, and literal characters ‘c’ refer to the ASCII fragment
`\u0000` – `\u007F`.

In Scala mode, _Unicode escapes_ are replaced by the corresponding
Unicode character with the given hexadecimal code.

```ebnf
UnicodeEscape ::= ‘\’ ‘u’ {‘u’} hexDigit hexDigit hexDigit hexDigit
hexDigit      ::= ‘0’ | … | ‘9’ | ‘A’ | … | ‘F’ | ‘a’ | … | ‘f’
```

<!--
TODO SI-4583: UnicodeEscape used to allow additional backslashes,
and there is something in the code `evenSlashPrefix` that alludes to it,
but I can't make it work nor can I imagine how this would make sense,
so I removed it for now.
-->

To construct tokens, characters are distinguished according to the following
classes (Unicode general category given in parentheses):

1. Whitespace characters. `\u0020 | \u0009 | \u000D | \u000A`.
1. Letters, which include lower case letters (`Ll`), upper case letters (`Lu`),
   titlecase letters (`Lt`), other letters (`Lo`), letter numerals (`Nl`) and the
   two characters `\u0024 ‘$’` and `\u005F ‘_’`, which both count as upper case
   letters.
1. Digits `‘0’ | … | ‘9’`.
1. Parentheses `‘(’ | ‘)’ | ‘[’ | ‘]’ | ‘{’ | ‘}’ `.
1. Delimiter characters ``‘`’ | ‘'’ | ‘"’ | ‘.’ | ‘;’ | ‘,’ ``.
1. Operator characters. These consist of all printable ASCII characters
   `\u0020` - `\u007F` which are in none of the sets above, mathematical
   symbols (`Sm`) and other symbols (`So`).

## Identifiers

```ebnf
op       ::=  opchar {opchar}
varid    ::=  lower idrest
plainid  ::=  upper idrest
           |  varid
           |  op
id       ::=  plainid
           |  ‘`’ stringLiteral ‘`’
idrest   ::=  {letter | digit} [‘_’ op]
```

There are three ways to form an identifier. First, an identifier can
start with a letter which can be followed by an arbitrary sequence of
letters and digits. This may be followed by underscore `‘_‘`
characters and another string composed of either letters and digits or
of operator characters.  Second, an identifier can start with an operator
character followed by an arbitrary sequence of operator characters.
The preceding two forms are called _plain_ identifiers.  Finally,
an identifier may also be formed by an arbitrary string between
back-quotes (host systems may impose some restrictions on which
strings are legal for identifiers).  The identifier then is composed
of all characters excluding the backquotes themselves.

As usual, a longest match rule applies. For instance, the string

```scala
big_bob++=`def`
```

decomposes into the three identifiers `big_bob`, `++=`, and
`def`. The rules for pattern matching further distinguish between
_variable identifiers_, which start with a lower case letter, and
{'fraction_non_alphanumeric': 0.1037431429493385, 'fraction_numerical': 0.016134236850596968, 'mean_word_length': 3.9081551860649246, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3483768', 'n_tokens_mistral': 1927, 'n_tokens_neox': 1782, 'n_words': 833}