<commit_before>USING: arrays assocs classes.tuple combinators command-line
effects generic generic.math generic.single help.markup
help.syntax io.pathnames kernel math parser sequences
vocabs.loader vocabs.parser words words.alias words.constant
words.symbol ;
IN: syntax

ARTICLE: "parser-algorithm" "Parser algorithm"
"At the most abstract level, Factor syntax consists of whitespace-separated tokens. The parser tokenizes the input on whitespace boundaries. The parser is case-sensitive and whitespace between tokens is significant, so the following three expressions tokenize differently:"
{ $code "2X+\n2 X +\n2 x +" }
"As the parser reads tokens it makes a distinction between numbers, ordinary words, and parsing words. Tokens are appended to the parse tree, the top level of which is a quotation returned by the original parser invocation. Nested levels of the parse tree are created by parsing words."
$nl
"The parser iterates through the input text, checking each character in turn. Here is the parser algorithm in more detail -- some of the concepts therein will be defined shortly:"
{ $list
    { "If the current character is a double-quote (\"), the " { $link POSTPONE: " } " parsing word is executed, causing a string to be read." }
    {
        "Otherwise, the next token is taken from the input. The parser searches for a word named by the token in the currently used set of vocabularies. If the word is found, one of the following two actions is taken:"
        { $list
            "If the word is an ordinary word, it is appended to the parse tree."
            "If the word is a parsing word, it is executed."
        }
    }
    "Otherwise if the token does not represent a known word, the parser attempts to parse it as a number. If the token is a number, the number object is added to the parse tree. Otherwise, an error is raised and parsing halts."
}
"Parsing words play a key role in parsing; while ordinary words and numbers are simply added to the parse tree, parsing words execute in the context of the parser, and can do their own parsing and create nested data structures in the parse tree. Parsing words are also able to define new words."
$nl
"While parsing words supporting arbitrary syntax can be defined, the default set is found in the " { $vocab-link "syntax" } " vocabulary and provides the basis for all further syntactic interaction with Factor." ;

ARTICLE: "syntax-comments" "Comments"
{ $subsections
    POSTPONE: !
    POSTPONE: #!
} ;

ARTICLE: "syntax-immediate" "Parse time evaluation"
"Code can be evaluated at parse time. This is a rarely-used feature; one use-case is " { $link "loading-libs" } ", where you want to execute some code before the words in a source file are compiled."
{ $subsections
    POSTPONE: <<
    POSTPONE: >>
} ;

ARTICLE: "syntax-integers" "Integer syntax"
"The printed representation of an integer consists of a sequence of digits, optionally prefixed by a sign."
{ $code
    "123456"
    "-10"
    "2432902008176640000"
}
"Integers are entered in base 10 unless prefixed with a base-changing prefix. " { $snippet "0x" } " begins a hexadecimal literal, " { $snippet "0o" } " an octal literal, and " { $snippet "0b" } " a binary literal. A sign, if any, goes before the base prefix."
{ $example
    "USE: prettyprint"
    "10 ."
    "0b10 ."
    "-0o10 ."
    "0x10 ."
    "10\n2\n-8\n16"
}
"More information on integers can be found in " { $link "integers" } "." ;

ARTICLE: "syntax-ratios" "Ratio syntax"
"The printed representation of a ratio is a pair of integers separated by a slash (" { $snippet "/" } "). A ratio can also be written as a proper fraction by following an integer part with " { $snippet "+" } " or " { $snippet "-" } " (matching the sign of the integer) and a ratio. No intermediate whitespace is permitted within a ratio literal. Here are some examples:"
{ $code
    "75/33"
    "1/10"
    "-5/-6"
    "1+1/3"
    "-10-1/7"
}
"More information on ratios can be found in " { $link "rationals" } "." ;

ARTICLE: "syntax-floats" "Float syntax"
"Floating point literals are specified when a literal number contains a decimal point or exponent. Exponents are marked by an " { $snippet "e" } " or " { $snippet "E" } ":"
{ $code
    "10.5"
    "-3.1456"
    "7e13"
    "1.0e-5"
    "1.0E+5"
}
"Literal numbers without a decimal point or an exponent always parse as integers:"
{ $example
    "1 float? ."
    "f"
}
{ $example
    "1. float? ."
    "t"
}
{ $example
    "1e0 float? ."
    "t"
}
"Literal floating point approximations of ratios can also be input by placing a decimal point in the denominator:"
{ $example
    "1/2. ."
    "0.5"
}
{ $example
    "1/3. ."
    "0.3333333333333333"
}
"The special float values have their own syntax:"
{ $table
{ "Positive infinity" { $snippet "1/0." } }
{ "Negative infinity" { $snippet "-1/0." } }
{ "Not-a-number" { $snippet "0/0." } }
}
"A Not-a-number literal with an arbitrary payload can also be input:"
{ $subsections POSTPONE: NAN: }
"Hexadecimal, octal and binary float literals are also supported. These consist of a hexadecimal, octal or binary literal with a decimal point and a mandatory base-two exponent expressed as a decimal number after " { $snippet "p" } " or " { $snippet "P" } ":"
{ $example
    "8.0 0x1.0p3 = ."
    "t"
}
{ $example
    "-1024.0 -0x1.0P10 = ."
    "t"
}
{ $example
    "10.125 0x1.44p3 = ."
    "t"
}
{ $example
    "10.125 0b1.01001p3 = ."
    "t"
}
{ $example
    "10.125 0o1.22p3 = ."
    "t"
}
"The normalized hex form " { $snippet "±0x1.MMMMMMMMMMMMMp±EEEE" } " allows any floating-point number to be specified precisely. The values of MMMMMMMMMMMMM and EEEE map directly to the mantissa and exponent fields of the binary IEEE 754 representation."
$nl
"More information on floats can be found in " { $link "floats" } "." ;

ARTICLE: "syntax-complex-numbers" "Complex number syntax"
"A complex number is given by two components, a “real” part and “imaginary” part. The components must either be integers, ratios or floats."
{ $code
    "C{ 1/2 1/3 }   ! the complex number 1/2+1/3i"
    "C{ 0 1 }       ! the imaginary unit"
}
{ $subsections POSTPONE: C{ }
"More information on complex numbers can be found in " { $link "complex-numbers" } "." ;

ARTICLE: "syntax-numbers" "Number syntax"
"If a vocabulary lookup of a token fails, the parser attempts to parse it as a number."<commit_msg>syntax: fix example in syntax-floats.<commit_after>USING: arrays assocs classes.tuple combinators command-line
effects generic generic.math generic.single help.markup
help.syntax io.pathnames kernel math parser sequences
vocabs.loader vocabs.parser words words.alias words.constant
words.symbol ;
IN: syntax

ARTICLE: "parser-algorithm" "Parser algorithm"
"At the most abstract level, Factor syntax consists of whitespace-separated tokens. The parser tokenizes the input on whitespace boundaries. The parser is case-sensitive and whitespace between tokens is significant, so the following three expressions tokenize differently:"
{ $code "2X+\n2 X +\n2 x +" }
"As the parser reads tokens it makes a distinction between numbers, ordinary words, and parsing words. Tokens are appended to the parse tree, the top level of which is a quotation returned by the original parser invocation. Nested levels of the parse tree are created by parsing words."
$nl
"The parser iterates through the input text, checking each character in turn. Here is the parser algorithm in more detail -- some of the concepts therein will be defined shortly:"
{ $list
    { "If the current character is a double-quote (\"), the " { $link POSTPONE: " } " parsing word is executed, causing a string to be read." }
    {
        "Otherwise, the next token is taken from the input. The parser searches for a word named by the token in the currently used set of vocabularies. If the word is found, one of the following two actions is taken:"
        { $list
            "If the word is an ordinary word, it is appended to the parse tree."
            "If the word is a parsing word, it is executed."
        }
    }
    "Otherwise if the token does not represent a known word, the parser attempts to parse it as a number. If the token is a number, the number object is added to the parse tree. Otherwise, an error is raised and parsing halts."
}
"Parsing words play a key role in parsing; while ordinary words and numbers are simply added to the parse tree, parsing words execute in the context of the parser, and can do their own parsing and create nested data structures in the parse tree. Parsing words are also able to define new words."
$nl
"While parsing words supporting arbitrary syntax can be defined, the default set is found in the " { $vocab-link "syntax" } " vocabulary and provides the basis for all further syntactic interaction with Factor." ;

ARTICLE: "syntax-comments" "Comments"
{ $subsections
    POSTPONE: !
    POSTPONE: #!
} ;

ARTICLE: "syntax-immediate" "Parse time evaluation"
"Code can be evaluated at parse time. This is a rarely-used feature; one use-case is " { $link "loading-libs" } ", where you want to execute some code before the words in a source file are compiled."
{ $subsections
    POSTPONE: <<
    POSTPONE: >>
} ;

ARTICLE: "syntax-integers" "Integer syntax"
"The printed representation of an integer consists of a sequence of digits, optionally prefixed by a sign."
{ $code
    "123456"
    "-10"
    "2432902008176640000"
}
"Integers are entered in base 10 unless prefixed with a base-changing prefix. " { $snippet "0x" } " begins a hexadecimal literal, " { $snippet "0o" } " an octal literal, and " { $snippet "0b" } " a binary literal. A sign, if any, goes before the base prefix."
{ $example
    "USE: prettyprint"
    "10 ."
    "0b10 ."
    "-0o10 ."
    "0x10 ."
    "10\n2\n-8\n16"
}
"More information on integers can be found in " { $link "integers" } "." ;

ARTICLE: "syntax-ratios" "Ratio syntax"
"The printed representation of a ratio is a pair of integers separated by a slash (" { $snippet "/" } "). A ratio can also be written as a proper fraction by following an integer part with " { $snippet "+" } " or " { $snippet "-" } " (matching the sign of the integer) and a ratio. No intermediate whitespace is permitted within a ratio literal. Here are some examples:"
{ $code
    "75/33"
    "1/10"
    "-5/-6"
    "1+1/3"
    "-10-1/7"
}
"More information on ratios can be found in " { $link "rationals" } "." ;

ARTICLE: "syntax-floats" "Float syntax"
"Floating point literals are specified when a literal number contains a decimal point or exponent. Exponents are marked by an " { $snippet "e" } " or " { $snippet "E" } ":"
{ $code
    "10.5"
    "-3.1456"
    "7e13"
    "1.0e-5"
    "1.0E+5"
}
"Literal numbers without a decimal point or an exponent always parse as integers:"
{ $example
    "1 float? ."
    "f"
}
{ $example
    "1. float? ."
    "t"
}
{ $example
    "1e0 float? ."
    "t"
}
"Literal floating point approximations of ratios can also be input by placing a decimal point in the denominator:"
{ $example
    "1/2. ."
    "0.5"
}
{ $example
    "1/3. ."
    "0.3333333333333333"
}
"The special float values have their own syntax:"
{ $table
{ "Positive infinity" { $snippet "1/0." } }
{ "Negative infinity" { $snippet "-1/0." } }
{ "Not-a-number" { $snippet "0/0." } }
}
"A Not-a-number literal with an arbitrary payload can also be input:"
{ $subsections POSTPONE: NAN: }
"Hexadecimal, octal and binary float literals are also supported. These consist of a hexadecimal, octal or binary literal with a decimal point and a mandatory base-two exponent expressed as a decimal number after " { $snippet "p" } " or " { $snippet "P" } ":"
{ $example
    "8.0 0x1.0p3 = ."
    "t"
}
{ $example
    "-1024.0 -0x1.0P10 = ."
    "t"
}
{ $example
    "10.125 0x1.44p3 = ."
    "t"
}
{ $example
    "10.125 0b1.010001p3 = ."
    "t"
}
{ $example
    "10.125 0o1.22p3 = ."
    "t"
}
"The normalized hex form " { $snippet "±0x1.MMMMMMMMMMMMMp±EEEE" } " allows any floating-point number to be specified precisely. The values of MMMMMMMMMMMMM and EEEE map directly to the mantissa and exponent fields of the binary IEEE 754 representation."
$nl
"More information on floats can be found in " { $link "floats" } "." ;

ARTICLE: "syntax-complex-numbers" "Complex number syntax"
"A complex number is given by two components, a “real” part and “imaginary” part. The components must either be integers, ratios or floats."
{ $code
    "C{ 1/2 1/3 }   ! the complex number 1/2+1/3i"
    "C{ 0 1 }       ! the imaginary unit"
}
{ $subsections POSTPONE: C{ }
"More information on complex numbers can be found in " { $link "complex-numbers" } "." ;

ARTICLE: "syntax-numbers" "Number syntax"
"If a vocabulary lookup of a token fails, the parser attempts to parse it as a number."
{'fraction_non_alphanumeric': 0.10322125558429344, 'fraction_numerical': 0.028607257622070694, 'mean_word_length': 3.4106463878326996, 'pattern_counts': {'":': 4, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 6, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2737300', 'n_tokens_mistral': 4114, 'n_tokens_neox': 3824, 'n_words': 1727}