<commit_before><chapter id="lexical">
    <title>Lexical structure</title>
    
    <para>Every Ceylon source file is a sequence of Unicode characters. Lexical
    analysis of the character stream, according to the grammar specified in this
    chapter, results in a stream of tokens. These tokens form the input of the 
    parser grammar defined in the later chapters of this specification. The
    Ceylon lexer is able to completely tokenize a character stream in a single 
    pass.</para> 
    
    <section id="whitespace">
        <title>Whitespace</title>
        
        <para><emphasis>Whitespace</emphasis> is composed of strings of Unicode 
        <literal>SPACE</literal>, <literal>CHARACTER TABULATION</literal>, 
        <literal>FORM FEED (FF)</literal>, <literal>LINE FEED (LF)</literal> and 
        <literal>CARRIAGE RETURN (CR)</literal> characters.</para>
        
        <synopsis>Whitespace: " " | Tab | Formfeed | Newline | Return</synopsis>
        
        <synopsis>Tab: "\{CHARACTER TABULATION}"</synopsis>
        <synopsis>Formfeed: "\{FORM FEED (FF)}"</synopsis>
        <synopsis>Newline: "\{LINE FEED (LF)}"</synopsis>
        <synopsis>Return: "\{CARRIAGE RETURN (CR)}"</synopsis>
        
        <para>Outside of a comment, string literal, or single quoted literal,
        whitespace acts as a token separator and is immediately discarded by
        the lexer. Whitespace is not used as a statement separator.</para>
        
        <para>Source text is divided into lines by <emphasis>line-terminating 
        character sequences</emphasis>. The following Unicode character sequences 
        terminate a line:</para>
        
        <itemizedlist>
            <listitem>
                <para><literal>LINE FEED (LF)</literal>,</para>
            </listitem>
            <listitem>
                <para><literal>CARRIAGE RETURN (CR)</literal>, and</para>
            </listitem>
            <listitem>
                <para><literal>CARRIAGE RETURN (CR)</literal> followed by
                <literal>LINE FEED (LF)</literal>.</para>
            </listitem>
        </itemizedlist>
        
        
    </section>
    
    <section id="comments">
        <title>Comments</title>
        
        <para>There are two kinds of comments:</para>
        
        <itemizedlist>
            <listitem>
                <para>a <emphasis>multiline comment</emphasis> begins with 
                <literal>/*</literal> and extends until <literal>*/</literal>, 
                and</para>
            </listitem>
            <listitem>
                <para>an <emphasis>end-of-line comment</emphasis> begins with 
                <literal>//</literal> or <literal>#!</literal> and extends until 
                the next line terminating character sequence.</para>
            </listitem>
        </itemizedlist>
        
        <para>Both kinds of comments can be nested.</para>
        
        <synopsis>LineComment: ("//"|"#!") ~(Newline | Return)* (Return Newline | Return | Newline)?</synopsis>
        
        <synopsis>MultilineComment: "/*" (MultilineCommentCharacter | MultilineComment)* "*/"</synopsis>
        
        <synopsis>MultilineCommentCharacter: ~("/"|"*") | ("/" ~"*") => "/" | ("*" ~"/") => "*"</synopsis>
        
        <para>The following examples are legal comments:</para>
        
        <programlisting>//this comment stops at the end of the line</programlisting>
        
        <programlisting>/*
   but this is a comment that spans
   multiple lines
*/</programlisting>
  
        <programlisting>#!/usr/bin/ceylon</programlisting>
        <commit_msg>Resolve specification rule name collision<commit_after><chapter id="lexical">
    <title>Lexical structure</title>
    
    <para>Every Ceylon source file is a sequence of Unicode characters. Lexical
    analysis of the character stream, according to the grammar specified in this
    chapter, results in a stream of tokens. These tokens form the input of the 
    parser grammar defined in the later chapters of this specification. The
    Ceylon lexer is able to completely tokenize a character stream in a single 
    pass.</para> 
    
    <section id="whitespace">
        <title>Whitespace</title>
        
        <para><emphasis>Whitespace</emphasis> is composed of strings of Unicode 
        <literal>SPACE</literal>, <literal>CHARACTER TABULATION</literal>, 
        <literal>FORM FEED (FF)</literal>, <literal>LINE FEED (LF)</literal> and 
        <literal>CARRIAGE RETURN (CR)</literal> characters.</para>
        
        <synopsis>Whitespace: " " | Tab | Formfeed | Newline | CarriageReturn</synopsis>
        
        <synopsis>Tab: "\{CHARACTER TABULATION}"</synopsis>
        <synopsis>Formfeed: "\{FORM FEED (FF)}"</synopsis>
        <synopsis>Newline: "\{LINE FEED (LF)}"</synopsis>
        <synopsis>CarriageReturn: "\{CARRIAGE RETURN (CR)}"</synopsis>
        
        <para>Outside of a comment, string literal, or single quoted literal,
        whitespace acts as a token separator and is immediately discarded by
        the lexer. Whitespace is not used as a statement separator.</para>
        
        <para>Source text is divided into lines by <emphasis>line-terminating 
        character sequences</emphasis>. The following Unicode character sequences 
        terminate a line:</para>
        
        <itemizedlist>
            <listitem>
                <para><literal>LINE FEED (LF)</literal>,</para>
            </listitem>
            <listitem>
                <para><literal>CARRIAGE RETURN (CR)</literal>, and</para>
            </listitem>
            <listitem>
                <para><literal>CARRIAGE RETURN (CR)</literal> followed by
                <literal>LINE FEED (LF)</literal>.</para>
            </listitem>
        </itemizedlist>
        
        
    </section>
    
    <section id="comments">
        <title>Comments</title>
        
        <para>There are two kinds of comments:</para>
        
        <itemizedlist>
            <listitem>
                <para>a <emphasis>multiline comment</emphasis> begins with 
                <literal>/*</literal> and extends until <literal>*/</literal>, 
                and</para>
            </listitem>
            <listitem>
                <para>an <emphasis>end-of-line comment</emphasis> begins with 
                <literal>//</literal> or <literal>#!</literal> and extends until 
                the next line terminating character sequence.</para>
            </listitem>
        </itemizedlist>
        
        <para>Both kinds of comments can be nested.</para>
        
        <synopsis>LineComment: ("//"|"#!") ~(Newline | CarriageReturn)* (CarriageReturn Newline | CarriageReturn | Newline)?</synopsis>
        
        <synopsis>MultilineComment: "/*" (MultilineCommentCharacter | MultilineComment)* "*/"</synopsis>
        
        <synopsis>MultilineCommentCharacter: ~("/"|"*") | ("/" ~"*") => "/" | ("*" ~"/") => "*"</synopsis>
        
        <para>The following examples are legal comments:</para>
        
        <programlisting>//this comment stops at the end of the line</programlisting>
        
        <programlisting>/*
   but this is a comment that spans
   multiple lines
*/</programlisting>
  
        <programlisting>#!/usr/bin/ceylon</programlisting>
        
{'fraction_non_alphanumeric': 0.12145196506550218, 'fraction_numerical': 0.0, 'mean_word_length': 2.2074398249452956, 'pattern_counts': {'":': 0, '<': 211, '<?xml version=': 0, '>': 215, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3269521', 'n_tokens_mistral': 2088, 'n_tokens_neox': 1933, 'n_words': 550}