<commit_before><?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
   "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

<book id="alex">

  <bookinfo>
    <date>2003-8-11</date>
    <title>Alex User Guide</title>
    <author>
      <firstname>Chris</firstname>
      <surname>Dornan</surname>
    </author>
    <author>
      <firstname>Isaac</firstname>
      <surname>Jones</surname>
    </author>
    <author>
      <firstname>Simon</firstname>
      <surname>Marlow</surname>
    </author>
    <address><email><EMAIL></email></address>
    <!--     <copyright> -->
    <!--       <year>1997-2001</year> -->
    <!--       <holder><NAME></holder> -->
    <!--     </copyright> -->

    <abstract>
      <para>Alex is a tool for generating lexical analysers in
      Haskell, given a description of the tokens to be recognised in
      the form of regular expressions.  It is similar to the tool
      <quote>lex</quote> or <quote>flex</quote> for C/C++.</para>
    </abstract>
  </bookinfo>

  <!-- Table of contents -->
  <toc></toc>

  <chapter id="about">
    <title>About Alex</title>

    <para>Alex can always be obtained from its <ulink
    url="http://www.haskell.org/alex">home page</ulink>.  The latest
    source code lives in the <literal>fptools</literal> CVS
    repository; instructions on accessing that repository are <ulink
    url="http://www.haskell.org/ghc/docs/latest/html/building/sec-cvs.html">here</ulink>.</para>

    <section id="relnotes-30">
      <title>Release Notes for version 3.0</title>

      <itemizedlist>
        <listitem>
          <para>Unicode support (contributed mostly by Jean-<NAME>, with help from <NAME>).</para>
          <itemizedlist>
            <listitem>
              <para>
                An Alex lexer now takes a UTF-8 encoded byte sequence
                as input (see <xref linkend="encoding" />.  If you are
                using the "basic" wrapper or one of the other wrappers
                that takes a Haskell <literal>String</literal> as
                input, the string is automatically encoded into UTF-8
                by Alex.  If your input is
                a <literal>ByteString</literal>, you are responsible
                for ensuring that the input is UTF-8 encoded.  The old
                8-bit behaviour is still available via
                the <option>--latin1</option> option.
              </para>
            </listitem>
            <listitem>
              <para><commit_msg>Change source code reference in the documentation<commit_after><?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
   "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

<book id="alex">

  <bookinfo>
    <date>2003-8-11</date>
    <title>Alex User Guide</title>
    <author>
      <firstname>Chris</firstname>
      <surname>Dornan</surname>
    </author>
    <author>
      <firstname>Isaac</firstname>
      <surname>Jones</surname>
    </author>
    <author>
      <firstname>Simon</firstname>
      <surname>Marlow</surname>
    </author>
    <address><email><EMAIL></email></address>
    <!--     <copyright> -->
    <!--       <year>1997-2001</year> -->
    <!--       <holder><NAME></holder> -->
    <!--     </copyright> -->

    <abstract>
      <para>Alex is a tool for generating lexical analysers in
      Haskell, given a description of the tokens to be recognised in
      the form of regular expressions.  It is similar to the tool
      <quote>lex</quote> or <quote>flex</quote> for C/C++.</para>
    </abstract>
  </bookinfo>

  <!-- Table of contents -->
  <toc></toc>

  <chapter id="about">
    <title>About Alex</title>

    <para>Alex can always be obtained from its <ulink
    url="http://www.haskell.org/alex">home page</ulink>.  The latest
    source code lives in the <ulink url="https://github.com/simonmar/alex">git
    repository</ulink> on <literal>GitHub</literal>.
    </para>

    <section id="relnotes-30">
      <title>Release Notes for version 3.0</title>

      <itemizedlist>
        <listitem>
          <para>Unicode support (contributed mostly by Jean-Philippe
            Bernardy, with help from <NAME>).</para>
          <itemizedlist>
            <listitem>
              <para>
                An Alex lexer now takes a UTF-8 encoded byte sequence
                as input (see <xref linkend="encoding" />.  If you are
                using the "basic" wrapper or one of the other wrappers
                that takes a Haskell <literal>String</literal> as
                input, the string is automatically encoded into UTF-8
                by Alex.  If your input is
                a <literal>ByteString</literal>, you are responsible
                for ensuring that the input is UTF-8 encoded.  The old
                8-bit behaviour is still available via
                the <option>--latin1</option> option.
              </para>
            </listitem>
            <listitem>
              <para>
{'fraction_non_alphanumeric': 0.13512978006736676, 'fraction_numerical': 0.013869625520110958, 'mean_word_length': 2.1160493827160494, 'pattern_counts': {'":': 0, '<': 178, '<?xml version=': 2, '>': 178, 'https://': 1, 'lorem ipsum': 0, 'www.': 5, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6362962', 'n_tokens_mistral': 1609, 'n_tokens_neox': 1476, 'n_words': 417}