<commit_before><?xml version="1.0" encoding="latin1" ?>
<!DOCTYPE erlref SYSTEM "erlref.dtd">

<erlref>
  <header>
    <copyright>
      <year>1996</year>
      <year>2011</year>
      <holder>Ericsson AB, All Rights Reserved</holder>
    </copyright>
    <legalnotice>
  The contents of this file are subject to the Erlang Public License,
  Version 1.1, (the "License"); you may not use this file except in
  compliance with the License. You should have received a copy of the
  Erlang Public License along with this software. If not, it can be
  retrieved online at http://www.erlang.org/.

  Software distributed under the License is distributed on an "AS IS"
  basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
  the License for the specific language governing rights and limitations
  under the License.

  The Initial Developer of the Original Code is Ericsson AB.
    </legalnotice>

    <title>unicode</title>
    <prepared></prepared>
    <docno></docno>
    <date></date>
    <rev></rev>
  </header>
  <module>unicode</module>
  <modulesummary>Functions for converting Unicode characters</modulesummary>
  <description>
  <p>This module contains functions for converting between different character representations. Basically it converts between iso-latin-1 characters and Unicode ditto, but it can also convert between different Unicode encodings (like UTF-8, UTF-16 and UTF-32).</p>
  <p>The default Unicode encoding in Erlang is in binaries UTF-8, which is also the format in which built in functions and libraries in OTP expect to find binary Unicode data. In lists, Unicode data is encoded as integers, each integer representing one character and encoded simply as the Unicode codepoint for the character.</p> 
  <p>Other Unicode encodings than integers representing codepoints or UTF-8 in binaries are referred to as &quot;external encodings&quot;. The iso-latin-1 encoding is in binaries and lists referred to as latin1-encoding.</p>
  <p>It is recommended to only use external encodings for communication with external entities where this is required. When working inside the Erlang/OTP environment, it is recommended to keep binaries in UTF-8 when representing Unicode characters. Latin1 encoding is supported both for backward compatibility and for communication with external entities not supporting Unicode character sets.</p>
  </description>

  <section>
    <title>DATA TYPES</title>
    <marker id="type-charlist"></marker>
    <code type="none">
unicode_binary() = binary() with characters encoded in UTF-8 coding standard
unicode_char() = integer() representing valid unicode codepoint

chardata() = charlist() | unicode_binary()

charlist() = [unicode_char() | unicode_binary() | charlist()]
  a unicode_binary is allowed as the tail of the list</code>

    <code type="none">
external_unicode_binary() = binary() 
  with characters coded in a user specified Unicode encoding other 
  than UTF-8 (UTF-16 or UTF-32)

external_chardata() = external_charlist() | external_unicode_binary()

external_charlist() = [unicode_char() | external_unicode_binary() | external_charlist()]
  an external_unicode_binary is allowed as the tail of the list</code>

    <code type="none">
latin1_binary() = binary() with characters coded in iso-latin-1
latin1_char() = integer() representing valid latin1 character (0-255)

latin1_chardata() = latin1_charlist() | latin1_binary()

latin1_charlist() = [latin1_char() | latin1_binary() | latin1_charlist()]
  a latin1_binary is allowed as the tail of the list</code>
  </section>
  <funcs>
    <func>
      <name>bom_to_encoding(Bin) -> {Encoding,Length}</name>
      <fsummary>Identify UTF byte order marks in a binary.</fsummary>
      <type>
        <v>Bin = binary() of byte_size 4 or more</v>
        <v>Encoding = latin1 | utf8 | {utf16,little} | {utf16,big} | {utf32,little} | {utf32,big}</v>
	<v>Length = int()</v>
      </type>
      <desc>

      <p>Check for a UTF byte order mark (BOM) in the beginning of a
      binary.  If the supplied binary <c>Bin</c> begins with a valid
      byte order mark for either UTF-8, UTF-16 or UTF-32, the function
      returns the encoding identified along with the length of the BOM
      in bytes.</p>

      <p>If no BOM is found, the function returns <c>{latin1,0}</c></p>
      </desc>
    </func>
    <func>
      <name>characters_to_list(Data) -> list() | {error, list(), RestData} | {incomplete, list(), binary()} </name>
      <fsummary>Convert a collection of characters to list of Unicode characters</fsummary>
      <type>
        <v>Data = latin1_chardata() | chardata() | external_chardata()</v>
        <v>RestData = latin1_chardata() | chardata() | external_chardata()</v>
      </type>
      <desc>
      <p>Same as characters_to_list(Data,unicode).</p>
      </desc>
    </func>
    <func>
      <name>characters_to_list(Data, InEncoding) -> list() | {error, list(), RestData} | {incomplete, list(), binary()} </name>
      <fsummary>Convert a collection of characters to list of Unicode characters</fsummary>
      <type>
        <v>Data = latin1_chardata() | chardata() | external_chardata()</v>
        <v>RestData = latin1_chardata() | chardata() | external_chardata()</v>
        <v>InEncoding = latin1 | unicode | utf8 | utf16 | utf32 | {utf16,little} | {utf16,big} | {utf32,little} | {utf32,big}</v>
      </type>
      <desc>

      <p>This function converts a possibly deep list of integers and
      binaries into a list of integers representing unicode
      characters. The binaries in the input may have characters
      encoded as latin1 (0 - 255, one character per byte), in which
      case the <c>InEncoding</c> parameter should be given as
      <c>latin1</c>, or have characters encoded as one of the
      UTF-encodings, which is given as the <c>InEncoding</c>
      parameter. Only when the <c>InEncoding</c> is one of the UTF
      encodings, integers in the list are allowed to be grater than
      255.</p>
      
      <p>If <c>InEncoding</c> is <c>latin1</c>, the <c>Data</c> parameter
      corresponds to the <c>iodata()</c> type, but for <c>unicode</c>,
      the <c>Data</c> parameter can contain integers greater than 255
      (unicode characters beyond the iso-latin-1 range), which would
      make it invalid as <c>iodata()</c>.</p>

      <p>The purpose of the function is mainly to be able to convert
      combinations of unicode characters into a pure unicode
      string in list representation for further processing. For
      writing the data to an external entity, the reverse function
      <seealso
      marker="#characters_to_binary/3">characters_to_binary/3</seealso>
      comes in handy.</p>
 
       <p>The option <c>unicode</c> is an alias for <c>utf8</c>, as this is the
      preferred encoding for Unicode characters in
      binaries. <c>utf16</c> is an alias for <c>{utf16,big}</c> and
      <c>utf32</c> is an alias for <c>{utf32,big}</c>. The <c>big</c>
      and <c>little</c> atoms denote big or little endian
      encoding.</p> 

      <p>If for some reason, the data cannot be converted, either
      because of illegal unicode/latin1 characters in the list, or
      because of invalid UTF encoding in any binaries, an error
      tuple is returned. The error tuple contains the tag
      <c>error</c>, a list representing the characters that could be
      converted before the error occurred and a representation of the
      characters including and after the offending integer/bytes. The
      last part is mostly for debugging as it still constitutes a
      possibly deep and/or mixed list, not necessarily of the same
      depth as the original data. The error occurs when traversing the
      list and whatever's left to decode is simply returned as is.</p>

      <p>However, if the input <c>Data</c> is a pure binary, the third
      part of the error tuple is guaranteed to be a binary as
      well.</p>

      <p>Errors occur for the following reasons:</p>
      <list type="bulleted">

           <item>Integers out of range - If <c>InEncoding</c> is
	   <c>latin1</c>, an error occurs whenever an integer greater
	   than 255 is found in the lists. If <c>InEncoding</c> is
	   of a Unicode type, error occurs whenever an integer greater than
	   <c>16#10FFFF</c> (the maximum unicode character) or in the
	   range <c>16#D800</c> to <c>16#DFFF</c> (invalid unicode
	   range) is found.</item>

	   <item>UTF encoding incorrect - If <c>InEncoding</c> is
	   one of the UTF types, the bytes in any binaries have to be valid
	   in that encoding. Errors can occur for various
	   reasons, including &quot;pure&quot; decoding errors 
	   (like the upper
	   bits of the bytes being wrong), the bytes are decoded to a<commit_msg>unicode: document 16#FFFE and 16#FFFF (non chars)<commit_after><?xml version="1.0" encoding="latin1" ?>
<!DOCTYPE erlref SYSTEM "erlref.dtd">

<erlref>
  <header>
    <copyright>
      <year>1996</year>
      <year>2011</year>
      <holder>Ericsson AB, All Rights Reserved</holder>
    </copyright>
    <legalnotice>
  The contents of this file are subject to the Erlang Public License,
  Version 1.1, (the "License"); you may not use this file except in
  compliance with the License. You should have received a copy of the
  Erlang Public License along with this software. If not, it can be
  retrieved online at http://www.erlang.org/.

  Software distributed under the License is distributed on an "AS IS"
  basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
  the License for the specific language governing rights and limitations
  under the License.

  The Initial Developer of the Original Code is Ericsson AB.
    </legalnotice>

    <title>unicode</title>
    <prepared></prepared>
    <docno></docno>
    <date></date>
    <rev></rev>
  </header>
  <module>unicode</module>
  <modulesummary>Functions for converting Unicode characters</modulesummary>
  <description>
  <p>This module contains functions for converting between different character representations. Basically it converts between iso-latin-1 characters and Unicode ditto, but it can also convert between different Unicode encodings (like UTF-8, UTF-16 and UTF-32).</p>
  <p>The default Unicode encoding in Erlang is in binaries UTF-8, which is also the format in which built in functions and libraries in OTP expect to find binary Unicode data. In lists, Unicode data is encoded as integers, each integer representing one character and encoded simply as the Unicode codepoint for the character.</p> 
  <p>Other Unicode encodings than integers representing codepoints or UTF-8 in binaries are referred to as &quot;external encodings&quot;. The iso-latin-1 encoding is in binaries and lists referred to as latin1-encoding.</p>
  <p>It is recommended to only use external encodings for communication with external entities where this is required. When working inside the Erlang/OTP environment, it is recommended to keep binaries in UTF-8 when representing Unicode characters. Latin1 encoding is supported both for backward compatibility and for communication with external entities not supporting Unicode character sets.</p>
  </description>

  <section>
    <title>DATA TYPES</title>
    <marker id="type-charlist"></marker>
    <code type="none">
unicode_binary() = binary() with characters encoded in UTF-8 coding standard
unicode_char() = integer() representing valid unicode codepoint

chardata() = charlist() | unicode_binary()

charlist() = [unicode_char() | unicode_binary() | charlist()]
  a unicode_binary is allowed as the tail of the list</code>

    <code type="none">
external_unicode_binary() = binary() 
  with characters coded in a user specified Unicode encoding other 
  than UTF-8 (UTF-16 or UTF-32)

external_chardata() = external_charlist() | external_unicode_binary()

external_charlist() = [unicode_char() | external_unicode_binary() | external_charlist()]
  an external_unicode_binary is allowed as the tail of the list</code>

    <code type="none">
latin1_binary() = binary() with characters coded in iso-latin-1
latin1_char() = integer() representing valid latin1 character (0-255)

latin1_chardata() = latin1_charlist() | latin1_binary()

latin1_charlist() = [latin1_char() | latin1_binary() | latin1_charlist()]
  a latin1_binary is allowed as the tail of the list</code>
  </section>
  <funcs>
    <func>
      <name>bom_to_encoding(Bin) -> {Encoding,Length}</name>
      <fsummary>Identify UTF byte order marks in a binary.</fsummary>
      <type>
        <v>Bin = binary() of byte_size 4 or more</v>
        <v>Encoding = latin1 | utf8 | {utf16,little} | {utf16,big} | {utf32,little} | {utf32,big}</v>
	<v>Length = int()</v>
      </type>
      <desc>

      <p>Check for a UTF byte order mark (BOM) in the beginning of a
      binary.  If the supplied binary <c>Bin</c> begins with a valid
      byte order mark for either UTF-8, UTF-16 or UTF-32, the function
      returns the encoding identified along with the length of the BOM
      in bytes.</p>

      <p>If no BOM is found, the function returns <c>{latin1,0}</c></p>
      </desc>
    </func>
    <func>
      <name>characters_to_list(Data) -> list() | {error, list(), RestData} | {incomplete, list(), binary()} </name>
      <fsummary>Convert a collection of characters to list of Unicode characters</fsummary>
      <type>
        <v>Data = latin1_chardata() | chardata() | external_chardata()</v>
        <v>RestData = latin1_chardata() | chardata() | external_chardata()</v>
      </type>
      <desc>
      <p>Same as characters_to_list(Data,unicode).</p>
      </desc>
    </func>
    <func>
      <name>characters_to_list(Data, InEncoding) -> list() | {error, list(), RestData} | {incomplete, list(), binary()} </name>
      <fsummary>Convert a collection of characters to list of Unicode characters</fsummary>
      <type>
        <v>Data = latin1_chardata() | chardata() | external_chardata()</v>
        <v>RestData = latin1_chardata() | chardata() | external_chardata()</v>
        <v>InEncoding = latin1 | unicode | utf8 | utf16 | utf32 | {utf16,little} | {utf16,big} | {utf32,little} | {utf32,big}</v>
      </type>
      <desc>

      <p>This function converts a possibly deep list of integers and
      binaries into a list of integers representing unicode
      characters. The binaries in the input may have characters
      encoded as latin1 (0 - 255, one character per byte), in which
      case the <c>InEncoding</c> parameter should be given as
      <c>latin1</c>, or have characters encoded as one of the
      UTF-encodings, which is given as the <c>InEncoding</c>
      parameter. Only when the <c>InEncoding</c> is one of the UTF
      encodings, integers in the list are allowed to be grater than
      255.</p>
      
      <p>If <c>InEncoding</c> is <c>latin1</c>, the <c>Data</c> parameter
      corresponds to the <c>iodata()</c> type, but for <c>unicode</c>,
      the <c>Data</c> parameter can contain integers greater than 255
      (unicode characters beyond the iso-latin-1 range), which would
      make it invalid as <c>iodata()</c>.</p>

      <p>The purpose of the function is mainly to be able to convert
      combinations of unicode characters into a pure unicode
      string in list representation for further processing. For
      writing the data to an external entity, the reverse function
      <seealso
      marker="#characters_to_binary/3">characters_to_binary/3</seealso>
      comes in handy.</p>
 
       <p>The option <c>unicode</c> is an alias for <c>utf8</c>, as this is the
      preferred encoding for Unicode characters in
      binaries. <c>utf16</c> is an alias for <c>{utf16,big}</c> and
      <c>utf32</c> is an alias for <c>{utf32,big}</c>. The <c>big</c>
      and <c>little</c> atoms denote big or little endian
      encoding.</p> 

      <p>If for some reason, the data cannot be converted, either
      because of illegal unicode/latin1 characters in the list, or
      because of invalid UTF encoding in any binaries, an error
      tuple is returned. The error tuple contains the tag
      <c>error</c>, a list representing the characters that could be
      converted before the error occurred and a representation of the
      characters including and after the offending integer/bytes. The
      last part is mostly for debugging as it still constitutes a
      possibly deep and/or mixed list, not necessarily of the same
      depth as the original data. The error occurs when traversing the
      list and whatever's left to decode is simply returned as is.</p>

      <p>However, if the input <c>Data</c> is a pure binary, the third
      part of the error tuple is guaranteed to be a binary as
      well.</p>

      <p>Errors occur for the following reasons:</p>
      <list type="bulleted">

           <item>Integers out of range - If <c>InEncoding</c> is
	   <c>latin1</c>, an error occurs whenever an integer greater
	   than 255 is found in the lists. If <c>InEncoding</c> is
	   of a Unicode type, an error occurs whenever an integer
	   <list type="bulleted">
	     <item>greater than <c>16#10FFFF</c>
	     (the maximum unicode character),</item>
	     <item>in the range <c>16#D800</c> to <c>16#DFFF</c>
	     (invalid unicode range)</item>
	     <item>or equal to 16#FFFE or 16#FFFF (non characters)</item>
	   </list>
	   is found.
	   </item>

	   <item>UTF encoding incorrect - If <c>InEncoding</c> is
	   one of the UTF types, the bytes in any binaries have to be valid
	   in that encoding. Errors can occur for various
	   reasons, including &quot;pure&quot; decoding errors 
	   (like the upper
	   bits of the bytes being wrong), the bytes are decoded to a
{'fraction_non_alphanumeric': 0.1010403709148479, 'fraction_numerical': 0.014474725771796901, 'mean_word_length': 3.4406226462465477, 'pattern_counts': {'":': 0, '<': 375, '<?xml version=': 2, '>': 381, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7366537', 'n_tokens_mistral': 5452, 'n_tokens_neox': 5061, 'n_words': 2067}