<commit_before>**********************************************************************
FGHI                                FIDONET GLOBAL HYPERTEXT INTERFACE
**********************************************************************
Status:         draft
Revision:       initial incomplete draft
Title:          Fidonet Unicode substrings
Author:         <NAME> Webmaster   (aka <NAME>, 2:50/88)
Revision Date:  28 Jul 2014
-+--------------------------------------------------------------------
Contents:
   1. Status of this document
   2. Introduction
   3. Key words to indicate requirement levels
   4. 8-bit encoding of a Fidonet message
      containing Unicode substrings
   5. Decoding of an 8-bit Fidonet message
      containing Unicode substrings
   6. Important notes
-+--------------------------------------------------------------------

1. Status of this document
-+------------------------

  This document is a draft of a Fidonet Standards Proposal (FSP).

  This document specifies an optional Fidonet standard
  that can be used in the Fidonet community.

  Implementation of the standard defined in this document is not
  mandatory, but all implementations are expected to adhere
  to this standard.

  Distribution of this document is unlimited, 
  provided that its text is not altered without notice.

2. Introduction
-+-------------

  Many classic Fidonet message editors (such as GoldED+, for example)
  were designed as 8-bit applications. They expect that each character
  in a Fidonet message is coded by one byte. Therefore they won't ever
  support Unicode UTF-8 or UTF-16 encoding.

  This situation is a problem of "chicken and egg" type. The messages
  in UTF-8 charset do not appear in Fidonet because they won't ever be
  read in any of the popular readers. On the other hand, the absence
  of such messages means that there's no need for the developers of
  popular readers to improve their soft, or for their users to upgrade
  their readers or to choose some newer (Unicode-supporting) readers.

  This document specifies a simple method that would allow Unicode
  substrings to appear (encoded and escaped) within 8-bit strings.

  The method of encoding is based on the UTF-7 format (RFC 2152).

  The method of escaping is inspired by HTML character references
  (HTML 4.01 subsection 5.3.1, subsection 5.3.2).

  By implementing this method, the following situation is achieved:

  *) The users of newer (Unicode-supporting) Fidonet applications
     can read and write Unicode substrings in 8-bit messages.

  *) The users of older (8-bit) Fidonet applications can read the
     8-bit parts of the message. The Unicode susbstrings remain
     unintelligible, but that's natural for an 8-bit application,
     and brings only a minor discomfort, and serves as a reason
     for an upgrade.

3. Key words to indicate requirement levels
-+-----------------------------------------

  The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
  "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY",
  and "OPTIONAL" in this document are to be interpreted as described
  in FTA-1006 (based on RFC 2119).

4. 8-bit encoding of a Fidonet message containing Unicode substrings
-+------------------------------------------------------------------

  First of all, the source (Unicode) text is split into an array of
  substrings following each other (successively), where the substrings
  that have even indices (0, 2, 4...) contain characters that can be
  encoded with the target encoding and the substrings that have odd
  indices (1, 3, 5...) contain characters that cannot be encoded with
  the target encoding. (Or vice versa; if a character that cannot be
  encoded with the target encoding appeared first, then its substring
  has zeroeth index, and all such substrings also have even indices.)

  The traditional 8-bit encoding is performed for the substrings that
  contain characters that can be encoded that way, i.e. each of such
  characters is represented by a byte.

  The other substrings ("Unicode substrings") are converted to the
  UTF-7 format (RFC 2152). For example, a string that consists of
  Unicode characters U+9802, U+5C16, U+5C0D, U+6C7A, U+4E4B, U+7A7F,
  U+8932, U+5B50, U+7BC7 is represented by the following string:

        +mAJcFlwNbHpOS3p/iTJbUHvH-

  However, the UTF-7 method of escaping (a plus before such string and
  a minus after) is not sufficient for Fidonet. Thus the minus MUST be
  replaced by a semicolon, the plus MUST be prepended by an ampersand.
  For example, a string that consists of Unicode characters U+9802,
  U+5C16, U+5C0D, U+6C7A, U+4E4B, U+7A7F, U+8932, U+5B50, U+7BC7 is
  represented by the following string:

        &+mAJcFlwNbHpOS3p/iTJbUHvH;

  Afterwards the traditional 8-but encoding is performed for these
  (ASCII-compatible) characters.

  The encoding results are concatenated (successively) in the order
  that substrings had in the initial array (i.e. in the order of
  their appearance in the source text).

5. Decoding of an 8-bit Fidonet message containing Unicode substrings
-+-------------------------------------------------------------------

  First of all, the message is decoded by a traditional 8-bit decoder,
  each byte is decoded to a character.

  Encoded Unicode substrings are then found in the message (using
  their unique form: an ampersand, then a plus, then one or more of
  base64 characters, then a semicolon) and replaced by their decoded
  equivalents.

  For finding that encoded forms, the following PECL (Perl-compatible
  regular expression) might be useful:

      /&\+[A-Za-z01-9+\/]+;/

  For decoding them, some RFC2152-compatible UTF-7 decoder MUST be
  used. (As explained in the previous section, Fidonet Unicode
  substrings use UTF-7 encoding and a different escaping. If the
  decoder expects RFC2152-compatible escaping, the ampersand MUST be
  removed and the semicolon MUST be replaced by a minus before the
  substring is given to the decoder.)

6. Important notes
-+----------------

  

**********************************************************************
EOTD                                               END OF THE DOCUMENT
**********************************************************************<commit_msg>add a note (note 1) to the “Important notes” section<commit_after>**********************************************************************
FGHI                                FIDONET GLOBAL HYPERTEXT INTERFACE
**********************************************************************
Status:         draft
Revision:       initial incomplete draft
Title:          Fidonet Unicode substrings
Author:         <NAME>   (aka <NAME>, 2:50/88)
Revision Date:  28 Jul 2014
-+--------------------------------------------------------------------
Contents:
   1. Status of this document
   2. Introduction
   3. Key words to indicate requirement levels
   4. 8-bit encoding of a Fidonet message
      containing Unicode substrings
   5. Decoding of an 8-bit Fidonet message
      containing Unicode substrings
   6. Important notes
-+--------------------------------------------------------------------

1. Status of this document
-+------------------------

  This document is a draft of a Fidonet Standards Proposal (FSP).

  This document specifies an optional Fidonet standard
  that can be used in the Fidonet community.

  Implementation of the standard defined in this document is not
  mandatory, but all implementations are expected to adhere
  to this standard.

  Distribution of this document is unlimited, 
  provided that its text is not altered without notice.

2. Introduction
-+-------------

  Many classic Fidonet message editors (such as GoldED+, for example)
  were designed as 8-bit applications. They expect that each character
  in a Fidonet message is coded by one byte. Therefore they won't ever
  support Unicode UTF-8 or UTF-16 encoding.

  This situation is a problem of "chicken and egg" type. The messages
  in UTF-8 charset do not appear in Fidonet because they won't ever be
  read in any of the popular readers. On the other hand, the absence
  of such messages means that there's no need for the developers of
  popular readers to improve their soft, or for their users to upgrade
  their readers or to choose some newer (Unicode-supporting) readers.

  This document specifies a simple method that would allow Unicode
  substrings to appear (encoded and escaped) within 8-bit strings.

  The method of encoding is based on the UTF-7 format (RFC 2152).

  The method of escaping is inspired by HTML character references
  (HTML 4.01 subsection 5.3.1, subsection 5.3.2).

  By implementing this method, the following situation is achieved:

  *) The users of newer (Unicode-supporting) Fidonet applications
     can read and write Unicode substrings in 8-bit messages.

  *) The users of older (8-bit) Fidonet applications can read the
     8-bit parts of the message. The Unicode susbstrings remain
     unintelligible, but that's natural for an 8-bit application,
     and brings only a minor discomfort, and serves as a reason
     for an upgrade.

3. Key words to indicate requirement levels
-+-----------------------------------------

  The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
  "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY",
  and "OPTIONAL" in this document are to be interpreted as described
  in FTA-1006 (based on RFC 2119).

4. 8-bit encoding of a Fidonet message containing Unicode substrings
-+------------------------------------------------------------------

  First of all, the source (Unicode) text is split into an array of
  substrings following each other (successively), where the substrings
  that have even indices (0, 2, 4...) contain characters that can be
  encoded with the target encoding and the substrings that have odd
  indices (1, 3, 5...) contain characters that cannot be encoded with
  the target encoding. (Or vice versa; if a character that cannot be
  encoded with the target encoding appeared first, then its substring
  has zeroeth index, and all such substrings also have even indices.)

  The traditional 8-bit encoding is performed for the substrings that
  contain characters that can be encoded that way, i.e. each of such
  characters is represented by a byte.

  The other substrings ("Unicode substrings") are converted to the
  UTF-7 format (RFC 2152). For example, a string that consists of
  Unicode characters U+9802, U+5C16, U+5C0D, U+6C7A, U+4E4B, U+7A7F,
  U+8932, U+5B50, U+7BC7 is represented by the following string:

        +mAJcFlwNbHpOS3p/iTJbUHvH-

  However, the UTF-7 method of escaping (a plus before such string and
  a minus after) is not sufficient for Fidonet. Thus the minus MUST be
  replaced by a semicolon, the plus MUST be prepended by an ampersand.
  For example, a string that consists of Unicode characters U+9802,
  U+5C16, U+5C0D, U+6C7A, U+4E4B, U+7A7F, U+8932, U+5B50, U+7BC7 is
  represented by the following string:

        &+mAJcFlwNbHpOS3p/iTJbUHvH;

  Afterwards the traditional 8-but encoding is performed for these
  (ASCII-compatible) characters.

  The encoding results are concatenated (successively) in the order
  that substrings had in the initial array (i.e. in the order of
  their appearance in the source text).

5. Decoding of an 8-bit Fidonet message containing Unicode substrings
-+-------------------------------------------------------------------

  First of all, the message is decoded by a traditional 8-bit decoder,
  each byte is decoded to a character.

  Encoded Unicode substrings are then found in the message (using
  their unique form: an ampersand, then a plus, then one or more of
  base64 characters, then a semicolon) and replaced by their decoded
  equivalents.

  For finding that encoded forms, the following PECL (Perl-compatible
  regular expression) might be useful:

      /&\+[A-Za-z01-9+\/]+;/

  For decoding them, some RFC2152-compatible UTF-7 decoder MUST be
  used. (As explained in the previous section, Fidonet Unicode
  substrings use UTF-7 encoding and a different escaping. If the
  decoder expects RFC2152-compatible escaping, the ampersand MUST be
  removed and the semicolon MUST be replaced by a minus before the
  substring is given to the decoder.)

6. Important notes
-+----------------

  Note 1. An ampersand, a semicolon, a plus and some of base64 codes
  (for example, capital Latin letters) might appear in UUE code blocks
  in Fidonet. If a Fidonet message reader interpretes UUE codes, then
  it MUST isolate and decode UUE before it applies the decoder of
  Fidonet Unicode substrings to the rest of the message. If a Fidonet
  message reader does not interprete UUE codes (i.e. just presents UUE
  as a big lump of human-unreadable codes), it MAY not care if some of
  these codes end up converted to Unicode substrings.

**********************************************************************
EOTD                                               END OF THE DOCUMENT
**********************************************************************
{'fraction_non_alphanumeric': 0.14563993303911124, 'fraction_numerical': 0.021457921168771876, 'mean_word_length': 2.8463564530289727, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5160074', 'n_tokens_mistral': 3683, 'n_tokens_neox': 3305, 'n_words': 1708}