<commit_before>.. @raise litre.TestsAreMissing

===================
Swift String Design
===================

Note:: This document represents the intended design of ``String``, not
its current implementation state.

.. contents:: Index

DaveZ's Proposed Document Structure
-----------------------------------

1. High level design rationale

Prose: words and thoughts like this: "deep reverence and respect for
the lessons learned from many languages and libraries". You don't
literally need to use the word "reverence" or "respect", but they do
help set the tone.

2. Examples and tutorials
3. Reference manual
4. High level comparison to NSString
5. Side by side comparisons with NSString
Take advantage of the back that you can reference back to sections 1,
2, and 3 in this outline. :-)

Chris' Outline of Important Points to Cover
-------------------------------------------

* Why having a builtin String is good

* The user model for Cocoa

  - Bridging
  - Extensions on String
  - Implicit Conversions
  - API?

* How it works?

  - Runtime Implementation Detail
  - How does Bridging Work
  - Measured Performance Cost
  - You can still use NSString

------------------------------------------------------------------------


General Principles
------------------

.. sidebar:: Rationale

   Making ``String`` a value type eliminates the need for defensive
   copying and a separate mutable string type.

   Unicode-awareness ensures that straightforward code written without
   attention to Unicode will not be badly broken in the presence of
   non-ASCII characters.  

   Locale-unawareness ensures that casual sorting or hashing of
   ``String``\ s is remains valid across changes in global system
   state.  Collections that automatically re-sort based on locale
   changes are firmly out-of-scope for Swift and in-scope for Cocoa.

   Encoding conversion is a neatly separable problem from other
   ``String`` issues, and—at least initially—we intend to refer users
   to Cocoa for that functionality.  When and if Swift itself acquires
   encoding support, it can go in a separate module with its own
   interfaces, e.g., ``utf16.encode(someString)``, thereby avoiding
   bloating the ``String`` interface.

   Strings are not indexable with ``Int`` because |Character|_\ s
   consume an arbitrary number of bytes, so they can't be randomly
   addressed with reasonable efficiency.

   .. _extending:

   The library uses restraint in extending ``String`` because
   ``String`` is a “vocabulary type” to and from which most other
   types are convertible.  Making these conversions members of
   ``String`` would quickly lead to an intolerably broad ``String``
   interface with intolerably slow code completion.

   The design of ``NSString`` is *very* different from the string
   designs of most modern programming languages, which all tend to be
   very similar to one another.  Although existing ``NSString`` users
   are a critical constituency today, current trends indicate that
   most of our *future* target audience will not be ``NSString``
   users. Absent compelling justification, it's important to make the
   Swift programming environment as familiar as possible for them.

* ``String`` is a mutable **value type**, just like ``Int``: each copy
  of a ``String`` is logically distinct from the original, and a
  ``String``’s value can be altered.

* ``String``\ 's primary interface is **Unicode-aware**.  For
  example::

    var x = "\u006E\u0303"   // Decomposed "ñ"
    var y = "\u00F1"         // Composed "ñ"
    assert(x == y)           // OK, passes

.. _locale-unaware:

* ``String``\ 's primary interface is **Locale-unaware**.  For any
  pair of strings ``s1`` and ``s2``, “``s1 == s2``” yields the same
  result regardless of the global locale. [#unaware]_

* The type currently called ``Char`` in Swift represents a Unicode
  code point.  This document refers to it as ``CodePoint``, in
  anticipation of renaming.

.. |Character| replace:: ``Character``
.. _Character:

* A new type called ``Character`` represents a Unicode `extended
  grapheme cluster`__ as defined by some Unicode segmentation
  algorithm.

  __ http://www.unicode.org/glossary/#extended_grapheme_cluster

* ``String`` elements are |Character|_\ s as defined by Unicode's
  `default segmentation`__ algorithm, rather than code units (as in
  Cocoa) or code points. [#elements]_ Secondary interfaces are
  provided for those who need to manipulate a ``String``\ 's
  ``CodePoint``\ s or its bytes.  For example::

    s                         // locale-independent Characters
    s.bytes                   // UInt8s (utf-8)
    s.codePoints              // CodePoints

  __ http://www.unicode.org/reports/tr29/#Default_Grapheme_Cluster_Table

* No ``String`` interfaces deal directly with encoding.

* ``String`` implements generic ``Indexable`` and ``Sliceable``
  protocols using a type conforming to ``BidirectionalIndex``, rather
  than ``Int``, as indices.

* When the user writes a string literal, she specifies a particular
  sequence of code points.  We guarantee that those code points are
  stored without change in the resulting ``String``.  The user
  can explicitly request normalization. [#normalizationbit]_

* The standard library does not extend ``String`` for each type with
  which it needs to interoperate.  Instead, types conforming to
  ``Printable`` have a ::

    func toString() -> String

  method, and types conforming to ``Parseable`` have a ::

    ``static func parse(s: String) -> This``

* ``String``\ 's interface design is primaily inspired by convention
  among “modern” programming languages such as Python, C#, Ruby, and
  Go.

Deviations from Unicode
-----------------------

Unicode is a pretty good standard.  Any deviation from what Unicode
specifies requires careful justification.  So far, we have found two
possible points of deviation for Swift ``String``:

1. The `Unicode Text Segmentation Specification`_ says, “`do not
   break between CR and LF`__.”  However, breaking between CR and LF
   may necessary if we wish ``String`` to “behave normally” for users
   of pure ASCII.  This point is still open for discussion.<commit_msg>[stdlib] StringDesign.rst: this => self<commit_after>.. @raise litre.TestsAreMissing

===================
Swift String Design
===================

Note:: This document represents the intended design of ``String``, not
its current implementation state.

.. contents:: Index

DaveZ's Proposed Document Structure
-----------------------------------

1. High level design rationale

Prose: words and thoughts like this: "deep reverence and respect for
the lessons learned from many languages and libraries". You don't
literally need to use the word "reverence" or "respect", but they do
help set the tone.

2. Examples and tutorials
3. Reference manual
4. High level comparison to NSString
5. Side by side comparisons with NSString
Take advantage of the back that you can reference back to sections 1,
2, and 3 in this outline. :-)

Chris' Outline of Important Points to Cover
-------------------------------------------

* Why having a builtin String is good

* The user model for Cocoa

  - Bridging
  - Extensions on String
  - Implicit Conversions
  - API?

* How it works?

  - Runtime Implementation Detail
  - How does Bridging Work
  - Measured Performance Cost
  - You can still use NSString

------------------------------------------------------------------------


General Principles
------------------

.. sidebar:: Rationale

   Making ``String`` a value type eliminates the need for defensive
   copying and a separate mutable string type.

   Unicode-awareness ensures that straightforward code written without
   attention to Unicode will not be badly broken in the presence of
   non-ASCII characters.  

   Locale-unawareness ensures that casual sorting or hashing of
   ``String``\ s is remains valid across changes in global system
   state.  Collections that automatically re-sort based on locale
   changes are firmly out-of-scope for Swift and in-scope for Cocoa.

   Encoding conversion is a neatly separable problem from other
   ``String`` issues, and—at least initially—we intend to refer users
   to Cocoa for that functionality.  When and if Swift itself acquires
   encoding support, it can go in a separate module with its own
   interfaces, e.g., ``utf16.encode(someString)``, thereby avoiding
   bloating the ``String`` interface.

   Strings are not indexable with ``Int`` because |Character|_\ s
   consume an arbitrary number of bytes, so they can't be randomly
   addressed with reasonable efficiency.

   .. _extending:

   The library uses restraint in extending ``String`` because
   ``String`` is a “vocabulary type” to and from which most other
   types are convertible.  Making these conversions members of
   ``String`` would quickly lead to an intolerably broad ``String``
   interface with intolerably slow code completion.

   The design of ``NSString`` is *very* different from the string
   designs of most modern programming languages, which all tend to be
   very similar to one another.  Although existing ``NSString`` users
   are a critical constituency today, current trends indicate that
   most of our *future* target audience will not be ``NSString``
   users. Absent compelling justification, it's important to make the
   Swift programming environment as familiar as possible for them.

* ``String`` is a mutable **value type**, just like ``Int``: each copy
  of a ``String`` is logically distinct from the original, and a
  ``String``’s value can be altered.

* ``String``\ 's primary interface is **Unicode-aware**.  For
  example::

    var x = "\u006E\u0303"   // Decomposed "ñ"
    var y = "\u00F1"         // Composed "ñ"
    assert(x == y)           // OK, passes

.. _locale-unaware:

* ``String``\ 's primary interface is **Locale-unaware**.  For any
  pair of strings ``s1`` and ``s2``, “``s1 == s2``” yields the same
  result regardless of the global locale. [#unaware]_

* The type currently called ``Char`` in Swift represents a Unicode
  code point.  This document refers to it as ``CodePoint``, in
  anticipation of renaming.

.. |Character| replace:: ``Character``
.. _Character:

* A new type called ``Character`` represents a Unicode `extended
  grapheme cluster`__ as defined by some Unicode segmentation
  algorithm.

  __ http://www.unicode.org/glossary/#extended_grapheme_cluster

* ``String`` elements are |Character|_\ s as defined by Unicode's
  `default segmentation`__ algorithm, rather than code units (as in
  Cocoa) or code points. [#elements]_ Secondary interfaces are
  provided for those who need to manipulate a ``String``\ 's
  ``CodePoint``\ s or its bytes.  For example::

    s                         // locale-independent Characters
    s.bytes                   // UInt8s (utf-8)
    s.codePoints              // CodePoints

  __ http://www.unicode.org/reports/tr29/#Default_Grapheme_Cluster_Table

* No ``String`` interfaces deal directly with encoding.

* ``String`` implements generic ``Indexable`` and ``Sliceable``
  protocols using a type conforming to ``BidirectionalIndex``, rather
  than ``Int``, as indices.

* When the user writes a string literal, she specifies a particular
  sequence of code points.  We guarantee that those code points are
  stored without change in the resulting ``String``.  The user
  can explicitly request normalization. [#normalizationbit]_

* The standard library does not extend ``String`` for each type with
  which it needs to interoperate.  Instead, types conforming to
  ``Printable`` have a ::

    func toString() -> String

  method, and types conforming to ``Parseable`` have a ::

    ``static func parse(s: String) -> Self``

* ``String``\ 's interface design is primaily inspired by convention
  among “modern” programming languages such as Python, C#, Ruby, and
  Go.

Deviations from Unicode
-----------------------

Unicode is a pretty good standard.  Any deviation from what Unicode
specifies requires careful justification.  So far, we have found two
possible points of deviation for Swift ``String``:

1. The `Unicode Text Segmentation Specification`_ says, “`do not
   break between CR and LF`__.”  However, breaking between CR and LF
   may necessary if we wish ``String`` to “behave normally” for users
   of pure ASCII.  This point is still open for discussion.
{'fraction_non_alphanumeric': 0.11390365716126953, 'fraction_numerical': 0.00467214435314967, 'mean_word_length': 3.177321668909825, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '473074', 'n_tokens_mistral': 3316, 'n_tokens_neox': 3094, 'n_words': 1560}