<story><title>UTF-8 Everywhere (2012)</title><url>http://utf8everywhere.org/</url></story><parent_chain><item><author>Animats</author><text>The Python problem is amusing. Python 3 has three representations of strings internally (1-byte, 2-byte, and 4-byte) and promotes them to a wider form when necessary. This is mostly to support string indexing. It probably would have been better to use UTF-8, and create an index array for the string when necessary.&lt;p&gt;You rarely need to index a string with an integer in Python.  FOR loops don&amp;#x27;t need to.  Regular expressions don&amp;#x27;t need to.  Operations that return a position into the string could return an opaque type which acts as a string index.  That type should support adding and subtracting integers (at least +1 and -1) by progressing through the string.  That would take care of most of the use cases. Attempts to index a string with an int would generate index arrays internally. (Or, for short strings, just start at the beginning every time and count.)&lt;p&gt;Windows and Java have big problems. They really are 16-bit char based. It&amp;#x27;s not Java&amp;#x27;s fault; they standardized when Unicode was 16 bits.</text></item></parent_chain><comment><author>johncolanduoni</author><text>I think it&amp;#x27;s even better to take this one step further and have your default &amp;quot;character&amp;quot; actually be a grapheme[1]. In almost any case where you&amp;#x27;re dealing with individual character boundaries you want to split things on the grapheme level, not the code-point level.&lt;p&gt;This doesn&amp;#x27;t matter much for (normalized) western European text, but if the language in question needs to use separate diacritical code points you&amp;#x27;ll likely end up with hanging accents in the like. Swift is the only language I know of that has grapheme clusters as the default unit of character, I&amp;#x27;d love to see it in more places.&lt;p&gt;[1]: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;unicode.org&amp;#x2F;reports&amp;#x2F;tr29&amp;#x2F;#Grapheme_Cluster_Boundaries&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;unicode.org&amp;#x2F;reports&amp;#x2F;tr29&amp;#x2F;#Grapheme_Cluster_Boundaries&lt;/a&gt;</text></comment>
<story><title>UTF-8 Everywhere (2012)</title><url>http://utf8everywhere.org/</url></story><parent_chain><item><author>Animats</author><text>The Python problem is amusing. Python 3 has three representations of strings internally (1-byte, 2-byte, and 4-byte) and promotes them to a wider form when necessary. This is mostly to support string indexing. It probably would have been better to use UTF-8, and create an index array for the string when necessary.&lt;p&gt;You rarely need to index a string with an integer in Python.  FOR loops don&amp;#x27;t need to.  Regular expressions don&amp;#x27;t need to.  Operations that return a position into the string could return an opaque type which acts as a string index.  That type should support adding and subtracting integers (at least +1 and -1) by progressing through the string.  That would take care of most of the use cases. Attempts to index a string with an int would generate index arrays internally. (Or, for short strings, just start at the beginning every time and count.)&lt;p&gt;Windows and Java have big problems. They really are 16-bit char based. It&amp;#x27;s not Java&amp;#x27;s fault; they standardized when Unicode was 16 bits.</text></item></parent_chain><comment><author>dietrichepp</author><text>There are a couple cases for string indexing, usually involving parsing or regular expressions.  You might want to slice the quotes off of a quoted string, or slice from one match of a regular expression to a match of a different regular expression starting at a different index.  These come up infrequently enough that it doesn&amp;#x27;t make sense to make a better API just for these use cases, but frequently enough that it would be a serious impediment if we didn&amp;#x27;t do &lt;i&gt;some&lt;/i&gt; kind of string indexing.&lt;p&gt;I agree, however, that it&amp;#x27;s completely irrelevant whether the indexes correspond to code units (i.e. byte offsets in UTF-8) or whether they correspond to code points (how it works in Python currently), as long as we have some way to store, compare, and otherwise manipulate locations within a string.&lt;p&gt;Some Rust developers at one point proposed making string indexes their own (opaque) type, as you suggest, so that they couldn&amp;#x27;t be confused with integers used for other purposes.  The extra complexity of such an API meant that this proposal was never really taken seriously, and it only prevents a small category of programming errors.&lt;p&gt;You might be interested in looking at some string APIs which are mostly without string indexing, like Haskell&amp;#x27;s Data.Text, which is one of the most well-designed string APIs ever made.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;hackage.haskell.org&amp;#x2F;package&amp;#x2F;text-1.2.2.1&amp;#x2F;docs&amp;#x2F;Data-Text.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;hackage.haskell.org&amp;#x2F;package&amp;#x2F;text-1.2.2.1&amp;#x2F;docs&amp;#x2F;Data-T...&lt;/a&gt;&lt;p&gt;As for Windows, my Windows apps use UTF-8 everywhere, and then convert to wchar_t at the last possible moment when interacting with the Windows API. I believe this is what UTF-8 Everywhere suggests.</text></comment>