<story><title>Mercurial’s journey to and reflections on Python 3</title><url>https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journey-to-and-reflections-on-python-3/</url></story><parent_chain><item><author>pdonis</author><text>&lt;i&gt;&amp;gt; The natural string type is the right type in many places&lt;/i&gt;&lt;p&gt;For many programs, yes. Not for a revision control system that needs to be sure it&amp;#x27;s working with the exact binary data that&amp;#x27;s stored in the repository. Repository data is bytes, not Unicode.&lt;p&gt;I think this article is an excellent illustration of the Python developers&amp;#x27; failure to properly recognize this use case in the 2 to 3 transition.</text></item><item><author>ploxiln</author><text>I&amp;#x27;ve been involved in multiple non-trivial libraries and frameworks that supported both python2 and python3 for many years with the same codebase ... and it really wasn&amp;#x27;t anything like this. The python3 &amp;quot;adaptation&amp;quot; effort for mercurial was just bungled by multiple terrible decisions.&lt;p&gt;First was the idea that normal feature contributors should not see any b&amp;quot;&amp;quot; or any sign of python3 support for the first couple years of the effort. Huge mistake. You need some b&amp;quot;&amp;quot;.&lt;p&gt;But you don&amp;#x27;t need all b&amp;quot;&amp;quot; everywhere. That was the second huge mistake. Don&amp;#x27;t just convert every natural string in the whole codebase to b&amp;quot;&amp;quot;. The natural string type is the right type in many places, both for python2 (bytes-like) and python3 (unicode-like). The helpers for converting kwargs keys to&amp;#x2F;from bytes is a sign that you are way off track. This guy got really hung up on the fact that the python2 natural string type is bytes-like, and tryied to force explicit bytes everywhere (dict keys, http headers, etc) and was really tilting at windmills for most of these past 5 years.&lt;p&gt;Yes, you pretty much had to wait for python-3.4 to be released and for python-2.6 to be mostly retired in favor of python-2.7. Then, starting in early 2014, it was pretty straightforward to make a clean codebase compatible with python-2.7 and python-3.4+, and I saw it done for Tornado, paramiko, and a few other smaller projects.</text></item></parent_chain><comment><author>jharsman</author><text>I was an early adopter of Mercurial and the teams insistence that file names were byte strings was the cause of lots of bugs when it came to Unicode support.&lt;p&gt;For example, when I converted our existing Subversion repository to Mercurial I had to rename a couple of files that had non ASCII characters in their names because Mercurial couldn&amp;#x27;t handle it. At least on Windows file names would either be broken in Explorer or in the command line.&lt;p&gt;In fact I just checked and it is STILL broken in Mercurial 4.8.2 which I happened to have installed on my work laptop with Windows. Any file with non ASCII characters in the name is shown as garbled in the command line interface on Windows.&lt;p&gt;I remember some mailing list post way back when where mpm said that it was very important that hg was 8-bit clean since a Makefile might contain some random string of bytes that indicated a file and for that Makefile to work the file in question had to have the exact same string of bytes for a name. Of course, if file names are just strings of bytes instead of text, you can&amp;#x27;t display them, or send them over the internet to a machine with another file name encoding or do hardly anything useful with them. So basic functionality still seems to be broken to support unix systems with non-ascii filenames that aren&amp;#x27;t in UTF-8.</text></comment>
<story><title>Mercurial’s journey to and reflections on Python 3</title><url>https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journey-to-and-reflections-on-python-3/</url></story><parent_chain><item><author>pdonis</author><text>&lt;i&gt;&amp;gt; The natural string type is the right type in many places&lt;/i&gt;&lt;p&gt;For many programs, yes. Not for a revision control system that needs to be sure it&amp;#x27;s working with the exact binary data that&amp;#x27;s stored in the repository. Repository data is bytes, not Unicode.&lt;p&gt;I think this article is an excellent illustration of the Python developers&amp;#x27; failure to properly recognize this use case in the 2 to 3 transition.</text></item><item><author>ploxiln</author><text>I&amp;#x27;ve been involved in multiple non-trivial libraries and frameworks that supported both python2 and python3 for many years with the same codebase ... and it really wasn&amp;#x27;t anything like this. The python3 &amp;quot;adaptation&amp;quot; effort for mercurial was just bungled by multiple terrible decisions.&lt;p&gt;First was the idea that normal feature contributors should not see any b&amp;quot;&amp;quot; or any sign of python3 support for the first couple years of the effort. Huge mistake. You need some b&amp;quot;&amp;quot;.&lt;p&gt;But you don&amp;#x27;t need all b&amp;quot;&amp;quot; everywhere. That was the second huge mistake. Don&amp;#x27;t just convert every natural string in the whole codebase to b&amp;quot;&amp;quot;. The natural string type is the right type in many places, both for python2 (bytes-like) and python3 (unicode-like). The helpers for converting kwargs keys to&amp;#x2F;from bytes is a sign that you are way off track. This guy got really hung up on the fact that the python2 natural string type is bytes-like, and tryied to force explicit bytes everywhere (dict keys, http headers, etc) and was really tilting at windmills for most of these past 5 years.&lt;p&gt;Yes, you pretty much had to wait for python-3.4 to be released and for python-2.6 to be mostly retired in favor of python-2.7. Then, starting in early 2014, it was pretty straightforward to make a clean codebase compatible with python-2.7 and python-3.4+, and I saw it done for Tornado, paramiko, and a few other smaller projects.</text></item></parent_chain><comment><author>masklinn</author><text>&amp;gt; For many programs, yes.&lt;p&gt;For all programs, for the simple reason that:&lt;p&gt;&amp;gt; Various standard library functionality now wanted unicode str and didn&amp;#x27;t accept bytes, even though the Python 2 implementation used the equivalent of bytes.&lt;p&gt;Much of the stdlib works with &lt;i&gt;native strings&lt;/i&gt; and will either blow up or misbehave if fed anything else[0], which means much of your codebase will necessarily be &lt;i&gt;native strings&lt;/i&gt;, with a subset being explicitly bytes or unicode.&lt;p&gt;&amp;gt; Repository data is bytes, not Unicode.&lt;p&gt;It&amp;#x27;s also mostly absent from the source code, and where it is present (e.g. placeholders or separators) it&amp;#x27;s easy to flag as explicitly bytes.&lt;p&gt;[0] though some e.g. the encoding layers or io module want either bytes or unicode depending what you&amp;#x27;re doing specifically, and not always the most sensible, like baseXY being bytes -&amp;gt; bytes conversions where 95% of the use case is to smuggle binary data through text… oh well</text></comment>