<story><title>Your code displays Japanese wrong</title><url>https://heistak.github.io/your-code-displays-japanese-wrong/</url></story><parent_chain><item><author>BoppreH</author><text>That&amp;#x27;s extremely interesting, if not depressing.&lt;p&gt;So, if I have to display user-entered text (usernames, posts, comments, messages, form data, etc), and I want to do The Right Thing™:&lt;p&gt;- I cannot rely on user locale, because it might be set to something generic like English, or the user may be bi-lingual.&lt;p&gt;- I cannot rely on location, because the user may be traveling to a different CJK region, or somewhere else altogether.&lt;p&gt;- I cannot set a single lang: attribute for the whole page because it&amp;#x27;ll be wrong for the other two languages.&lt;p&gt;- The string alone is not sufficient to identify the language because you can write valid sentences in different CJK languages with the same codepoints.&lt;p&gt;- I cannot have a per-user language setting, because users may be bi-lingual.&lt;p&gt;What does that leave me? A dropdown list &amp;quot;C&amp;#x2F;J&amp;#x2F;K&amp;#x2F;Other&amp;quot; besides &lt;i&gt;every single text field&lt;/i&gt;?&lt;p&gt;I&amp;#x27;m chucking this on my pile of examples of software development being hopelessly broken by design, along with &amp;quot;unix time is non-monotonic and discontinuous at random&amp;quot; (hint: what&amp;#x27;s the unix time exactly 1e8 seconds, ~3 years, from now? Answer: it&amp;#x27;s up to the astronomers[1]!).&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Unix_time#Leap_seconds&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Unix_time#Leap_seconds&lt;/a&gt;&lt;p&gt;Edit: actually, even the dropdown list is insufficient because it only allows one language per string! How is a Japanese user asking for help learning Chinese supposed to write?</text></item></parent_chain><comment><author>Asooka</author><text>Trying to do something smart is usually the wrong approach and leads to users tangling themselves in invisible state that they do not understand and can&amp;#x27;t change. The best thing would have been for Unicode to not do Han unification. The second best would be to provide alternate glyphs now. The third best is to display the characters either in the language they&amp;#x27;re written in, when you know that for sure (usually for text that you wrote yourself), or in the user&amp;#x27;s most likely locale, when you don&amp;#x27;t. For locale I would go down this list of traits and pick the first that matches:&lt;p&gt;1) The language setting on your website if you have one and have translated it to C&amp;#x2F;J&amp;#x2F;K. You may use different TLDs for the different languages and discern that way, too.&lt;p&gt;2) The list of preferred languages from the browser. This is usually unreliable, but if someone has gone to the trouble of inputting &amp;quot;english=1;japanese=.9;chinese=.8&amp;quot;, then it&amp;#x27;s a fair bet they want Japanese Kanji usually and will be understanding if you use them in place of Chinese Han characters.&lt;p&gt;3) The country to which the user&amp;#x27;s IP belongs. The least ideal option, but if you&amp;#x27;re in Korea and reading a random string of Hanzi, you probably expect them to look like Hanzi.&lt;p&gt;You will show the wrong characters to some users, but the behaviour is understandable. &amp;quot;Oh, the site is showing me Korean characters because I&amp;#x27;m in Korea.&amp;quot; is a lot easier to grasp than &amp;quot;The site is showing me Chinese characters because I clicked a dropdown one time that I forgot about and now I have no idea why my name is written wrong!&amp;quot;&lt;p&gt;You can argue about point 2) that some users might set their language preferences and forget about it, but so far I have never observed a user who doesn&amp;#x27;t know about them messing with the setting.</text></comment>
<story><title>Your code displays Japanese wrong</title><url>https://heistak.github.io/your-code-displays-japanese-wrong/</url></story><parent_chain><item><author>BoppreH</author><text>That&amp;#x27;s extremely interesting, if not depressing.&lt;p&gt;So, if I have to display user-entered text (usernames, posts, comments, messages, form data, etc), and I want to do The Right Thing™:&lt;p&gt;- I cannot rely on user locale, because it might be set to something generic like English, or the user may be bi-lingual.&lt;p&gt;- I cannot rely on location, because the user may be traveling to a different CJK region, or somewhere else altogether.&lt;p&gt;- I cannot set a single lang: attribute for the whole page because it&amp;#x27;ll be wrong for the other two languages.&lt;p&gt;- The string alone is not sufficient to identify the language because you can write valid sentences in different CJK languages with the same codepoints.&lt;p&gt;- I cannot have a per-user language setting, because users may be bi-lingual.&lt;p&gt;What does that leave me? A dropdown list &amp;quot;C&amp;#x2F;J&amp;#x2F;K&amp;#x2F;Other&amp;quot; besides &lt;i&gt;every single text field&lt;/i&gt;?&lt;p&gt;I&amp;#x27;m chucking this on my pile of examples of software development being hopelessly broken by design, along with &amp;quot;unix time is non-monotonic and discontinuous at random&amp;quot; (hint: what&amp;#x27;s the unix time exactly 1e8 seconds, ~3 years, from now? Answer: it&amp;#x27;s up to the astronomers[1]!).&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Unix_time#Leap_seconds&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Unix_time#Leap_seconds&lt;/a&gt;&lt;p&gt;Edit: actually, even the dropdown list is insufficient because it only allows one language per string! How is a Japanese user asking for help learning Chinese supposed to write?</text></item></parent_chain><comment><author>makeitdouble</author><text>For what it&amp;#x27;s worth, mainstream OSes will also have poor handling of these cases, which will eliminate the most tricky cases by the sheer inconvenience it causes.&lt;p&gt;As far as I know textfields only have one font applied, so entering both languages in a single field won&amp;#x27;t be optimal. And if you&amp;#x27;re not doing anything fancy with your fields, they will all take the same font as well.&lt;p&gt;So even at the input level, the user switching languages will already be mildly screwed, and the best solution would probably be to change pages for each language.</text></comment>