<story><title>Brian Kernighan adds Unicode support to Awk</title><url>https://github.com/onetrueawk/awk/commit/9ebe940cf3c652b0e373634d2aa4a00b8395b636</url></story><parent_chain><item><author>tialaramex</author><text>The choice to use UTF-32 (ie Unicode code points as integers, which might as well be 32-bit since your CPU definitely doesn&amp;#x27;t have a suitably sized integer type) is unexpected, as I had seen so many other systems just choose to work entirely in UTF-8 for this problem.&lt;p&gt;Now, Brian obviously has much better instincts about performance than I do and may even have tried some things and benchmarked them, but my guess would have been that you should stay in UTF-8 because it&amp;#x27;s always faster for the typical cases.</text></item></parent_chain><comment><author>simias</author><text>He mentions that &amp;quot;The amount of actual change isn&amp;#x27;t too great, so I think this might be ok&amp;quot; so I wonder if part of the equation has more to do with avoiding messing with legacy code rather than raw performance. If the current code expects all codepoints to have a constant-width representation, it may be complicated to add UTF-8 into the mix.&lt;p&gt;A complete guess on my part though, I never looked into AWK&amp;#x27;s source code.</text></comment>
<story><title>Brian Kernighan adds Unicode support to Awk</title><url>https://github.com/onetrueawk/awk/commit/9ebe940cf3c652b0e373634d2aa4a00b8395b636</url></story><parent_chain><item><author>tialaramex</author><text>The choice to use UTF-32 (ie Unicode code points as integers, which might as well be 32-bit since your CPU definitely doesn&amp;#x27;t have a suitably sized integer type) is unexpected, as I had seen so many other systems just choose to work entirely in UTF-8 for this problem.&lt;p&gt;Now, Brian obviously has much better instincts about performance than I do and may even have tried some things and benchmarked them, but my guess would have been that you should stay in UTF-8 because it&amp;#x27;s always faster for the typical cases.</text></item></parent_chain><comment><author>_0w8t</author><text>The code only uses UTF-32 in regular expressions where I suppose it was much simpler to  adopt the older code. The rest uses UTF-8.</text></comment>