<story><title>Show HN: Faster UTF-8 validator</title><url>https://github.com/zwegner/faster-utf8-validator</url></story><parent_chain></parent_chain><comment><author>andrewf</author><text>Hi! I took a shot at vectorized UTF-8 validation in 2012. I just put this faster validator, and Daniel Lemire&amp;#x27;s code, into my test harness at &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;andrewffff&amp;#x2F;utf8fuzz&amp;#x2F;tree&amp;#x2F;2019_compare&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;andrewffff&amp;#x2F;utf8fuzz&amp;#x2F;tree&amp;#x2F;2019_compare&lt;/a&gt;&lt;p&gt;On an i7-7800X (clang  6.0.0-1ubuntu2, WSL, don&amp;#x27;t trust my numbers) my benchmarks showed about the same relationship between SSE4, AVX2 and Lemire&amp;#x27;s code, as your benchmarks did. My own attempt is about half as fast. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;raw.githubusercontent.com&amp;#x2F;andrewffff&amp;#x2F;utf8fuzz&amp;#x2F;2019_compare&amp;#x2F;rough-benchmark.png&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;raw.githubusercontent.com&amp;#x2F;andrewffff&amp;#x2F;utf8fuzz&amp;#x2F;2019_c...&lt;/a&gt;&lt;p&gt;A few examples of invalid UTF-8 from Markus Kuhn&amp;#x27;s suite pass this validator right now, specifically 4.1.3, 4.2.3 and 4.3.3. My randomized tests, which compare the results from different validators, also fail a small percentage of the time, I&amp;#x27;d guess for the same reason. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.cl.cam.ac.uk&amp;#x2F;~mgk25&amp;#x2F;ucs&amp;#x2F;examples&amp;#x2F;UTF-8-test.txt&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.cl.cam.ac.uk&amp;#x2F;~mgk25&amp;#x2F;ucs&amp;#x2F;examples&amp;#x2F;UTF-8-test.txt&lt;/a&gt;&lt;p&gt;I&amp;#x27;m really interested in the different approaches taken here. Fortunately both Daniel and you&amp;#x27;ve communicated what you were doing, I think it&amp;#x27;s going to take longer for me to re-comprehend my own approach!&lt;p&gt;Thanks for sharing this.</text></comment>
<story><title>Show HN: Faster UTF-8 validator</title><url>https://github.com/zwegner/faster-utf8-validator</url></story><parent_chain></parent_chain><comment><author>BeeOnRope</author><text>Good stuff.&lt;p&gt;The key to a solid claim, especially for something as bold as &amp;quot;fastest in the world&amp;quot;, is a complete specification of the inputs to the benchmark.&lt;p&gt;You mention random ASCII bytes and &amp;quot;random UTF-8 bytes&amp;quot;. The former is definitely an really important case but also the least interesting. I can write on a napkin a UTF-8-but-is-actually-ASCII decoder that approaches 256 bytes a cycle cached (with a fallback routine when the ASCII assumption fails).&lt;p&gt;So then what about the random UTF-8 bytes. What does it mean? Do you generate random bytes and then exclude invalid sequences? Do you generate a uniform random codepoint in the 21-bit codepoint space and then concert it to utf-8?&lt;p&gt;At a minimum it would good to see a benchmark with random distributions that approximate common languages.&lt;p&gt;The true fastest decoders on such realistic data will be adaptive and more complicated than yours.</text></comment>