<story><title>Curl doesn’t spew binary anymore</title><url>https://daniel.haxx.se/blog/2017/06/17/curl-doesnt-spew-binary-anymore/</url></story><parent_chain><item><author>baby</author><text>I just looked at the commit and I was surprised at how nice the code is! The relevant line to my question is:&lt;p&gt;&lt;pre&gt;&lt;code&gt;  if(isatty &amp;amp;&amp;amp; (outs-&amp;gt;bytes &amp;lt; 2000) &amp;amp;&amp;amp; terminal_binary_ok) {
    if(memchr(buffer, 0, bytes)) {
&lt;/code&gt;&lt;/pre&gt;
memchr() returns the first occurrence of the byte 0 (your second argument), or NULL.&lt;p&gt;So a few things:&lt;p&gt;* what if your output is more than 2000 bytes?&lt;p&gt;* what if your output is binary but doesn’t contain a byte 0?&lt;p&gt;* what if your output is a normal UTF-8 string but contains a byte 0? ( see &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;stackoverflow.com&amp;#x2F;questions&amp;#x2F;6907297&amp;#x2F;can-utf-8-contain-zero-byte&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;stackoverflow.com&amp;#x2F;questions&amp;#x2F;6907297&amp;#x2F;can-utf-8-contai...&lt;/a&gt; )&lt;p&gt;This is interesting to me as I am developing a tool that parses files in search for bugs and I need to ignore binary files. What I am doing right now: when checking a line, if it is not a valid UTF-8 string I skip the file. It&amp;#x27;s not really nice as I am doing this verification for every file&amp;#x27;s lines...</text></item></parent_chain><comment><author>arghwhat</author><text>1. It is not sensible to check more than a small chunk of data, as it would result in significant performance penalties, both in high resource consumption (memory or temporary files) and in blocked pipeline (cURL streams data as it is received). Imagine `curl &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;somesite&amp;#x2F;10GBoftext&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;somesite&amp;#x2F;10GBoftext&lt;/a&gt; | grep &amp;quot;rarely-occurring-prefix&amp;quot; | do-something-with-found-data`—with a full scan, curl would use 10GB of memory, checking every byte before it sent things to grep and do-something. Without a full scan, do-something processes data live, and curl uses negligible resources.&lt;p&gt;2. Then the check yields a false negative, which is not a problem.&lt;p&gt;3. Then your UTF-8 string is unprintable, and the check will yield a true positive. The UTF-8&amp;#x2F;ASCII NUL character is not printable, despite being valid.&lt;p&gt;If one only assumes ASCII&amp;#x2F;UTF-8&amp;#x2F;Shift-JIS&amp;#x2F;similar, then a blob containing a null byte is guaranteed to be unprintable, while a blob not containing a null byte may be printable. That&amp;#x27;s good enough for a warning, telling you that you&amp;#x27;re doing something that you might not have intended.&lt;p&gt;Given that UTF-8 has become standard, it means that you will never realistically get a false warning, but may still get bonkers output. You can always overrule if you have a fetish for UTF-32.</text></comment>
<story><title>Curl doesn’t spew binary anymore</title><url>https://daniel.haxx.se/blog/2017/06/17/curl-doesnt-spew-binary-anymore/</url></story><parent_chain><item><author>baby</author><text>I just looked at the commit and I was surprised at how nice the code is! The relevant line to my question is:&lt;p&gt;&lt;pre&gt;&lt;code&gt;  if(isatty &amp;amp;&amp;amp; (outs-&amp;gt;bytes &amp;lt; 2000) &amp;amp;&amp;amp; terminal_binary_ok) {
    if(memchr(buffer, 0, bytes)) {
&lt;/code&gt;&lt;/pre&gt;
memchr() returns the first occurrence of the byte 0 (your second argument), or NULL.&lt;p&gt;So a few things:&lt;p&gt;* what if your output is more than 2000 bytes?&lt;p&gt;* what if your output is binary but doesn’t contain a byte 0?&lt;p&gt;* what if your output is a normal UTF-8 string but contains a byte 0? ( see &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;stackoverflow.com&amp;#x2F;questions&amp;#x2F;6907297&amp;#x2F;can-utf-8-contain-zero-byte&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;stackoverflow.com&amp;#x2F;questions&amp;#x2F;6907297&amp;#x2F;can-utf-8-contai...&lt;/a&gt; )&lt;p&gt;This is interesting to me as I am developing a tool that parses files in search for bugs and I need to ignore binary files. What I am doing right now: when checking a line, if it is not a valid UTF-8 string I skip the file. It&amp;#x27;s not really nice as I am doing this verification for every file&amp;#x27;s lines...</text></item></parent_chain><comment><author>jrgv</author><text>&lt;i&gt;what if your output is more than 2000 bytes?&lt;/i&gt;&lt;p&gt;The announcement says that &amp;quot;curl will inspect the beginning of each download&amp;quot;, and I think that comparison just turns off the check after at least 2000 bytes have already been output (see a few lines below the change you quoted, where outs-&amp;gt;bytes is incremented by the amount of bytes that were output).&lt;p&gt;&lt;i&gt;what if your output is binary but doesn’t contain a byte 0?&lt;/i&gt;&lt;p&gt;I guess curl will incorrectly recognize the binary as text.&lt;p&gt;&lt;i&gt;what if your output is a normal UTF-8 string but contains a byte 0?&lt;/i&gt;&lt;p&gt;I guess curl will incorrectly recognize the text as binary, and you can use `-o -` to override that and output to the terminal anyway.</text></comment>