<story><title>Debunking Cloudflare’s recent performance tests</title><url>https://www.fastly.com/blog/debunking-cloudflares-recent-performance-tests</url></story><parent_chain><item><author>kentonv</author><text>I do not believe I suggested doing anything against the ToS. I think you&amp;#x27;re misinterpreting the clause. But not being a lawyer, I don&amp;#x27;t really want to get into that discussion.&lt;p&gt;If I didn&amp;#x27;t state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.</text></item><item><author>dcow</author><text>Sure, but you self-identified as someone responsible for the project and then suggested people go violate the ToS by running their own benchmarks. I think the community here calling out the double standards and asking for an update&amp;#x2F;response is entirely fair. If you didn&amp;#x27;t want the flak then leave off the &amp;quot;I&amp;#x27;m &lt;i&gt;the&lt;/i&gt; tech lead on CF workers&amp;quot; intro. Seems to me like the ball is in your court to at least try and make your advice actionable.</text></item><item><author>kentonv</author><text>To be clear my comment here is not in any way, shape, or form, Cloudflare&amp;#x27;s response. I am here representing only myself.</text></item><item><author>dwwoelfel</author><text>It has been two weeks since Fastly&amp;#x27;s VP of Eng called out your ToS and errors in your benchmark in her original tweet thread [1]. I would hope that Cloudflare would have a better response than this that directly addresses Fastly&amp;#x27;s claims.&lt;p&gt;Are you going to remove the ToS clause or issue a correction on the blog post?&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;lxt&amp;#x2F;status&amp;#x2F;1462896850055352320&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;lxt&amp;#x2F;status&amp;#x2F;1462896850055352320&lt;/a&gt;</text></item><item><author>kentonv</author><text>OK...&lt;p&gt;So, I&amp;#x27;m the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don&amp;#x27;t know anything about either beyond what&amp;#x27;s publicly visible.&lt;p&gt;But as long as we&amp;#x27;re already mud slinging, I&amp;#x27;d like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers&amp;#x27; 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn&amp;#x27;t seem like an interesting number?&lt;p&gt;We never called this out, it didn&amp;#x27;t seem worth arguing over. But excuse me if I&amp;#x27;m not impressed by claims of false comparisons...&lt;p&gt;On a serious note, I&amp;#x27;ve been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can&amp;#x27;t tell anything about how &lt;i&gt;your&lt;/i&gt; use case will perform unless you actually test that use case.  So, I would encourage everyone to run your own test and don&amp;#x27;t just go on other people&amp;#x27;s numbers. It&amp;#x27;s great that Fastly has opened up C@E for self-service testing so that people can actually try it out.</text></item></parent_chain><comment><author>justinpombrio</author><text>&amp;gt; I think you&amp;#x27;re misinterpreting the clause. But not being a lawyer, I don&amp;#x27;t really want to get into that discussion.&lt;p&gt;The clause says &amp;quot;Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;&amp;quot;[1]&lt;p&gt;IANAL, but this seems to very unambiguously prohibit benchmarking Cloudflare&amp;#x27;s services unless you have written permission. I know you don&amp;#x27;t want to get into an argument on HN, but could you like... bring it up to someone inside of CloudFlare who would be capable of changing it? You can point to this thread about how this clause is generating negative publicity.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.cloudflare.com&amp;#x2F;terms&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.cloudflare.com&amp;#x2F;terms&amp;#x2F;&lt;/a&gt; section 2.2</text></comment>
<story><title>Debunking Cloudflare’s recent performance tests</title><url>https://www.fastly.com/blog/debunking-cloudflares-recent-performance-tests</url></story><parent_chain><item><author>kentonv</author><text>I do not believe I suggested doing anything against the ToS. I think you&amp;#x27;re misinterpreting the clause. But not being a lawyer, I don&amp;#x27;t really want to get into that discussion.&lt;p&gt;If I didn&amp;#x27;t state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.</text></item><item><author>dcow</author><text>Sure, but you self-identified as someone responsible for the project and then suggested people go violate the ToS by running their own benchmarks. I think the community here calling out the double standards and asking for an update&amp;#x2F;response is entirely fair. If you didn&amp;#x27;t want the flak then leave off the &amp;quot;I&amp;#x27;m &lt;i&gt;the&lt;/i&gt; tech lead on CF workers&amp;quot; intro. Seems to me like the ball is in your court to at least try and make your advice actionable.</text></item><item><author>kentonv</author><text>To be clear my comment here is not in any way, shape, or form, Cloudflare&amp;#x27;s response. I am here representing only myself.</text></item><item><author>dwwoelfel</author><text>It has been two weeks since Fastly&amp;#x27;s VP of Eng called out your ToS and errors in your benchmark in her original tweet thread [1]. I would hope that Cloudflare would have a better response than this that directly addresses Fastly&amp;#x27;s claims.&lt;p&gt;Are you going to remove the ToS clause or issue a correction on the blog post?&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;lxt&amp;#x2F;status&amp;#x2F;1462896850055352320&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;lxt&amp;#x2F;status&amp;#x2F;1462896850055352320&lt;/a&gt;</text></item><item><author>kentonv</author><text>OK...&lt;p&gt;So, I&amp;#x27;m the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don&amp;#x27;t know anything about either beyond what&amp;#x27;s publicly visible.&lt;p&gt;But as long as we&amp;#x27;re already mud slinging, I&amp;#x27;d like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers&amp;#x27; 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn&amp;#x27;t seem like an interesting number?&lt;p&gt;We never called this out, it didn&amp;#x27;t seem worth arguing over. But excuse me if I&amp;#x27;m not impressed by claims of false comparisons...&lt;p&gt;On a serious note, I&amp;#x27;ve been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can&amp;#x27;t tell anything about how &lt;i&gt;your&lt;/i&gt; use case will perform unless you actually test that use case.  So, I would encourage everyone to run your own test and don&amp;#x27;t just go on other people&amp;#x27;s numbers. It&amp;#x27;s great that Fastly has opened up C@E for self-service testing so that people can actually try it out.</text></item></parent_chain><comment><author>hn_throwaway_99</author><text>First off, just want to say thanks for your posts, I found they give useful context and I really appreciate them.&lt;p&gt;I don&amp;#x27;t want the following to come off as unnecessarily argumentative, but regarding the ToS, I&amp;#x27;m not a lawyer either, but my &amp;quot;ability to read English&amp;quot; interpretation of the section on &amp;quot;&lt;i&gt;perform&lt;/i&gt; or publish benchmarks...&amp;quot; certainly sounds like it is prohibiting folks from doing their own side-by-side comparisons. Which is, of course, nonsense, because any engineer worth their salt would do their own analysis, even if they didn&amp;#x27;t publish it.&lt;p&gt;Just sounds to me like the CloudFlare lawyers got a little too aggressive to the point of absurdity, but I still think it&amp;#x27;s fair to call out CloudFlare for this.</text></comment>