<story><title>Surprisingly Slow</title><url>https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/</url></story><parent_chain><item><author>peter_d_sherman</author><text>&amp;gt;&amp;quot;Closing File Handles on Windows&lt;p&gt;Many years ago I was profiling Mercurial to help improve the working directory checkout speed on Windows, as users were observing that checkout times on Windows were much slower than on Linux, even on the same machine.&lt;p&gt;I thought I could chalk this up to NTFS versus Linux filesystems or general kernel&amp;#x2F;OS level efficiency differences. What I actually learned was much more surprising.&lt;p&gt;When I started profiling Mercurial on Windows, I observed that most I&amp;#x2F;O APIs were completing in a few dozen microseconds, maybe a single millisecond or two ever now and then. Windows&amp;#x2F;NTFS performance seemed great!&lt;p&gt;Except for CloseHandle(). These calls were often taking 1-10+ milliseconds to complete. It seemed odd to me that file writes - even sustained file writes that were sufficient to blow past any write buffering capacity - were fast but closes slow. It was even more perplexing that CloseHandle() was slow even if you were using completion ports (i.e. async I&amp;#x2F;O). This behavior for completion ports was counter to what the MSDN documentation said should happen (the function should return immediately and its status can be retrieved later).&lt;p&gt;While I didn&amp;#x27;t realize it at the time, the cause for this was&amp;#x2F;is Windows Defender. Windows Defender (and other anti-virus &amp;#x2F; scanning software) typically work on Windows by installing what&amp;#x27;s called a filesystem filter driver. This is a kernel driver that essentially hooks itself into the kernel and receives callbacks on I&amp;#x2F;O and filesystem events. It turns out the close file callback triggers scanning of written data. And this scanning appears to occur synchronously, blocking CloseHandle() from returning. This adds milliseconds of overhead.&amp;quot;&lt;p&gt;PDS: Observation: In an OS, if I&amp;#x2F;O (or more generally, API calls) are initially written to run and return quickly -- this doesn&amp;#x27;t mean that they won&amp;#x27;t degrade (for whatever reason), as the OS expands and&amp;#x2F;or underlying hardware changes, over time...&lt;p&gt;For any OS writer, present or future, a key aspect of OS development is writing I&amp;#x2F;O (and API) performance tests, running them regularly, and &lt;i&gt;immediately halting development to understand&amp;#x2F;fix the root cause&lt;/i&gt; -- if and when performance anomalies are detected... in large software systems, in large codebases, it&amp;#x27;s usually much harder to gain back performance several versions after performance has been lost (i.e., Browsers), than to be disciplined, constantly test performance, and halt development (and understand&amp;#x2F;fix the root cause) the instant any performance anomaly is detected...</text></item></parent_chain><comment><author>greggman3</author><text>Related, If you copy a file via the OS&amp;#x27;s copy function the system knows the file was scanned and you get fast copies. If you copy the file by opening a new destination file for write, opening the source file for read, and copying bytes, then of course you trigger the virus scanner.&lt;p&gt;So for example I was using a build system and part of my build needed to copy ~5000 files of assets to the &amp;quot;out&amp;quot; folder. It was taking 5 seconds on other OSes and 2 minutes on Windows. Turned out the build system was copying using the &amp;quot;make a new file and copy bytes&amp;quot; approach instead of calling the their language&amp;#x27;s library copy function, which, at least on Windows, calls the OS copyfile function. I filed a bug and submitted a PR. Unfortunately while they acknowledged the issue they did not take the PR nor fix it on their side. My guess is they don&amp;#x27;t really care about devs that use Windows.&lt;p&gt;Note that python&amp;#x27;s copyfile does this wrong on MacOS. It also uses the open, read bytes, write bytes to new file method instead of calling into the OS. While it doesn&amp;#x27;t have the virus scanning issue (yet) it does mean files aren&amp;#x27;t actually &amp;quot;copied&amp;quot; so metadata is lost.</text></comment>
<story><title>Surprisingly Slow</title><url>https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/</url></story><parent_chain><item><author>peter_d_sherman</author><text>&amp;gt;&amp;quot;Closing File Handles on Windows&lt;p&gt;Many years ago I was profiling Mercurial to help improve the working directory checkout speed on Windows, as users were observing that checkout times on Windows were much slower than on Linux, even on the same machine.&lt;p&gt;I thought I could chalk this up to NTFS versus Linux filesystems or general kernel&amp;#x2F;OS level efficiency differences. What I actually learned was much more surprising.&lt;p&gt;When I started profiling Mercurial on Windows, I observed that most I&amp;#x2F;O APIs were completing in a few dozen microseconds, maybe a single millisecond or two ever now and then. Windows&amp;#x2F;NTFS performance seemed great!&lt;p&gt;Except for CloseHandle(). These calls were often taking 1-10+ milliseconds to complete. It seemed odd to me that file writes - even sustained file writes that were sufficient to blow past any write buffering capacity - were fast but closes slow. It was even more perplexing that CloseHandle() was slow even if you were using completion ports (i.e. async I&amp;#x2F;O). This behavior for completion ports was counter to what the MSDN documentation said should happen (the function should return immediately and its status can be retrieved later).&lt;p&gt;While I didn&amp;#x27;t realize it at the time, the cause for this was&amp;#x2F;is Windows Defender. Windows Defender (and other anti-virus &amp;#x2F; scanning software) typically work on Windows by installing what&amp;#x27;s called a filesystem filter driver. This is a kernel driver that essentially hooks itself into the kernel and receives callbacks on I&amp;#x2F;O and filesystem events. It turns out the close file callback triggers scanning of written data. And this scanning appears to occur synchronously, blocking CloseHandle() from returning. This adds milliseconds of overhead.&amp;quot;&lt;p&gt;PDS: Observation: In an OS, if I&amp;#x2F;O (or more generally, API calls) are initially written to run and return quickly -- this doesn&amp;#x27;t mean that they won&amp;#x27;t degrade (for whatever reason), as the OS expands and&amp;#x2F;or underlying hardware changes, over time...&lt;p&gt;For any OS writer, present or future, a key aspect of OS development is writing I&amp;#x2F;O (and API) performance tests, running them regularly, and &lt;i&gt;immediately halting development to understand&amp;#x2F;fix the root cause&lt;/i&gt; -- if and when performance anomalies are detected... in large software systems, in large codebases, it&amp;#x27;s usually much harder to gain back performance several versions after performance has been lost (i.e., Browsers), than to be disciplined, constantly test performance, and halt development (and understand&amp;#x2F;fix the root cause) the instant any performance anomaly is detected...</text></item></parent_chain><comment><author>chrisweekly</author><text>&amp;gt; &amp;quot;For any OS writer, present or future, a key aspect of OS development is writing I&amp;#x2F;O (and API) performance tests, running them regularly, and immediately halting development to understand&amp;#x2F;fix the root cause -- if and when performance anomalies are detected... in large software systems, in large codebases, it&amp;#x27;s usually much harder to gain back performance several versions after performance has been lost (i.e., Browsers), than to be disciplined, constantly test performance, and halt development (and understand&amp;#x2F;fix the root cause) the instant any performance anomaly is detected...&amp;quot;&lt;p&gt;Yes, this! And not just OS writers, but authors of any kind of software. Performance is like a living thing; vigilance is required.</text></comment>