<story><title>How fast are Linux pipes anyway? (2022)</title><url>https://mazzo.li/posts/fast-pipes.html</url></story><parent_chain><item><author>crabbone</author><text>I have a very long list of things that were good, worked well, and ended up rejected because the team didn&amp;#x27;t want to put in effort to learn how they work.&lt;p&gt;My conclusion so far is that if you want to make things work well, you shouldn&amp;#x27;t be working on a commercial project, use a unpopular language with a steep learning curve to filter out those who&amp;#x27;d be a drag on your project.  Maybe you don&amp;#x27;t have to be a jerk, but being blunt helps.&lt;p&gt;Below are some examples of initiatives that were meant to improve things and how they failed due to other programmers being lazy and &amp;#x2F; or ignorant.&lt;p&gt;When ActionScript was a thing it competed with HaXe.  A similar (also ECMAScript-related) language with a small but dedicated community, a compiler that was hugely superior to to MXMLC (official Adobe compiler for AS3) and a bunch of features intended to improve code correctness and performance.&lt;p&gt;I was hired by a company making a &amp;quot;PowerPoint online&amp;quot; kind of product.  The main system component was a large Flex (AS3) applet that was hugely inefficient especially in terms of how it utilized network.  It had to load huge shared libraries with assets (mostly clip art) every time users wanted to either edit or watch a presentation.  The AWS bill was growing dangerously big.  My mission was to find a solution to reduce the network activity.&lt;p&gt;My idea was to create a separate player component that would extract the relevant assets from the libraries server-side, compile them into individual SWFs. The reason was that the final presentations were loaded a lot more often and by first-time users (i.e. no caching).  HaXe was the ideal language because it already had a library that could generate a large subset of SWF, and it could compile both to AS3 and to C++, so that generation could also be done on a server using a more efficient implementation.&lt;p&gt;After several month of work, I produced a set of programs that could generate SWFs both server-side and client side and showed how this would improve the network activity.  The other programmers on the AS3 team, who earlier promised to get familiar with HaXe, since they had to incorporate the new player component into the existing Flex applet... didn&amp;#x27;t hold their part of the bargain.  No matter the amount of help I provided, they simply wouldn&amp;#x27;t do anything to incorporate the new component, instead making claims that grew more bizarre and more untrue as time went by.&lt;p&gt;Having spent more time trying to convince the team to adopt my code rather than writing it, I decided to look for a different place to work at.  In the end, this entire effort went down the drain.&lt;p&gt;----&lt;p&gt;In a very similar way, I had to solve a problem created by using Google&amp;#x27;s Protobuf Python bindings which required generating Python modules in order to function.  We needed to have an API server that could simultaneously serve multiple versions of the same Protobuf API from similarly named modules.  Since Google&amp;#x27;s implementation didn&amp;#x27;t allow this, I wrote my own (while my manager was on maternity leave). I improved parsing speed, network load, reduced the amount of maintenance the component needed by making it possible to add new Protobuf message definitions at run time...&lt;p&gt;The problem was I wrote the parser in C.  This is what enabled good performance.  When my manager came back to work, she realized she declared that she doesn&amp;#x27;t know C and will never learn (even though she wasn&amp;#x27;t related directly to the project), and the project was thrown to the dogs.&lt;p&gt;----&lt;p&gt;I have a similar story about extracting and aggregating Web interface from a RoR app, producing a Swagger definition... written in Prolog, which was also thrown away because Prolog.  Similarly, had written an I&amp;#x2F;O tester for distributed filesystem in Prolog, which was thrown away for the same reason...  And this answer will eventually hit the character limit if I keep listing things that were discarded simply because programmers didn&amp;#x27;t want to learn how to do their job.</text></item><item><author>db48x</author><text>Not necessarily. Good comments go a long way.</text></item><item><author>winternewt</author><text>And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.</text></item><item><author>Too</author><text>So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time.&lt;p&gt;Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 20x slower than what is possible.&lt;p&gt;Exceptionally written article btw.</text></item></parent_chain><comment><author>__turbobrew__</author><text>The part you left out is where you inevitably leave the company and now there is this bespoke snowflake which nobody  knows about doing god knows what transpiling stuff under the hood. With the average tenure of a software engineer being a few years maintainability and standardization are important above all. This is one of the reasons why opinionated and simple languages like golang have gained large popularity.&lt;p&gt;I do agree that genuine software excellence and craft is rare in corporate settings. If you want to make a piece of excellent software like curl or sqlite you will be doing it on your own, and you most likely will not be making a living off of it either.</text></comment>
<story><title>How fast are Linux pipes anyway? (2022)</title><url>https://mazzo.li/posts/fast-pipes.html</url></story><parent_chain><item><author>crabbone</author><text>I have a very long list of things that were good, worked well, and ended up rejected because the team didn&amp;#x27;t want to put in effort to learn how they work.&lt;p&gt;My conclusion so far is that if you want to make things work well, you shouldn&amp;#x27;t be working on a commercial project, use a unpopular language with a steep learning curve to filter out those who&amp;#x27;d be a drag on your project.  Maybe you don&amp;#x27;t have to be a jerk, but being blunt helps.&lt;p&gt;Below are some examples of initiatives that were meant to improve things and how they failed due to other programmers being lazy and &amp;#x2F; or ignorant.&lt;p&gt;When ActionScript was a thing it competed with HaXe.  A similar (also ECMAScript-related) language with a small but dedicated community, a compiler that was hugely superior to to MXMLC (official Adobe compiler for AS3) and a bunch of features intended to improve code correctness and performance.&lt;p&gt;I was hired by a company making a &amp;quot;PowerPoint online&amp;quot; kind of product.  The main system component was a large Flex (AS3) applet that was hugely inefficient especially in terms of how it utilized network.  It had to load huge shared libraries with assets (mostly clip art) every time users wanted to either edit or watch a presentation.  The AWS bill was growing dangerously big.  My mission was to find a solution to reduce the network activity.&lt;p&gt;My idea was to create a separate player component that would extract the relevant assets from the libraries server-side, compile them into individual SWFs. The reason was that the final presentations were loaded a lot more often and by first-time users (i.e. no caching).  HaXe was the ideal language because it already had a library that could generate a large subset of SWF, and it could compile both to AS3 and to C++, so that generation could also be done on a server using a more efficient implementation.&lt;p&gt;After several month of work, I produced a set of programs that could generate SWFs both server-side and client side and showed how this would improve the network activity.  The other programmers on the AS3 team, who earlier promised to get familiar with HaXe, since they had to incorporate the new player component into the existing Flex applet... didn&amp;#x27;t hold their part of the bargain.  No matter the amount of help I provided, they simply wouldn&amp;#x27;t do anything to incorporate the new component, instead making claims that grew more bizarre and more untrue as time went by.&lt;p&gt;Having spent more time trying to convince the team to adopt my code rather than writing it, I decided to look for a different place to work at.  In the end, this entire effort went down the drain.&lt;p&gt;----&lt;p&gt;In a very similar way, I had to solve a problem created by using Google&amp;#x27;s Protobuf Python bindings which required generating Python modules in order to function.  We needed to have an API server that could simultaneously serve multiple versions of the same Protobuf API from similarly named modules.  Since Google&amp;#x27;s implementation didn&amp;#x27;t allow this, I wrote my own (while my manager was on maternity leave). I improved parsing speed, network load, reduced the amount of maintenance the component needed by making it possible to add new Protobuf message definitions at run time...&lt;p&gt;The problem was I wrote the parser in C.  This is what enabled good performance.  When my manager came back to work, she realized she declared that she doesn&amp;#x27;t know C and will never learn (even though she wasn&amp;#x27;t related directly to the project), and the project was thrown to the dogs.&lt;p&gt;----&lt;p&gt;I have a similar story about extracting and aggregating Web interface from a RoR app, producing a Swagger definition... written in Prolog, which was also thrown away because Prolog.  Similarly, had written an I&amp;#x2F;O tester for distributed filesystem in Prolog, which was thrown away for the same reason...  And this answer will eventually hit the character limit if I keep listing things that were discarded simply because programmers didn&amp;#x27;t want to learn how to do their job.</text></item><item><author>db48x</author><text>Not necessarily. Good comments go a long way.</text></item><item><author>winternewt</author><text>And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.</text></item><item><author>Too</author><text>So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time.&lt;p&gt;Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 20x slower than what is possible.&lt;p&gt;Exceptionally written article btw.</text></item></parent_chain><comment><author>Nathanba</author><text>wouldnt it have been easier to port the Haxe code to AS3? afaik both language are very similar. Then you wouldn&amp;#x27;t have a problem and you wouldnt have to rely on them to learn your codebase. Likewise I doubt that the parser being written in C was the problem, the problem was probably that (I assume?) you didnt also provide the convenient python bindings. Otherwise it&amp;#x27;s hard to believe that someone would throw away finished code. I myself was in a similar situation once though, I bound to an external java code via JNI and it worked fine. But they ended up rewriting via HTTP calls anyway. Sure it&amp;#x27;s easier that way but I mean damn.. not even keeping it around as an option or as a benchmark seems a bit too lazy. But I still understood why they did it, it&amp;#x27;s just a source of possible errors for them and they don&amp;#x27;t want to deal with it. Also I wouldn&amp;#x27;t want to inherit Prolog code either because it&amp;#x27;s an ancient niche language, IDE and docs and everything else is probably terrible.</text></comment>