<story><title>The Beauty of Unix Pipelines</title><url>https://prithu.xyz/posts/unix-pipeline/</url></story><parent_chain><item><author>tsimionescu</author><text>The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world.&lt;p&gt;The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.</text></item><item><author>gorgoiler</author><text>To criticize sh semantics without acknowledging that C was &lt;i&gt;always there when you needed something serious&lt;/i&gt; is a bit short sighted.&lt;p&gt;There are two uses of the Unix “api”:&lt;p&gt;[A] Long lived tools for other people to use.&lt;p&gt;[B] Short lived tools one throws together oneself.&lt;p&gt;The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools designed for others to use, in A.&lt;p&gt;The ubiquity of the C APIs of course solved [A] use cases in the past, when it was unconscionable to operate a system without cc(1).  It’s part of why they get first class treatment in the Unix man pages, as old fashioned as that seems nowadays.</text></item><item><author>jcranmer</author><text>&amp;gt; (1) everything is text&lt;p&gt;And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you&amp;#x27;re invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them.</text></item><item><author>gorgoiler</author><text>Pipes are wonderful!  In my opinion you can’t extol them by themselves.  One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix:&lt;p&gt;(1) everything is &lt;i&gt;text&lt;/i&gt;&lt;p&gt;(2) everything (ish) is a &lt;i&gt;file&lt;/i&gt;&lt;p&gt;(3) including &lt;i&gt;pipes&lt;/i&gt; and &lt;i&gt;fds&lt;/i&gt;&lt;p&gt;(4) every piece of software is accessible as a file, invoked at the &lt;i&gt;command line&lt;/i&gt;&lt;p&gt;(5) ...with local &lt;i&gt;arguments&lt;/i&gt;&lt;p&gt;(6) ...and persistent globals in the &lt;i&gt;environment&lt;/i&gt;&lt;p&gt;A lot of understanding comes once you know what execve does, though such knowledge is of course not necessary.  It just helps.&lt;p&gt;Unix is seriously uncool with young people at the moment.  I intend to turn that around and articles like this offer good material.</text></item></parent_chain><comment><author>mbreese</author><text>Pipelines are mainly for short-lived, one-off quick scripts. But they are also really useful for when you control the input data. For example, if you need a summary report based on the results of a sql query.&lt;p&gt;If you control the inputs and you need to support quotes, spaces, non-white space delimiters, etc... in shell script, then that’s on you.&lt;p&gt;If you don’t control the inputs, then shell scripts are generally a poor match. For example, if you need summary reports from a client, but they sometimes provide the table in xlxs or csv format — shell might not be a good idea.&lt;p&gt;Might be controversial, but I think you can tell who works with shell pipes the most by looking at who uses CSV vs tab-delimited text files. Tabs can still be a pain if you have spaces in data.  But if you mix shell scripts with CSV, you’re just asking for trouble.</text></comment>
<story><title>The Beauty of Unix Pipelines</title><url>https://prithu.xyz/posts/unix-pipeline/</url></story><parent_chain><item><author>tsimionescu</author><text>The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world.&lt;p&gt;The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.</text></item><item><author>gorgoiler</author><text>To criticize sh semantics without acknowledging that C was &lt;i&gt;always there when you needed something serious&lt;/i&gt; is a bit short sighted.&lt;p&gt;There are two uses of the Unix “api”:&lt;p&gt;[A] Long lived tools for other people to use.&lt;p&gt;[B] Short lived tools one throws together oneself.&lt;p&gt;The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools designed for others to use, in A.&lt;p&gt;The ubiquity of the C APIs of course solved [A] use cases in the past, when it was unconscionable to operate a system without cc(1).  It’s part of why they get first class treatment in the Unix man pages, as old fashioned as that seems nowadays.</text></item><item><author>jcranmer</author><text>&amp;gt; (1) everything is text&lt;p&gt;And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you&amp;#x27;re invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them.</text></item><item><author>gorgoiler</author><text>Pipes are wonderful!  In my opinion you can’t extol them by themselves.  One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix:&lt;p&gt;(1) everything is &lt;i&gt;text&lt;/i&gt;&lt;p&gt;(2) everything (ish) is a &lt;i&gt;file&lt;/i&gt;&lt;p&gt;(3) including &lt;i&gt;pipes&lt;/i&gt; and &lt;i&gt;fds&lt;/i&gt;&lt;p&gt;(4) every piece of software is accessible as a file, invoked at the &lt;i&gt;command line&lt;/i&gt;&lt;p&gt;(5) ...with local &lt;i&gt;arguments&lt;/i&gt;&lt;p&gt;(6) ...and persistent globals in the &lt;i&gt;environment&lt;/i&gt;&lt;p&gt;A lot of understanding comes once you know what execve does, though such knowledge is of course not necessary.  It just helps.&lt;p&gt;Unix is seriously uncool with young people at the moment.  I intend to turn that around and articles like this offer good material.</text></item></parent_chain><comment><author>necrotic_comp</author><text>I&amp;#x27;ve been scripting stuff on the pipeline for over a decade and haven&amp;#x27;t really run into this much.&lt;p&gt;You can define a field separator in the command line with the environment variable IFS - i.e. &amp;#x27;IFS=$(echo -en &amp;quot;\n\b&amp;quot;);&amp;#x27; for newlines - which takes care of the basic cases like spaces in filenames&amp;#x2F;directory names when doing a for loop, and if I have other highly structured data that is heavily quoted or has some other sort of structure to it, then I either normalize it in some fashion or, as you suggest, write a perl script.&lt;p&gt;I haven&amp;#x27;t found it too much of a burden, even when dealing with exceptionally large files.</text></comment>