<story><title>Language and shell in Go with 92% test coverage and instant CI/CD [video]</title><url>https://www.youtube.com/watch?v=YzIiUjgnSsA</url></story><parent_chain><item><author>throwaway2016a</author><text>This seems like a cool project.&lt;p&gt;This is meant as additional information not criticism. I skimmed the transcript really fast so if this is in there and I missed it, please correct me, but two things I think are helpful for people creating projects like this to be aware of:&lt;p&gt;- This video seems to combine the concepts of lexing and parsing. It is usually beneficial to separate these two steps and lex the input into tokens before passing to the parser.&lt;p&gt;- Go actually has a pure Go implementation of Yacc in the toolset and I&amp;#x27;ve used it in several projects to make parses. Dealing with the Yacc file is often much easier than dealing with code directly since it takes care of writing the actual parser. There is a lot of boiler plate that goes into parsers that when you use Yacc it &amp;quot;just works&amp;quot;.&lt;p&gt;Edit: there are also some tools for writing parsers in Lex&amp;#x2F;Flex like syntax (re2c comes to mind) but I&amp;#x27;ve found hand writing lexers to be effective in Go if your language doesn&amp;#x27;t have many different types of tokens.</text></item></parent_chain><comment><author>xiaq</author><text>Right, I may have forgot to mention that lexerless parsers are somewhat unusual.&lt;p&gt;I didn&amp;#x27;t have much time in the talk to go into the reason, so here it is:&lt;p&gt;- You&amp;#x27;ll need a more complex lexer to parse a shell-like syntax. For example, one common thing you do with lexers is get rid of whitespaces, but shell syntax is whitespace sensitive: &amp;quot;a$x&amp;quot; and &amp;quot;a $x&amp;quot; (double quotes not part of the code) are different things: the first is a single word containing a string concatenation, the second is two separate words.&lt;p&gt;- If your parser backtracks a lot, lexing can improve performance: you&amp;#x27;re not going back characters, only tokens (and there are fewer tokens than characters). Elvish&amp;#x27;s parser doesn&amp;#x27;t backtrack. (It does use lookahead fairly liberally.)&lt;p&gt;Having a lexerless parser does mean that you have to constantly deal with whitespaces in every place though, and it can get a bit annoying. But personally I like the conceptual simplicity and not having to deal with silly tokens like LBRACE, LPAREN, PIPE.&lt;p&gt;I have not used parser generators enough to comment about the benefits of using them compared to writing a parser by hand. The handwritten one works well so far :)</text></comment>
<story><title>Language and shell in Go with 92% test coverage and instant CI/CD [video]</title><url>https://www.youtube.com/watch?v=YzIiUjgnSsA</url></story><parent_chain><item><author>throwaway2016a</author><text>This seems like a cool project.&lt;p&gt;This is meant as additional information not criticism. I skimmed the transcript really fast so if this is in there and I missed it, please correct me, but two things I think are helpful for people creating projects like this to be aware of:&lt;p&gt;- This video seems to combine the concepts of lexing and parsing. It is usually beneficial to separate these two steps and lex the input into tokens before passing to the parser.&lt;p&gt;- Go actually has a pure Go implementation of Yacc in the toolset and I&amp;#x27;ve used it in several projects to make parses. Dealing with the Yacc file is often much easier than dealing with code directly since it takes care of writing the actual parser. There is a lot of boiler plate that goes into parsers that when you use Yacc it &amp;quot;just works&amp;quot;.&lt;p&gt;Edit: there are also some tools for writing parsers in Lex&amp;#x2F;Flex like syntax (re2c comes to mind) but I&amp;#x27;ve found hand writing lexers to be effective in Go if your language doesn&amp;#x27;t have many different types of tokens.</text></item></parent_chain><comment><author>ridiculous_fish</author><text>Shells have somewhat unusual parsing requirements. For example &amp;quot;if&amp;quot; is a keyword when used as `if echo` but not `echo if`.&lt;p&gt;So you either need to implement the lexer hack, or have a &amp;quot;string&amp;quot; token type which is disambiguated by the parser (which is what fish-shell does).&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Lexer_hack&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Lexer_hack&lt;/a&gt;</text></comment>