A Lisp REPL as my main shell
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Neidhardt, Pierre
The popular but aging shells (Bash and the like) suffer from many design flaws: lack of structured data, pipes are hard-to-debug blackboxes, lack of interactivity, while the user interfaces are mostly poor and limiting. High time we moved on away from this cruft, starting with a top-notch interactive language boasting full-fledged introspection and debugging. For this experiment I've used Common Lisp and SLY, a Common Lisp REPL and development environment for Emacs.
 Hello, I'm Pierre Nighthart, and I'm going to showcase my development environment, which is based on a rather unconventional shell. There are many features to demo, so without further ado, let's get started! First thing first, what's wrong with the current state of affairs? Let me replay a shell session that I went through the other day. I wanted to collect the checksums of all the files with the name Nix. First, I switched to the right directory. I find that having to type the CD command is tedious, even with completion. Now I list the desired files recursively. Ok, let's collect the checksums. Whoops, first bit for, I forgot to surround a variable name with double quotes. But let's grab the first result this time. Uh oh. Wait, this wasn't there before. What's going on? Uh huh. I have a file with a new line, and it seems that the textual representation of the new line differs between LS, find, and grep. Grep mismatches the new line for an N. Now what if we use another tool, like find? This returns almost what I want. But if we were not for show and sum, displaying a spurious backslash for the file with the new line. Finally, if I want to copy this result, I am forced to use the mouse for the lack of better keyboard controls. Let's review the list of issues I've been trying to address. First thing is text manipulation. Maybe the most basic feature is to be able to search the whole shell buffer just like we would do in any other text editor. We should be able to copy and paste using keyboard shortcuts, not just the mouse. Maybe a bit pancier. I'd like to be able to navigate the prompts, but after all, isn't it the most natural kind of sectioning for a shell? Then performance. First, bash and france can be very slow, but maybe most importantly, the idiomatic way to filter and process outputs is to spawn a system process for each unique's tools, such as grep, sort, etc. And last, the typical unique's shell workflow becomes problematic when you want to reprocess an output. What you would do is rerun the command that produced the output in a different pipeline. This can be a showstopper if the initial command is very slow. Referring to the output using back references is one solution to this problem. Debugging. Wouldn't it be nice to have state-of-the-art debugging tools, like interactive stack traces and introspectable objects? As but not least, handle structured data. List hash tables, objects, you name it. In particular, in pipelines. In bash, the file list is typically passed around as a string buffer. Ideally, we should be able to pass lists of files objects. Let's get down to the demo. This is Sly, my common list rebel. It displays a typical multiline prompt with the current working directory at the top and to the left there is an index, here, 0, which is the back reference index. I will come back to it in a minute. Let's start with a simple example, say a calculation of a big power. Ta-da! That was fast. Evidently, common lists can handle large arbitrary precision numbers out of the box. So it's pretty handy as a pocket calculator, even for more advanced math. Conversely, bash is pretty bad, even for simple calculations. Let's move on to a more typical task. Compute the share one sum of a file. Let's take this.png file here. Notice that I have tried completion. Okay, now what if I want to compare it with the value that was recorded in the.sum file? Here. I could compare them visually, but that would be a bit tedious. Instead I can leverage the language together with a very important concept I'm going to present here, back references. On pressing hash V, all the previous results get prefixed with the index which I can complete upon. On pressing zero, the non-zero prefixes disappear and my hash V turns to orange, thus validating the match. Let's repeat it with the second result. This returns T for true, meaning the strings are identical and the checksum is valid. Previously, I mentioned the issue of search and copy paste. Since my repo is actually an Emacs, I can use all the features of the text editor, including search. Say I want to look up the term. Term, they all get highlighted as I type. Then I can cycle to the results. When the cursor is on the previous prompt and I press enter, it paste the whole command to the current prompt, ready for editing. Of course, I can also move around with a keyboard, select an proportional text and copy it. Next, I'd like to introduce the selector library I've worked on. When working on multiple projects, each of them using multiple shells, it's only too easy to get overwhelmed with the amount of open shells and keep stumbling around to find a desired shell. To the right, the list of open shells, only one is open at the moment. Let's open a new shell and call it project. The new shell popped up at the bottom. This shell shares the same underlying list process as the one at the top. So if I were to define a global variable, checksum, holding the value of the previously computed shell one. They will be understood in both shells. What if we want to open a new shell using a new unrelated list process? Here I have a list of predefined list processes. Some contain preloaded libraries for working on the next project. Some use a different list compiler altogether. I can even run the list process in a container with limited file system access. Let's pick the first one. In this new shell, if I enter checksum, the shell will expectantly complain that the symbol is undefined. Shell management goes beyond just that. I can rename the shell, say, to my project. If I review the list of shells, I see that I have three of them. I can fuzzy search, narrow down, or even select multiple shells. If I press enter on a selection, it displays all the selected shells on screen. Let's showcase the prompt navigation feature. I can navigate the prompt up and down like I did before. Not only that, but I can fuzzy search all the prompts in all open shells. So if I were to search the term checksum, the view gets updated to the shell containing a matching prompt. Let's enter, confirm the selection, and brings me to the desired prompt. Another itch that I've been scratching for too long, directory switching. Instead of typing CD, I want to use my regular file manager. Here it's Enix Helm, but it could be your GNOME or KDE file manager. I can fuzzy match against directories. Then pressing a keyboard shortcut brings me to the selected directory in my shell. It's just as easy to go anywhere, say to the root directory, and from there, I can invoke the history of selected or visited directories, which is also fuzzy searchable. And in a matter of just a few keystrokes, I'm back to where I started. Similarly, the history of commands is also fuzzy searchable. Say I were to search the previously computed checksum command, notice how it narrows down live, and how it successfully matches regardless of the order of the terms. While this shell is primary list, it also understands string interpolation, just like in bash, say I were to get the PID of the list process. Now I could query the command that started it by inspecting the press entry corresponding to it. So I can dollar escape the PID variable, just like I would do in bash. And here I get the full command. What's interesting about this is that this is not standard common list. Here the syntax of the language has been extended to understand the special strings. In the same way, the syntax can be extended at will to make common writing as convenient as possible. Say you don't want to write any parenthesis to execute a simple command. No problem. I can write any valid bash command directly by prefixing it with a special shabang. Again, this is not standard common list. This is an extension. Let's consider the more complex case of pipeline. I want to return the three most frequent terms in a dictionary of words containing duplicates. Here is the dictionary. Now I filter it by the unique command, sort, and finally get the three most frequent words. Now what if we wanted a more less basic text? Would it be two verbose? I believe it does not have to be. With the right commands and keyboard shortcuts, it can be just as fast to type. I complete here and insert the pipeline, which is this column dash. Here I'm using a keyboard shortcut to automatically insert double quotes and the column dash character, which represents a pipe. This allows me to type the whole command very fast with about the same amount of key presses compared to the pure bash version. It gets more interesting when I start intermingling common lists with the rest of the pipeline. For instance, let's change the number of results we want by fetching the value from any max buffer. Let's call it ZZZ. I insert two here, come back to the shell, and I have the two most frequent words. Now this is where it gets a bit mind-boggling. Everything starts mixing up together. The shell, common list, and the text editor. All that said, we're still mimicking the old pipeline behavior. These pipelines are in effect black boxes. You can't expect what's wrong if something breaks in the middle. So what if we took a different approach? Let's ask my package manager to build, say, Emacs. See here, the command returns two values. The first one is the stream output, and the second one is the process itself, which I can inspect in this new window to monitor the status. Here I see that the process is currently running, but if I refresh this window, now it has exited. So now that the process is over, I can indeed inspect the output stream, which gives me another output stream which can be further passed to another asynchronous command, like a pipe, and the content of the stream itself, which is the path produced by the Geeks package manager. This is the first building block of a paradigm in which we process pipeline data in an iterative fashion. Let's consider a more complex real-life example. I have a messy music directory with random files scattered across subdirectories. I'd like to clean it up, say, remove low-quality artwork and music files, and maybe remove the music from a specific artist, but say, un-lib the quality is too low. The finder command returns a list of all the file objects randomly cursively in all subdirectories. Finder is extensible, so I wrote a specialized version called media finder, which returns media file objects instead. These objects have extra media information. Now what's interesting is that I can pass additional predicates to further filter the list with the requirements. Say I want to match files which name matches cover and of the JPEG extension. Okay, so I can now inspect the result. Say I choose this file. This is inspector. It shows me all kinds of UNIX metadata, but also the MIME type, and I can inspect the media stream, which gives me valuable information such as the resolution here. By using regular common list, I can filter out the good quality artworks. That is to save those with the height above, say, 700, only keeping the bad quality finds. Now I can refer to the previous list. And pass it the height. Okay, so I have a narrow down list now. Previously, I've used filters like match name and match extension, but unlike the find UNIX tool, where the search rules are hard coded, here I can extend them and define my own rules to match files with a given bitrate or say songs by a specific artist. Let's define new bitrate in failure function, which will return a predicate. Now let's compare the bitrate of the file parameter. We get this bitrate from the media format slot and compare it with the argument. Okay. Now let's do the same with match artist. Again, we return a predicate. So lambda. We compare the string, which we get from the tags stored in media format. Oops. And we compare it with the argument again. Okay. So now what if I want to narrow down my search again? I call media finder again, this time with using the above predicate. So say I want to list the song with low quality bitrate, with a low bitrate. So I would say above 150k, matching the Sponger artist. And there you go. So now I have two lists, and I would like to put them together. So here I have the previous result. And if I want to put this result, so that would be number three. Oops. Okay. So I have concatenated the list. And this is the final list, which is the last opportunity that I have to inspect the list of facts I want to process before making any final mistake. And that would be it for the demo. Whoa. This was a lot to take in. I hope it did not lose you. Despite all these novelties, there is still room for improvement. I'd love to have better job control, such as to add a convenient way to stop and continue processes, detach them to the background, record the time they ran, and maybe more statistics, such as CPU and memory usage. Pipeline introspection can be improved, such that we would easily inspect the input and output streams of individual sub-processes, or even build a processed graph. Previously, I underlined the importance of using an extensible language for a shell. Racket, the programmable programming language, is an obvious contender to this role. Besides, there is already a rash, the racket shell, while the Dr. Racket IDE could provide a graphical interface. All I've demoed today is only available to E-max users, which is a shame, so it'd be interesting to have it ported to other more accessible interfaces. A popular option would be Jupyter, which is in many ways already a rebel, only lacking some more advanced developer tools like an interactive stack trace or an inspector. Another option would be the Nix web browser, which already features a graphical commonness rebel. It was a long journey for me to reach here. I was a long-term user of other shells and tools, going from the terminal-based FZE-F to E-max E-shell and Meta-X shell. The recent developments presented in this talk partly got inspired by Harvard's talk on Piper, who introduced similar concepts, but with a rather different approach. You will find most of my setup in my.files linked here. Finally, if you'd like to know more about the setup and implementation details, check out my website. I'm going to publish an extensive article soon, covering all this. I hope my talk inspired you. Thanks for watching and happy hacking. Pierre? Are you here? Hello. Yes, I'm here. Hi. Hi. Yeah. So, again, you answered a lot of these questions in text during the presentation. But let's go through them here as well so that they will be added to the recorded video. Sure. You could talk, by the way. Thanks. Glad you liked it. Yeah. So, the most highly voted question was, how do you find the existing commands? Is it just... Yeah. Yeah. So, I guess there are two types of commands you want to find here. One is the common list command. You can look them up using the traditional tools in slide, like a slide app for both, or just simple completion. Other commands you want to look up would be the external executables. And this, you can look up using an extra completion system, which you can install with a new Max package, or simply... Yeah, that would be mostly it. Yeah. That sort of answered the next question as well. So, are you able to use Unix-style commands? Yes. Something like LS or... Yeah. I think this is a very important point because the strength of shells like Bash and France is that how easy it is to just call a program and pass arguments to it. This is what you want to do, right? And in any programming language, if you want to do it, you have to call it function and write strings. It's not convenient. But the thing that's really powerful with these languages is that you can program the interpreter to change how it understands what you pass to it. So, here what I've done is that I've programmed the Shadang symbol to interpret what I would pass to it as a Bash command. It would directly execute it in Bash. So, this allows me to execute any command as I would do in Bash just as fast, well, with just two extra characters. Yeah. So, you're spinning that off to an actual Bash process in the background? So, yeah. Well, I like to do it here because you can actually do two things. One thing is that you can send it to Shell, which is very convenient if you want to run a snippet that's shared by your friends or on the website. And there's so much Bash around that we need to do this all the time, right? Or if you want something a bit more robust that doesn't have a string interpolation or white cards, then you can effectively, you can use a different symbol which tells the interpreter execute this command with these flags and this will not be passed to Shell. Interesting. Yeah. And then also some comparisons. So how does it compare to E-Shell and how does it compare to Babashka? So, the short answer about Babashka is that I've never used it. I heard about it from what I've read on paper, it's very similar. So I think you can achieve more or less the same things. Might be a bit more memory consuming. So if you need to fire up multiple Babashka instances, that may be a bit heavier memory. Well, that's all I can tell. E-Shell, however, I've used it for many years. I've hacked it, I've improved it, I've done so many things with it. I think E-Shell has a couple of shortcomings that might be a bit limiting. For instance, it's not so good with, I mean, Emax doesn't have thread management, doesn't support threading, and this is kind of limiting. So E-Shell, I think the biggest blocker for me using E-Shell is that it doesn't make the distinction between standard output and standard error. And this means that it can actually break numerous scripts or make it impossible to achieve some tests. So there are a couple of more things, but well, then I wrote extensively about it in a few articles on my website, so I invite you to check them out. Cool, cool. Yeah, I guess those were most of the questions. Yeah, pretty much. I was a bit curious, like how does it, I'm guessing starting something interactive, like VIM or even like a get-commit kind of situation would be tricky with this, since you're running it inside of Emax. But the good news here is that it's Emax, so you don't really need it to start with VIM. You already have, you already are in the editor. And for Git, well, in Git you can really, you can tell which editor you want to use, so same thing, if you would write Git, commit, whatever, it will prompt the editor for the commit message, and then you can tell it to use Emax client, which will automatically pop up an Emax buffer in your running session. So it won't be disturbing here. But maybe what you're asking here is what if we want to run NCurses program or something like Htop that uses these terminal visual libraries, sort of. Here I think, well, you can do this, obviously, because this is not a terminal, and there. So what, there is one workaround that Echelle is already using, which is to intercept the command and scan for known visual commands. So if it detects that you're trying to run Htop, it can automatically fire up, so it can deflect the call and fire up the terminal running Htop. And you can do this, so it cannot streamline the call to any command so that it won't ever break. So that's pretty cool. But obviously you lose a bit of the integration here. In my opinion, I think that we should not tie ourselves to the past and keep trying to run VT, 100 terminals. I think we can do better with more sophisticated and modern UIs. So maybe it's time that we wrote Htop in something a bit more modern.
 Hello, I'm Pierre Nighthart, and I'm going to showcase my development environment, which  is based on a rather unconventional shell.  There are many features to demo, so without further ado, let's get started!  First thing first, what's wrong with the current state of affairs?  Let me replay a shell session that I went through the other day.  I wanted to collect the checksums of all the files with the name Nix.  First, I switched to the right directory.  I find that having to type the CD command is tedious, even with completion.  Now I list the desired files recursively.  Ok, let's collect the checksums.  Whoops, first bit for, I forgot to surround a variable name with double quotes.  But let's grab the first result this time.  Uh oh.  Wait, this wasn't there before.  What's going on?  Uh huh.  I have a file with a new line, and it seems that the textual representation of the new  line differs between LS, find, and grep.  Grep mismatches the new line for an N.  Now what if we use another tool, like find?  This returns almost what I want.  But if we were not for show and sum, displaying a spurious backslash for the file with the  new line.  Finally, if I want to copy this result, I am forced to use the mouse for the lack of  better keyboard controls.  Let's review the list of issues I've been trying to address.  First thing is text manipulation.  Maybe the most basic feature is to be able to search the whole shell buffer just like  we would do in any other text editor.  We should be able to copy and paste using keyboard shortcuts, not just the mouse.  Maybe a bit pancier.  I'd like to be able to navigate the prompts, but after all, isn't it the most natural  kind of sectioning for a shell?  Then performance.  First, bash and france can be very slow, but maybe most importantly, the idiomatic way  to filter and process outputs is to spawn a system process for each unique's tools,  such as grep, sort, etc.  And last, the typical unique's shell workflow becomes problematic when you want to reprocess  an output.  What you would do is rerun the command that produced the output in a different pipeline.  This can be a showstopper if the initial command is very slow.  Referring to the output using back references is one solution to this problem.  Debugging.  Wouldn't it be nice to have state-of-the-art debugging tools, like interactive stack traces  and introspectable objects?  As but not least, handle structured data.  List hash tables, objects, you name it.  In particular, in pipelines.  In bash, the file list is typically passed around as a string buffer.  Ideally, we should be able to pass lists of files objects.  Let's get down to the demo.  This is Sly, my common list rebel.  It displays a typical multiline prompt with the current working directory at the top and  to the left there is an index, here, 0, which is the back reference index.  I will come back to it in a minute.  Let's start with a simple example, say a calculation of a big power.  Ta-da!  That was fast.  Evidently, common lists can handle large arbitrary precision numbers out of the box.  So it's pretty handy as a pocket calculator, even for more advanced math.  Conversely, bash is pretty bad, even for simple calculations.  Let's move on to a more typical task.  Compute the share one sum of a file.  Let's take this.png file here.  Notice that I have tried completion.  Okay, now what if I want to compare it with the value that was recorded in the.sum file?  Here.  I could compare them visually, but that would be a bit tedious.  Instead I can leverage the language together with a very important concept I'm going to  present here, back references.  On pressing hash V, all the previous results get prefixed with the index which I can complete  upon.  On pressing zero, the non-zero prefixes disappear and my hash V turns to orange, thus validating  the match.  Let's repeat it with the second result.  This returns T for true, meaning the strings are identical and the checksum is valid.  Previously, I mentioned the issue of search and copy paste.  Since my repo is actually an Emacs, I can use all the features of the text editor, including  search.  Say I want to look up the term.  Term, they all get highlighted as I type.  Then I can cycle to the results.  When the cursor is on the previous prompt and I press enter, it paste the whole command  to the current prompt, ready for editing.  Of course, I can also move around with a keyboard, select an proportional text and copy it.  Next, I'd like to introduce the selector library I've worked on.  When working on multiple projects, each of them using multiple shells, it's only too  easy to get overwhelmed with the amount of open shells and keep stumbling around to find  a desired shell.  To the right, the list of open shells, only one is open at the moment.  Let's open a new shell and call it project.  The new shell popped up at the bottom.  This shell shares the same underlying list process as the one at the top.  So if I were to define a global variable, checksum, holding the value of the previously  computed shell one.  They will be understood in both shells.  What if we want to open a new shell using a new unrelated list process?  Here I have a list of predefined list processes.  Some contain preloaded libraries for working on the next project.  Some use a different list compiler altogether.  I can even run the list process in a container with limited file system access.  Let's pick the first one.  In this new shell, if I enter checksum, the shell will expectantly complain that the symbol  is undefined.  Shell management goes beyond just that.  I can rename the shell, say, to my project.  If I review the list of shells, I see that I have three of them.  I can fuzzy search, narrow down, or even select multiple shells.  If I press enter on a selection, it displays all the selected shells on screen.  Let's showcase the prompt navigation feature.  I can navigate the prompt up and down like I did before.  Not only that, but I can fuzzy search all the prompts in all open shells.  So if I were to search the term checksum, the view gets updated to the shell containing  a matching prompt.  Let's enter, confirm the selection, and brings me to the desired prompt.  Another itch that I've been scratching for too long, directory switching.  Instead of typing CD, I want to use my regular file manager.  Here it's Enix Helm, but it could be your GNOME or KDE file manager.  I can fuzzy match against directories.  Then pressing a keyboard shortcut brings me to the selected directory in my shell.  It's just as easy to go anywhere, say to the root directory, and from there, I can invoke  the history of selected or visited directories, which is also fuzzy searchable.  And in a matter of just a few keystrokes, I'm back to where I started.  Similarly, the history of commands is also fuzzy searchable.  Say I were to search the previously computed checksum command, notice how it narrows down  live, and how it successfully matches regardless of the order of the terms.  While this shell is primary list, it also understands string interpolation, just like  in bash, say I were to get the PID of the list process.  Now I could query the command that started it by inspecting the press entry corresponding  to it.  So I can dollar escape the PID variable, just like I would do in bash.  And here I get the full command.  What's interesting about this is that this is not standard common list.  Here the syntax of the language has been extended to understand the special strings.  In the same way, the syntax can be extended at will to make common writing as convenient  as possible.  Say you don't want to write any parenthesis to execute a simple command.  No problem.  I can write any valid bash command directly by prefixing it with a special shabang.  Again, this is not standard common list.  This is an extension.  Let's consider the more complex case of pipeline.  I want to return the three most frequent terms in a dictionary of words containing duplicates.  Here is the dictionary.  Now I filter it by the unique command, sort, and finally get the three most frequent words.  Now what if we wanted a more less basic text?  Would it be two verbose?  I believe it does not have to be.  With the right commands and keyboard shortcuts, it can be just as fast to type.  I complete here and insert the pipeline, which is this column dash.  Here I'm using a keyboard shortcut to automatically insert double quotes and the column dash character,  which represents a pipe.  This allows me to type the whole command very fast with about the same amount of key presses  compared to the pure bash version.  It gets more interesting when I start intermingling common lists with the rest of the pipeline.  For instance, let's change the number of results we want by fetching the value from  any max buffer.  Let's call it ZZZ.  I insert two here, come back to the shell, and I have the two most frequent words.  Now this is where it gets a bit mind-boggling.  Everything starts mixing up together.  The shell, common list, and the text editor.  All that said, we're still mimicking the old pipeline behavior.  These pipelines are in effect black boxes.  You can't expect what's wrong if something breaks in the middle.  So what if we took a different approach?  Let's ask my package manager to build, say, Emacs.  See here, the command returns two values.  The first one is the stream output, and the second one is the process itself, which I  can inspect in this new window to monitor the status.  Here I see that the process is currently running, but if I refresh this window, now it has exited.  So now that the process is over, I can indeed inspect the output stream, which gives me  another output stream which can be further passed to another asynchronous command, like  a pipe, and the content of the stream itself, which is the path produced by the Geeks package  manager.  This is the first building block of a paradigm in which we process pipeline data in an iterative  fashion.  Let's consider a more complex real-life example.  I have a messy music directory with random files scattered across subdirectories.  I'd like to clean it up, say, remove low-quality artwork and music files, and maybe remove  the music from a specific artist, but say, un-lib the quality is too low.  The finder command returns a list of all the file objects randomly cursively in all subdirectories.  Finder is extensible, so I wrote a specialized version called media finder, which returns  media file objects instead.  These objects have extra media information.  Now what's interesting is that I can pass additional predicates to further filter the  list with the requirements.  Say I want to match files which name matches cover and of the JPEG extension.  Okay, so I can now inspect the result.  Say I choose this file.  This is inspector.  It shows me all kinds of UNIX metadata, but also the MIME type, and I can inspect the  media stream, which gives me valuable information such as the resolution here.  By using regular common list, I can filter out the good quality artworks.  That is to save those with the height above, say, 700, only keeping the bad quality finds.  Now I can refer to the previous list.  And pass it the height.  Okay, so I have a narrow down list now.  Previously, I've used filters like match name and match extension, but unlike the find  UNIX tool, where the search rules are hard coded, here I can extend them and define  my own rules to match files with a given bitrate or say songs by a specific artist.  Let's define new bitrate in failure function, which will return a predicate.  Now let's compare the bitrate of the file parameter.  We get this bitrate from the media format slot and compare it with the argument.  Okay.  Now let's do the same with match artist.  Again, we return a predicate.  So lambda.  We compare the string, which we get from the tags stored in media format.  Oops.  And we compare it with the argument again.  Okay.  So now what if I want to narrow down my search again?  I call media finder again, this time with using the above predicate.  So say I want to list the song with low quality bitrate, with a low bitrate.  So I would say above 150k, matching the Sponger artist.  And there you go.  So now I have two lists, and I would like to put them together.  So here I have the previous result.  And if I want to put this result, so that would be number three.  Oops.  Okay.  So I have concatenated the list.  And this is the final list, which is the last opportunity that I have to inspect the list  of facts I want to process before making any final mistake.  And that would be it for the demo.  Whoa.  This was a lot to take in.  I hope it did not lose you.  Despite all these novelties, there is still room for improvement.  I'd love to have better job control, such as to add a convenient way to stop and continue  processes, detach them to the background, record the time they ran, and maybe more statistics,  such as CPU and memory usage.  Pipeline introspection can be improved, such that we would easily inspect the input and  output streams of individual sub-processes, or even build a processed graph.  Previously, I underlined the importance of using an extensible language for a shell.  Racket, the programmable programming language, is an obvious contender to this role.  Besides, there is already a rash, the racket shell, while the Dr. Racket IDE could provide  a graphical interface.  All I've demoed today is only available to E-max users, which is a shame, so it'd be  interesting to have it ported to other more accessible interfaces.  A popular option would be Jupyter, which is in many ways already a rebel, only lacking  some more advanced developer tools like an interactive stack trace or an inspector.  Another option would be the Nix web browser, which already features a graphical commonness  rebel.  It was a long journey for me to reach here.  I was a long-term user of other shells and tools, going from the terminal-based FZE-F  to E-max E-shell and Meta-X shell.  The recent developments presented in this talk partly got inspired by Harvard's talk  on Piper, who introduced similar concepts, but with a rather different approach.  You will find most of my setup in my.files linked here.  Finally, if you'd like to know more about the setup and implementation details, check  out my website.  I'm going to publish an extensive article soon, covering all this.  I hope my talk inspired you.  Thanks for watching and happy hacking.  Pierre?  Are you here?  Hello.  Yes, I'm here.  Hi.  Hi.  Yeah.  So, again, you answered a lot of these questions in text during the presentation.  But let's go through them here as well so that they will be added to the recorded video.  Sure.  You could talk, by the way.  Thanks.  Glad you liked it.  Yeah.  So, the most highly voted question was, how do you find the existing commands?  Is it just... Yeah.  Yeah.  So, I guess there are two types of commands you want to find here.  One is the common list command.  You can look them up using the traditional tools in slide, like a slide app for both,  or just simple completion.  Other commands you want to look up would be the external executables.  And this, you can look up using an extra completion system, which you can install with a new  Max package, or simply... Yeah, that would be mostly it.  Yeah.  That sort of answered the next question as well.  So, are you able to use Unix-style commands?  Yes.  Something like LS or... Yeah.  I think this is a very important point because the strength of shells like Bash and France  is that how easy it is to just call a program and pass arguments to it.  This is what you want to do, right?  And in any programming language, if you want to do it, you have to call it function and  write strings.  It's not convenient.  But the thing that's really powerful with these languages is that you can program the  interpreter to change how it understands what you pass to it.  So, here what I've done is that I've programmed the Shadang symbol to interpret what I would  pass to it as a Bash command.  It would directly execute it in Bash.  So, this allows me to execute any command as I would do in Bash just as fast, well,  with just two extra characters.  Yeah.  So, you're spinning that off to an actual Bash process in the background?  So, yeah.  Well, I like to do it here because you can actually do two things.  One thing is that you can send it to Shell, which is very convenient if you want to run  a snippet that's shared by your friends or on the website.  And there's so much Bash around that we need to do this all the time, right?  Or if you want something a bit more robust that doesn't have a string interpolation or  white cards, then you can effectively, you can use a different symbol which tells the  interpreter execute this command with these flags and this will not be passed to Shell.  Interesting.  Yeah.  And then also some comparisons.  So how does it compare to E-Shell and how does it compare to Babashka?  So, the short answer about Babashka is that I've never used it.  I heard about it from what I've read on paper, it's very similar.  So I think you can achieve more or less the same things.  Might be a bit more memory consuming.  So if you need to fire up multiple Babashka instances, that may be a bit heavier memory.  Well, that's all I can tell.  E-Shell, however, I've used it for many years.  I've hacked it, I've improved it, I've done so many things with it.  I think E-Shell has a couple of shortcomings that might be a bit limiting.  For instance, it's not so good with, I mean, Emax doesn't have thread management, doesn't  support threading, and this is kind of limiting.  So E-Shell, I think the biggest blocker for me using E-Shell is that it doesn't make the  distinction between standard output and standard error.  And this means that it can actually break numerous scripts or make it impossible to achieve some  tests.  So there are a couple of more things, but well, then I wrote extensively about it in  a few articles on my website, so I invite you to check them out.  Cool, cool.  Yeah, I guess those were most of the questions.  Yeah, pretty much.  I was a bit curious, like how does it, I'm guessing starting something interactive, like  VIM or even like a get-commit kind of situation would be tricky with this, since you're running  it inside of Emax.  But the good news here is that it's Emax, so you don't really need it to start with  VIM.  You already have, you already are in the editor.  And for Git, well, in Git you can really, you can tell which editor you want to use,  so same thing, if you would write Git, commit, whatever, it will prompt the editor for the  commit message, and then you can tell it to use Emax client, which will automatically  pop up an Emax buffer in your running session.  So it won't be disturbing here.  But maybe what you're asking here is what if we want to run NCurses program or something  like Htop that uses these terminal visual libraries, sort of.  Here I think, well, you can do this, obviously, because this is not a terminal, and there.  So what, there is one workaround that Echelle is already using, which is to intercept the  command and scan for known visual commands.  So if it detects that you're trying to run Htop, it can automatically fire up, so it  can deflect the call and fire up the terminal running Htop.  And you can do this, so it cannot streamline the call to any command so that it won't ever  break.  So that's pretty cool.  But obviously you lose a bit of the integration here.  In my opinion, I think that we should not tie ourselves to the past and keep trying to  run VT, 100 terminals.  I think we can do better with more sophisticated and modern UIs.  So maybe it's time that we wrote Htop in something a bit more modern.
lisprepl_0_2543.png lisprepl_2544_5567.png lisprepl_5568_6695.png lisprepl_6696_6743.png lisprepl_6744_7031.png lisprepl_7032_7451.png lisprepl_7452_7667.png lisprepl_7668_8399.png lisprepl_8400_8807.png lisprepl_8808_9011.png lisprepl_9012_9479.png lisprepl_9480_9695.png lisprepl_9696_10439.png lisprepl_10440_11087.png lisprepl_11088_11567.png lisprepl_11568_13067.png lisprepl_13068_13535.png lisprepl_13536_14495.png lisprepl_14496_14723.png lisprepl_14724_14903.png lisprepl_14904_15119.png lisprepl_15120_17243.png lisprepl_17244_17471.png lisprepl_17472_17723.png lisprepl_17724_18179.png lisprepl_18180_18623.png lisprepl_18624_18671.png lisprepl_18672_21179.png lisprepl_21180_23255.png lisprepl_23256_23627.png lisprepl_23628_23711.png lisprepl_23712_23855.png lisprepl_23856_23903.png lisprepl_23904_24851.png lisprepl_24852_25451.png lisprepl_25452_25535.png lisprepl_25536_25895.png lisprepl_25896_25991.png lisprepl_25992_42993.png