Reproducible research with GNU Emacs and Org-mode
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Lestang, Thibault
This presentation illustrates how the GNU Emacs text editor provides an powerful integrated environment for reproducible research, effectively bypassing the need for juggling different software in order to write and execute code, manage data or write papers. GNU Emacs as a long history, and is still widely used and supported by a very active community of users and developers. A very popular feature of GNU Emacs is Org-mode which, at its core, offers a markup language similar to Markdown. Following a brief introduction to Org-mode, this presentation demonstrates its use for reproducible research: straightforward mixing of prose and code, execution of code blocks as well as display of the results. With Org-mode, GNU Emacs is turned into a computational notebook which functionalities goes well beyond popular alternatives such as Jupyter. Code blocks are not restricted to a particular programming language and data can be passed between them: generate data in C, analyse it in Python, visualise it with R, all in one single executable document. Moreover, Org-mode documents are nothing but plain text, making them inherently portable, sustainable, and suited to version control - crucial qualities for academic research. Moving on, I illustrate the export of Org-mode documents to richer formats: PDF, ODT, HTML and many more - all from within GNU Emacs. Lastly, I broaden the scope of this presentation and discuss the open nature of GNU Emacs itself. Indeed, GNU Emacs is free (as in freedom) software under a copyleft license. This ensures that GNU Emacs remains sustainable, community-owned software: GNU Emacs will never be "discontinued" or its features reduced inside a "community edition". If GNU Emacs is widely used among GNU/Linux users, it is a rather unusual component of nowadays researchers' toolbox. This presentation is an opportunity for attendees to (re)discover GNU Emacs, not so much as a code editor, but as a powerful tool for reproducible research, from day-to-day data collection and analysis, all the way to paper publication. Despite being over 30 years old, GNU Emacs comes with with very modern ideas, highly relevant to today's discussions of openness and reproducibility in scientific research. Beyond the tool itself, this presentation is an opportunity to sparkle discussions on the nature of research tooling. GNU Emacs has always been developed and maintained in the open, and its license ensures that it will always be. GNU Emacs maintenance and development is a collective, not-for-profit effort of thousands of developers worldwide and anyone wanting to make modifications to it is welcome to do so. This ensures both reliability and sustainability, which are key characteristics for any tool at the core of a reproducible research practice.
 Hi everyone, my name is Thibault Estan, I'm a research software engineer at the University of Oxford and also a junior research fellow at Kellogg College. Today, I'd like to give you an overview of Aug Mode, which is a set of functionalities that live inside New Emacs and that makes for a very convenient and powerful environment for research. Particularly, I'm going to try and show you how to bundle software data and figures in a single plaintext executable document that's just going to be one keystroke away from exporting it to PDF or maybe HTML. All this is going to happen inside software called GNU Emacs, which is mostly known for being a powerful text editor. In my opinion, there are mainly two reasons why Emacs is a powerful tool. The first reason is technical and the fact that Emacs goes actually beyond being a text editor. It will be more accurate to describe it as an interpreter for EmacsLisp, which is a dialect of the Lisp programming language with a focus on editing text. That means that Emacs is fully programmable and if you know EmacsLisp, well, you can make Emacs do virtually anything. The second reason is that Emacs is free software in the sense that anybody has the freedom to modify and contribute to Emacs. It's been maintained and extended by its own community of users over more than 30 years now and it's not showing any sign of stopping soon. Actually, quite the opposite. One of the major contributions to Emacs is certainly one of the most popular is AugMode, which was started by Carsten Dominik, an astrophysicist from the University of Amsterdam, to help him with his daily work as a researcher. Now fast forward nearly about 20 years, AugMode nowadays provides a huge range of functionalities and basically no two org users use org in the same way. But today I want to focus on a very specific use case of AugMode, which is reproducible research. Because, well, the big challenge in everyday research nowadays is that, well, it is made of many different activities. You have data collection or maybe data generation, data processing, making figures, making table, writing papers. And all of these steps, they have a dependency on the previous one. So reproducible research or making sure that the research is reproducible is about preserving those dependencies so that if you give your data to someone, they can reproduce the figures in the paper. And if you give them the paper, well, they can go back all the way to the data or maybe the software that produced the data. And obviously, this level of reproducibility is very hard to achieve because it implies that these dependencies are well documented. For instance, what data came from where, goes where, is processed by which version, of which software. And therefore when something is changed upstream, maybe software data, well, that change must be propagated all the way to the figure of maybe the paper. And that's very error prone if it's done manually. And it's often is. It also may involve a collection of different tools. For instance, you have a text editor that you used to program, maybe Python to execute the code and maybe, I don't know, or Relief to write the latex and compile it into your PDF. And as always, the risk of depending on software format, it's not going to be available under the same conditions in the future. So what I want to do now is to give you a taste of how Org Mode can help with these issues. It's not magic. It has limitations. But in my opinion, it's worth having a serious look at. So let's do this now. So here's a file opened in Org Mode inside Emacs. So Org Mode at its core is, well, defines a markup language that's going to be very similar to markdown or restricted text, for instance. So you can define, for instance, headlines or at different levels. You can define paragraphs, code blocks. You can define emphasized texts, links, lists, nested lists, and even equations. And so the benefit of editing Org in Emacs is that you benefit from Org Mode, and this gives you the ability to very efficiently manipulate this kind of document. So for instance, you could fold sections. You can move from one to another very quickly. You can manipulate lists very efficiently and many, many other things that I don't really have time to describe it. I certainly don't know anything. So it's about manipulating these kind of documents very efficiently. So let's start with this first section that is about simulating some data, so generating some data. So what we're going to do is that we're going to simulate this equation of motion here that corresponds to something called an Einstein-Nuland-Bek process. And it's going to be a one-dimensional random process. So it's basically the one-dimensional movement of some kind of random variable. So here we just describe a little bit about the process. And then here there's a subsection about the actual code that's going to simulate or generate the data. And so here what you have is a code block very similar to Macdown. And this code block has a name, an associated language, which is going to be C++ and some attributes that I'm going to quickly describe later. What's amazing about writing code in ARC mode is that here I'm in ARC mode in this kind of window. I can very quickly open a separate window, a temporary window, that lets me edit the code with the corresponding mode. So for instance, in this window I'm in C++ mode and I'm going to get proper syntax highlighting and for instance auto-completion. So the data will go for instance distribution. So I'm going to get all that I want to write C++ efficiently. And then with a single keystroke I'll go back to the code block and the code block is automatically updated. So here we have these attributes to code blocks that can be either specific to the language, so for instance here it's the include list for the C++ but also other attributes, for instance, saying that I want to cache the result of that code block so that I don't have to re-execute it each time to get the results. And also there's this no web here that lets me put this kind of syntax here and this is a reference to another code block that is actually in the section above that describes the parameters for the simulation. So there's a bit of C++ here that when I execute this code, and I'm going to show that in a minute, this bit of syntax is going to be replaced by the content of that code block. And that's useful when you want to, for instance, discuss the value of parameters in a section above and if I ever make a modification to these parameters, well I don't have to manually go and duplicate that term, it's going to be automatic. So now I can just execute this small code with just a single keystroke again, like Ctrl CC and this is going to be compiled and yes, because I modified the content and now if I hit Ctrl CC again, it's going to compile and execute that code and it's going to catch the output of that code block and display the result in a table here. And so what we have is that we have a first column which is the value of the time and the second column is going to be the value of the random variable at this corresponding moment in time and you have 100 values here. So cool with ARMA, we just wrote the code and then executed directly from Emacs. Now let's just maybe visualize this time series. And so what we're going to do is that we're going to now plot the data using Python. So here's a new code block in Python, I could do the same, I could open another window in Python mode and I'm going to avoid messing with that this time. And so we import numpy, map.lib and then we basically generate a plot and we could execute that code again and then what's going to happen this time is that it's going to generate a file, it's an image file that contains the plot and it's going to output a link to that file. And so what we're going to export that document is going to put the image where it should be. So basically to show you that you can actually use different languages inside the same document. And now lastly we can imagine that now that we have the data, we have a good idea of what the data looks like, we might want to do some statistics for instance computing the average value of the time series. What you can do in org mode is that you can define code blocks but use them as functions. So for instance here I have a code block that is going to expect some kind of input with the name x, sorry, and then it's going to return, it's going to expect that this input is an array and it's going to return the average of the second column of the array. And that's going to be useful because for instance if we were to discuss the average of the time series for instance, we can then avoid hard coding the value and then make a call, an inline call to that code block passing the data that we generated with the C++ block. So instead of hard coding the value here we have a call and then if we change something above and for instance the parameters or the data itself then we don't have to go back and manually make the change and therefore we have a constant document. Now that we're done, the only thing that we have to do is export that into a Merch Tretor format for maybe sharing that with colleagues so we could do a whole range of different formats, for instance a PDF through latex, so I just hit hell and then we want a PDF file and open it and that's going to actually open within Emacs but you couldn't open it with any other software and here it is. You could probably compute so that you have syntax aliting for code and then also configure the latex properties. Something I do quite often for my personal loads for instance is generating HTML files so I can also generate some HTML and open it with a web browser and here we go. And so we can see that latex is properly rendered, we have nice syntax aliting and that's basically nice to share notes with other people for instance or maybe me, myself down the line and we can see here that the value of the advert was correctly replaced. And so you could also generate all the other formats like for instance if you work with people that use Microsoft Word or LibreOffice you can generate an ODE file in the same way. So here it is, there's much more to OGMODE than what I just showed you and there's actually much more to working with source code but I hope it's giving you a good taste of the functionality of OGMODE and what it allows you to do for your research. OGMODE has a very strong and active community but I don't think the bulk of it is actually made of people in academic research because when I look around I see much less Emacs and OGMODE than I see triple notebooks, plain latex or maybe Word processes. There are probably many reasons for that but I could think of three main ones that keep coming back in discussions about Emacs. The first one is that Emacs comes with very bare bones defaults. The second one is that it's keyboard oriented and that can be quite off-putting for people who are used to using a mouse. And the third reason is that Emacs is highly customizable which is great but using a dialect of Lisp which is not the most popular language in academia. What's interesting though with these three things is that these could be either drawbacks or advantages depending on who you talk to. But why is certain is that for a researcher who's not particularly well versed in computing maybe well there's a steep learning curve for Emacs at least compared to using for instance a chiptate notebook. But using Emacs is all about the return on investment because what you get in the end is a full-blown work environment that is really well suited for research. And also it's not especially hard to get started with the basics of Emacs and then expanding that use in incremental steps. And for instance just going through the introductory tutorial already can take you a rather long way. And so beyond the functionality and convenience that Emacs and Org mode bring an important aspect of it all is that it's all plain text. So you can actually read and write Org using any text editor on any platform and that's obviously very important for the sustainability of your research. And lastly the second aspect I want to focus on and I'm going to conclude with it is that Emacs and Org mode are both free software, free as in free beer but also free as in freedom. So anybody has the freedom to use Emacs and Org mode and that means that anyone can legally study and build upon your research. And then anybody also has the freedom to modify Emacs and share their modifications with other users of Emacs and this partly explains the impressive longevity of Emacs and that makes it sustainable software, one that can be relied upon reasonably far in the future. So as research students, postdocs, maybe PIs, research software engineers or research facilitators I really encourage you to have a look at Emacs and Org mode, maybe rediscover Emacs and check how it can be useful for you, for your friends and for your colleagues. Thank you. Where does the reproducibility come in? I will add a few more on this question which is a bit rushed. At some point can you actually do the whole process automatically, like even go to the exports in HTML or in PDF automatically so we can run in CLI for instance without opening Emacs at some point. Is that possible? Is that something that you can do like doing the whole pipeline from getting the data to getting the figures down and everything is spotted? Is that something you can streamline? The short and say is yes, you can do that. What is typically done is, well, you use Emacs, right? So at some point the idea is that you will write your paper in Emacs and basically when you are done, when you export your old document, your plain text document to say PDF, then it is going to go very much like you would execute a Jupyter notebook for instance or any kind of computational notebook. It is going to take the first code block and execute that and if there are any dependencies it is going to resolve it. So it is going to go basically from top to bottom and then export it. Now obviously people have been trying to extend that and as you said, it is really useful to have some kind of something you could use in the command line for instance that could do that process for you with them without you having to actually open Emacs. I don't remember the name but if you just look for org exporter, the Myers might also be several projects that just do that. You just org export in the command line, it is just a wrapper around the functionality that Emacs uses. So yes, you can do that. So next question is about Macdown. So do you know any tools that can convert Macdown back into org files if collaborators comment or change the MD export because they want the no best Macdown than the org mode? Yeah, I don't know. One way is working, right? You can go from org to anything. The other way is always a bit more complex. I think org is supported by Pandoc. Pretty sure it is actually. So maybe you can use that to go back to org. There are also, I don't remember the name of the, there are a few projects at the moment that really try to make conversions between formats very easy. And so I think at the moment it's a bit fiddly. It's not easy to do, but I think it's going to come in the near future. Let's hope so. A question from Mathieu. Does it work on Windows or macOS because he asks because open source is a ladder and the magic that execute code makes me wonder if someone else will be able to run your code, your doc. How did you use Emax or more than other systems and Linux? So using, so there are two layers to that question, I think. So using Emax and onward in other operating systems is very straightforward because Emax is available basically everywhere. That's one of the very good reasons to use Emax is that it's available literally everywhere. And it should be in the future as well. It's been around for like 30 years and it should be hopefully in the next 30 as well. So in terms of like accessing, in terms of portability, it's just great for that. And then because org mode, well, if you redo the introduction of the argument of making org mode with an Emax is that then it means you can access org mode from everywhere. Now obviously if you want to run your code when I run that C++ sample, you have to have a C++ compiler and then you have to care about. Which brings to the next question, which I had too, which is about the packaging, the dependencies because you can, okay, you can edit the code directly there in Emax. Of course there you need, still you need your system to have the binary somewhere. Yeah. You can keep the code, right? But org mode doesn't manage that. It's something you have to deal outside of org mode, which leads to, I guess, so that's the first question. The second question is how do you handle then the version dependencies that you might have? Well, I think yes. I mean, in terms of the software that will be called, whether it's, I don't know, the Python interpreter or the version of your compiler, then you are going to, I don't think org can do anything for you. In terms of versioning the document itself, obviously because it's plain text, it's just great. But in terms of the software, the backend, if you will, yeah, you're going to have to do this. Yeah. Okay. Do you happen to have experience? You do. Or collaboratively? Would you generally just a standard git based flow or something else? Sorry, my connection is a little bit off, but I think you asked me about, can you just repeat the question? Yes. Do you happen to have experience with using org mode collaboratively? Do you generally just git based flow or something else? Yeah. Yeah. I think I would. The main problem with org mode is that your colleagues don't use it. And so I very rarely had to, well, have I ever collaborated directly with somebody in org mode? But I expect that it would be just like you would write later document or Mac down document. So yeah. Okay. Using a virtual system. Okay. So Jonas just mentioned that you could use nix or j u i x for reproducible execution of arguments. I don't know. You know those? I know the name, never used nix or gigs. But yeah, I think, and I would, I don't know it. So it's a wild guess, but I would, I would imagine that gigs would integrate pretty well with, like new gigs would integrate well with new emacs. And so, yeah, it's a good, it's a good thing to explore in order to manage the dependencies on top or with, with org mode. Yeah. Okay. So I've got a question myself. So you mentioned that you can use a cache for code blocks. So can you just please follow a little bit how, how the cache works? Like, if you use a cache, it will not reload automatically if that data or anything before this is updated, right? So that is the mechanism? Yeah. I'm not sure of the exact, how exactly works, but basically in that example, that C++ block was basically spitting out a bunch of numbers. And so when you say cache, you know, you have this like results drawer or results container and then these containers are actually a versioned inside of, or more versions of these. So it just assigns a hash to, and so whenever you, you ask org to cache it, then it just, whenever there's a result, it's just not going to run it again, just directly take the numbers out. Okay. Roughly, I think. In that case, it will just rerun if you do manually the, yes. Yeah. Yeah. Okay, there were a lot of discussion about trolling about Veeam and Emacs. So I have to ask you this question. How do you quit Emacs? I ask control, I don't know. It's muscle memory. It's control like C. No, in terms, it's an interesting thing. And just like, I know it's about to end, but you can use the benefit of plaintext. You can use other editors. I would encourage people to, if they like other editors, like try and find your way. And org is actually accessible and there's like org for VS code, there's org for Veeam. It's abandoned, I know, but it might start, it might start again. Let's hope so. So, and even though it's not a good thing, it's not a good thing. That's why it's not. Okay.
 Hi everyone, my name is Thibault Estan, I'm a research software engineer at the University  of Oxford and also a junior research fellow at Kellogg College.  Today, I'd like to give you an overview of Aug Mode, which is a set of functionalities  that live inside New Emacs and that makes for a very convenient and powerful environment  for research.  Particularly, I'm going to try and show you how to bundle software data and figures in  a single plaintext executable document that's just going to be one keystroke away from exporting  it to PDF or maybe HTML.  All this is going to happen inside software called GNU Emacs, which is mostly known for  being a powerful text editor.  In my opinion, there are mainly two reasons why Emacs is a powerful tool.  The first reason is technical and the fact that Emacs goes actually beyond being a text  editor.  It will be more accurate to describe it as an interpreter for EmacsLisp, which is a dialect  of the Lisp programming language with a focus on editing text.  That means that Emacs is fully programmable and if you know EmacsLisp, well, you can make  Emacs do virtually anything.  The second reason is that Emacs is free software in the sense that anybody has the freedom  to modify and contribute to Emacs.  It's been maintained and extended by its own community of users over more than 30 years  now and it's not showing any sign of stopping soon.  Actually, quite the opposite.  One of the major contributions to Emacs is certainly one of the most popular is AugMode,  which was started by Carsten Dominik, an astrophysicist from the University of Amsterdam, to help  him with his daily work as a researcher.  Now fast forward nearly about 20 years, AugMode nowadays provides a huge range of functionalities  and basically no two org users use org in the same way.  But today I want to focus on a very specific use case of AugMode, which is reproducible  research.  Because, well, the big challenge in everyday research nowadays is that, well, it is made  of many different activities.  You have data collection or maybe data generation, data processing, making figures, making table,  writing papers.  And all of these steps, they have a dependency on the previous one.  So reproducible research or making sure that the research is reproducible is about preserving  those dependencies so that if you give your data to someone, they can reproduce the figures  in the paper.  And if you give them the paper, well, they can go back all the way to the data or maybe  the software that produced the data.  And obviously, this level of reproducibility is very hard to achieve because it implies  that these dependencies are well documented.  For instance, what data came from where, goes where, is processed by which version,  of which software.  And therefore when something is changed upstream, maybe software data, well, that change must  be propagated all the way to the figure of maybe the paper.  And that's very error prone if it's done manually.  And it's often is.  It also may involve a collection of different tools.  For instance, you have a text editor that you used to program, maybe Python to execute  the code and maybe, I don't know, or Relief to write the latex and compile it into your  PDF.  And as always, the risk of depending on software format, it's not going to be available under  the same conditions in the future.  So what I want to do now is to give you a taste of how Org Mode can help with these issues.  It's not magic.  It has limitations.  But in my opinion, it's worth having a serious look at.  So let's do this now.  So here's a file opened in Org Mode inside Emacs.  So Org Mode at its core is, well, defines a markup language that's going to be very similar  to markdown or restricted text, for instance.  So you can define, for instance, headlines or at different levels.  You can define paragraphs, code blocks.  You can define emphasized texts, links, lists, nested lists, and even equations.  And so the benefit of editing Org in Emacs is that you benefit from Org Mode, and this  gives you the ability to very efficiently manipulate this kind of document.  So for instance, you could fold sections.  You can move from one to another very quickly.  You can manipulate lists very efficiently and many, many other things that I don't really  have time to describe it.  I certainly don't know anything.  So it's about manipulating these kind of documents very efficiently.  So let's start with this first section that is about simulating some data, so generating  some data.  So what we're going to do is that we're going to simulate this equation of motion here that  corresponds to something called an Einstein-Nuland-Bek process.  And it's going to be a one-dimensional random process.  So it's basically the one-dimensional movement of some kind of random variable.  So here we just describe a little bit about the process.  And then here there's a subsection about the actual code that's going to simulate or generate  the data.  And so here what you have is a code block very similar to Macdown.  And this code block has a name, an associated language, which is going to be C++ and some  attributes that I'm going to quickly describe later.  What's amazing about writing code in ARC mode is that here I'm in ARC mode in this kind  of window.  I can very quickly open a separate window, a temporary window, that lets me edit the  code with the corresponding mode.  So for instance, in this window I'm in C++ mode and I'm going to get proper syntax highlighting  and for instance auto-completion.  So the data will go for instance distribution.  So I'm going to get all that I want to write C++ efficiently.  And then with a single keystroke I'll go back to the code block and the code block is automatically  updated.  So here we have these attributes to code blocks that can be either specific to the language,  so for instance here it's the include list for the C++ but also other attributes, for  instance, saying that I want to cache the result of that code block so that I don't  have to re-execute it each time to get the results.  And also there's this no web here that lets me put this kind of syntax here and this is  a reference to another code block that is actually in the section above that describes  the parameters for the simulation.  So there's a bit of C++ here that when I execute this code, and I'm going to show that  in a minute, this bit of syntax is going to be replaced by the content of that code block.  And that's useful when you want to, for instance, discuss the value of parameters in a section  above and if I ever make a modification to these parameters, well I don't have to manually  go and duplicate that term, it's going to be automatic.  So now I can just execute this small code with just a single keystroke again, like Ctrl  CC and this is going to be compiled and yes, because I modified the content and now if  I hit Ctrl CC again, it's going to compile and execute that code and it's going to catch  the output of that code block and display the result in a table here.  And so what we have is that we have a first column which is the value of the time and  the second column is going to be the value of the random variable at this corresponding  moment in time and you have 100 values here.  So cool with ARMA, we just wrote the code and then executed directly from Emacs.  Now let's just maybe visualize this time series.  And so what we're going to do is that we're going to now plot the data using Python.  So here's a new code block in Python, I could do the same, I could open another window in  Python mode and I'm going to avoid messing with that this time.  And so we import numpy, map.lib and then we basically generate a plot and we could execute  that code again and then what's going to happen this time is that it's going to generate  a file, it's an image file that contains the plot and it's going to output a link to that  file.  And so what we're going to export that document is going to put the image where it should be.  So basically to show you that you can actually use different languages inside the same document.  And now lastly we can imagine that now that we have the data, we have a good idea of what  the data looks like, we might want to do some statistics for instance computing the average  value of the time series.  What you can do in org mode is that you can define code blocks but use them as functions.  So for instance here I have a code block that is going to expect some kind of input with  the name x, sorry, and then it's going to return, it's going to expect that this input  is an array and it's going to return the average of the second column of the array.  And that's going to be useful because for instance if we were to discuss the average  of the time series for instance, we can then avoid hard coding the value and then make  a call, an inline call to that code block passing the data that we generated with the  C++ block.  So instead of hard coding the value here we have a call and then if we change something  above and for instance the parameters or the data itself then we don't have to go back  and manually make the change and therefore we have a constant document.  Now that we're done, the only thing that we have to do is export that into a Merch  Tretor format for maybe sharing that with colleagues so we could do a whole range of  different formats, for instance a PDF through latex, so I just hit hell and then we want  a PDF file and open it and that's going to actually open within Emacs but you couldn't  open it with any other software and here it is.  You could probably compute so that you have syntax aliting for code and then also configure  the latex properties.  Something I do quite often for my personal loads for instance is generating HTML files  so I can also generate some HTML and open it with a web browser and here we go.  And so we can see that latex is properly rendered, we have nice syntax aliting and that's basically  nice to share notes with other people for instance or maybe me, myself down the line  and we can see here that the value of the advert was correctly replaced.  And so you could also generate all the other formats like for instance if you work with  people that use Microsoft Word or LibreOffice you can generate an ODE file in the same way.  So here it is, there's much more to OGMODE than what I just showed you and there's actually  much more to working with source code but I hope it's giving you a good taste of the  functionality of OGMODE and what it allows you to do for your research.  OGMODE has a very strong and active community but I don't think the bulk of it is actually  made of people in academic research because when I look around I see much less Emacs and  OGMODE than I see triple notebooks, plain latex or maybe Word processes.  There are probably many reasons for that but I could think of three main ones that keep  coming back in discussions about Emacs.  The first one is that Emacs comes with very bare bones defaults.  The second one is that it's keyboard oriented and that can be quite off-putting for people  who are used to using a mouse.  And the third reason is that Emacs is highly customizable which is great but using a dialect  of Lisp which is not the most popular language in academia.  What's interesting though with these three things is that these could be either drawbacks  or advantages depending on who you talk to.  But why is certain is that for a researcher who's not particularly well versed in computing  maybe well there's a steep learning curve for Emacs at least compared to using for instance  a chiptate notebook.  But using Emacs is all about the return on investment because what you get in the end  is a full-blown work environment that is really well suited for research.  And also it's not especially hard to get started with the basics of Emacs and then expanding  that use in incremental steps.  And for instance just going through the introductory tutorial already can take you a rather long  way.  And so beyond the functionality and convenience that Emacs and Org mode bring an important  aspect of it all is that it's all plain text.  So you can actually read and write Org using any text editor on any platform and that's  obviously very important for the sustainability of your research.  And lastly the second aspect I want to focus on and I'm going to conclude with it is that  Emacs and Org mode are both free software, free as in free beer but also free as in freedom.  So anybody has the freedom to use Emacs and Org mode and that means that anyone can legally  study and build upon your research.  And then anybody also has the freedom to modify Emacs and share their modifications with other  users of Emacs and this partly explains the impressive longevity of Emacs and that makes  it sustainable software, one that can be relied upon reasonably far in the future.  So as research students, postdocs, maybe PIs, research software engineers or research facilitators  I really encourage you to have a look at Emacs and Org mode, maybe rediscover Emacs and check  how it can be useful for you, for your friends and for your colleagues.  Thank you.  Where does the reproducibility come in?  I will add a few more on this question which is a bit rushed.  At some point can you actually do the whole process automatically, like even go to the  exports in HTML or in PDF automatically so we can run in CLI for instance without opening  Emacs at some point.  Is that possible?  Is that something that you can do like doing the whole pipeline from getting the data to  getting the figures down and everything is spotted?  Is that something you can streamline?  The short and say is yes, you can do that.  What is typically done is, well, you use Emacs, right?  So at some point the idea is that you will write your paper in Emacs and basically when  you are done, when you export your old document, your plain text document to say PDF, then  it is going to go very much like you would execute a Jupyter notebook for instance or  any kind of computational notebook.  It is going to take the first code block and execute that and if there are any dependencies  it is going to resolve it.  So it is going to go basically from top to bottom and then export it.  Now obviously people have been trying to extend that and as you said, it is really useful  to have some kind of something you could use in the command line for instance that could  do that process for you with them without you having to actually open Emacs.  I don't remember the name but if you just look for org exporter, the Myers might also  be several projects that just do that.  You just org export in the command line, it is just a wrapper around the functionality  that Emacs uses.  So yes, you can do that.  So next question is about Macdown.  So do you know any tools that can convert Macdown back into org files if collaborators comment  or change the MD export because they want the no best Macdown than the org mode?  Yeah, I don't know.  One way is working, right?  You can go from org to anything.  The other way is always a bit more complex.  I think org is supported by Pandoc.  Pretty sure it is actually.  So maybe you can use that to go back to org.  There are also, I don't remember the name of the, there are a few projects at the moment  that really try to make conversions between formats very easy.  And so I think at the moment it's a bit fiddly.  It's not easy to do, but I think it's going to come in the near future.  Let's hope so.  A question from Mathieu.  Does it work on Windows or macOS because he asks because open source is a ladder and  the magic that execute code makes me wonder if someone else will be able to run your code,  your doc.  How did you use Emax or more than other systems and Linux?  So using, so there are two layers to that question, I think.  So using Emax and onward in other operating systems is very straightforward because Emax  is available basically everywhere.  That's one of the very good reasons to use Emax is that it's available literally everywhere.  And it should be in the future as well.  It's been around for like 30 years and it should be hopefully in the next 30 as well.  So in terms of like accessing, in terms of portability, it's just great for that.  And then because org mode, well, if you redo the introduction of the argument of making  org mode with an Emax is that then it means you can access org mode from everywhere.  Now obviously if you want to run your code when I run that C++ sample, you have to have  a C++ compiler and then you have to care about.  Which brings to the next question, which I had too, which is about the packaging, the  dependencies because you can, okay, you can edit the code directly there in Emax.  Of course there you need, still you need your system to have the binary somewhere.  Yeah.  You can keep the code, right?  But org mode doesn't manage that.  It's something you have to deal outside of org mode, which leads to, I guess, so that's  the first question.  The second question is how do you handle then the version dependencies that you might have?  Well, I think yes.  I mean, in terms of the software that will be called, whether it's, I don't know, the  Python interpreter or the version of your compiler, then you are going to, I don't think  org can do anything for you.  In terms of versioning the document itself, obviously because it's plain text, it's just  great.  But in terms of the software, the backend, if you will, yeah, you're going to have to  do this.  Yeah.  Okay.  Do you happen to have experience?  You do.  Or collaboratively?  Would you generally just a standard git based flow or something else?  Sorry, my connection is a little bit off, but I think you asked me about, can you just  repeat the question?  Yes.  Do you happen to have experience with using org mode collaboratively?  Do you generally just git based flow or something else?  Yeah.  Yeah.  I think I would.  The main problem with org mode is that your colleagues don't use it.  And so I very rarely had to, well, have I ever collaborated directly with somebody  in org mode?  But I expect that it would be just like you would write later document or Mac down document.  So yeah.  Okay.  Using a virtual system.  Okay.  So Jonas just mentioned that you could use nix or j u i x for reproducible execution  of arguments.  I don't know.  You know those?  I know the name, never used nix or gigs.  But yeah, I think, and I would, I don't know it.  So it's a wild guess, but I would, I would imagine that gigs would integrate pretty well  with, like new gigs would integrate well with new emacs.  And so, yeah, it's a good, it's a good thing to explore in order to manage the dependencies  on top or with, with org mode.  Yeah.  Okay.  So I've got a question myself.  So you mentioned that you can use a cache for code blocks.  So can you just please follow a little bit how, how the cache works?  Like, if you use a cache, it will not reload automatically if that data or anything before  this is updated, right?  So that is the mechanism?  Yeah.  I'm not sure of the exact, how exactly works, but basically in that example, that C++ block  was basically spitting out a bunch of numbers.  And so when you say cache, you know, you have this like results drawer or results container  and then these containers are actually a versioned inside of, or more versions of these.  So it just assigns a hash to, and so whenever you, you ask org to cache it, then it just,  whenever there's a result, it's just not going to run it again, just directly take the numbers  out.  Okay.  Roughly, I think.  In that case, it will just rerun if you do manually the, yes.  Yeah.  Yeah.  Okay, there were a lot of discussion about trolling about Veeam and Emacs.  So I have to ask you this question.  How do you quit Emacs?  I ask control, I don't know.  It's muscle memory.  It's control like C.  No, in terms, it's an interesting thing.  And just like, I know it's about to end, but you can use the benefit of plaintext.  You can use other editors.  I would encourage people to, if they like other editors, like try and find your way.  And org is actually accessible and there's like org for VS code, there's org for Veeam.  It's abandoned, I know, but it might start, it might start again.  Let's hope so.  So, and even though it's not a good thing, it's not a good thing.  That's why it's not.  Okay.
open_research_emacs_orgmode_0_4739.png open_research_emacs_orgmode_4740_5003.png open_research_emacs_orgmode_5004_5459.png open_research_emacs_orgmode_5460_6047.png open_research_emacs_orgmode_6048_6131.png open_research_emacs_orgmode_6132_6191.png open_research_emacs_orgmode_6192_6371.png open_research_emacs_orgmode_6372_6791.png open_research_emacs_orgmode_6792_10139.png open_research_emacs_orgmode_10140_11531.png open_research_emacs_orgmode_11532_12203.png open_research_emacs_orgmode_12204_12323.png open_research_emacs_orgmode_12324_12443.png open_research_emacs_orgmode_12444_12647.png open_research_emacs_orgmode_12648_12719.png open_research_emacs_orgmode_12720_12947.png open_research_emacs_orgmode_12948_13163.png open_research_emacs_orgmode_13164_13247.png open_research_emacs_orgmode_13248_13703.png open_research_emacs_orgmode_13704_15815.png open_research_emacs_orgmode_15816_15863.png open_research_emacs_orgmode_15864_17519.png open_research_emacs_orgmode_17520_18815.png open_research_emacs_orgmode_18816_20063.png open_research_emacs_orgmode_20064_20111.png open_research_emacs_orgmode_20112_20183.png open_research_emacs_orgmode_20184_20243.png open_research_emacs_orgmode_20244_22451.png open_research_emacs_orgmode_22452_26795.png open_research_emacs_orgmode_26796_35459.png open_research_emacs_orgmode_35460_35603.png open_research_emacs_orgmode_35604_36059.png open_research_emacs_orgmode_36060_36176.png