<jupyter_start><jupyter_text>Literate PrototypingAs scholars in Software Engineering, and as such, we teach others how to build software.  When I get asked what the most important rules for great and long-lasting software are, I usually tell these:1. Have a **repository** for your code where people can contribute2. Make sure that **anyone can build** your software automatically from the source3. Have **automatic tests** that you can run anytime to prevent errors after changes4. Be **open for extension** such that your code can be extended and reused without having to touch core functionality5. Have **examples and tutorials** that allow newcomers to quickly grasp how to use your code.Following these rules will get you the most important benefits of software engineering.  We know this, as we apply them in the software we build ourselves.  Yet, for decades, we have ran into the same problems again and again, eventually rendering all the above efforts useless.  Which are these problems?  Research Software: Smelling Heaps of Wire and Duct TapeMuch of our research consists of building software.  Not only do we build software for _conducting_ research; the software that we produce and its evaluation actually is the *result* of research.  If we have an idea on how to automatically test software, for instance, we evaluate whether the idea works by implementing the idea in a piece of software and running it on a number of (software) subjects.You would assume that we have some plan in the beginning, craft some elegant algorithm, and then build the software according to plan.  That might be true.  Problem is, though, that the original plan seldom works perfectly, and we need to come up with new, different ideas – which again have to be implemented.  Since these were not planned for in the original software design, the software structure quickly deteriorates, and this becomes a problem.  For decades, our typical software life cycle went like this:1. A PhD student builds some tool for a "classic" language, say Java or C.2. While building the tool, research frequently changes course.  Some ideas don't work.  We try out other ideas, which were not planned initially.  We extend the code again and again, but make little effort to get it to work beyond our subject programs.3. We eventually have a tool with several 10,000 lines of code, badly documented, and not very well structured.4. Every time a new student comes along, the PhD student spends days and weeks introducing the new student to her code.  The new student spends weeks just figuring out how to do things.  Eventually, she manages to get things done, but then, there are only two weeks left before the thesis is due.  She spends one week on the evaluation, and one week on the write up.  None of the results are reusable.5. When the PhD student leaves, her code is effectively dead.This is how it has always been.  Ask me for the source code of some project that one of my students wrote five years ago.  I can send it to you, no problem, but you won't get it to run; the language has evolved, the tools have evolved – and there is noone left who can tell you how to do things.  Sorry.Is there something we can do to improve things?  Literate Programming to the Rescue?The problem of passing knowledge about code has been around for decades.  Tools like [Javadoc]() provide standards on how to document code (or at least its function interfaces).  Knuth's concept of [literate programming]() allows to have a single document that contains both code as well as an explanation of all its components and how they fit together, resulting in beautifully typeset books that describe all details of the code.<jupyter_code># Have a cweb example!<jupyter_output><empty_output><jupyter_text>The problem with these techniques is that they describe the _static_ structure of the code, without pointing out any of its _dynamic_ features.  Yet, programmers do not only learn code by reading and understanding it, but also by _executing_ it, by means of _examples_ that illustrate how code is supposed to be used, and the typical results it produces.  If code produces more than just raw data, we need means to put these into diagrams, graphs, and pictures, features any good textbook would use.    Notebooks Join Code, Text, and ExamplesThis is where the concept of _notebooks_ comes into play.  In a single document, Jupyter Notebooks bring together1. **Code definitions** (functions, constants, classes, ...)2. **Text** (with rich typesetting and \LaTeX formulas)3. **Code examples** (to be executed right as you enter them)4. **Output** (of the above examples, can be anything that JavaScript produces)5. **Tests** (that check the above outputs against expected values)Like literate programming, Jupyter Notebooks allow to join (and document) code pieces with illustrative text.  Unlike literate programming, however, they also allow to include _examples_, lots of examples, which make use of the definitions just entered, and whose rich outputs are visualized right away. Let me illustrate these features with – an example.  The `triangle()` program classifies a triangle of lengths $a$, $b$, $c$ into one of three categories – equilateral, scalene, or isosceles.  We can include its definition right within the notebook:<jupyter_code>def triangle(a, b, c):
    if a == b:
        if b == c:
            return 'equilateral'
        else:
            return 'isosceles'
    else:
        if b == c:
            return 'isosceles'
        else:
            if a == c:
                return 'isosceles'
            else:
                return 'scalene'<jupyter_output><empty_output><jupyter_text>In a notebook, we can now go and illustrate how `triangle()` works, showing both an expression and its result:<jupyter_code>triangle(2, 2, 3)<jupyter_output><empty_output><jupyter_text>We can also integrate a set of tests to ensure that `triangle()` does the right thing.  These checks would be executed every time the program runs.<jupyter_code>assert triangle(1, 1, 1) == 'equilateral'
assert triangle(3, 4, 5) == 'scalene'<jupyter_output><empty_output><jupyter_text>We can also include a function that _draws_ a triangle.<jupyter_code>import drawSvg as draw
d = draw.Drawing(120, 120, origin='center')
triangle = draw.Path(stroke_width=2, stroke='green', fill='black', fill_opacity=0.1)
triangle.M(0, 0)
triangle.l(0, 100)
triangle.l(100, 100)
triangle.Z()
d.append(triangle)
d<jupyter_output><empty_output><jupyter_text>Compared to Knuth's literate programming, notebooks do not provide a "weaving" feature that allows you to compose a function from several small code pieces, all defined individually.  However, in a notebook, you can use programming language features to decompose a function into several small helpers, which than can all be illustrated with examples individually.  And while one can have examples in literate programming, these have to be created and maintained manually, including their results (and tests).  Only notebooks combine all of the above in a single, self-contained package.  PrototypingNotebooks by themselves are language-agnostic; you can use them with any language you like.  Jupyter Notebooks as Productivity BoostersOne particular nice feature of Python is that it is _interactive_ – you create a function, try it out, alter it a bit, try it out again.  As they allow you to keep track of all your interactions and their results, Jupyter notebooks are a great way of developing a prototype step by step.  As shown in the above examples, output is not limited to strings; you can produce arbitrary graphics and diagrams.  And you can (and should) add text and examples that explain and illustrate what you do. These examples also serve as _tests_, as they would execute the code you just tried out.  You can throw in _assertions_ to check code against expectations:<jupyter_code>assert triangle(1, 1, 2) == 'isosceles'
assert triangle(2, 2, 2) == 'equilateral'<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05335499187804573, 'fraction_numerical': 0.0069973759840059975, 'mean_word_length': 4.493479752916953, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '443812', 'license': 'MIT', 'n_tokens_mistral': 2034, 'n_tokens_neox': 1926, 'n_words': 1190, 'path': 'notebooks/LiteratePrototyping.ipynb', 'repo_name': 'unibw-patch/fuzzingbook', 'size': '8003'}