On the Programmatic Generation of Reproducible Documents

Reproducible document standards, like R Markdown, facilitate the programmatic creation of documents whose content is itself programmatically generated. While these documents are generally not complete in the sense that they will not include prose content, generated by an author to provide context, a narrative, etc., programmatic generation can provide substantial efficiencies for structuring and constructing documents. This paper explores the programmatic generation of reproducible by distinguishing components than can be created by computational means from those requiring human-generated prose, providing guidelines for the generation of these documents, and identifying a use case in clinical trial reporting. These concepts and use case are illustrated through the listdown package for the R programming environment, which is is currently available on the Comprehensive R Archive Network (CRAN).


Background and concepts
R Markdown (Baumer, Cetinkaya-Rundel, Bray, Loi, and Horton 2014) facilitates the construction of computationally reproducible documents by allowing authors to insert R code for data processing, exploration, analysis, table-making, and visualization directly into structured, electronic documents. The resulting documents are made up of these chunks of R code, which we will refer to as computational components since they are generated by computational means, as well as narrative components, which (in scientific writing) is prose intended to contextualize computational components, provide background, define goals, establish themes, and convey the results of the document. The integration of narrative and computational components was identified as "Literate Programming" by Knuth (1984) and software tools, like Sweave (Leisch 2002), have supported this functionality for almost two decades.
R Markdown is particularly popular with its success likely being driven by two factors. The first is the relative ease with which these documents can be constructed. While L A T E X is more expressive, it is relatively technical and requires an investment in time to become proficient. By contrast R Markdown documents are easier to create and format and, when the document is used to create L A T E X, formatting can be passed through to the underlying .tex file. The second factor driving adoption is likely its support for creating modifiable documents, namely Microsoft Word documents. Researchers and analysts, especially those creating applied statistical analyses, often collaborate with domain experts with less technical knowledge. In these cases, the analyst focuses on creating the computational components and narrative components related to results and interpretation. After this initial document is created, the domain expert is free to develop narrative components directly in the document without needing to go through the analyst.
Since computational components are, by definition, computationally derived objects and R Markdown is a well-defined standard, it is possible to programmatically create R Markdown documents with computational components, which is the focus of this paper. Generating documents in this manner has two appealing characteristics. First, it allows us to distinguish the presentation of analytical results from other steps in a data science or data processing pipeline. These other steps including cleaning and analysis often require their own environment and configuration with requirements very different than the computational needs of creating a presentation. By separating these components each can be developed independently. At the same time, by specifying a contract for the output of those objects, we can establish a consistent means by which processed data can be passed to systems for presentating those data in a structured way. The second reason for programmatic creation of R Markdown documents is convenience. In collaborative environments, especially in the early stages, large numbers of graphs and tables are generated and discussed. By collecting these artifacts and structuring them consistently, we can quickly iterate upon and restructure the resulting documents to more clearly present the data without needing to spend time on the creation of the presentation document.
The listdown package provides functions to programmatically create R Markdown files from named lists. It is intended for data analysis pipelines where the presentation of the results is separated from their creation. For this use case, a data processing (or analysis) is performed and the results are provided in a single named list, organized heirarchically. List element names denote sections, subsections, subsubsection, etc. and the list elements contain the data structure to be presented including graphs and tables. The package has native support for workflowr (Blischak, Carbonetto, and Stephens 2019), pdf, word, or html document along with functions allowing a user to easily extend to other types of supported documents. The goal of the package is to create a documents with all tables and visualization that will appear (computational components). This serves as a starting point from which a user can organize outputs, describe a study, discuss results, and provide conclusions (narrative components).
listdown therefore provides a reproducible means for producing a document with specified computational components. It is most compatible with data analysis pipelines where the data format is fixed but the analyses are either being updated, which may affect narrative components including the result discussion and conclusion, or where the experiment is different, which affects all narrative components, but the data format and processing is consistent. An example of the former is provided later in this paper.
One area where we have found listdown is particuarly useful is in the reporting and research of clinical trial data. These collaborations tend to be between (bio)statisticians and clinicians either analyzing past trial data to formulate a new trial or in trial monitoring where trial telemetry (enrollment, responses, etc.) is reported and initial analyses are conveyed to a clinician. The associated presentations require very little context -clinicians often have a better understanding of the data collected than the statistician -often eliminating the need for narrative components. At the same time, a large number of hierarchical, heterogenous artifacts (tables and multiple types of plots) need to be conveyed thereby making the manual creation of R Markdown documents inconvenient.
In this case, data presentation can be fixed across trials. This is especially true in the initial stages, which focus on patient demographics and enrollment. This approach has made it convenient for our group to quickly generate standardize and complete reports for multiple trials concurrently. To date, we have used listdown to report on five clinical trials, with another two currently in process. Results are disseminated using the workflowr package, usually with nine tabs conveying aspects of the data from collection through several different analyses, and each tab containing approxiately five to thirty tables, plots, or other artifacts including trelliscopejs (Hafen and Schloerke 2020) environments which may hold hundreds of graphs. By generating many presentation artifacts we are able to address data-drive questions and issues during collaborative sessions and by carefully structuring these elements we allowing all members to participate in the process.
The listdown package itself is relatively simple with 6 distinct methods that can be easily incorporated into existing analysis pipelines for automatically creating documents that can be used for data exploration and reviewing analysis results as well as a starting point for a more formal write up. These methods include: The rest of this paper is structured as follow. The next section goes over basic usage and commentary. This section is meant to convey the basic approach used by the package and shows how to describe an output document using listdown, create a document, and change how the presentation of computational components can be specialized using listdown decorators. With the user accustomed to the package's basic usage, section 3 describes the design of the package. Section 4 goes over advanced usage of the package including adding initialization code to and outputted document as well as how to control chunk-level options. Section 5 provides a simplified case study of how the package is currently being used in clinical trial reporting. Section 6 concludes the paper with a few final remarks on the general types of applications where listdown has been shown effective.

Basic usage and overview
Suppose we have just completed and analysis and have collected all of the results into a list where the list elements are roughly in the order we would like to present them in a document. It may be noted that this is not always how computational components derived from data analyses are collated. Often individual components are stored in multiple locations on a single machine or across machines. However, it is important to realize that even for analyses on large-scale data, the digital artifacts to be presented are relatively small. Centralizing them makes it easier to access them, since they don't need to be found in mulitple locations. Also, storing them as a list provides a hierarchical structure that translates directly to a document as we will see below.
As a first example, we will consider the a list of visualizations from the anscombe data set. The list is composed of four ggplot2 (Wickham 2016)elements (named Linear, Non Linear, Outlier Vertical, and Outlier Horizontal) each containing a scatter plot from the famous Anscome Quartet -made available in the datasets package (R Core Team 2012). From the computational_components list, we would like to create a document with four sections with names corresponding to the list names, each containing their respective visualizations.

Creating a document with listdown
Creating a document from the computational_components requires two steps. First, we will create a listdown object that specifies how the computational_components object will be loaded into the document, which libraries and code needs to be included, and how the list elements will be presented in the output R markdown document.
R> library(listdown) R> R> ld <-listdown(load_cc_expr = readRDS("comp-comp.rds"), R+ package = "ggplot2") The ld object, along with the computational components in the comp-comp.rds file are sufficient to to create the sections, subsections, and R chunks of a document. The only other thing requires to create the document is the header. The listdown package currently supports regular R Markdown and workflowr as yml objects from the yaml package (Stephens, Simonov, Xie, Dong, Wickham, Horner, reikoch, Beasley, O'Connor, and Warnes 2020). These objects are stored as named lists in R and are easily modified to accomodate document parameters. A complete document can then be written to the console using the code shown below. It could easily be written to a file for rendering using the writeLines() function.
R> doc <-c( R+ as.character(ld_rmarkdown_header("Anscombe's Quartet", R+ author = "Francis Anscombe", R+ date = "1973")), R+ ld_make_chunks (ld) The listdown() function provides document-wide R chunk options for displaying computational components. The chunk options are exactly the same as those in the R markdown document and can be used to tailor the default presentation for a variety of needs. The complete set of options can be found in the R Markdown Reference Guide (RStudio 2020). As a concrete example, the code used to create present the plots could be hidden in the output document using the following code.

Decorators
The first example is simple in part because the ggplot objects both contains the data we want to display and, at the same time, provides the mechanism for presenting them -rendering them in a graph. However, this is not always the case. The objects being stored in the list of computational components may not translate directly to the presentation in a document. In these cases, a function is needed that takes the list component and returns an object to be displayed. For example, suppose that, along with showing graphs from the Anscombe Quartet, we would like to include the data themselves. We could add the data to the computational_components list and then create the document with: R> computational_components$Data <-anscombe R> saveRDS(computational_components, "comp-comp.rds") R> ld_make_chunks(ld)[32:36] [1] "# Data" [2] "" [3] " {r echo = FALSE}" [4] "cc_list$Data" [5] " " In this case, the listdown package will show the entire data set as is the default specified. However, suppose we do not want to show the entire data set in the document. This is common, especially when the data set is large and requires too much vertical space in the outputted document resulting in too much or irrelevant data being shown. Instead, we would like to output to an html document where the data is shown in a datatable thereby controlling the amount of real-estate needed to present the data and, at the same time, providing the user with interactivity to sort and search the data set.
In listdown, a function or method that implements the presentation of a computational component is referred to as a decorator since if follows the classic decorator pattern described in Gamma, Helm, Johnson, and Vlissides (1995). A decorator takes the element that will be presented as an argument and returns an object for presentation in the output directory. A decorator is specified using the decorator parameter of the listdown() function using a named list where the name corresponds to the type and the element correspond to the function or method that will decorate an object of that type. For example, the anscombe data set can be decorated with the DT::datatable() function (Xie, Cheng, and Tan 2020) as: R> ld <-listdown(load_cc_expr = readRDS("comp-comp.rds"), R+ package = c("ggplot2", "DT"), R+ decorator = list(data.frame = datatable)) R> R> ld_make_chunks(ld)[33:37] [1] "# Data" [2] "" [3] " {r}" [4] "datatable(cc_list$Data)" [5] " " List names in the decorator argument provide a key to which a function or method is mapped. The underlying decorator resolution is implemented for a given computational component by going through decorator names sequentially to see if the component inherits from the name using the inherits() function. The function or method is selected from the corresponding name which the element first inherits from. This means that when customizing the presentation of objects that inherit from a common class, the more abstract classes should appear at the end of the list. This will ensure that specialized classes will be encountered first in the resolution process. It should be noted that an object's type is first checked against the decorator name list and then checked to see if it is a list. This allows a user to both decorate a list and retain "list" in its class attributes.
A separate argument, default_decorator, allows the user to specify the default decorator for an object whose type does not appear in the decorator list. This allows the user to specify any class name for the decorator and avoids a potential type name collision with a default decorator whose name is determined by convention. By default, this argument is set to identity but it can be use to not display a computational component by default if the argument is set to NULL.

Design
A listdown object specifies the location of a list of computational components and options for presenting those components in an R Markdown document. The list is a hierarchical data structure that also provides the structure of the outputted document. A corresponding document has two sections "Iris" and "Sepal.Length". The latter has three subsections "Sepal.Width", "Petal.Length", and "Colored". The "Colored" subsection has two sub-subsections, "Sepal.Width" and "Petal.Length". The structure can be seen using the ld_cc_dendro() function. aes(x = Sepal.Length, y = Petal.Length, R+ color = Species)) + geom_point()))) R> R> # Create the dendrogram. R> ld_cc_dendro(comp_comp2) Both the ld_cc_dendro() and ld_make_chunks() functions work by recursively descending the computational components list depth-first. If the list containing and element has a name, it is written to the output as a section, subsection, subsubsection, etc. to a return string. If the visited list element is itself a list, then the same procedure is called on the child list through a recursive call. If the element is not a list, then it is outputted inside an R Markdown chunk in the return string using the appropropriate decorator.

Advanced usage 4.1. Initialization code
The listdown() function facilitates the insertion of initialization code through the init_expr argument. Code is inserted immediately after the libraries are loaded in the R Markdown document. In general, it is suggested that the number of initial expressions be kept small so that the R Markdown document is easy to read. If a large number of functions are required by the target R Markdown document then they can be put into a file and sourced using the initial expression. As an example, suppose we are creating an html document and presenting data using the datatable() function. However, we do not want to include the search capabilities provided. This can be easily accomplished by creating a new function, datatable_no_search(), created using the partial() function Henry and Wickham (2020) to partially apply list(dom = 't') to the options argument of datatable.

R code chunk customization
The listdown package supports also supports capabilities to further customize the presentation by specifying R code chunk options in the R Markdown document in two distinct ways. The first is used when the options we would like to specify is tied to type of the object being presented. This can be though of as a chunk-option decorator. The second is use for changing the options for an individual chunk in an ad-hoc manner.
With three different modes of chunk customization it should be noted that the increasing priority of the chunk options specification is document-wide, decorator-wide, and ad hoc. That is, decorator-wide chunk options take priority over document-wide chunk options and ad-hod options take priority over decorator-wide options. In addition, it should be noted that the use of the lowest priority scheme that accomplishes the presentation goals is preferred because it lends itself to greater code and maintenance efficiency.

Controlling options associated with a decorator
The document-wide chunk option specification provides the default chunk options for output documents generated using listdown. However, the presentation of a data object often varies by type. For example, we may want to specify the height and width of a graph, but not a table. This is accomplished in the listdown package when a listdown object is created using the decorator_chunk_opts option in the listdown() function. For example, associating all ggplot objects with R chunks having a width of 100 and a height of 200 can be accomplished with the following code and it can be seen that only chunk options associated with a plot are modified.

Controlling chunk-level options
Along with providing decorator-wide chunk options, it is also possible to control individual chunk options. The capability is distinct from the document-wide and decorator-wide specification of options in that it must be applied to the computational component list element whose associated options will be modified, rather than the listdown object. This is because the listdown only specifies how classes of objects should be presented. To modify the chunk options associated with a specific list element the list element is provided with a set of attributes that can be queried by the ld_chunk_opts() function as the output document is being generated. Because of the ad hoc nature of this capability, its use is discouraged. A better solution, that maintains the behavior is to add class information to the list element and specify decorator-wide chunk options for the new class. This maintains the separation of the computational component list, which maintains the document structure and data for presentation from the specification of how the document will be created and rendered.
R> comp_comp2$Iris <-ld_chunk_opts(comp_comp2$Iris, echo = TRUE) R> saveRDS(comp_comp2, "comp-comp2.rds") R> ld_make_chunks(ld)[12:16] [1] "# Iris" [2] "" [3] " {r echo = TRUE}" [4] "cc_list$Iris" [5] " " 5. A simple example: Reporting on the gtsummary::trial dataset As mentioned before, we have found the listdown package particularly helpful for reporting the results of clinical trials thereby creating a basis for discussion and collaboration between (bio)statisticians and the clinicians running the trials. For this use case, the context and data collection procedures are well-understood and as a result very few narrative components are needed. It is also the case that the modes of presentation (tables, scatterplots, survivalplots, consort diagrams, etc.) are standardized. The goal in presenting the trial characteristics is to identify problems in the data, monitor trial enrollment and response, quantify known relationships among the data, and test hypotheses about a therapy's efficacy.
In practice we generally separate the data cleaning, exploration, analysis, monitoring, and presentation components. The computational component list has tens of elements with thousands of visualizations. These facts, coupled with the privacy constraints make a complete example difficult for the purposes of this paper. So, below we provide a simple example of the types of documents we provide using the gtsummary::trial dataset. While it is not complete, it does convey they types of reports we are currently generating for completed and ongoing clinical trials.

Conclusion
While the programmtic generation of reproducible documents has appealing qualities, it also has fundamental limitations that should be kept in mind when deciding if tools like listdown should be employed. First and foremost, without narrative components a document has very little context. Quantitative analyses require research questions, hypotheses, reviews, interpretations, and conclusions. Computational components are necessary but generally not sufficient for constructing an analysis. This means that if narrative components must be conveyed in a document, then listdown may make the generation of their presentation more convenient. However, it does not relieve the burden of the author to create prose developing a narrative.
Second, it is difficult if not impossible to construct computational components for an arbitary analyses. Analyses themselves have context and are built with a set of assumptions and goals. Our experience shows that listdown is easiest to used for a fixed data format. This means a standard set of table and visualizations for similarly formatted data. In particular, data that is periodicaly updated, without changes to the format, is a case that is particularly amenable to document generation reuse.
Keeping these limitations in mind, listdown can be used to effectively reduce the the difficulty of generating documents in a variety of contexts and fits readily in data processing and analysis pipelines.