<commit_before># mfnf-pdf-export
mfnf-pdf-export is a set of tools to create documents from MediaWiki articles. Target formats are currently LaTeX and PDF. 
Our main design goals are extensibility and simplicity. 

The heart of this repository is a collection of Makefiles, which build the target file step-by-step using small, unix-like tools.

## Glossary

* Target: An output format, like LaTeX, PDF or HTML
* Subtarget: A specific configuration of a target. E.g. PDF for print (pdf.print), verbose LaTeX (latex.all), ...

## Getting started

This tool is only tested on linux so far, so these instructions are for linux only. 
Make sure you have the following software installed:

* python3
* bash
* python3-virtualenv
* cargo ([rust package manager](https://www.rust-lang.org/en-US/install.html))
* inkscape
* qrencode
* lualatex (e.g. via texlive-full)
* ocaml
* convert (by image magick)
* sed

In the root of this repository, create a new python virtual environment:
``` sh
virtualenv -p python3.6 venv
```
Source the virtual environment:
``` sh
source venv/bin/activate
```
Now run the init script. This will install some python libraries in your virtual environments and build some tools needed for article parsing / export. 
``` sh
make init
```
Done!

## Usage

To build a file with make, simply supply the path to the target file and let make do its magic. The directory structure of article exports is as follows:
``` sh
article_exports/<target>/<subtarget>/<escaped article name>/<revision id>.<extension>
```
This looks complicated, but actually looks quite beatiful when running `tree` on it ;)

* `<target>`: (latex, pdf, ...) The target format format. Targets and subtargets are defined in `config/mfnf.yml`.
* `<subtarget>`: (all, print, ...) The subtarget. Configuration see `<target>`
* `<escaped article name>`: Article names can be extracted from the article url: `https://de.wikibooks.org/wiki/Mathe_für_Nicht-Freaks:_Beispielkapitel:_Grundlegende_Formatierungen -> Mathe_für_Nicht-Freaks:_Beispielkapitel:_Grundlegende_Formatierungen`.
Unfortunately, make needs some special characters escaped (see table below)
This means: `Mathe_für_Nicht-Freaks:_Beispielkapitel:_Grundlegende_Formatierungen` becomes `Mathe_für_Nicht-Freaks@COLON@_Beispielkapitel@COLON@_Grundlegende_Formatierungen`.
* `<revision_id>`: Revision ids can be found on the history page of an article. 
* `<extension>`: File extension of your target format. (e.g. `pdf` for PDF, `tex` for LaTeX)

| original | escaped   |
|-----|-----------|
| `:` | `@COLON@` |
| ` ` | `_`       |
| `(` | `@LBR@`   |
| `)` | `@RBR@`   |

Thus, to build the example article we have to tell make to "make" the file:
``` sh
make article_exports/pdf/all/Mathe_für_Nicht-Freaks@COLON@_Beispielkapitel@COLON@_Grundlegende_Formatierungen/843164.pdf
```
Now we can use `tree article_exports` to print the directory structure:
```
article_exports
├── latex <-- pdf depends on latex, so latex is also built.
│   └── all
│       └── Mathe_für_Nicht-Freaks@COLON@_Beispielkapitel@COLON@_Grundlegende_Formatierungen
│           ├── 843164.dep
│           └── 843164.tex
└── pdf
    └── all
        └── Mathe_für_Nicht-Freaks@COLON@_Beispielkapitel@COLON@_Grundlegende_Formatierungen
            ├── 843164.aux
            ├── 843164.fdb_latexmk
            ├── 843164.fls
            ├── 843164.log
            ├── 843164.out
            ├── 843164.pdf <-- Our pdf file is here
            ├── 843164.tex
            └── 843164.yml
```<commit_msg>update readme for new build system<commit_after># mfnf-pdf-export
mfnf-pdf-export is a set of tools to create documents from MediaWiki articles. Target formats are currently LaTeX, PDF and HTML (an a statistics target), more are planned.
Building on our previous experiences, the main design goals of this project are simplicity and extensibility. 

The heart of this repository is a collection of Makefiles, which build the target file step-by-step using small, unix-like tools.
We try to use existing / standard tools where possible. Most of the helper programs we develop ourselves are written in [Rust](https://www.rust-lang.org)
or [Python 3](https://www.python.org).

## Glossary

* Target: An output format, like LaTeX, PDF or HTML
* Subtarget: A specific configuration of a target. E.g. PDF for print (pdf.print), minimalistic HTML (html.minimal), ...
* Article: A document dealing with a certain topic, equivalent to articles in [WikiBooks](https://de.wikibooks.org/wiki/Hauptseite).
* Sitemap: A special article describing the composition of a book.
* Book: A collection of articles with some modifications like excluded headings or added meta information pages.
* Section: In our code, section usually refers to parts of an article marked with the `<section begin="..." />` tag.
* Marker: Annotation on a Part or Chapter of a book

## Getting started

This tool is only tested on linux so far, so these instructions are for linux only. 
Make sure you have the following software installed:

* python3
* python3-virtualenv
* cargo ([rust package manager](https://www.rust-lang.org/en-US/install.html))
* inkscape
* qrencode
* jq
* cmark
* lualatex (e.g. via texlive-full)
* ocaml
* convert (by image magick)
* sed

In the root of this repository, create a new python virtual environment:
``` sh
virtualenv -p python3.6 venv
```
Source the virtual environment:
``` sh
source venv/bin/activate
```
Now run the init script. This will install some python libraries in your virtual environments and build some tools needed for article parsing / export. 
``` sh
make init
```
Done!

## Usage

Whenever an article name or path is passed to `make`, it must be escaped:
 
| original | escaped   |
|-----|-----------|
| ` ` |  `_` |
| `:` |  `@COLON@` |
| `(` |  `@LBR@` |
| `)` |  `@RBR@` |
| `/` |  `@SLASH@` |
| `'` |  `@SQUOTE@` |
| `"` |  `@DQUOTE@` |
| `*` |  `@STAR@` |
| `=` |  `@EQ@` |
| `$` |  `@DOLLAR@` |
| `#` |  `@SHARP@` |
| `%` |  `@PERC@` |

### Book export

To build a file with make, simply supply the path to the target file and let make do its magic. The directory structure of book exports is as follows:
``` sh
make exports/<sitemap>/<sitemap revision>/<target>/<subtarget>/<sitemap revision>.book.<extension>
```
where
* `<sitemap>`: Name of the sitemap article. (escaped as described above)
* `<sitemap revision>`: Revision of the sitemap article to use. Can be *latest*.
* `<target>`: (latex, pdf, ...) The target format format. Targets and subtargets are defined in `config/mfnf.yml`.
* `<subtarget>`: (all, print, ...) The subtarget. Configuration see `<target>`
* `<extension>`: File extension of your target format. (e.g. `pdf` for PDF, `tex` for LaTeX, `html` for HTML, `stats.html` or `stats.yml` for stats)

This looks complicated, but actually looks quite beatiful when running `tree` on it ;)

### Article export

Exporting only a specific article works similarly, but the export of an article normally depends on its context in the book (e.g. for link targets). When we export a single article, we usually want it exported as if it was its own book. To force this behaviour, we have to export articles in the context of a "dummy book", called *articles*:

``` sh
make exports/articles/latest/<target>/<subtarget>/<article name>/<article revision>.<extension>
```
* `<article name>`: Article names can be extracted from the article url: `https://de.wikibooks.org/wiki/Mathe_für_Nicht-Freaks:_Beispielkapitel:_Grundlegende_Formatierungen -> Mathe_für_Nicht-Freaks@COLON@_Beispielkapitel@COLON@_Grundlegende_Formatierungen`.
* `<article revision>`: Revision IDs of an article can be found on its history page. (or use *latest*)
* see [Book export](#book-export)

### Example

Thus, to build the example article we have to tell make to `make` the file:
``` sh
make exports/articles/latest/html/all/Mathe_für_Nicht-Freaks@COLON@_Beispielkapitel@COLON@_Grundlegende_Formatierungen/latest.html
```
Make will fetch the latest article revisions and media files included in the article and will continue to export it to HTML.

Now we can use `tree exports` to print the directory structure:
```
exports
└── articles
    ├── dummy
    │   └── html
    │       └── all
    │           ├── dummy.book.dep
    │           └── Mathe_für_Nicht-Freaks@COLON@_Beispielkapitel@COLON@_Grundlegende_Formatierungen
    │               ├── 848617.anchors
    │               ├── 848617.html
    │               ├── 848617.markers
    │               ├── 848617.media-dep
    │               ├── 848617.raw_html
    │               ├── 848617.section-dep
    │               └── latest.html -> 848617.html
    └── latest -> dummy
```
{'fraction_non_alphanumeric': 0.12004621606008088, 'fraction_numerical': 0.013518197573656845, 'mean_word_length': 3.7404162102957286, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 38, 'https://': 7, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2029252', 'n_tokens_mistral': 2890, 'n_tokens_neox': 2730, 'n_words': 1005}