<commit_before># Master Parse Tool

## Introduction

The master parse tool parses Databricks notebooks (in source form) and,
based on metadata embedded within the notebooks, produces (possibly multiple)
output notebooks.

It can be used as a command or as a library.

The tool is usable from either Python 2 or Python 3.

## Installation

To install, run

```
python setup.py install
```

The setup file installs all required packages, as well as `master_parse`.

## Command line use

Once installed, the master parse tool can be invoked as `master_parse`.
It takes arguments in several groups, and at least one argument from
each group is required.

### Notebook type group

* `-db` or `--databricks`: Generate Databricks notebook(s)
* `-ip` or `--ipython`: Generate IPython (i.e., Jupyter) notebook(s)

More than one option may be specified.

### Notebook language group

* `-sc` or `--scala`: Generate Scala notebooks
* `-py` or `--python`: Generate Python notebooks
* `-r` or `--rproject`: Generate R notebooks
* `-sq` or `--sql`: Generate SQL notebooks

More than one option may be specified.

### Target audience group

* `-in` or `--instructor`: Generate instructor notebooks (notebooks ending
  with `_instructor`). Instructor notebooks contain all instructor note cells
  and all answer cells. Exercise cells (those marked `TODO`) are omitted.

* `-an` or `--answers`: Generate answer notebooks (notebooks ending
  with `_answers`). Answer notebooks contain all answer cells. Exercise cells 
  (those marked `TODO`) are omitted, as are instructor note cells. Thus, an
  answer notebook is the same as the corresponding notebook _minus_ the 
  instructor notes.

* `-ex` or `--exercises`: Generate exercise notebooks (notebooks with 
  `_exercises`). These notebooks are the primary student notebooks. They omit
  answer cells and instructor notes and contain all cells marked `TODO`.

More than one option may be specified.

### Miscellaneous options

* `-cc`: Add a Markdown cell at the top of the notebook that identifies the 
  notebook as being released under a Creative Commons
  [Attribution-NonCommercial-NoDerivatives 4.0 International](https://creativecommons.org/licenses/by-nc-nd/4.0/) 
  (CC BY-NC-ND 4.0) license.
* `-d OUTPUT_DIR`: Specify the output directory for the build artifacts. 
  Defaults to `build_mp`
* `-ei ENCODING`: Specifies the encoding of the input file. Defaults to
  "cp1252", though a more useful default might be "UTF-8".
* `-eo ENCODING`: Specifies the desired encoding of the output files.
  Defaults to "UTF-8".
* `-nh PATH` or `--notebook-heading PATH`: Specifies the path to a file
  containing Markdown and/or HTML to be used as the top-of-notebook heading.
  If not specified, internally coded default HTML is used.
* `--heading`: By default, even if you specify `-nh`, the `master_parse`
  tool does _not_ add the notebook heading at the top of the generated
  notebooks. You need to specify `--heading` to force that behavior.
  (This preserves historical behavior better.)

### Filename group

* `<filename>` specifies the notebook file to parse.
* `-v` or `--version` tells `master_parse` to display its version and exit.
  This option is mutually exclusive with `<filename>`.


## Library use

```python
import master_parse
master_parse.process_notebooks(args)
```

The arguments correspond to the command line parameters. See the source for
details.

## Notebook metadata

The tool looks for various labels, as well as language-specific tokens, within
notebook cells.

### Language tokens

By default, the master parse tool processes `%scala`, `%python`, `%r` and
`%sql` cells specially. How it handles those cells is best described by
example.

Suppose you've run the tool on a Python notebook (i.e., a file ending in `.py`),
and the notebook also contains some `%scala` and `%r` cells.

* If you've specified `--scala` (on the command line) or passed
  `scala=True` (to `master_parse.process_notebooks()`), then the tool will
  create Scala notebooks that contain all non-code cells (Markdown cells,
  `%fs` and `%sh` cells, etc.) in the original, as well as any `%scala` cells.
  All other language code cells will be stripped from the Scala notebooks.
* If you've specified `--rproject` (on the command line) or passed
  `r=True` (to `master_parse.process_notebooks()`), then the tool will
  create R notebooks that contain all non-code cells (Markdown cells,
  `%fs` and `%sh` cells, etc.) in the original, as well as any `%r` cells.
  All other language code cells will be stripped from the R notebooks.
* If you've specified `--python` (on the command line) or passed
  `python=True` (to `master_parse.process_notebooks()`), then the tool will
  create Python notebooks that contain all non-code cells (Markdown cells,
  `%fs` and `%sh` cells, etc.) in the original, as well as any Python
  cells. Since the file ends in `.py`, Python cells are assumed to be
  cells with explicit `%python` magic or any non-decorated (i.e., normal)
  code cells.

You can modify this behavior somewhat, using the labels below.

### Master parse labels

Master parse labels are cells that are marked with special tokens that only
the master parse tool recognizes. Some labels make sense only in code cells.
Others can be used in code cells, Markdown cells, etc.

All  labels _must_ be preceded by a single command character, and the comment
character must be correct for the notebook type. That is, a Python notebook
(ending in `.py`) always uses "#" as the comment character. This is true
even in Markdown cells.

Cells not marked with any label appear in _all_ outputs.

#### Examples

In a Scala code cell:

```scala
// ANSWER
// Scala answer goes here<commit_msg>Doc clarification.<commit_after># Master Parse Tool

## Introduction

The master parse tool parses Databricks notebooks (in source form) and,
based on metadata embedded within the notebooks, produces (possibly multiple)
output notebooks.

It can be used as a command or as a library.

The tool is usable from either Python 2 or Python 3.

## Installation

To install, run

```
python setup.py install
```

The setup file installs all required packages, as well as `master_parse`.

## Command line use

Once installed, the master parse tool can be invoked as `master_parse`.
It takes arguments in several groups, and at least one argument from
each group is required.

### Notebook type group

* `-db` or `--databricks`: Generate Databricks notebook(s)
* `-ip` or `--ipython`: Generate IPython (i.e., Jupyter) notebook(s)

More than one option may be specified.

### Notebook language group

* `-sc` or `--scala`: Generate Scala notebooks
* `-py` or `--python`: Generate Python notebooks
* `-r` or `--rproject`: Generate R notebooks
* `-sq` or `--sql`: Generate SQL notebooks

More than one option may be specified.

### Target audience group

* `-in` or `--instructor`: Generate instructor notebooks (notebooks ending
  with `_instructor`). Instructor notebooks contain all instructor note cells
  and all answer cells. Exercise cells (those marked `TODO`) are omitted.

* `-an` or `--answers`: Generate answer notebooks (notebooks ending
  with `_answers`). Answer notebooks contain all answer cells. Exercise cells 
  (those marked `TODO`) are omitted, as are instructor note cells. Thus, an
  answer notebook is the same as the corresponding notebook _minus_ the 
  instructor notes.

* `-ex` or `--exercises`: Generate exercise notebooks (notebooks with 
  `_exercises`). These notebooks are the primary student notebooks. They omit
  answer cells and instructor notes and contain all cells marked `TODO`.

More than one option may be specified.

### Miscellaneous options

* `-cc`: Add a Markdown cell at the top of the notebook that identifies the 
  notebook as being released under a Creative Commons
  [Attribution-NonCommercial-NoDerivatives 4.0 International](https://creativecommons.org/licenses/by-nc-nd/4.0/) 
  (CC BY-NC-ND 4.0) license.
* `-d OUTPUT_DIR`: Specify the output directory for the build artifacts. 
  Defaults to `build_mp`
* `-ei ENCODING`: Specifies the encoding of the input file. Defaults to
  "cp1252", though a more useful default might be "UTF-8".
* `-eo ENCODING`: Specifies the desired encoding of the output files.
  Defaults to "UTF-8".
* `-nh PATH` or `--notebook-heading PATH`: Specifies the path to a file
  containing Markdown and/or HTML to be used as the top-of-notebook heading.
  If not specified, internally coded default HTML is used.
* `--heading`: By default, even if you specify `-nh`, the `master_parse`
  tool does _not_ add the notebook heading at the top of the generated
  notebooks. You need to specify `--heading` to force that behavior.
  (This preserves historical behavior better.)

### Filename group

* `<filename>` specifies the notebook file to parse.
* `-v` or `--version` tells `master_parse` to display its version and exit.
  This option is mutually exclusive with `<filename>`.


## Library use

```python
import master_parse
master_parse.process_notebooks(args)
```

The arguments correspond to the command line parameters. See the source for
details.

## Notebook metadata

The tool looks for various labels, as well as language-specific tokens, within
notebook cells.

### Language tokens

By default, the master parse tool processes `%scala`, `%python`, `%r` and
`%sql` cells specially. How it handles those cells is best described by
example.

Suppose you've run the tool on a Python notebook (i.e., a file ending in `.py`),
and the notebook also contains some `%scala` and `%r` cells.

* If you've specified `--scala` (on the command line) or passed
  `scala=True` (to `master_parse.process_notebooks()`), then the tool will
  create Scala notebooks that contain all non-code cells (Markdown cells,
  `%fs` and `%sh` cells, etc.) in the original, as well as any `%scala` cells.
  All other language code cells will be stripped from the Scala notebooks.
* If you've specified `--rproject` (on the command line) or passed
  `r=True` (to `master_parse.process_notebooks()`), then the tool will
  create R notebooks that contain all non-code cells (Markdown cells,
  `%fs` and `%sh` cells, etc.) in the original, as well as any `%r` cells.
  All other language code cells will be stripped from the R notebooks.
* If you've specified `--python` (on the command line) or passed
  `python=True` (to `master_parse.process_notebooks()`), then the tool will
  create Python notebooks that contain all non-code cells (Markdown cells,
  `%fs` and `%sh` cells, etc.) in the original, as well as any Python
  cells. Since the file ends in `.py`, Python cells are assumed to be
  cells with explicit `%python` magic or any non-decorated (i.e., normal)
  code cells.

You can modify this behavior somewhat, using the labels below.

### Master parse labels

Master parse labels are cells that are marked with special tokens that only
the master parse tool recognizes. Some labels make sense only in code cells.
Others can be used in code cells, Markdown cells, etc.

All labels _must_ be preceded by a comment sequence. For instance:

```
# TODO
// TODO
-- TODO
```

Labels must appear on a line by themselves. Thus, use:

```
%md
// SCALA_ONLY
```

not 

```
%md // SCALA_ONLY
```


#### Unlabeled

Cells not marked with any label are handled specially, depending on the cell
type:

* `%md`, `%md-sandbox`: Markdown cells appear in all output notebooks, unless
  suppressed, for example, with `SCALA_ONLY`, `PYTHON_ONLY`,
  `INSTRUCTOR_NOTE`, etc.
  
* `%fs` and `%sh` cells appear in all output notebooks, unless explicitly
  suppressed.
  
* Code cells only appear in the output notebook for their language, unless
  marked with `ALL_NOTEBOOKS`. Thus, a Scala cell only shows up in Scala 
  notebooks, unless marked with `ALL_NOTEBOOKS`.

#### Examples

In a Scala code cell:

```scala
// ANSWER
// Scala answer goes here
{'fraction_non_alphanumeric': 0.09186417885358884, 'fraction_numerical': 0.002353336695242898, 'mean_word_length': 4.104676104676105, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2935091', 'n_tokens_mistral': 3544, 'n_tokens_neox': 3367, 'n_words': 1669}