<commit_before>Grammar Specification
=====================

The grammar follows that of [Markdown](http://daringfireball.net/projects/markdown/syntax), with elements from [Scholmd](https://github.com/scholmd/scholmd/wiki/_pages) and [Github-flavored Markdown](https://help.github.com/articles/github-flavored-markdown).

It deviates from the original Markdown as follows:

Inline Blocks
-------------
Inline code supports a schema like so: `tag, actual code`, where tag can be used for various things such as syntax highlighting.
For instance, code highlighting and explicit inclusion of raw latex could be denoted by this.
As well, perhaps it could be used as a method to inline scripts for text generation?

Inline Math
-----------
Inline math equations can be denoted by $ signs. \$ will place a literal $ in either text or inlined math equations.

Code Blocks
-----------
These also support tags in the same fashion that inline blocks do, and should be handled in the same way.

Citations
---------
Citations need to handle bibtex-style citations. This also means we'll need a method for including bibtex files.

Table Syntax
------------
LaTeX's table syntax sucks.
The original Markdown doesn't provide a spec for tables, but Reddit's does, so that may be worth investigating.

Current State of the Grammar
============================

Completed Features
------------------
* Headings
* Paragraphs and Lines
* Inline blocks
* Inline math
* Code blocks
<commit_msg>Better docs for current state<commit_after>Grammar Specification
=====================

The grammar follows that of [Markdown](http://daringfireball.net/projects/markdown/syntax), with elements from [Scholmd](https://github.com/scholmd/scholmd/wiki/_pages) and [Github-flavored Markdown](https://help.github.com/articles/github-flavored-markdown).

It deviates from the original Markdown as follows:

Inline Blocks
-------------
Inline code supports a schema like so: `` `tag, actual code` ``, where tag can be used for various things such as syntax highlighting.
For instance, code highlighting and explicit inclusion of raw latex could be denoted by this.
As well, perhaps it could be used as a method to inline scripts for text generation?

Inline Math
-----------
Inline math equations can be denoted by $ signs. \$ will place a literal $ in either text or inlined math equations.

Code Blocks
-----------
These also support tags in the same fashion that inline blocks do, and should be handled in the same way.

Citations
---------
Citations need to handle bibtex-style citations. This also means we'll need a method for including bibtex files.

Table Syntax
------------
LaTeX's table syntax sucks.
The original Markdown doesn't provide a spec for tables, but [Reddit's](http://www.reddit.com/wiki/commenting) does, so that may be worth investigating.

Current State of the Grammar
============================

Completed Features
------------------
* Headings
* Paragraphs and Lines
* Inline math
* Code blocks

Inline Blocks
-------------
Mostly done. They escape `` ` `` with `` \` ``, but do not yet support multiple-backticks like [markdown](http://daringfireball.net/projects/markdown/syntax#code) does.

{'fraction_non_alphanumeric': 0.14245371823031064, 'fraction_numerical': 0.0, 'mean_word_length': 3.6676427525622253, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5721011', 'n_tokens_mistral': 840, 'n_tokens_neox': 800, 'n_words': 378}