<commit_before>texts.js
========

**texts.js** is a JavaScript library (for Node and for browser) that implements
conversion between various “rich” text formats. It does the conversion via
universal text model (TextJSON, see below) that is rich enough for most of the
texts, yet simple.

Current version of texts.js supports reading of Text (Markdown subset) and
writing of Text, HTML5 and XeLaTeX formats.

Usage
-----

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ npm install texts
$ texts --from text --to html5
*Hello* world!
^D
<p><em>Hello<em> world!</p>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For information on options use the following command.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
texts --help
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Text Format
-----------

Text format produced and recognized by texts.js is similar to Markdown but is
more general. It is not tied to generation of HTML. The purpose of Text format
to represent a structure that can be published in any format (provided that the
format offers enough of expressive means). The major differences from Markdown
are:

-   no support for inline HTML,

-   explicit format definition (in form of PEG grammar),

-   single markup for each style.

The following paragraph styles are supported:

-   headings,

-   bulleted and numbered lists,

-   quotations,

-   verbatim (code) blocks,

-   math formulas,

-   images,

-   footnotes,

-   hyperlinks.

There is also intention to make Text format a bit more rational. For example,
indentation in Markdown can represent either structure (second paragraph in a
list) or style (verbatim block). In Text format indentation represents text
structure only.

Files in the Text format can be edited in any plain text editor or in a WYSIWYM
editor [Texts][1].

[1]: <http://www.texts.io/>

TextJSON
--------

TextJSON is a format of universal text model in **text.js**. It is based on
JSONML, a format for structured data. Any TextJSON document is a valid JSON
document and thus can be easily produced or consumed in any programming
language.

Each TextJSON object is a 2-level structure—an array of blocks, where each block
is an array of spans. Plus a formal root element “text”. Each block has
attribute “level” with zero being the level of the main text, positive levels
for nested elements and negative levels for headings.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ texts --from text --to json<commit_msg>Minor corrections to README.<commit_after>texts.js
========

**texts.js** is a JavaScript library (for Node and for browser) that implements
conversion between various “rich” text formats. It does the conversion via
universal text model (TextJSON, see below) that is rich enough for most of the
texts, yet simple.

Current version of texts.js supports reading of Text (Markdown subset), Pandoc
JSON and writing of Text, HTML5, XeLaTeX and Pandoc JSON formats.

Usage
-----

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ npm install texts
$ texts --from text --to html5
*Hello* world!
^D
<p><em>Hello<em> world!</p>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For information on options use the following command.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
texts --help
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Text Format
-----------

Text format produced and recognized by texts.js is similar to Markdown but is
not tied to the generation of HTML. The purpose of Text format to represent a
structure that can be published in any text format. The major differences from
Markdown are:

-   no support for inline HTML,

-   explicit format definition (in form of PEG grammar),

-   one variation of markup for each style.

The following paragraph styles are supported:

-   headings,

-   bulleted and numbered lists,

-   quotations,

-   verbatim (code) blocks,

-   math formulas,

-   images,

-   footnotes,

-   hyperlinks.

There is also intention to make Text format a bit more rational. For example,
indentation in Markdown can represent either structure (second paragraph in a
list) or style (verbatim block). In Text format indentation represents text
structure only.

Files in the Text format can be edited in any plain text editor or in a WYSIWYM
editor [Texts][1].

[1]: <http://www.texts.io/>

TextJSON
--------

TextJSON is the format of the universal text model in **text.js**. It is based
on [JsonML][2], format for structured data. Any TextJSON document is a valid
JSON document and thus can be easily produced or consumed in any programming
language.

[2]: <http://www.jsonml.org/>

Each TextJSON object is a 2-level structure—an array of blocks, where each block
is an array of spans. Above them is a formal root element “text”. Each block has
attribute “level” with zero being the level of the main text, positive levels
for nested elements and negative levels for headings.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ texts --from text --to json
{'fraction_non_alphanumeric': 0.21514922553834528, 'fraction_numerical': 0.0022667170381564035, 'mean_word_length': 4.498442367601246, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2571919', 'n_tokens_mistral': 1337, 'n_tokens_neox': 1258, 'n_words': 661}