<commit_before>.. _transforms:

Transforms
==========

Each Kibitzr transform modifies content and passes it forward.
Transforms can be divided into following groups: HTML, plain text, JSON.

HTML
----

1. ``tag: tagname`` - crop HTML to contents of the first matching HTML tag.
2. ``css: selector`` - crop HTML to the first encountered outer HTML matching passed `CSS selector`_.
3. ``css-all: selector`` - crop HTML to the concatenated list of all matching elements.
4. ``xpath: path`` - crop HTML to contents of the passed `XPath`_.
5. ``text`` - strip all HTML tags and return only text.

Plain text
----------

1. ``changes`` - Compare to the previous version of the content and return difference report
2. ``changes: verbose`` - Same as ``changes``, but in human-friendly format
3. ``sort`` - Sort lines of text alphabetically
4. ``cut: N`` - Remove lines after ``N``'th
5. ``python: script`` - Execute arbitrary Python code on passed content. See :ref:`Python support` for details.
6. ``bash: script`` - Execute arbitrary bash code on passed content. Call ``grep``, ``awk`` or ``sed``, for example.

JSON (for APIs)
---------------

1. ``json`` - Pretty print JSON content.
2. ``jq`` - Apply jq JSON transformation (`jq`_ must be installed).

Examples
--------

Here is a sequence of transformations, that will

1. Crop HTML page to CSS selector ``#plugin-description > div > p > a``
2. Transform it's contents to text<commit_msg>Removed cut and sort transforms from doc<commit_after>.. _transforms:

Transforms
==========

Each Kibitzr transform modifies content and passes it forward.
Transforms can be divided into following groups: HTML, plain text, JSON.

HTML
----

1. ``tag: tagname`` - crop HTML to contents of the first matching HTML tag.
2. ``css: selector`` - crop HTML to the first encountered outer HTML matching passed `CSS selector`_.
3. ``css-all: selector`` - crop HTML to the concatenated list of all matching elements.
4. ``xpath: path`` - crop HTML to contents of the passed `XPath`_.
5. ``text`` - strip all HTML tags and return only text.

Plain text
----------

1. ``changes`` - Compare to the previous version of the content and return difference report
2. ``changes: verbose`` - Same as ``changes``, but in human-friendly format
5. ``python: script`` - Execute arbitrary Python code on passed content. See :ref:`Python support` for details.
6. ``bash: script`` - Execute arbitrary bash code on passed content. Call ``grep``, ``awk`` or ``sed``, for example.

JSON (for APIs)
---------------

1. ``json`` - Pretty print JSON content.
2. ``jq`` - Apply jq JSON transformation (`jq`_ must be installed).

Examples
--------

Here is a sequence of transformations, that will

1. Crop HTML page to CSS selector ``#plugin-description > div > p > a``
2. Transform it's contents to text
{'fraction_non_alphanumeric': 0.141687433250267, 'fraction_numerical': 0.009967960128159488, 'mean_word_length': 3.524959742351047, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '467210', 'n_tokens_mistral': 800, 'n_tokens_neox': 770, 'n_words': 397}