<commit_before># Watney

[![Circle CI](https://circleci.com/gh/philippkueng/watney.svg?style=svg)](https://circleci.com/gh/philippkueng/watney)
[![Join the chat at https://gitter.im/philippkueng/watney](https://badges.gitter.im/philippkueng/watney.svg)](https://gitter.im/philippkueng/watney?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

HTML to Markdown converter

## Tests

```bash
lein midje
```

Or let the tests run automatically on file changes during development.

```bash
lein repl
user=> (use 'midje.repl)
user=> (autotest)
```
<commit_msg>declaring the current limitations of converting HTML to Markdown<commit_after># Watney

[![Circle CI](https://circleci.com/gh/philippkueng/watney.svg?style=svg)](https://circleci.com/gh/philippkueng/watney)
[![Join the chat at https://gitter.im/philippkueng/watney](https://badges.gitter.im/philippkueng/watney.svg)](https://gitter.im/philippkueng/watney?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

HTML to Markdown converter

## Tests

```bash
lein midje
```

Or let the tests run automatically on file changes during development.

```bash
lein repl
user=> (use 'midje.repl)
user=> (autotest)
```

## Limitations

Since both HTML and Markdown comes in a variety of ways, there are some constraints that were introduced to make the generated Markdown file as correct as possible.

* Unordered Lists are always converted to `* ` with a single space.
* List indentations are always 2 spaces.
* Ordered Lists are always converted to `1. ` with a single space.
* Between every change of HTML element there will be a newline. `<h1>H1</h1><h3>H3</h3> => # H1\n##H3` 
{'fraction_non_alphanumeric': 0.13285457809694792, 'fraction_numerical': 0.005984440454817474, 'mean_word_length': 5.53125, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 12, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1904514', 'n_tokens_mistral': 622, 'n_tokens_neox': 572, 'n_words': 149}