<commit_before>require 'polyglot'
require 'treetop'

Treetop.load "laydown"

parser = LayDownParser.new

text = <<'END'
this text is in a beautiful paragraph and there's nothing you can do about it.

And this is another `inline, inlined code` paragraph.

```CODE
BUT THIS IS A CODE BLOCK
```

INTERVENING PARAGRAPH

    And here is another
	and it might have a couple
	different ways of handling identation
    which vim makes kind of confusing

you all can suck a dick.

END

p text
line = 0
text.split("\n").each { |l|
	print line += 1
	print ' '
	puts l
}

result = parser.parse(text)<commit_msg>Indenting<commit_after>require 'polyglot'
require 'treetop'

Treetop.load "laydown"

parser = LayDownParser.new

text = <<'END'
this text is in a beautiful paragraph and there's nothing you can do about it.

And this is another `inline, inlined code` paragraph.

```CODE
BUT THIS IS A CODE BLOCK
```

INTERVENING PARAGRAPH

		And here is another
	and it might have a couple
	different ways of handling identation
		which vim makes kind of confusing

you all can suck a dick.

END

p text
line = 0
text.split("\n").each { |l|
	print line += 1
	print ' '
	puts l
}

result = parser.parse(text)
{'fraction_non_alphanumeric': 0.08403361344537816, 'fraction_numerical': 0.0033613445378151263, 'mean_word_length': 3.783132530120482, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3452737', 'n_tokens_mistral': 419, 'n_tokens_neox': 390, 'n_words': 167}