<commit_before>overridden declarations

(foo %bar) where bar is empty

case where entity contains just "|" or "," (maybe don't unexpand these)

undefined elements<commit_msg>Many changes.<commit_after>Encoding detection

expand general entity references for default values

attval normalization of default values

general entities

notations

Error reporting.

Don't unexpand entities that are not properly nested.

Don't expand empty entities that might be pasted.

overridden declarations

(foo %bar) where bar is empty

case where entity contains just "|" or ","

undefined elements


empty param entities at the group level
ignored sections
included sections
external general entities


something for case where param entity was not preserved

<!ENTITY % g "(x|y)">
<!ATTLIST foo bar NOTATION %g; #IMPLIED>
{'fraction_non_alphanumeric': 0.05925925925925926, 'fraction_numerical': 0.0, 'mean_word_length': 5.190839694656488, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4393808', 'n_tokens_mistral': 219, 'n_tokens_neox': 205, 'n_words': 103}