<commit_before>Crawler-Commons Change Log

Current Development 1.3-SNAPSHOT (yyyy-mm-dd)
- Migrate Continuous Integration from Travis to GitHub Actions (Valery Yatsynovich) #333
- Upgrade dependencies (dependabot) #334, #339, #345, #346, #347
- Upgrade Maven plugins (dependabot, <NAME>, sebastian-nagel) #328, #329, #330, #331, #335, #336, #337, #338, #340, #341, #343
- Enable Dependabot (<NAME>) #327

Release 1.2 (2021-10-06)
- [Sitemaps] Avoid calling java.net.URL::equals in equals method of sitemaps and extensions (sebastian-nagel) #322
- [URLs] Provide a builder class to configure the URL normalizer (aecio) #321, #324
- [URLs] Make normalization of IDNs configurable (to ASCII or Unicode) via builder (aecio, sebastian-nagel) #324
- [Sitemaps] Fix XXE vulnerability in Sitemap parser (kovyrin) #323
- [URLs] Sorting the Query Parameters (aecio) #246, #309
- [URLs] Allows to (optionally) remove common irrelevant query parameters (aecio) #309
- [Sitemaps] Allow to normalize URLs in sitemaps (murderinc, sebastian-nagel) #305<commit_msg>Updates changelog with latest additions / changes<commit_after>Crawler-Commons Change Log

Current Development 1.3-SNAPSHOT (yyyy-mm-dd)
- Migrate Continuous Integration from Travis to GitHub Actions (<NAME>) #333
- Upgrade dependencies (dependabot) #334, #339, #345, #346, #347, #350
- Upgrade Maven plugins (dependabot, <NAME>, sebastian-nagel) #328, #329, #330, #331, #335, #336, #337, #338, #340, #341, #343, #356, #363. #364, #366
- Enable Dependabot (<NAME>) #327
- Removes test dependency towards mockito-core (<NAME>) #367

Release 1.2 (2021-10-06)
- [Sitemaps] Avoid calling java.net.URL::equals in equals method of sitemaps and extensions (sebastian-nagel) #322
- [URLs] Provide a builder class to configure the URL normalizer (aecio) #321, #324
- [URLs] Make normalization of IDNs configurable (to ASCII or Unicode) via builder (aecio, sebastian-nagel) #324
- [Sitemaps] Fix XXE vulnerability in Sitemap parser (kovyrin) #323
- [URLs] Sorting the Query Parameters (aecio) #246, #309
- [URLs] Allows to (optionally) remove common irrelevant query parameters (aecio) #309
- [Sitemaps] Allow to normalize URLs in sitemaps (murderinc, sebastian-nagel) #305
{'fraction_non_alphanumeric': 0.12211668928086838, 'fraction_numerical': 0.09226594301221167, 'mean_word_length': 5.230985915492957, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1938873', 'n_tokens_mistral': 922, 'n_tokens_neox': 753, 'n_words': 273}