<jupyter_start><jupyter_text>Text ProcessingThere are a number of generalized steps in processing various text values. Unicode NormalizationUnicode encoding supports a number of "normal forms" and the concept of ["Unicode equivalence"](https://en.wikipedia.org/wiki/Unicode_equivalence). In order to make simple comparisons between strings possible, and efficient, all strings involved should be [normalized](https://en.wikipedia.org/wiki/Text_normalization) to a known standard, either combined (utilizing characters / code points that combine accents with the letter they modify) or utilizing [combining characters](https://en.wikipedia.org/wiki/Combining_character).Additionally, there is an excellent library called [FTFY](https://ftfy.readthedocs.org/) (it _Fixes Text For You_) that can resolve incorrect decoding referred to as [mojibake](https://en.wikipedia.org/wiki/Mojibake), and other issues.<jupyter_code>from typing import List, Union
from unicodedata import combining, normalize
def uninorm(value:Union[bytes,str], encoding:str='utf8', fallback:str='Windows-1252') -> str:
	if isinstance(value, bytes):
		try:
			value = value.decode(encoding)
		except UnicodeDecodeError:
			value = value.decode(fallback)
	
    # value = fix_text(value)  # If utilizing the excellent FTFY library.
	value = normalize('NFC', value)  # Combined form.
	value = value.replace('\r\n', '\n')  # Also eliminate extraneous Windows end-of-line markers.
	
	return value<jupyter_output><empty_output><jupyter_text>Eliminating Combining CharactersSometimes you just want ASCII out the other side. However, most "accented" characters can be represented without their accent without too much loss of fidelity; wetware is quite good at correcting minor deficciencies like that. Luckily, by decomposing into a combining character form we can very easily run through the resulting characters and throw away those identified as combining.This has the added benefit of eliminating most "[Zalgo text](https://en.wikipedia.org/wiki/Combining_characterZalgo_text)", a disruptive abuse of stackable combining characters to force rendering to escape a container and obscure surrounding text.<jupyter_code>def noncombining(value:str) -> str:
	return "".join(c for c in normalize('NFKD', value) if not combining(c))
noncombining("<NAME>-McGregor")<jupyter_output><empty_output><jupyter_text>URL-Safe "Slugification"An extremely common pattern is that of "slugification", that is, transforming a textual label into a value suitable for use within a URI, e.g. as a path element. Typically used to support "SEO-friendly URLs" where the address can be memorable and not just an abstract identifier.<jupyter_code>from re import compile as re, Pattern
def urlsafe(name:str, collection:List[str]=[], replacement:str='-', unsafe:Pattern=re(r'\W+')) -> str:
    base = unsafe.sub(replacement, uninorm(name)).strip(replacement)
    suffix = 0
    
    while True:
        if (result := f"{base}{replacement if suffix else ''}{suffix if suffix else ''}") not in collection:
            break
        suffix += 1
    
    return result
urlsafe("<NAME>")
# If you have a collection of values, to ensure collisions are avoided, pass it in.
urlsafe("Words", ["Words", "Words-1"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.094859241126071, 'fraction_numerical': 0.0024479804161566705, 'mean_word_length': 5.372319688109162, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 18, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '536303', 'license': 'MIT', 'n_tokens_mistral': 913, 'n_tokens_neox': 869, 'n_words': 341, 'path': 'Input Processing/Text Processing.ipynb', 'repo_name': 'amcgregor/python-samples', 'size': '3295'}