<jupyter_start><jupyter_text>Text Processing Primer<jupyter_code>text1 = "Julia is a high-level high-performance dynamic programming language for numerical computing and Julia is used ..."

len(text1)
text2 = text1.split(' ')
text2
len(text2)
[w for w in text2 if len(w) > 3] # Words that are greater than 3 letters long in text2
[w for w in text2 if w.istitle()] # Capitalized words in text2
[w for w in text2 if w.endswith('l')] # Words in text2 that end in 'l'<jupyter_output><empty_output><jupyter_text>We can find unique words using `set()`.<jupyter_code>len(set(text2))
set(text2)
set([w.lower() for w in text2])<jupyter_output><empty_output><jupyter_text>Processing free-text<jupyter_code>text3 = 'Demystifying Dynamic Programming @freecamp @bostongroup @ NY #algorithms'
text4 = text3.split(' ')

text4<jupyter_output><empty_output><jupyter_text>Finding hastags:<jupyter_code>[w for w in text4 if w.startswith('#')]<jupyter_output><empty_output><jupyter_text>Finding callouts:<jupyter_code>[w for w in text4 if w.startswith('@')]<jupyter_output><empty_output><jupyter_text>**Problem**Finds also single '@'.**Solution**Regular expressions for more complex parsing. For example `'@[A-Za-z0-9_]+'` will return all words that: * start with `'@'` and are followed by at least one: * capital letter (`'A-Z'`)* lowercase letter (`'a-z'`) * number (`'0-9'`)* or underscore (`'_'`)<jupyter_code>import re # for regular expressions

[w for w in text4 if re.search('@[A-Za-z0-9_]+', w)]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14474553866490417, 'fraction_numerical': 0.01982815598149372, 'mean_word_length': 6.685279187817259, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '279092', 'license': 'MIT', 'n_tokens_mistral': 574, 'n_tokens_neox': 548, 'n_words': 151, 'path': 'nbs/6/1-1-Text-Processing-Primer.ipynb', 'repo_name': 'juliaeis/informatica-pfr-2017', 'size': '1513'}