<jupyter_start><jupyter_text>Introduction to Regular ExpressionsWhen you are parsing, what you are ultimately doing is splitting the text into various parts as well as cleaning or extracting different pieces. To do any of this, you need to be able to tell your programme where to apply the scissors. Below, we will learn how "regular expressions operations" can be very useful in this endeavour.Regular Expressions operations ("RegEx") are a very flexible version of the text search function that you find in most text processing software. In those regular search functions, you press CTRL+F (or COMMAND+F) and type in the search phrase you are looking for e.g. "Congress". If your software finds an exact match for your search phrase ("Congress"), it jumps to its position in the text and you can take it from there.Thinking a bit more abstract about this, "Congress" is nothing else than a very specific search. In it, we ask the search function to report the position where it finds a capital "C" followed seven lower case letters ("o", "n", "g", "r", "e","s","s"), all in a specific order. Depending on your text, it may have been sufficient to let your search function look for all words starting with the captial letter "C", or for those words starting with "C" and ending with "ess". This kind of flexibility is exactly what RegEx provides.RegEx is more flexible than the customary search function as it does not restrict you to spell out the literal word, number or phrase you are looking for. Rather, in RegEx you can describe the necessary characteristics for a match. You can enter these characteristics based on rules and special characters that make RegEx what it is. Literal charactersAs described above, RegEx is more flexible than the customary search function. Still, the customary search function is a special case of RegEx and we can use RegEx just like it.RegEx is just like the customary search function when we supply it exclusively with literal characters i.e. spell out the exact phrase we want to find. Let's search for "Congress" in an example text and see what RegEx comes back with.<jupyter_code>import re

example="The GPO published the transcripts of more than 20000 hearings held in the US Congress since 1995. Hearing transcripts are complete for the Congressional Sessions 105 to 114."

print("Our example text: '",example,"'", sep="")

print("Our search result: '",re.search("Congress", example),"'", sep="")<jupyter_output>Our example text: 'The GPO published the transcripts of more than 20000 hearings held in the US Congress since 1995. Hearing transcripts are complete for the Congressional Sessions 105 to 114.'
Our search result: '<_sre.SRE_Match object; span=(77, 85), match='Congress'>'<jupyter_text>What we receive is an object that contains two pieces of information: the location of the match and the matched phrase. The location provided are the character numbers of the first and the last character of our match. In the above text, the word "Congress" starts at the 78th character and ends on the 86th (recall that Python starts counting at 0). The matched phrase we received is "Congress". Unsuprising, since this is precisely what we asked RegEx to search for with our specific query. Python RegEx methods and their differencesBefore we go on, we first need to learn a few of the methods Python has to work with a regular expression. Above, we used the ``.search()`` method. Let's contrast it to a few others before we go on:- ``.match()``: Checks whether the pattern occurs at the start of your text. If so, it returns the position and the phrase.- ``.search()``: Checks whether your pattern appears somewhere inside your text (including the start). If so, it returns the position and the phrase of the first match.- ``.findall()``: Checks whether your pattern appears somewhere inside your text (including the start). If so, it returns all phrases that matched your pattern, but not their position.- ``.finditer()``: Checks whether your pattern appears somewhere inside your text (including the start). If so, it returns all phrases as an iterator object. An iterator object includes a stream of elements, in our case all information about each matching phrase (position and the phrase itself).- ``.sub()``: Substitute all occurances of your pattern.- ``.split()``: Split your string wherever your pattern appears. In the split, your pattern will be deleted.Let's contrast the first three. First, the match:<jupyter_code>re.match("Congress", example)<jupyter_output><empty_output><jupyter_text>The method ``.match()`` does not return anything. Why? Because it is only interested in the beginning of our string. Our example text begins with "The GPO published ..." and thus nothing that matches our pattern.Let's check ``.findall()`` next:<jupyter_code>re.findall("Congress", example)<jupyter_output><empty_output><jupyter_text>Now this time, our search found two matches. Just like with a customary search function, a RegEx query for "Congress" also returns parts of our example text where "Congress" does not stand alone e.g. in "Congressional". This is not entirely surprising since to the computer the entire string is just one sequence of characters. No matter whether it includes letters, digits, punctuation or whitespace. To receive only the phrase "Congress" by itself, we have to be more precise in the eyes of the computer. In our case, adding whitespace at the end would do the trick.<jupyter_code>re.findall("Congress ", example)<jupyter_output><empty_output><jupyter_text>But let's return to our initial result that returned two matches. Using ``.findall()`` only gave us the phrases we matched, but not their position. To get their positions, we have to use the ``.finditer()`` method. ``.finditer()`` creates an iteration object that includes all matches. For each element in the object, we can retrieve the attributes matched phrase (by using ``.group()``) as well as their positions (by using ``.span()`` or ``.start()``/``.end()``) . Applying it here we can print all the matched phrases and their positions as follows:<jupyter_code>for result in re.finditer("Congress", example):
    print("'",result.group(),"' ", result.span(), sep="")<jupyter_output>'Congress' (77, 85)
'Congress' (139, 147)<jupyter_text>MetacharactersTo go beyond the limits of literal character queries, RegEx provides a set of so-called metacharacters. Let's understand them in turn. The dotThe dot is the most greedy of them all. Adding a dot into a RegEx query is equivalent to searching for "any character". This includes letters, digits, whitespace etc. Having such a flexible metacharacter is often useful, but has to be employed with care. If one uses the dot without being very precise about the start and end point of the desired string, one can quickly end up with large portions of the document being extracted.Still, to see the dot at aork, in our example query, we can replace any literal character with a dot and still get the same result:<jupyter_code>re.findall("C.ngress", example)<jupyter_output><empty_output><jupyter_text>Or one could replace two characters:<jupyter_code>re.findall("C.ngr.ss", example)<jupyter_output><empty_output><jupyter_text>Note that in both examples, the matched phrase is "Congress" without the metacharacter we used here.  Flexible pattern length with the quantifiers *, +, ? and {}Now we wrote earlier, one could search for "Congress" by asking RegEx to return any word that starts with a "C" and ends with "ess". If we just had the dot, then our query would have to be "``C....ess``" which is not so different from the literal search we started with. RegEx provides three metacharacters that allow us to search for flexible pattern length. The most greedy of them is the asterisk (``*``). Adding an asterisk to your query demands that the preceding character has to appear zero or more times, a feature which is useful to navigate typos and the like. A close relative to the asterisk is the plus sign (``+``) which demands that the preceding character appears at least once.We can already use the asterisk to implement our "C" to "ess" query. All we ask is that there be "zero or more characters of any type" between "C" and "ess":<jupyter_code>re.findall("C.*ess", example)<jupyter_output><empty_output><jupyter_text>... or so we thought. The greediness of the dot got in the way of a useful result. To avoid extracting large chunks of our text unwantigly, we can employ the last quantifier available in RegEx: the curly brackets ``{}``.Both the asterisk and the plus sign are greedy in the sense that they will capture as many characters as fulfill the stated RegEx query. One way to mitigate their appetite is to add the question mark (``?``). The question mark has two roles in RegEx queries. If it is put behind a quantifier, it makes the quantifier lazy. A lazy quantifier will no longer grab as many characters as fulfill the RegEx query. Rather it will grab as few as are necessary to create a match.The question mark has another meaning when it is put behind a character. A question marks makes the preceding character optional i.e. the RegEx query "colou?r" would match both "colour" and "color". Making our asterisk lazy in the above example yields:<jupyter_code>re.findall("C.*?ess", example)<jupyter_output><empty_output><jupyter_text>*Hurray.*There is a second way to restrict the number of characters a RegEx query may return. The curly brackets allow us to specify the minimum and maximum number of repetitions for the preceding character. For example, we could search for the number of times the letter "s" appears alone or in a pair using the RegEx query "``s{1,2}``". The first argument inside the curly bracket specifies the minimum, the second the maximum. We can also use curly brackets to search for minimal ("``{1,}``"), maximal ("``{,2}``") or exact repetitions ("``{2}``").Now in our toy example, we can mitigate the greediness of the dot by supplying a maximal number of repetitions "any character" can occur.<jupyter_code>re.findall("C.{,4}ess", example)<jupyter_output><empty_output><jupyter_text>Tangent: Finding a metacharacter inside the textBut what do I do when I want to find the punction period that separates the two sentences? For any of the metacharacters, if you want to find it inside the string, add a backward slash in front of it.<jupyter_code>print(example)
re.search("\.", example)<jupyter_output><empty_output><jupyter_text>Finding digits, alphanumeric characters and whitespace (or not)As less greedy versions of the dot, RegEx provides metacharacters to search for all digits ("``\d``"), alphanumeric characters ("``\w``") and/or whitespace ("``\s``"). RegEx also supports the negation of those metacharacters i.e. "any character that is not a digit" can be found by using a capital "D" instead i.e. "``\D``".Here are all the numbers used in the example text:<jupyter_code>print(example)
re.findall("\d+", example)<jupyter_output><empty_output><jupyter_text>Now, "alphanumeric" means anything that is either a letter or a digit. Using this in the above query would return a list of all words and digits in the example text (as long as one assumes that a whitespace separates two words and two numbers).Returning to our "C..gress" example, we can find various lengths of the word using:<jupyter_code>re.findall("C\w+ess", example)<jupyter_output><empty_output><jupyter_text>or, if one wanted to extract the complete word starting with "Congress":<jupyter_code>re.findall("C\w+ess\w*", example)<jupyter_output><empty_output><jupyter_text>Playing with a negation ("``\S``") for once:<jupyter_code>re.findall("C\S+ess\w*", example)<jupyter_output><empty_output><jupyter_text>Note that the metacharacter "\s" includes more than just the result of your spacebar. It further captures the start of a new line ("``\n``"), the horizontal tab ("``\t``"), the vertical tab ("``\v``") and the carriage return ("``\r``", &crarr;). All of these are their own metacharacters in RegEx as well.  Character setsThe above example is not exactly accurate. It is true that "C\w+ess" would return "Congress" from our example text. But "\w" means alphanumeric and it would thus also return typos like "C56gress". This may be desirable in some applications, of course.Still to be more precise, one can use so-called character sets. RegEx allows its users to specify a set of characters that could appear in our desired result. We can supply such a set in square brackets ("[]"). RegEx will still produce a match as long as *at least one of the characters inside the brackets* is found. For example, using the RegEx command "[Congress]" without a quantifier would return every appearance of the individual letters "C", "o", "n", "g" etc.While one can write any combination of characters into a set, RegEx provides two shortcuts for digits and letters. To receive any digit, use the character set "[0-9]". To receive any character, use "[a-z]" for small type or "[A-Z]" for capital letters. We can also combine the letter shortcuts to receive every word in our example.<jupyter_code>print(example)
re.findall("[A-Za-z]+", example)<jupyter_output><empty_output><jupyter_text>Negating a character set is also possible. This is done by adding a hat ("^") at the start of your bracket. To extract anything that has at least two characters and is not a letter, type:<jupyter_code>re.findall("[^A-Za-z]{2,}", example)<jupyter_output><empty_output><jupyter_text>Exploiting locationIn one of the rare confusions inside RegEx, the metacharacter "^" appears in two separate functions. As just described above, the hat can be used to negate a character set. At the same time, the hat can also be used to describe the location of a sought pattern.There are two ways to describe the location of a sought pattern in RegEx: Looking for a pattern at the start of a string is done using the hat again e.g. "^(Congress)". The end is designated with a dollar sign i.e. "(Congress)$". Note that these commands will look at our example text as one string and thus not return a result.<jupyter_code>re.findall("^(Congress)", example)<jupyter_output><empty_output><jupyter_text>... but it would work in a different setting:<jupyter_code>re.findall("^(Congress)", "Congress is in session this week.")<jupyter_output><empty_output><jupyter_text>The other way to use locations in RegEx is to extract the characters before or after a specified pattern. The lookahead query is provided by "``(?=pattern)``". Looking behind can be done by "``(?<=pattern)``".Both can also be used in a negation i.e  "``(?!pattern)``" or  "``(?<!pattern)``". This is how to extract the alphanumeric characters before the word "Congress" in our example:<jupyter_code>re.findall("\w+(?= Congress)", example)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08134585500979663, 'fraction_numerical': 0.004053780149989866, 'mean_word_length': 5.1444582814445825, 'pattern_counts': {'":': 2, '<': 82, '<?xml version=': 0, '>': 80, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '534736', 'license': 'Apache-2.0', 'n_tokens_mistral': 4006, 'n_tokens_neox': 3778, 'n_words': 2148, 'path': 'parsing/2 - Introduction to Regular Expressions (RegEx).ipynb', 'repo_name': 'Goodly/LTD', 'size': '14801'}