<jupyter_start><jupyter_text>LECTURE OVERVIEW --- By the end of the lecture, you'll be able to:- pattern match with string methods- pattern match regular expressions using the `re` module- pattern match filenames using the `glob` and `pathlib` modules- handle the complexities of dates and times using the `datetime` and `dateutil` modules- archive and unarchive files using the `zipfile` and `shutil` modules- serialize and deserialize Python objects using the `pickle` module PATTERN MATCHING---How often do you Google for something you are looking for in the web of the internet? How often do you search for text in a document or webpage by pressing `Ctlr-F` and typing in the words you're looking for? These technologies all use some sort of pattern matching to solve a problem and there are many ways to go about pattern matching in Python.For instance, you've learned to use the `in` operator to see if a given string is a substring of another string:<jupyter_code>'apples' in "I love apples!"<jupyter_output><empty_output><jupyter_text>You have also learned how to index and split strings that can used with comparison operators:<jupyter_code>"apples"[0] == 'z'
"apples"[-2] != 's'<jupyter_output><empty_output><jupyter_text>String Methods for Pattern Matching By the end of the lecture, you'll be able to:- **pattern match with string methods**- pattern match regular expressions using the re module- pattern match filenames using the glob and pathlib modules- handle the complexities of dates and times using the datetime and dateutil modules- archive and unarchive files using the zipfile and shutil modules- serialize and deserialize Python objects using the pickle modulePython has several built-in functions associated with the string datatype. We'll go over several different functions that we can use for pattern matching.Specifically, the string methods that will evaluate to a `Boolean` value. These methods are useful when, for example, we are creating forms for users to fill in. There are a number of string methods that will return `Boolean` values:- `str.isalnum()`: String consists of only alphanumeric characters (no symbols)- `str.isalpha()`: String consists of only alphabetic characters (no symbols)- `str.islower()`: String’s alphabetic characters are all lower case- `str.isnumeric()`: String consists of only numeric characters- `str.isspace()`: String consists of only whitespace characters- `str.istitle()`: String is in title case- `str.isupper()`: String’s alphabetic characters are all upper case- `str.startswith(val)`: String starts with the specified value- `str.endswith(val)`: String ends with the specified value- `str.find(substring, start=None, end=None)`: Searches the target string for a given substringLet's see a few of these in action:<jupyter_code>"5".isnumeric()
"abcdef".isnumeric()<jupyter_output><empty_output><jupyter_text>Checking whether characters are lower case, upper case, or title case, can help us to sort our data appropriately. It can also provide us with the opportunity to standardize data we collect by checking and then modifying strings as needed:<jupyter_code>movie = "2012: THE KINGS OF BBQ BARBECUE KUWAIT"
book = "The Barbecue! Bible"
poem = "fred could bbq before he could read"

print(f"movie.islower(): {movie.islower()}")
print(f"movie.isupper(): {movie.isupper()}")
 
print(f"book.istitle(): {book.istitle()}")
print(f"book.isupper(): {book.isupper()}")
 
print(f"poem.istitle(): {poem.istitle()}")
print(f"poem.islower(): {poem.islower()}")<jupyter_output><empty_output><jupyter_text>We can also chain these with logical operators:<jupyter_code>print(
    f"Does the movie title start with '2012' and is upper case?: {movie.startswith('2012') & movie.isupper()}"
)

print(
    f"Is 'fred' in the poem and ends with 'city'?: {('fred' in poem) & (poem.endswith('city'))}"
)<jupyter_output><empty_output><jupyter_text>To find if a string contains a particular substring you can use the `str.find()` method that will return the lowest index in the string that the substring is found:<jupyter_code># show readable indexes
for i in range(len(book)):
    print(f'{i:2d} {book[i]}')

print(f"book.find('Bible'): {book.find('Bible')}")<jupyter_output><empty_output><jupyter_text>To learn more about string methods, check out the [documentation](https://docs.python.org/3.8/library/stdtypes.htmlstring-methods).  Regular Expressions**Regular expressions**, also known as **regexes** (pronounced reg-EX-is), is a sequence of characters that can be used to define a search pattern for finding text. To use regex, you sepcify the rules for the set of possible strings that you want to match and then ask youself:*"Does this string match the pattern?"* *"Is there a match for the pattern anywhere in this string?"* SyntaxSince a regex is a pattern that aims to match an input string, here are some special characters and patterns you can use to match strings:| Metacharacter | Description ||:---------------:|-------------|| `[] ` | Specifies a character class. For example, `[abc]` matches "a", "b", or "c". || `()` | Creates a group. For example, `(ab)*` matches "", "ab", "abab", "ababab", and so on. || `*` | Matches zero or more repetitions. For example, `ab*c` matches "ac", "abc", "abbbc", etc. || `+` | Matches one or more repetitions. For example, `ab+c` matches "abc", "abbc", "abbbc", and so on, but not "ac". || `?` | Matches the preceding element zero or one time. For example, `ab?c` matches only "ac" or "abc". Additionally, it matches the ending position of the string or the position just before a string-ending newline, like the `endsWith()` function. In line-based tools, it matches the ending position of any line.|| `\|` | Designates alternation. For example, `abc\|def` can match either "abc" or "def". || `.` | Matches any single character except newline. For example, `a.c` matches "abc", etc., but `[a.c]` matches only "a", ".", or "c". || `^` | Matches the starting position in the string, like the `startsWith()` function. In line-based tools, it matches the starting position of any line. || `{}` | Matches an explicity specified number of repetitions. || `` | Creates a named group. || `$` | Anchors a match at the end of a string. || `\`| Escapes a metacharacter of its special meaning, introduces a special character class, or introduces a grouping backreference. |If you want to use regex metacharacters in a Python string, you must insert a `r` before the string:```pythonregex_pattern = r'[abc]'```There are also a list of special sequences that consist of `\` and a character from the list below:| Metacharacter | Description ||:-------------:|-------------|| `\number` | Matches groups of the same `number` || `\A` | Matches only at the start of the string || `\b` | Matches a empty string, but only at the beginning or end of a word || `\B` | Matches the empty string, but only when it is not at the beginning or end of a word || `\d` | Matches digits || `\D` | Matches character that is not a digit || `\s` | Matches whitespace characters || `\S` | Matches non-whitespace characters || `\w` | Matches words || `\W` | Matches non-words || `\Z` | Matches only at the end of the string |You can read more about regex [here](https://en.wikipedia.org/wiki/Regular_expression). The `re` Module By the end of the lecture, you'll be able to:- pattern match with string methods- **pattern match regular expressions using the `re` module**- pattern match filenames using the glob and pathlib modules- handle the complexities of dates and times using the datetime and dateutil modules- archive and unarchive files using the zipfile and shutil modules- serialize and deserialize Python objects using the pickle moduleThere are several methods that are available with the built-in `re` module and we are going to discuss the most commonly used methods.<jupyter_code>import re<jupyter_output><empty_output><jupyter_text>Note, most of these methods return a [Match object](https://docs.python.org/3/library/re.htmlmatch-objects) which always has a boolean value of `True`. This makes it convenient to test if there is a match with a simple `if` statement.1. ```pythonre.match(pattern, string, ...)```: return `Match` object if `pattern` is a match at the *beginning* of a `string`<jupyter_code>string = 'I LOVE BBQ!'

result1 = re.match('BBQ', string)
print(result1)

result2 = re.match('I', string)
print(result2)<jupyter_output><empty_output><jupyter_text>You can pass in a start position and end position parameters to `re.match()` but you might as well use the `re.search()` method that allows you to locate a match string anywhere.To read more about `re.match()`, check out help output:<jupyter_code>help(re.match)<jupyter_output><empty_output><jupyter_text>2. ```pythonre.search(pattern, string, ...)```: returns `Match` object if `pattern` matches *anywhere* in a given `string`<jupyter_code>string = 'Mark is bringing his 3 kids to the BBQ.'

if re.search('kids', string):
    print('Kids at the BBQ!')
else:
    print('No kids at the BBQ.')<jupyter_output><empty_output><jupyter_text>To read more about `re.find()`, check out help output:<jupyter_code>help(re.search)<jupyter_output><empty_output><jupyter_text>3. ```pythonre.findall(pattern, string, ...)```: finds and retrieves a list of all occurances of `pattern` in a given `string`<jupyter_code>string = "Mark is bringing 3 kids, Joanne is bringing 0 kids, and Georgie is bringing 1 kid."

re.findall(r'(\d+)[^\d]+[kid]', string)<jupyter_output><empty_output><jupyter_text>To read more about `re.findall()`, check out help output:<jupyter_code>help(re.findall)<jupyter_output><empty_output><jupyter_text>4. ```pythonre.sub(pattern, replace, string, ...)```: search `string` for a `pattern` and substitute with a new string `replace` if `pattern` occurs<jupyter_code>string = 'I will have chicken, mashed potatoes, and a beer from the BBQ menu. Please and thank you!'
pattern = 'chicken'
repl = 'pulled pork'

re.sub(pattern, repl, string)<jupyter_output><empty_output><jupyter_text>To read more about `re.sub()`, check out help output:<jupyter_code>help(re.sub)<jupyter_output><empty_output><jupyter_text>5. ```pythonre.split(pattern, string, ...)```: identical to the `split()` function in Python, will split a `string` at the location in which a `pattern` occurs and return the text of all groups as list<jupyter_code>string = "Mark is bringing 3 kids, Joanne is bringing 0 kids, and Georgie is bringing 1 kid."

re.split(', ', string)<jupyter_output><empty_output><jupyter_text>To read more about `re.split()`, check out help output:<jupyter_code>help(re.split)<jupyter_output><empty_output><jupyter_text>6. ```pythonre.compile(pattern, ...)```: compiles a regex `pattern` into a regular expression objectWe can directly call the functions we have learned above on this compiled object that can save time time since parsing/handling regex strings can be computationally expensive.<jupyter_code>string = "Mark is bringing 3 kids, Joanne is bringing 0 kids, and Georgie is bringing 1 kid."
pattern = re.compile(r'(\d+)[^\d]+[kid]')
pattern.findall(string)<jupyter_output><empty_output><jupyter_text>To read more about `re.compile()`, check out help output:<jupyter_code>help(re.compile)<jupyter_output><empty_output><jupyter_text>As you've seen, regular expressions can be rather complex so use them sparingly. It can be tempting to write one massive, super regular expression, but it will most likely not be the best practice to do. So, as a rule of thumb, use the `in` operator or string methods.For a more in-depth tutorial on regular expressions, check [this](https://realpython.com/regex-python/) out.To learn more about the `re` module, see the [documentation](https://docs.python.org/3/library/re.htmlre.Pattern.match). ** Exercise**Write a program that finds all the sub-strings where there is a single character between `a` and `c`.For example:```pythonstring = "abcadsfearc"```Expected output:    `['abc', 'adc', 'arc']`<jupyter_code># TODO: insert solution here<jupyter_output><empty_output><jupyter_text>Filename Pattern MatchingWhen you are working within your filesystem in your OS, you'll often times want to quickly retrieve the filenames with the system. Additionally, you might only want a subset of the filenames based on a given pattern. To accomplish this, there are a few built-in Python modules to help out.  The `glob` module By the end of the lecture, you'll be able to:- pattern match with string methods- pattern match regular expressions using the re module- **pattern match filenames using the `glob` and `pathlib` modules**- handle the complexities of dates and times using the datetime and dateutil modules- archive and unarchive files using the zipfile and shutil modules- serialize and deserialize Python objects using the pickle moduleThe `glob` module (short fot global) is useful for finding all the pathnames matching a specific pattern.We can use `glob` to search for a specific file pattern, or even search for files where the filenames matches a certain pattern using wildcards such as:- `*`: matches zero or more characters- `?`: matches exactly one character<jupyter_code>import glob<jupyter_output><empty_output><jupyter_text>For exmple, we can use the```pythonglob.glob(pathname, recursive=False)```method to search for all the IPython Jupyter Notebook (`.ipynb` extension) source files in the current directory and returns a list of the filenames:<jupyter_code>glob.glob('*.ipynb')<jupyter_output><empty_output><jupyter_text>Here's another example where we find all the files with a `.ipynb` extension that contains `lecture` in the filename:<jupyter_code>for filename in glob.glob('*lecture*.ipynb'):
    print(filename)<jupyter_output><empty_output><jupyter_text>The `glob` module also makes it easy to search for files recursively in subdirectories too:<jupyter_code>for file in glob.glob('**/*.txt', recursive=True):
    print(file)<jupyter_output><empty_output><jupyter_text>To read more about `glob.glob()`, check out help output:<jupyter_code>help(glob.glob)<jupyter_output><empty_output><jupyter_text>A similar method,```pythonglob.iglob(pathname, recursive=False)```, returns an iterator with the same values as `glob.glob()`.Searching a large number of directories could take a long time and use a lot of memory. A solution to this is to use `glob.iglob()`.For example:<jupyter_code>glob.iglob('**/*.txt', recursive=True)
for file in glob.iglob('**/*.txt', recursive=True):
    print(file)<jupyter_output><empty_output><jupyter_text>To read more about `glob.iglob()`, check out help output:<jupyter_code>help(glob.iglob)<jupyter_output><empty_output><jupyter_text>To read more about the `glob` module, check out the [documentation](https://docs.python.org/3/library/glob.html).  The `pathlib` ModuleIn previous lectures, we've learned how to use the `pathlib` module to navigate around our filesystem. It can also be used for pattern matching of filenames. The module contains similar `glob` functionality for making flexible file listings.<jupyter_code>from pathlib import Path<jupyter_output><empty_output><jupyter_text>For example, when we import `Path` from `pathlib`, we can use the```pythonPath.glob(pattern)```or```pythonPath.rglob(pattern)```(recursive glob) function to list file types that contain the number `2` that returns a generator object that points to all the files in the current directory that fits our match:<jupyter_code>p = Path('.')
for name in p.glob('*2*'):
    print(name)<jupyter_output><empty_output><jupyter_text>To read more about `Path.glob()` and `Path.rglob()`, check out help output:<jupyter_code>help(Path.glob)
help(Path.rglob)<jupyter_output><empty_output><jupyter_text>** Exercise**Recall, in day 3 lab we did a scavenger hunt for coins scatter across directories starting at `day_3_assets/lab` that have the format `{coin type}_COIN.txt` for the filename.Find all the coins and print out the pathnames.<jupyter_code># TODO: insert solution here<jupyter_output><empty_output><jupyter_text>DATES AND TIMES---Working with dates and times can be quite a tricky challenge since you have to deal with time zones, daylight savings time, and various written date formats. Fortunately, Python has built-in libraries that can help manage all these complexities.  How Computers Count TimeAlmost all computers count time form an instant called the [Unix epoch](https://en.wikipedia.org/wiki/Unix_time), which occured on January 1, 1970, at 00:00:00 UTC (or [Coordinated Universal Time](https://en.wikipedia.org/wiki/Coordinated_Universal_Time)).By definition, Unix time elapses at the same rate as UTC (i.e., a one-second step in UTC is equivalent to a one-second step in Unix time). Nearly all programming languages, including Python, incorporate the concept of Unix time.For instancem we can use the `time` module to show the number of seconds since the Unix epoch (excluding leap seconds):<jupyter_code>import time
time.time()<jupyter_output><empty_output><jupyter_text>As you can see from the example above, Unix time is nearly impossible for humans to parse so instead, it is typically converted to UTC, which can then be converted into a local time zone. Date FormatsInstead of trying to parse Unix time, we can work in terms of years, months, days, and so forth. But even with these conventions, another layer of complexity stems from the fact that different languages and cultures have different ways of writing the date.For instance, in the United States, dates are usually written starting with the month, then day, then year. For example, January 31, 2020, is written as **01-31-2020**. However, most of Europe and many other areas write the date starting with the day, then the month, then the year. This means that January 31, 2020 is written as **31-01-2020**. These differences can cause all sorts of confusion when communicating across cultures. To help avoid communication mistakes, the International Organization for Standardization (ISO) developed a standard (called [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601)) that specifies that all dates should be written in order of most-to-least-significant data. For instance, the format is year (`YYYY`), month (`MM`), day (`DD`), hour (`hh`), minute (`mm`), and second (`ss`) is formatted as:```YYYY-MM-DD hh:mm:ss```This format has no ambiguity and you'll see how the ISO 8601 format is used in the `datetime` and `dateutil` modules. The `datetime` and `dateutil` Modules By the end of the lecture, you'll be able to:- pattern match with string methods- pattern match regular expressions using the re module- pattern match filenames using the glob and pathlib modules- **handle the complexities of dates and times using the `datetime` and `dateutil` modules**- archive and unarchive files using the zipfile and shutil modules- serialize and deserialize Python objects using the pickle moduleFortunately, you don't need to implement these complicated features from scratch. We will show you how to to navigate working with dates and times using the `datetime` and `dateutil` modules.<jupyter_code>import datetime<jupyter_output><empty_output><jupyter_text>Creating `datetime` InstancesThe `datetime` module provides three classes that make up the high-level interface that most people will use:1. `datetime.date`: idealized date that assumes the Gregorian calendar. This object stores the `year`, `month`, and `day` as attributes2. `datetime.time`: idealized time that assumes there are 86,400 seconds per day with no leap seconds. This object stores `hour`, `minute`, `second`, and `microsecond`.3. `datetime.datetime`: combination of `datetime.date` and `datetime.time` classes. It has all the attributes of both classes.<jupyter_code>datetime.date(year=2020, month=1, day=31)
datetime.time(hour=13, minute=14, second=31)
datetime.datetime(year=2020, month=1, day=31, hour=13, minute=14, second=31)<jupyter_output><empty_output><jupyter_text>To read more about the `datetime()` classes, check out help output:<jupyter_code>help(datetime.date)
help(datetime.time)
help(datetime.datetime)<jupyter_output><empty_output><jupyter_text>The `datetime` module also provides several other ways to create `datetime` instances that don't require you to specify each attribute:1. `.date.today()`: creates `datetime.time` instance with the current local date2. `.datetime.now()`: creates a `datetime.datetime` instance with the current local date and time3. `.datetime.combine()`: combines instances of `datetime.date` and `datetime.time` into a single `datetime.datetime` instanceFor example:<jupyter_code>today = datetime.date.today()
today
now_time = datetime.datetime.now()
now_time
now = datetime.datetime.now()
current_time = datetime.time(now_time.hour, now_time.minute, now_time.second)
datetime.datetime.combine(today, current_time)<jupyter_output><empty_output><jupyter_text>Using Strings to Create `datetime` instancesIf you have a `date_string` with the date in ISO 8601 format, you can create a `date` instance from the `date_string` by using the```python.fromisoformat(date_string)```function.For example:<jupyter_code>datetime.date.fromisoformat("2020-01-21")<jupyter_output><empty_output><jupyter_text>But what if you have a string that represents a date and time that isn't in the ISO 8601 format? Fortunately, `datetime` provides a method called```python.strptime(date_string, format_string)```to handle this.First, you need to tell Python what each part of the string represents using formatting codes:<jupyter_code>date_string = "01-31-2020 14:45:37"
format_string = "%m-%d-%Y %H:%M:%S"<jupyter_output><empty_output><jupyter_text>Now that `date_string` and `format_string` are defined, we can use them to create a `datetime` instance using `.strptime()`:<jupyter_code>datetime.datetime.strptime(date_string, format_string)<jupyter_output><empty_output><jupyter_text>To find a complete list of format codes, see the [documentation](https://docs.python.org/3/library/datetime.htmlstrftime-strptime-behavior). Working With Time ZonesTo ensure that your code is correct, it is important to know the time zone in which a date occurs. The `datetime` module does not provide a direct way to interact with the time zone database. Thus, the `datetime` module recommends using a third-party package called `dateutil` for this.<jupyter_code>import dateutil<jupyter_output><empty_output><jupyter_text>We can use the ```pythondateutil.tz.tzlocal()```method to get the local time zone.For example:<jupyter_code>now = datetime.datetime.now(tz=dateutil.tz.tzlocal())
now.tzname()<jupyter_output><empty_output><jupyter_text>You can also create time zones that are not the same as the time zone reported by your computer. To do this, we will use the```pythondateutil.tz.gettz(zone_string)```method where `zone_string` is an offical [IANA name](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for the time zone you're interested in.For example:<jupyter_code>london_tz = dateutil.tz.gettz("Europe/London")
now = datetime.datetime.now(tz=london_tz)
now<jupyter_output><empty_output><jupyter_text>We can also use the `.tzname()` method to print the name of the time zone, which is `'BST'` or British Standard Time.<jupyter_code>now.tzname()<jupyter_output><empty_output><jupyter_text>To read more about `dateutil.tz`, check out help output:<jupyter_code>help(dateutil.tz)<jupyter_output><empty_output><jupyter_text>Arithmetic With `datetime`Using the a ```pythondatetime.timedelta(...)```instance, we can do all sorts of arithmetic on our `datetime` instances.For example:<jupyter_code>now = datetime.datetime.now()
now
delta_pos_1_day = datetime.timedelta(days=+1)
tomorrow = now + delta_pos_1_day
tomorrow<jupyter_output><empty_output><jupyter_text>`datetime.timedelta` instances also support negative values.For example:<jupyter_code>now = datetime.datetime.now()
now
delta_neg_1_day = datetime.timedelta(days=-1)
yesterday = now + delta_neg_1_day
yesterday<jupyter_output><empty_output><jupyter_text>You can also provide a mix of positive and negative arguments.For example:<jupyter_code>now = datetime.datetime.now()
now
delta = datetime.timedelta(days=+3, hours=-4)
now + delta<jupyter_output><empty_output><jupyter_text>To read more about `datetime.timedelta()`, check out help output:<jupyter_code>help(datetime.timedelta)<jupyter_output><empty_output><jupyter_text>`datetime.timedelta` is very useful, but is limited because it cannot add or subtract intervals larger than a day. Fortunately, `dateutil` provides a more powerfull replacement called```pythondateutil.relativedelta(...)```that is very similar to `datetime.timedelta`.For example:<jupyter_code>now = datetime.datetime.now()
now
from dateutil.relativedelta import relativedelta

delta = relativedelta(years=+5, months=+1, days=+3, hours=-4, minutes=-30)
now + delta<jupyter_output><empty_output><jupyter_text>You can also use `.relativedelta()` to calculate the difference between two `datetime` instances.For example:<jupyter_code>now = datetime.datetime.now()
now
tomorrow = now + datetime.timedelta(days=+1)
relativedelta(now, tomorrow)<jupyter_output><empty_output><jupyter_text>To read more about `.relativedelta()`, check out help output:<jupyter_code>help(relativedelta)<jupyter_output><empty_output><jupyter_text>To learn more about the `datetime` module, see the [documentation](https://docs.python.org/3/library/datetime.htmlmodule-datetime).To learn more about the `dateutil` module, see the [documentation](https://dateutil.readthedocs.io/en/stable/). ** Exercise**This years Community BBQ will start on July 17, 2021 in Providence, RI at 10am. Next years Community BBQ will start on August 16, 2021 in Providence, RI at 10:30am.Do and answer the following:- create a `datetime` instance of the 2021 Community BBQ named `bbq_date_2021`- create a `datetime` instance of the 2022 Community BBQ named `bbq_date_2022`- what is date and time difference between `bbq_date_2021` and `bbq_date_2022`?<jupyter_code># TODO: insert solution here<jupyter_output><empty_output><jupyter_text>ARCHIVING---Archiving is a convenient way to package several files into one. The two most common archive types are ZIP and TAR but we will mostly cover working with ZIP archive types. By the end of the lecture, you'll be able to:- pattern match with string methods- pattern match regular expressions using the re module- pattern match filenames using the glob and pathlib modules- handle the complexities of dates and times using the datetime and dateutil modules- **archive and unarchive files using the `zipfile` and `shutil` modules**- serialize and deserialize Python objects using the pickle module Reading ZIP FilesThe `zipfile` module makes it easy to open and extract ZIP files. Similar to using `open()`, you can create a `ZipFile` object with the support of a `with` statement.<jupyter_code>from zipfile import ZipFile<jupyter_output><empty_output><jupyter_text>For example, using the ```pythonZipFile.namelist()```method we can retrieve a list of files in the archive:<jupyter_code>from pprint import pprint
from pathlib import Path

with ZipFile(Path.cwd() / 'day_4_assets' / 'event_tables.zip', 'r') as zip_obj:
    pprint(zip_obj.namelist())<jupyter_output><empty_output><jupyter_text>You can also retrieve information about the files in the archive using the```pythonZipFile.getinfo(filename)```method that returns a `ZipInfo` object.For example:<jupyter_code>with ZipFile(Path.cwd() / 'day_4_assets' / 'event_tables.zip', 'r') as zip_obj:
    bbq_event_2_csv = [name for name in zip_obj.namelist() if 'bbq_event_2.csv' in name][0]
    print(zip_obj.getinfo(bbq_event_2_csv))<jupyter_output><empty_output><jupyter_text>Extracting ZIP FilesWe can extract one or more files from ZIP archives through the```pythonZipFile.extract(member, path=None, pwd=None)```and```pythonZipFile.extractall(path=None, members=None, pwd=None)```methods. They both extract files to the current directory by default but you can pass in a `path` that allows you to specify a different directory to extract files to. If the directory does not exist, it is automatically created.For example:<jupyter_code>with ZipFile(Path.cwd() / 'day_4_assets' / 'event_tables.zip', 'r') as zip_obj:
    extract_path = [name for name in zip_obj.namelist() if 'bbq_event_2.csv' in name][0]
    unzip_path = Path.cwd() / 'day_4_assets' / 'unzipped_with_extract'
    zip_obj.extract(extract_path, path=unzip_path)<jupyter_output><empty_output><jupyter_text>We will take our `tree()` function from when a previous lecture using the `pathlib` module to view the directory tree of our archive results:<jupyter_code>def tree(directory):
    print(f'+ {directory}')
    for path in sorted(directory.rglob('*')):             # list subdirectories
        depth = len(path.relative_to(directory).parts)    # use .relative_to() to get how far we are from the root
        spacer = '    ' * depth
        print(f'{spacer}+ {path.name}')
tree(Path.cwd() / 'day_4_assets' / 'unzipped_with_extract')
with ZipFile(Path.cwd() / 'day_4_assets' / 'event_tables.zip', 'r') as zip_obj:
    unzip_path = Path.cwd() / 'day_4_assets' / 'unzipped_with_extractall'
    zip_obj.extractall(path=unzip_path)
tree(Path.cwd() / 'day_4_assets' / 'unzipped_with_extractall')<jupyter_output><empty_output><jupyter_text>Some misc. cleanup:<jupyter_code>import shutil
shutil.rmtree(Path.cwd() / 'day_4_assets' / 'unzipped_with_extract')
shutil.rmtree(Path.cwd() / 'day_4_assets' / 'unzipped_with_extractall')<jupyter_output><empty_output><jupyter_text>Creating New ZIP ArchivesTo create a new ZIP archive, you simply open a `ZipFile` object in write mode and add the files you want to archive using the ```pythonZipFile.write(filename, arcname=None, ...)```method:<jupyter_code>recipe_files = ['pulled_pork_recipe.txt', 'smoked_mac_and_cheese_recipe.txt']
with ZipFile(Path.cwd() / 'day_4_assets' / 'recipes.zip', 'w') as recipe_zip:
    for name in recipe_files:
        filename = Path.cwd() / 'day_4_assets' / 'recipes' / name
        recipe_zip.write(filename, arcname=name)<jupyter_output><empty_output><jupyter_text>Note, we must pass in the `arcname` otherwise, the file will be saved with the full path within our archive file.<jupyter_code>with ZipFile(Path.cwd() / 'day_4_assets' / 'recipes.zip', 'r') as zip_obj:
    pprint(zip_obj.namelist())<jupyter_output><empty_output><jupyter_text>You can also add files to an existing archive using the append mode:<jupyter_code>new_recipe_files = ['cornbread_recipe.txt']
with ZipFile(Path.cwd() / 'day_4_assets' / 'recipes.zip', 'a') as recipe_zip:
    for name in new_recipe_files:
        filename = Path.cwd() / 'day_4_assets' / 'recipes' / name
        recipe_zip.write(filename, arcname=name)
with ZipFile(Path.cwd() / 'day_4_assets' / 'recipes.zip', 'r') as zip_obj:
    pprint(zip_obj.namelist())<jupyter_output><empty_output><jupyter_text>Some misc. cleanup:<jupyter_code>import os
os.remove(Path.cwd() / 'day_4_assets' / 'recipes.zip')<jupyter_output><empty_output><jupyter_text>To read more about `ZipFile()`, check out help output:<jupyter_code>help(ZipFile)<jupyter_output><empty_output><jupyter_text>To learn more about `zipfile`, check out the [documentation](https://docs.python.org/3/library/zipfile.htmlmodule-zipfile).  An Easier Way of Creating ArchivesThe `shutil` module also supports creating, reading, and extracting TAR and ZIP archives using high-level methods. We will also use the `pathlib` module for best practices when creating file paths.<jupyter_code>import shutil
from pathlib import Path<jupyter_output><empty_output><jupyter_text>To create an archive using `shutil`, you will use the```pythonshutil.make_archive(base_name, archive_format, root_dir=current_directory, ...)```method where it supports `archive_format`s of `zip`, `tar`, `bztar`, and `gztar`.For example:<jupyter_code>base_name = Path.cwd() / 'day_4_assets' / 'archived_recipes'
root_dir = Path.cwd() / 'day_4_assets' / 'recipes/'
shutil.make_archive(base_name, 'zip', root_dir)
with ZipFile(Path.cwd() / 'day_4_assets' / 'archived_recipes.zip', 'r') as zip_obj:
    pprint(zip_obj.namelist())<jupyter_output><empty_output><jupyter_text>To read more about `shutil.make_archive()`, check out help output:<jupyter_code>help(shutil.make_archive)<jupyter_output><empty_output><jupyter_text>To extract the archive, call the```pythonshutil.unpack_archive(filename, extract_dir=current_directory)```method.For example:<jupyter_code>filename = Path.cwd() / 'day_4_assets' / 'archived_recipes.zip'
extract_dir = Path.cwd() / 'day_4_assets' / 'unpacked_recipes/'
shutil.unpack_archive(filename, extract_dir)
tree(Path.cwd() / 'day_4_assets' / 'unpacked_recipes')<jupyter_output><empty_output><jupyter_text>To read more about `shutil.unpack_archive()`, check out help output:<jupyter_code>help(shutil.unpack_archive)<jupyter_output><empty_output><jupyter_text>Some misc. cleanup:<jupyter_code>import os
shutil.rmtree(Path.cwd() / 'day_4_assets' / 'unpacked_recipes')
os.remove(Path.cwd() / 'day_4_assets' / 'archived_recipes.zip')<jupyter_output><empty_output><jupyter_text>To learn more about archiving operations from the `shutil` module, check out the [documentation](https://docs.python.org/3/library/shutil.htmlarchiving-operations). OBJECT PERSISTENCE---The term **pickling** is a popular method of preserving food for later consumption. By placing a product in a specific solution, it is possible to increase its shelf life.As a Python developer, you might one day find yourself in need of a way to store your Python objects for later use. What if I told you that you can pickle Python objects too? Serialization**Serialization** is the process of transforming objects (or data structures) into **byte streams**. A byte is composed of 8 bits of zeros and ones, which can be then stored and transferred easily. This allows for develpers to save configuration data or a program's progress, and then store it on a disk or send it to another location.Unlike pickling vegetables, where the pickled food's flavor and texture change, pickled (or serialized) Python objects can be easily unpickled back to their original form. This process is know as **deserialization**.Python objects can be serialized and deserialized using the native `pickle` module.<jupyter_code>import pickle<jupyter_output><empty_output><jupyter_text>Pickling should not be confused with archiving (i.e., compression). Pickling translates data into a format that can be transferred from RAM to disk. Compression, on the other hand, is the process of encoding data using fewer bits to save disk space.For example, think of serialization as the process of saving and loading your progress in a video game from your storage disk. Pickle vs JSONRecall from previous lectures, JavaScript Object Notation (JSON) lets us save and transmit objects encoded as strings in a human-readable, language-independent schema. Although converting Python objects to JSON might be faster than pickling, the JSON format does have limitations. Most importantly, only a limited subset of Python built-in data structures can be respresented by JSON. With pickling in Python, we can easily serialize a very large range of Python types, and custom classes. What can be Pickled and UnpickledThe following types can be serialized and deserialized using the `pickle` module:- All native datatypes in Python (e.g., `booleans`, `None`, `integers`, `floats`, `strings`, etc)- Python containers (e.g., `dictionary`, `set`, `list`, `tuples`) - as long as the container has pickleable objects- Functions and classes that are defined at the top level of a module    IMPORTANT NOTES:    Your pickled data can only be unpickled using Python    Using a different Python version when pickling and unpickling can cause many problems    Make sure that the environment where the function is unpickled is able to import the function, otherwise, an exception will be raised    Unpickling data from an untrusted source can result in the execution of malicious code By the end of the lecture, you'll be able to:- pattern match with string methods- pattern match regular expressions using the re module- pattern match filenames using the glob and pathlib modules- handle the complexities of dates and times using the datetime and dateutil modules- archive and unarchive files using the zipfile and shutil modules- **serialize and deserialize Python objects using the `pickle` module** PicklingTo pickle a Python object, use the```pythonpickle.dump(obj, file_obj, ...)```method paired with an `open()` method in write binary mode and a `with` statement. For example:<jupyter_code>from pathlib import Path

food_lst = ['cucumber', 'pumpkin', 'carrot']

with open(Path.cwd() / 'day_4_assets' / 'food_lst.pkl', 'wb') as f:
    pickle.dump(food_lst, f)
    
tree(Path.cwd() / 'day_4_assets')<jupyter_output><empty_output><jupyter_text>Note, that the `.pkl` extension is not necessary but it's good practice to include the extension for readability purposes.To read more about `pickle.dump()`, check out help output:<jupyter_code>help(pickle.dump)<jupyter_output><empty_output><jupyter_text>UnpicklingTo unpickle the contents we just pickled, use the```pythonpickle.load(pickle_filename, ...)```method in read binary mode.For example:<jupyter_code>with open(Path.cwd() / 'day_4_assets' / 'food_lst.pkl', 'rb') as f:
    unpickled_lst = pickle.load(f)
    
print(unpickled_lst)<jupyter_output><empty_output><jupyter_text>To read more about `pickle.load()`, check out help output:<jupyter_code>help(pickle.load)<jupyter_output><empty_output><jupyter_text>Pickling and Unpickling Custom ObjectsAs mentioned, you can serialize your custom objects.For example:<jupyter_code>class Veggy():
    def __init__(self):
        self.color = ''
    def set_color(self, color):
        self.color = color

cucumber = Veggy()
cucumber.set_color('green')

with open(Path.cwd() / 'day_4_assets' / 'cucumber.pkl', 'wb') as f:
    pickle.dump(cucumber, f)

with open(Path.cwd() / 'day_4_assets' / 'cucumber.pkl', 'rb') as f:
    unpickled_cucumber = pickle.load(f)

print(unpickled_cucumber.color)<jupyter_output><empty_output><jupyter_text>**Caution**We can only unpickle the object in an environment where the class `Veggy` is either defined or imported. Otherwise, we'll get an `AttributeError`.<jupyter_code>del Veggy
with open(Path.cwd() / 'day_4_assets' / 'cucumber.pkl', 'rb') as f:
    unpickled_cucumber = pickle.load(f)

print(unpickled_cucumber.color)<jupyter_output><empty_output><jupyter_text>Some misc. cleanup:<jupyter_code>import os
os.remove(Path.cwd() / 'day_4_assets' / 'food_lst.pkl')
os.remove(Path.cwd() / 'day_4_assets' / 'cucumber.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10465176312664567, 'fraction_numerical': 0.006815013681656255, 'mean_word_length': 5.839512711864407, 'pattern_counts': {'":': 0, '<': 309, '<?xml version=': 0, '>': 309, 'https://': 15, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '77', 'dir': 'jupyter-structured-clean-dedup', 'id': '474205', 'license': 'MIT', 'n_tokens_mistral': 11898, 'n_tokens_neox': 11441, 'n_words': 4310, 'path': 'week_3/day_4_lecture.ipynb', 'repo_name': 'brown-ccv/workshop_python_2021', 'size': '38738'}