<jupyter_start><jupyter_text>Data structures We learned a lot about processing data in Python so far, but our emphasis has been to bring data into `pandas` as a `DataFrame` object, in other words: **tabular data**. But data is not always tabular and in this notebook we focus on some semi-structured data formats. Two things stand out the most about semi-structured data:1. There is usually no pre-set schema so we need to be extra careful when processing the data and be ready to catch different kinds of errors.1. The data is usually **hierarchical**, in other words information can be **nested** at various levels and we need to find the right way to **flatten** the data if we want to turn it into tabular data. One of the most common types of semi-structured data is JSON. It has a very minimialist syntax, making it easy to explore. In Python, the natural object to represent JSON data is a Python dictionary, because of its flexibility and ability to nest information, and because it's a **key-value** store. So we can **load** a JSON file into a Python dictionary, or go the other around and **dump** a Python dictionary into a JSON file. JSON files also have support for **arrays**, which is the equivalent of a Python list. Here's an example:  Exercise In the following example, we create a Python dictionary<jupyter_code>dd = {"a": 1, 
      "b": [12, "55", ["hello", "hi"]], # we can have lists and nested lists
      "c": {"A": "bonjour", "B": "hola"}, # we can have nested information
     }

print(type(dd))<jupyter_output><empty_output><jupyter_text>Let's first see how we can extract the information contained in it. Drill down into the dictionary to extract the following items:- the number 55- the string "hi"- the string "hola" We now dump the dictionary's content into a JSON file using the `json` library. Run the cell and open the file in your editor to examine it. Does the JSON file and the dictionary look almost identical?<jupyter_code>import json

with open('delete_me.json', 'w') as outfile:
    json.dump(dd, outfile) # you can set sort_keys = True if you wish<jupyter_output><empty_output><jupyter_text>However, note that we need to be careful not to equate the two: A dictionary can have many things in it, including for example a Python function. Return to the code above, and add a new element to it whose key is `"d"` and whose value is a Python function, like `print` or a function you wrote yourself. Then try to run the cell and report what error message you get, if any. End of exercise Let's now go in the opposite direction. Since we just dumped the content of the dictionary into a JSON file, let's now read the file and load its content back into a dictionary. We use `load` to do this.<jupyter_code>with open('delete_me.json', encoding = 'utf-8') as infile:
    dd = json.load(infile)

print(type(dd))
print(dd)<jupyter_output><empty_output><jupyter_text>We can also load directly from a string in Python, using the `loads` function (with an `s`), like the following example:<jupyter_code>json_str = '{"drinks": ["coffee", "tea", "water"]}'
dd = json.loads(json_str)
print(type(dd)) # the JSON string becomes a Python dictionary
print(type(dd["drinks"])) # the JSON array becomes a Python list
print(dd)<jupyter_output><empty_output><jupyter_text>And once again we can go in the other direction by using `dumps` (with an `s`):<jupyter_code>json_str = json.dumps(dd)
print(json_str)
type(json_str)<jupyter_output><empty_output><jupyter_text>Let's go over one practical application for `loads` and `dumps`. One question that we may run into is what to do when the quantity of data is large and reading it all at once into our Python session is not an option. Is there a more light-weight approach to reading data gradually? The answer is yes, but we need to talk about reading raw text files in Python first. Say we have a large file, like the following. One important thing to notice about this file is that **each line is a JSON object, but the file as a whole is not**. It would be if we wrap it in square brackets and separate lines by commas, but we don't want to do that here, because we're not interested in reading the file all at once, but rather line by line.<jupyter_code>%%writefile delete_me.txt
{"drinks": ["coffee", "tea", "water"]}
{"drinks": ["coffee", "juice", "rum", "beer"]}
{"drinks": ["vodka", "tapioca", "lulo"]}
{"drinks": ["milk", "kambucha", "water"]}<jupyter_output><empty_output><jupyter_text>ExerciseLet's now read the above file. In our earlier example, we read files using the `with` statement, which has the advantage of closing files automatically for us, so we don't have to do our own cleanup. This time we're going to read our file without `with` to show the difference.<jupyter_code>file = open("delete_me.txt", "r")
print(file)<jupyter_output><empty_output><jupyter_text>Uncomment and run the line below, you'll see that we cannot load the file using `json.load` because as we said the file as a whole is not JSON.<jupyter_code># json_data = json.load(file)<jupyter_output><empty_output><jupyter_text>To read the content of the file above, we have two functions we can use. Let's first begin with `readlines` (with an `s`), which reads all the lines at once.<jupyter_code>file.readlines()<jupyter_output><empty_output><jupyter_text>What kind of Python object did `readlines` return? You should be able to answer just from looking at it. Now run the above cell one more time. What is the output you see? Can you guess what happened? HINT: Go up and read the file one more time and then run the above cell.Write a code that reads the file and then write a loop that iterates over every element returned by `readlines` and prints it. Functions like `readlines` are called **iterators** because they read some content broken up into pieces (how a text file is made up of lines) and return it as one big list. Now let's read the file again and this time we learn about another function: `readline` (without an `s`).<jupyter_code>file = open("delete_me.txt", "r")<jupyter_output><empty_output><jupyter_text>Run the following line a few times, and report what you see every time you run it.<jupyter_code>file.readline()<jupyter_output><empty_output><jupyter_text>Functions like `readline` are called **generators** because instead of reading the content all at once and returning one big list with everything, they return the content piecemeal (one line at a time in our case). Unlike iterators, generators don't need to load the entire content at once, and this means we can keep a low memory footprint when working with large objects in Python. And just like iterators, generators can be looped over. End of exercise Now we read the file again and write a loop to read each line of the file and print it. Note that this is not as easy as working with iterators. One way to do it is use a `while True` statement and define a condition for breaking it.<jupyter_code>file = open("delete_me.txt", "r")

while True:
    line = file.readline()
    if line != "": # we assume that if line == "" we've reached EOF
        print(line)
    else:
        break<jupyter_output><empty_output><jupyter_text>Time to put it all together. We write a loop that does the following at each iteration:- read the next line of the input file- use `json.loads` to load it as a dictionary- calculate the number of drinks and add it to the dictionary using `size` as key- dump the content to an output file<jupyter_code>in_file = open("delete_me.txt", "r")
out_file = open("delete_me_out.txt", "w") # change w to a if you want to append

while True:
    line = in_file.readline()
    if line != "": # we assume that if line == "" we've reached EOF
        line_dict = json.loads(line)
        line_dict['size'] = len(line_dict['drinks'])
        json.dump(line_dict, out_file) # dump content into file
        out_file.write('\n') # add a line break
    else:
        break
        
out_file.close() # don't forget to close the file when you're done<jupyter_output><empty_output><jupyter_text>Make sure to open the file and check the results. Another common data structure with a very minimalistic syntax and similarities with JSON is YAML.<jupyter_code>import yaml
from pprint import pprint

with open("data/spen.yaml", encoding = 'utf-8') as file:
    en_sp = yaml.load(file, Loader = yaml.FullLoader)

pprint(en_sp)<jupyter_output><empty_output><jupyter_text>One advantage of YAML is its support for references, called **anchors** by YAML, but that's a topic we won't get into. Suffice it to say that for this reason YAML is a popular format for configuration files, like the **conda** environment we're using for the course.<jupyter_code># %load config/environment.yml
name: uwdatasci400

channels:
  - defaults
  - conda-forge

dependencies:
  - python=3.6
  - scikit-learn=0.20.3
  - seaborn=0.10.0
  - category_encoders=2.0.0
  - joblib=0.14.1
  - jupyter=1.0.0
  - ipykernel=5.1.4
  - mlxtend=0.17.2
  - graphviz=2.38
  - pip=19.0.1
  - requests=2.23.0
  - pyyaml=5.3.1

  - pip:
    - pylint
    - pandas==1.0.1
    - bs4==4.9.0<jupyter_output><empty_output><jupyter_text>ExerciseRead the file `config/environment.yml` into Python and write a loop to print all the packages defined in our conda environment, including the `pip` packages.  End of exercise JSON and YAML are relatively new data structures, and quickly gaining in popularity. Prior to that, XML was very common, and it's sytnax is definitely more verbose. Here is an example of an XML file.<jupyter_code>import requests

url = 'https://www.w3schools.com/xml/cd_catalog.xml'
document = requests.get(url)

print(document.text[:249]) # show a small chunk of the data<jupyter_output><empty_output><jupyter_text>Data in XML as we can see is also hierarchical, and each piece of data is surrounded in **tags**, which are identified by `` and `` where `TAG` in our case is be `CD`, `TITLE`, etc. The tags in XML are similar to the keys in a JSON file. So we can see the above data is a CD catalogue, which contains a title, artist name, etc. To read this type of data we use the `BeautifulSoup` function in the `bs4` package, but keep in mind that there are alternative packages we could have used, such as `lxml`, which offers support for **XPath**, which is a query language for XML documents. We do not cover XPath here, but encourage you to look into it.<jupyter_code>from bs4 import BeautifulSoup
soup = BeautifulSoup(document.content,"lxml-xml")<jupyter_output><empty_output><jupyter_text>`BeautifulSoup` does not support searching an XML file using XPath, but it does have its own functionality for searching XML. We can use `find` to find the **first** occurence of a tag and use `find_all` to find all the occurences of a tag.<jupyter_code>print(soup.find("TITLE"))<jupyter_output><empty_output><jupyter_text>If we wish to extract the text itself without the tag, we can use the `text` attribute of the object.<jupyter_code>print(soup.find("TITLE").text)<jupyter_output><empty_output><jupyter_text>ExerciseUse `find_all` to find all the occurences of the `TITLE` tag, and write a loop to extract and print the text attribute of each.  End of exercise If an XML file looks familiar it's because it's format is almost the same as HTML, which what web pages use to store their content. In fact, HTML and XML are practically the same thing, with the difference that **with HTML the tags are pre-defined** whereas XML tags are whatever we choose. This makes XML a repository for any data, whereas HTML is a repository for a web page. **Web scraping** refers to writing programs that extract information from websites. It shouldn't surprise you that information found in websites is very disorganized and disparate. As such, there's a lot of work that go into cleaning it once we read it. We introduce you to some libraries that make this easier, but at the end of the day a lot of this functionality depends on the website we're trying to scrape, so it involves a lot of trial and error. But the tooling that we learned for XML can in most cases be directly applied to HTML as we're about to see.Let's send an HTTP request to scrape wikipedia.<jupyter_code>url = "https://wiki.python.org/moin/IntroductoryBooks" 
response = requests.get(url)
print(response.headers)

content = response.content
print(content[:30]) # print a small section of the result<jupyter_output><empty_output><jupyter_text>As we can see, the above information is just in raw text format. To clean it up, we can use the `bs4` library. Since websites requests are sent back as HTML files, the job of `BeautifulSoup` function is to **parse** the HTML for us.<jupyter_code>soup = BeautifulSoup(content, "html")<jupyter_output><empty_output><jupyter_text>We can use the `prettify` function to print the HTML in a more readable format where the hierarchy of nested **HTML tags** can be seen. Some tags contain **elements**. For example `...` is the `input` tag and `id` is one of its elements.<jupyter_code>print(soup.prettify()[:500])<jupyter_output><empty_output><jupyter_text>The soup object can help us extract specific tags out. Because tags for an HTML file are pre-defined (unlike XML), there are some shortcuts for searching HTML. For example, if we want the `title` tag, we can just use the `title` attribute:<jupyter_code>print(soup.title)<jupyter_output><empty_output><jupyter_text>And if we just want the content of the `title` tag, without the tag wrapper, we can use the `text` attribute just like we did with XML.<jupyter_code>print(soup.title.text)<jupyter_output><empty_output><jupyter_text>Many tags show up not once but many times in the HTML document, so we can use `soup.find_all` to search tags.<jupyter_code>all_a = soup.find_all("a") # the a tag contains an element called href for hyperlinks

for x in all_a[:20]: # print the top 20
    print(x)<jupyter_output><empty_output><jupyter_text>ExerciseLet's print of the lines form above here:<jupyter_code>x = all_a[20]
print(x)<jupyter_output><empty_output><jupyter_text>Copy the output of the above cell and paste it into a new Markdown cell in your notebook. What do you see when you render the cell? You can also extract just the URL by using the `get` method and passing it the name of the element, `href` in our case.<jupyter_code>print(x.text) # this is the title of the URL (what you see when you click on it)
print(x.get("href")) # this is the URL (the link you go to)<jupyter_output><empty_output><jupyter_text>End of exercise With `find_all` we can also narrow our search by finding for example all `a` tags that have an element called `https`:<jupyter_code>all_a_https = soup.find_all("a", "https")   

for x in all_a_https[:20]: # print the top 20
    print(x)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08333897960566435, 'fraction_numerical': 0.005691442509655126, 'mean_word_length': 4.767878077373974, 'pattern_counts': {'":': 12, '<': 113, '<?xml version=': 0, '>': 113, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '193886', 'license': 'MIT', 'n_tokens_mistral': 4380, 'n_tokens_neox': 4254, 'n_words': 2077, 'path': 'lesson_6.ipynb', 'repo_name': 'sethmott/datasci400', 'size': '14759'}