<jupyter_start><jupyter_text>Now You Code In Class: Extracting Hyperlinks from a WebpageLet's write a program which extracts the hyperlinks from a Webpage. There are 3 webpage files you can try with this program:    httpbin-org.html    ischool-directory.html    ist256-com.html    wikipedia-President-of-the-United-States.htmlRun this command to download these three files:<jupyter_code>!curl https://httpbin.org/ -o httpbin-org.html
!curl https://ischool.syr.edu/directory/?cat=all -o ischool-directory.html
!curl https://ist256.com -o ist256-com.html   
!curl https://en.wikipedia.org/wiki/President_of_the_United_States  -o wikipedia-president-of-the-united-states.html<jupyter_output><empty_output><jupyter_text>In addition you can download any file you wish by copying the `!curl` command. Place the URL after `curl` and the filename after `-o`. StrategyHere's the basic strategy. We must read in the contents all at once since there is not line structure within the HTML. So as one big string we should be looking for the following token in the text of the file `<a href="` when you find this token the link is all characters until the second `"` character. For example: `` the Hyperlink would be: `http://ist256.github.io`.In addition there are 4 types of links:- **absolute** links are URL's like `https://www.google.com`. These begin with `http://` or `https://`- **email** links like `mailto:<EMAIL>`. This begin with `mailto:`- **bookmarks** are links to places on the same page. `about`. These begin with ``. - **relative** links (to pages on the same site). Consider all other links this type of link. Recommended approach : Bottom UpWe will use an approach called Bottom Up to solve this problem. It works by writing the alogrithm at a high level, then for each step in the algorithm which is non-trivial (more than a line of code) we write the step as its own function. We test the functions then assemble the final program code by calling the functions we made.THE ALGORITHM    1. read the file into a string    2. loop    3.    if string has link    4.       extract link from string returning link and remaining string    5.       determine link type    6.       print link and link type (external / internal / email / bookmark)    7.    else    8.       stop looping Which steps must be written as functions?Ask yourself which lines of the alogirthm can be implemented trivially in Python.PROMPT 1  Step 1a: Problem Analysis for `readFile()`Inputs:     PROMPT 2Outputs:     PROMPT 3Algorithm (Steps in Program):    PROMPT 4How many tests are required and why?    PROMPT 5 Step 1b: Code the function<jupyter_code># PROMPT 6 - write function<jupyter_output><empty_output><jupyter_text>Step 2: Write testsLet's make sure the function works<jupyter_code># PROMPT 7 Test(s) for function<jupyter_output><empty_output><jupyter_text>Step 3a: Problem Analysis for `extractLink()`Inputs:     PROMPT 8Outputs:     PROMPT 9Algorithm (Steps in Program):    PROMPT 10 (Having Trouble? Let's discuss an approach. Write then re-factor into a function.)How many tests are required and why?    PROMPT 11 Step 3b: Code the function<jupyter_code># PROMPT 12 - write function<jupyter_output><empty_output><jupyter_text>Step 4: Write testsLet's make sure the function works<jupyter_code># PROMPT 13 - write test(s) for function<jupyter_output><empty_output><jupyter_text>Step 5a: Problem Analysis for `linkType()`Inputs:     PROMPT 14Outputs:     PROMPT 15Algorithm (Steps in Program):    PROMPT 16 How many tests are required and why?    PROMPT 17 Step 5b: Code the function<jupyter_code># PROMPT 18 - write function<jupyter_output><empty_output><jupyter_text>Step 6: Write testsLet's make sure the function works<jupyter_code># PROMPT 19 - write tests for function<jupyter_output><empty_output><jupyter_text>Final ProgramWith functions written, return to the original algorithm and imeplement. Here's a layout to get it working in ipython interact.<jupyter_code>from ipywidgets import interact_manual
files = ['httpbin-org.html','ischool-directory.html','ist256-com.html','wikipedia-president-of-the-united-states.html']

print("Link Extractor")
@interact_manual( choose_file=files)
def main(choose_file):
    #TODO Write code here

# run this code to turn in your work!
from coursetools.submission import Submission
Submission().submit_now()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08213223518645618, 'fraction_numerical': 0.014413177762525738, 'mean_word_length': 4.626769626769627, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 34, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '154515', 'license': 'MIT', 'n_tokens_mistral': 1358, 'n_tokens_neox': 1246, 'n_words': 525, 'path': 'content/lessons/07-Files/SmallGroup-Files.ipynb', 'repo_name': 'IST256/learn-python', 'size': '4388'}