<jupyter_start><jupyter_text>We can use `requests` to fetch data from websites.Sometimes this data is just the HTML itself:<jupyter_code>res = requests.get("https://cmsc320.github.io")
len(res.text)<jupyter_output><empty_output><jupyter_text>It's useful to separate out the `url` which is the service we are requesting data from, and the `parameters` which is the query we are making.A useful list of public APIs for practicing can be found at [https://github.com/public-apis/public-apis](https://github.com/public-apis/public-apis)For `gutendex` the result is not HTML, but JSON. You can find more information about the Gutendex API at their homepage: [gutendex.com](https://gutendex.com/)<jupyter_code>url = "https://gutendex.com/books"

params = { "search": "Bhagavad" }
res = requests.get( url, params )<jupyter_output><empty_output><jupyter_text>You can use the `.json()` method on the result to turn the response into a dictionary. (Note: this only works for well-formed JSON, some websites may not provide well-formed JSON!)<jupyter_code>d = res.json()
type( d )<jupyter_output><empty_output><jupyter_text>You can then look up information about the resulting dictionary:<jupyter_code>d.keys()
d['count']
first = d['results'][0]
first['title']
first<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11906614785992217, 'fraction_numerical': 0.0031128404669260703, 'mean_word_length': 6.144444444444445, 'pattern_counts': {'":': 1, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '25090', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 422, 'n_tokens_neox': 408, 'n_words': 132, 'path': 'notebooks/lec03-basic-web-scraping.ipynb', 'repo_name': 'ak840206/lecture-code', 'size': '1285'}