<jupyter_start><jupyter_text>Gathering data from the web - Problems **Author:** <NAME> ([Personal Website](https://www.tiesdekok.com))  **Last updated:** June 2020  **Python version:** Python 3.6+     **License:** MIT License  **Recommended environment: `limpergPython`**<jupyter_code>import os
recommendedEnvironment = 'limpergPython'
if os.environ['CONDA_DEFAULT_ENV'] != recommendedEnvironment:
    print('Warning: it does not appear you are using the {0} environment, did you run "conda activate {0}" before starting Jupyter?'.format(recommendedEnvironment))<jupyter_output><empty_output><jupyter_text>Introduction  Make sure to open up the respective tutorial notebook(s)!  That is what you are expected to use as primariy reference material.   Relevant tutorial notebooks:1) [`0_python_basics.ipynb`](https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/0_python_basics.ipynb)  2) [`2_handling_data.ipynb`](https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/2_handling_data.ipynb)  3) [`4_web_scraping.ipynb`](https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/4_web_scraping.ipynb)   Part 1    Note: feel free to add as many cells as you'd like to answer these problems, you don't have to fit it all in one cell.  The goal of these problems is to get hands-on experience with gathering data from the Web using `Requests` and `Requests-HTML`. The tasks below are split up into two sections:  1. API tasks  2. Web scraping tasks    Import required packages<jupyter_code>import requests
from requests_html import HTMLSession
import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Also run the code below, it solves a couple of minor problems that you don't need to worry about<jupyter_code>from IPython.display import HTML
import time
def show_image(url):
    return HTML('<img src="{}?{}"></img>'.format(url, int(time.time())))
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10207939508506617, 'fraction_numerical': 0.010396975425330813, 'mean_word_length': 6.350694444444445, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '491298', 'license': 'MIT', 'n_tokens_mistral': 686, 'n_tokens_neox': 645, 'n_words': 192, 'path': 'problems/day_3/web_gathering_problems.ipynb', 'repo_name': 'ashercurtis/MPAcc-Advanced-NLP', 'size': '2121'}