<jupyter_start><jupyter_text>Normal scrapingBy now we all know how to scrape normal sites (kind of, mostly, somewhat).<jupyter_code># Grab the NYT's homepage
response = requests.get("http://nytimes.com")
doc = BeautifulSoup(response.text)
# Snag all of the headlines (h3 tags with 'story-heading' class)
headlines = doc.find_all("h3", {'class': 'story-heading'})
# Getting the headline text out using list comprehensions
# is a lot more fun but I guess you just learned those
# like a day ago, so we'll go ahead and use a for loop.
# But for the curious:
#   [headline.text.strip() for headline in headlines]

# Print the text of the headlines
for headline in headlines:
    print(headline.)<jupyter_output><empty_output><jupyter_text>But... forms!So the issue is that sometimes you need to submit *forms* on a web site. Why? Well, let's look at an example.This example is going to come from [<NAME>](https://twitter.com/dancow)'s incredible [Search, Script, Scrape](https://github.com/compjour/search-script-scrape), 101 scraping exercises.> The number of FDA-approved, but now discontinued drug products that contain Fentanyl as an active ingredient>> Related URL: [http://www.accessdata.fda.gov/scripts/cder/ob/docs/queryai.cfm](http://www.accessdata.fda.gov/scripts/cder/ob/docs/queryai.cfm)When you visit that URL, you're going to type in "Fentanyl," and select "Disc (Discontinued Drug Products)." Then you'll hit **search**.Hooray, results! Now **look at the URL.**> http://www.accessdata.fda.gov/scripts/cder/ob/docs/tempai.cfmDoes anything about that URL say "Fentanyl" or "Discontinued Drug Products"? Nope! And if you [straight up visit it](http://www.accessdata.fda.gov/scripts/cder/ob/docs/tempai.cfm) (might need to open an Incognito window) you'll end up being redirected back to a different page. This means **`requests.get` just isn't going to cut it.** If you tell `requests` to download that page it's going to get a whooole lot of uselessness.Be my guest if you want to try it!<jupyter_code>from selenium import webdriver<jupyter_output><empty_output><jupyter_text>Submitting forms with `requests`There are two kinds of forms, `GET` forms and `POST` forms (...this is 99% true). `GET` formsA `GET` form is one where you can see parameters in the URL. For example, if you searched for images of animals surfing on Bing you'd end up here:> `http://www.bing.com/images/search?q=animals+surfing&FORM=HDRSC2`It has a couple parameters - `q` and `FORM`. `FORM` is some sort of weird analytics thing that doesn't affect the page, but `q` is definitely the term you're searching for. **With a `GET` form, the data you put into the form is kept in the URL.**Just for kicks, if we looked at the HTML for a `GET` form it might look like this:````html````It might also leave the whole `method` part off, too - `GET` is the default.A fun part about `GET` forms you can share the URL to share the results. If you don't believe me, visit [http://www.bing.com/images/search?q=animals+surfing&FORM=HDRSC2](http://www.bing.com/images/search?q=animals+surfing&FORM=HDRSC2) to see animals surfing.`GET` is how most web pages work. You've used it every time you invoke the unholy powers of `requests.get`.````pyrequests.get("https://api.spotify.com/v1/search?query=90s&offset=20&limit=20&type=playlist")`````GET` is nice. `GET` is easy. **But `GET` is not all there is.** `POST` formsThe other kind of forms are `POST` forms. `POST` forms are not friendly!Unlike `GET` forms, **you can't share the URL to get the same information.** The parameters - the `q` for your search, for example - aren't in the URL, they're hidden in *the actual request*.What this means is that **every time you request something from a `POST`-based form, you have to pretend you filled out the form and clicked the button.** Grabbing the parametersFirst we need to find out what parameters we're going to hunt down. To do this, first make your way to the form, then **get prepared.**1) In Chrome, `View` > `Developer` > `Developer Tools`2) Click the **Network** tab3) Fill the form out, and submit it4) Scroll up to the top of the Network pane, select the segment of the URL you're at (I'm at `tempai.cfm`)5) Click it6) Select **Headers** on the right7) Scroll down until you see **Form Data**Okay, that seemed like a *lot of work,* but I promise it was actually simple and easy and you're living life in a grand grand way. Two parameters are listed for the search we're doing:````Generic_Name:Fentanyltable1:OB_Disc````Seems simple enough! Now let's put them to work. Submitting `POST` forms with `requests.post`This is going to be *so easy* you might have a *heart attack* as a result of your body being *so amazed* that it doesn't have to do anything strenuous. All you have to do is````pyrequests.get("http://whatever.com/url/to/something", { "param1": "val1", "param2": "val2" })````and treat it like a normal response! Here, I'll prove it.<jupyter_code># Just in case you didn't run it up there, I'll import again
import requests


# Using .select instead of .find is a little more
# readable to people from the web dev world, maybe?<jupyter_output><empty_output><jupyter_text>It's magic, I swear! But then...Sometimes `requests.get` just isn't enough. Why? It mostly has to do with JavaScript or complicated forms - when a site reacts and changes without loading a new page, you can't use `requests` for that (think "Load more" buttons on Instagram).For those sites you need **Selenium!** Selenium = you put your browser on autopilot. As in, literally, it takes control over your browser. There are "headless" versions that use invisible browsers but if you don't like to install a bunch of stuff, the normal version is usually fine. Installing Selenium Selenium isn't just a Python package, but you'll need to install **python bindings** in order to have Python talk to Selenium.````pip install selenium````You'll also need the [Firefox browser](https://www.mozilla.org/en-US/firefox), since that's the browser we're going to be controlling.Selenium is built on **WebDrivers**, which are libraries that let you... drive a browser. I believe it comes with a Firefox WebDriver, whereas Safari/Chrome/etc take a little more effort to set up.  Using Selenium<jupyter_code># Imports, of course
!pip install selenium
# Initialize a Firefox webdriver
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWai

driver = webdriver.Firefox()

# Grab the web page

# You'll use selenium.webdriver.support.ui.Select
# that we imported above to grab the Seelct element called 
# t_web_lookup__license_type_name, then select Acupuncturists

# We use .find_element_by_name here because we know the name
# We use .find_element_by_id here because we know the id

# Then we'll fake typing into it
# Now we can grab the search button and click it

# Instead of using requests.get, we just look at .page_source of the driver

# We can feed that into Beautiful Soup

# It's a tricky table, but this grabs the linked names inside of the A
#rows = doc.select("#datagrid_results tr")<jupyter_output><empty_output><jupyter_text>Closing the webdriverOnce we have all the data we want, we can close our webdriver.<jupyter_code># Close the webdriver<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09714673913043478, 'fraction_numerical': 0.003940217391304348, 'mean_word_length': 4.89823717948718, 'pattern_counts': {'":': 2, '<': 22, '<?xml version=': 0, '>': 29, 'https://': 4, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '104457', 'license': 'MIT', 'n_tokens_mistral': 2227, 'n_tokens_neox': 2080, 'n_words': 998, 'path': 'scraping_form_submissions.ipynb', 'repo_name': 'kbennion/foundations-hw', 'size': '7364'}