<commit_before># Chapter 4: Web Scraping
## Summary:

Big data. Big deal. In this chapter, we're going to take a gander at a subject
known as "Web Scraping." What's that you ask? Is that when you crawl the
Internet for days trying to escape the dark web? No, that's called
poor judgement. Web scraping is when a script or program retrieves information
from a website for processing, analyzing, or organizing.

So why did I mention big data? Well, since the computer is retrieving the data
from the website(s), it can get that data and crunch through it pretty quickly.
So, naturally, web scraping is popular among those seeking the usage and analytics
of "big data." You could grab the numbers and statistics from a large collection
of websites, store it in a neat data structure, and produce whatever your heart
desires--graphs, charts, tables, etc. manufactured from the data.

Maybe handling a sea of data seems daunting to you though. Perhaps we could
study the more common applications of web scraping? Sure! Web scraping can be
used for myriad tasks, such as:

  - Sports statistics
  - Social media trends
  - Online product comparisons
  - Stock data aggregation
  - Crime statistics

So, in this chapter we'll get to look at the basics of scraping the web using
a pretty sweet third-party Python package called *BeautifulSoup*.

---

## Content:

Alright, so first we need to install *BeautifulSoup* for Python. The newest
version, as of the writing of this book, is v4.4.1, which supports both Python 2.6+
and Python 3. You can install the package with whatever package manager you prefer,
such as *pip* or *easy_install*, like so:

```
pip install beautifulsoup4
```

```
easy_install beautifulsoup4
```

Or, if you don't like that method, you can always visit the website for the package
at [https://www.crummy.com/software/BeautifulSoup/bs4/download/](https://www.crummy.com/software/BeautifulSoup/bs4/download/).

To test that you installed the package successfully, you can simply start the
Python interpreter, and run the following import command:

```python
from bs4 import BeautifulSoup
```

If that import command completed without error, you're setup and ready to scrape!

---

Now then, before we actually start using the *bs4* package, let's first take
a look at the general process of web scraping. <commit_msg>Chapter 4 initial draft - content - process<commit_after># Chapter 4: Web Scraping
## Summary:

Big data. Big deal. In this chapter, we're going to take a gander at a subject
known as "Web Scraping." What's that you ask? Is that when you crawl the
Internet for days trying to escape the dark web? No, that's called
poor judgement. Web scraping is when a script or program retrieves information
from a website for processing, analyzing, or organizing.

So why did I mention big data? Well, since the computer is retrieving the data
from the website(s), it can get that data and crunch through it pretty quickly.
So, naturally, web scraping is popular among those seeking the usage and analytics
of "big data." You could grab the numbers and statistics from a large collection
of websites, store it in a neat data structure, and produce whatever your heart
desires--graphs, charts, tables, etc. manufactured from the data.

Maybe handling a sea of data seems daunting to you though. Perhaps we could
study the more common applications of web scraping? Sure! Web scraping can be
used for myriad tasks, such as:
  - Sports statistics
  - Social media trends
  - Online product comparisons
  - Stock data aggregation
  - Crime statistics

So, in this chapter we'll get to look at the basics of scraping the web using
a pretty sweet third-party Python package called *BeautifulSoup*.

---

## Content:

Alright, so first we need to install *BeautifulSoup* for Python. The newest
version, as of the writing of this book, is v4.4.1, which supports both Python 2.6+
and Python 3. You can install the package with whatever package manager you prefer,
such as *pip* or *easy_install*, like so:

```
pip install beautifulsoup4
```

```
easy_install beautifulsoup4
```

Or, if you don't like that method, you can always visit the website for the package
at [https://www.crummy.com/software/BeautifulSoup/bs4/download/](https://www.crummy.com/software/BeautifulSoup/bs4/download/).

To test that you installed the package successfully, you can simply start the
Python interpreter, and run the following import command:

```python
from bs4 import BeautifulSoup
```

If that import command completed without error, you're setup and ready to scrape!

---

Now then, before we actually start using the *bs4* package, let's first take
a look at the general process of web scraping, as follows:
  1. Study the format of target webpage
  2. Retrieve content of webpage (an HTTP request)
  3. Search the content for specific data based off the content format
  4. Do stuff with the data found in the search
{'fraction_non_alphanumeric': 0.06120374135827572, 'fraction_numerical': 0.0063033753558357055, 'mean_word_length': 4.255341880341881, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2923308', 'n_tokens_mistral': 1360, 'n_tokens_neox': 1250, 'n_words': 742}