<jupyter_start><jupyter_text>BeautifulSoup 1: scraping my pageBeautifulSoup is a powerful Python library used for pulling data out of HTML documents. In this notebook we will use the **requests** library to get the HTML document from my website and then use **BeautifulSoup** to get data from that document. Unlike **regular expressions**, which deal all HTML document as a string/text, **BeautifulSoup** distinguishes between simple/plain text and HTML tags/attributes which is very helpful for scraping.If you do not have BeautifulSoup installed, then open a completely new command prompt (black window) and type the following command:```pip install beautifulsoup```Okay, let's start from importing abovementioned libraries and selecting the url to scrape.<jupyter_code>import requests
# import everything from BeautifulSoup
from BeautifulSoup import *
url = "https://hrantdavtyan.github.io/"<jupyter_output><empty_output><jupyter_text>Once we have the libraries imported and the url selected, we should use the **get()** function from the **requests** library to get the website content as a response and then, convert it to text.<jupyter_code>response = requests.get(url)
my_page = response.text
print(response)
type(my_page)<jupyter_output><Response [200]><jupyter_text>In order to be able to initiate several function available from BeautifulSoup library, we need to pass **my_page** as an argument to **BeautifulSoup()** function. The content will still remain the same, yet the object type will change which will let us to use some nice methods.<jupyter_code>soup = BeautifulSoup(my_page)
type(soup)
print(soup)<jupyter_output><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title><NAME></title>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<link rel="stylesheet" type="text/css" href="css/reset.css" />
<link rel="stylesheet" type="text/css" href="css/style.css" />
<link rel="stylesheet" type="text/css" href="css/fancybox.css" />
<link rel="stylesheet" type="text/css" href="http://fonts.googleapis.com/css?family=Open+Sans:400,600,300,800,700,400italic|PT+Serif:400,400italic" />
<script type="text/javascript" src="js/jquery.min.js"></script>
<script type="text/javascript" src="js/jquery.easytabs.min.js"></script>
<script type="text/javascript" src="js/respond.min.js"></script>
<script type="text/javascript" src="js/j[...]<jupyter_text>Fine. Let's now try to find all the a tags from my page.<jupyter_code>a_tags = soup.findAll('a')
type(a_tags)
len(a_tags)<jupyter_output><empty_output><jupyter_text>As you can see above, we received a list as an output with 29 elements. The 29 elements are the 29 a tags from my website. We can print the outcome to see them.<jupyter_code>print(a_tags)<jupyter_output>[<a href="#" class="social-text">SOCIAL PROFILES</a>, <a href="https://www.facebook.com/HrantDavtyan" class="social-facebook"></a>, <a href="https://www.linkedin.com/in/hrantdavtyan" class="social-in"></a>, <a href="#profile" class="tab-profile">Profile</a>, <a href="#resume" class="tab-resume">Resume</a>, <a href="#portfolio" class="tab-portfolio">Portfolio</a>, <a href="#contact" class="tab-contact">Contact</a>, <a href="http://aua.am/">American University of Armenia (AUA)</a>, <a href="http://isifa.am/en/ifa-dfi/">French-German University</a>, <a href="https://www.cerge-ei.cz/">Cerge-EI</a>, <a href="http://www.fao.org/armenia/en/">United Nations Food and Agriculture Organization (UN FAO)</a>, <a href="http://www.metric.am/">METRIC</a>, <a href="http://ahpc.am/?lang=en">Armenian Harvest Promotion Center</a>, <a href="http://www.ucl.ac.uk/">University College London (UCL)</a>, <a href="http://iset.tsu.ge/">International School of Economics (ISET)</a>, <a href="" class="current" data-[...]<jupyter_text>If you were interested in finding only the very first a tag, then the **find()** function could be useful instead of **findAll()**. This function already strings the a tag and its content as a string, rather than a list.<jupyter_code>a_tag = soup.find('a')
type(a_tag)
print(a_tag)<jupyter_output><a href="#" class="social-text">SOCIAL PROFILES</a><jupyter_text>As you can see above, although this is just s string, its type is a **BeautifulSoup.Tag** which will helps us to use some other methods on it. For example, we can get the link inside the a tag (**href**) by using a **get()** function. As the links are always inside a **href** attribute, we will try to get the value of **href** as follows:<jupyter_code>print(a_tag.get('href'))<jupyter_output>#<jupyter_text>If we want to get links from all a_tags (the latter was a list), then we should iterate over the list and get the **href** value from each element of the list as follows:<jupyter_code>for i in a_tags:
    print(i.get("href"))<jupyter_output>#
https://www.facebook.com/HrantDavtyan
https://www.linkedin.com/in/hrantdavtyan
#profile
#resume
#portfolio
#contact
http://aua.am/
http://isifa.am/en/ifa-dfi/
https://www.cerge-ei.cz/
http://www.fao.org/armenia/en/
http://www.metric.am/
http://ahpc.am/?lang=en
http://www.ucl.ac.uk/
http://iset.tsu.ge/






https://github.com/HrantDavtyan/HrantDavtyan.github.io\teaching\jdocs\Business Analytics\html\index.html
http://clearwaterbeachassoc.com/wp-content/uploads/2016/01/Under-construction.png
http://www.armstat.am/en/?nid=661
http://clearwaterbeachassoc.com/wp-content/uploads/2016/01/Under-construction.png
http://clearwaterbeachassoc.com/wp-content/uploads/2016/01/Under-construction.png
http://clearwaterbeachassoc.com/wp-content/uploads/2016/01/Under-construction.png
http://clearwaterbeachassoc.com/wp-content/uploads/2016/01/Under-construction.png
http://clearwaterbeachassoc.com/wp-content/uploads/2016/01/Under-construction.png<jupyter_text>Similarly, one can get all the p_tags from my page by just searching for All p-s as follows:<jupyter_code>p_tags = soup.findAll('p')
print(p_tags)<jupyter_output>[<p>I am a Data Enthusiast, teaching Business Analytics and providing consultancy on Statistics, Economics and IT. Feel free to take a look around my webpage.</p>, <p align="justify">I achieved an average feedback score of 4.7 (out of 5) teaching Business Analytics,
									Business Mathematics and Macroeconomics courses to overall 111 students.</p>, <p align="justify">Students with no prior knowledge of programming successfully learned Python, R, Stata and IBM SPSS
									and applied their skills by participating to a Kaggle competition as a final course project</p>, <p align="justify">I was awarded a one year fellowship to teach modern economics at local universities.
									During the fellowship tenure I taught around 230 students at 3 different universities.</p>, <p align="justify">Developed a software (using R, HTML, CSS) for Post Disaster Needs Assessment.
									Consulted contracted companies regarding Ministry staff appraisal and training needs assessment.
									[...]<jupyter_text>If you are interested only in paragraphs (text without tags) then you should again (as above in case of a_tags) iterate over the list and for each element of the list, get the text/string out of it as follows:<jupyter_code>for i in p_tags:
    print(i.text)<jupyter_output>I am a Data Enthusiast, teaching Business Analytics and providing consultancy on Statistics, Economics and IT. Feel free to take a look around my webpage.
I achieved an average feedback score of 4.7 (out of 5) teaching Business Analytics,
									Business Mathematics and Macroeconomics courses to overall 111 students.
Students with no prior knowledge of programming successfully learned Python, R, Stata and IBM SPSS
									and applied their skills by participating to a Kaggle competition as a final course project
I was awarded a one year fellowship to teach modern economics at local universities.
									During the fellowship tenure I taught around 230 students at 3 different universities.
Developed a software (using R, HTML, CSS) for Post Disaster Needs Assessment.
									Consulted contracted companies regarding Ministry staff appraisal and training needs assessment.
									Consulted National Statistical Service regarding Census (and post census) activities
Led several maj[...]
{'fraction_non_alphanumeric': 0.12375886524822695, 'fraction_numerical': 0.013002364066193853, 'mean_word_length': 5.239675516224189, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 94, 'https://': 8, 'lorem ipsum': 0, 'www.': 15, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '118541', 'license': 'Apache-2.0', 'n_tokens_mistral': 2781, 'n_tokens_neox': 2509, 'n_words': 863, 'path': 'Week 4/B_soup_1_(my_page).ipynb', 'repo_name': 'HrantDavtyan/Data_Scraping', 'size': '8467'}