<jupyter_start><jupyter_text>Web Scraping Yahoo! Finance using PythonA detailed guide for web scraping https://finance.yahoo.com/ using **Requests**, **BeautifulSoup**, **Selenium**, **HTML tags** & embedded **JSON** data.![](https://i.imgur.com/V1bzyMs.png)  Introduction**What is Web scraping?**Web scraping is the process of extracting and parsing data from websites in an automated fashion using a computer program. It's a useful technique for creating datasets for research and learning.**Objective**The main objective of this tutorial is to showcase different web scraping methods which can be applied to any web page. This is for educational purposes only. Please read the Terms & Conditions carefully for any website to see whether you can legally use the data. In this project, we will perform web scraping using the following 3 techniques based on the problem statement.* use `Requests`, `BeautifulSoup` and `HTML tags` to extract web page* use `Selenium` to scrape data from dynamically loading websites * use embedded `JSON` data to scrape website  **The problem statement**1. Web Scraping **Stock Market News** (url : https://finance.yahoo.com/topic/stock-market-news/)    This web page shows the latest **news** related to **stock market**, we will try to extract data from this web page and store it in a `CSV` (comma-separated values) file. The file layout would be as mentioned below.![](https://i.imgur.com/5I5SVFA.png)![](https://i.imgur.com/TnIWrXq.png) 2. Web Scraping **Cryptocurrencies** (url : https://finance.yahoo.com/cryptocurrencies)    This Yahoo! finance web page shows list of trending **Cryptocurrencies** in tabular format, we will perform the web scraping to retrieve first 10 columns for top 100 **Cryptocurrencies** in following `CSV` format.![](https://i.imgur.com/VInpTU8.png)![](https://i.imgur.com/KhPbg7S.png)             3. Web Scraping **Market Events Calendar** (url : https://finance.yahoo.com/calendar)     This page shows **date-wise market events**, user have the option to select the date and choose any one of the following market events **Earnings**, **Stock Splits**, **Economic Events** & **IPO**. Our aim is to create a script which can be run for any single date and market event which grabs the data and loads in `CSV` format as shown below.![](https://i.imgur.com/6ECn0AG.png)![](https://i.imgur.com/OAYlZ9I.png) **Prerequisites*** Knowledge of Python* Basic knowledge of HTML although it is not necessary **How to run the Code**You can execute the code using "Run" button on the top of this page and selecting **"Run on Colab"** or **"Run Locally"** **Setup and Tools**Run on Colab :     You will need to provide the Google login to run this notebook on Colab.Run Locally : Download and install [Anaconda](https://www.anaconda.com/) framework, We will be using Jupyter Notebook for writing & executing code. **Version control**You can make changes and save your version of the notebook to [Jovian](https://jovian.ai/) by executing following cells.<jupyter_code>!pip install jovian --quiet
import jovian
# Execute this to save new versions of the notebook
jovian.commit(project="yahoo-finance-web-scraper")<jupyter_output><empty_output><jupyter_text>1. Web Scraping Stock Market NewsIn this section we will learn a basic Python web scraping technique using `Requests`, `BeautifulSoup` and `HTML tags`. The objective here is to perform web scraping of [Yahoo! finance Stock Market News](https://finance.yahoo.com/topic/stock-market-news/)![](https://i.imgur.com/1I0Btau.jpg)Let's kick-start with the first objective. Here's an outline of the steps we'll follow**1.1 Download & Parse web page using `Requests` and `BeautifulSoup`****1.2 Exploring and locating Elements****1.3 Extract & Compile the information into python list****1.4 Save the extracted information to a CSV file**  1.1 Download & Parse webpage using Requests and BeautifulSoupFirst step is to install [`requests`](https://docs.python-requests.org/en/latest/) & [`beautifulsoup4`](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) Libraries using `pip`.<jupyter_code>!pip install requests --quiet
!pip install beautifulsoup4 --quiet

import requests
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>The libraries are installed and imported.To download the page, we can use `requests.get`, which returns a response object. We can access the content of the web page using `response.text`Also the `response.ok` & [`response.status_code`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) can be used for error trapping &  tracking. Finally, we can use `BeautifulSoup` to parse the HTML data. This will return `bs4.BeautifulSoup` object.<jupyter_code>my_url = 'https://finance.yahoo.com/topic/stock-market-news/'
response = requests.get(my_url)
print("response.ok : {} , response.status_code : {}".format(response.ok , response.status_code))
print("Preview of response.text : ", response.text[:500])<jupyter_output>Preview of response.text :  <!DOCTYPE html><html data-color-theme="light" id="atomic" class="NoJs chrome desktop failsafe" lang="en-US"><head prefix="og: http://ogp.me/ns#"><script>window.performance && window.performance.mark && window.performance.mark('PageStart');</script><meta charset="utf-8" /><title>Latest Stock Market News</title><meta name="keywords" content="401k, Business, Financial Information, Investing, Investor, Market News, Stock Research, Stock Valuation, business news, economy, finance, investment tools, m<jupyter_text>Let's create a function to perform this step.<jupyter_code>def get_page(url):
    """Download a webpage and return a beautiful soup doc"""
    response = requests.get(url)
    if not response.ok:
        print('Status code:', response.status_code)
        raise Exception('Failed to load page {}'.format(url))
    page_content = response.text
    doc = BeautifulSoup(page_content, 'html.parser')
    return doc<jupyter_output><empty_output><jupyter_text>calling function `get_page` and analyzing the output.<jupyter_code>doc = get_page(my_url)
print('Type of doc: ',type(doc))<jupyter_output>Type of doc:  <class 'bs4.BeautifulSoup'><jupyter_text>You can access different properties of HTML web page from doc. Following example will display Title of the web page.<jupyter_code>doc.find('title')<jupyter_output><empty_output><jupyter_text>We can use the function `get_page` to download any web page and parse it using beautiful soup.  1.2 Exploring and locating ElementsNow its time to explore the elements to find the required data point from the web page. Web pages are written in a language called HTML (Hyper Text Markup Language).  HTML is a fairly simple language comprised of *tags*  (also called *nodes* or *elements*) e.g. `Go to Yahoo! Finance`. An HTML tag has three parts:1. **Name**: (`html`, `head`, `body`, `div`, etc.) Indicates what the tag represents and how a browser should interpret the information inside it.2. **Attributes**: (`href`, `target`, `class`, `id`, etc.) Properties of tag used by the browser to customize how a tag is displayed and decide what happens on user interactions.3. **Children**: A tag can contain some text or other tags or both between the opening and closing segments, e.g., `Some content`. Let's inspect the webpage source code by right-clicking and selecting the "Inspect" option. First, we need to identify the tag which represents the news listing.![](https://i.imgur.com/pGwXU1J.gif) In this case we can see the `` tag having class name `"Ov(h) Pend(44px) Pstart(25px)"` is representing news listing. We can apply `find_all` method to grab this information<jupyter_code>div_tags = doc.find_all('div', {'class': "Ov(h) Pend(44px) Pstart(25px)"})<jupyter_output><empty_output><jupyter_text>Total elements in the `` tag list matching with the numbers of news displaying on the webpage, so we are heading towards the right direction.<jupyter_code>len(div_tags)<jupyter_output><empty_output><jupyter_text>Next step to inspect the individual `` tag and try to find more information. I am using "Visual Studio Code", but you can use any tool as simple as notepad.<jupyter_code>print(div_tags[1])<jupyter_output><div class="Ov(h) Pend(44px) Pstart(25px)"><div class="C(#959595) Fz(11px) D(ib) Mb(6px)">Bloomberg</div><h3 class="Mb(5px)"><a class="js-content-viewer wafer-caas Fw(b) Fz(18px) Lh(23px) LineClamp(2,46px) Fz(17px)--sm1024 Lh(19px)--sm1024 LineClamp(2,38px)--sm1024 mega-item-header-link Td(n) C(#0078ff):h C(#000) LineClamp(2,46px) LineClamp(2,38px)--sm1024 not-isInStreamVideoEnabled" data-uuid="ee555bdf-8ced-3ec5-8527-f11ab477237d" data-wf-caas-prefetch="1" data-wf-caas-uuid="ee555bdf-8ced-3ec5-8527-f11ab477237d" href="/news/bond-yields-jump-asia-stocks-223246963.html"><u class="StretchedBox"></u>U.S. Index Futures Drop as Yields, Netflix Watched: Markets Wrap</a></h3><p class="Fz(14px) Lh(19px) Fz(13px)--sm1024 Lh(17px)--sm1024 LineClamp(2,38px) LineClamp(2,34px)--sm1024 M(0)">(Bloomberg) -- U.S. index futures fell as investors weighed the impact of rising real yields on the appeal of riskier assets and Netflix Inc. damped the earnings-season outlook. European stocks rose with focus t[...]<jupyter_text>![](https://i.imgur.com/ncnfg0z.png)Luckily, most of the required data points are available in this ``, so we can use `find` method to grab each item.<jupyter_code>print("Source: ", div_tags[1].find('div').text)
print("Head Line : {}".format(div_tags[1].find('a').text))<jupyter_output>Source:  Bloomberg
Head Line : U.S. Index Futures Drop as Yields, Netflix Watched: Markets Wrap<jupyter_text>If any tag is not accessible directly, then you can use methods like `findParent()` or `'findChild()` to point to the required tag.![](https://i.imgur.com/OnOAtT2.png)<jupyter_code>print("Image URL: ",div_tags[1].findParent().find('img')['src'])<jupyter_output>Image URL:  https://s.yimg.com/uu/api/res/1.2/sLZ4pDqUIX51l.JGdlyleg--~B/Zmk9c3RyaW07aD0xMjM7cT04MDt3PTIyMDthcHBpZD15dGFjaHlvbg--/https://s.yimg.com/uu/api/res/1.2/HAUckSPP2QERhFOj_FkbiQ--~B/aD0xMzM0O3c9MjAwMDthcHBpZD15dGFjaHlvbg--/https://media.zenfs.com/en/bloomberg_markets_842/25526e5283a4af65d4c90846e175bb89.cf.jpg<jupyter_text>Key Takeout from this exercise is to identify the optimal tag which will provide us required information. Mostly this is straight forward, but sometimes you will have to perform a little more research.    1.3 Extract & Compile the information into python listWe've identified all the required tags and information. Let's put this together in the functions.<jupyter_code>def get_news_tags(doc):
    """Get the list of tags containing news information"""
    news_class = "Ov(h) Pend(44px) Pstart(25px)" ## class name of div tag 
    news_list  = doc.find_all('div', {'class': news_class})
    return news_list<jupyter_output><empty_output><jupyter_text>sample run of the function `get_news_tags`<jupyter_code>my_news_tags = get_news_tags(doc)<jupyter_output><empty_output><jupyter_text>we will create one more function, to parse individual `` tags and return the information in dictionary form<jupyter_code>BASE_URL = 'https://finance.yahoo.com' #Global Variable 

def parse_news(news_tag):
    """Get the news data point and return dictionary"""
    news_source = news_tag.find('div').text #source
    news_headline = news_tag.find('a').text #heading
    news_url = news_tag.find('a')['href'] #link
    news_content = news_tag.find('p').text #content
    news_image = news_tag.findParent().find('img')['src'] #thumb image
    return { 'source' : news_source,
            'headline' : news_headline,
            'url' : BASE_URL + news_url,
            'content' : news_content,
            'image' : news_image
           }<jupyter_output><empty_output><jupyter_text>Testing the `parse_news` function for first `` tag<jupyter_code>parse_news(my_news_tags[0])<jupyter_output><empty_output><jupyter_text>We can use the `get_news_tags` & `parse_news` functions to pars news.  1.4 Save the extracted information to a CSV fileThis is the last step of this section. We are going to use Python library [`pandas`](https://pandas.pydata.org/docs/) to save the data in CSV format. Install and then import the pandas Library.<jupyter_code>!pip install pandas --upgrade --quiet

import pandas as pd<jupyter_output><empty_output><jupyter_text>Creating wrapper function which will call previously created helper functions.The `get_page` function will download HTML page, then we can pass the result in `get_news_tags` to identify list of `` tags for news.After that we will use [List Comprehension](https://www.w3schools.com/python/python_lists_comprehension.asp) technique to parse each `` tag using `parse_news`, the output will be in the form of `lists` of `dictionaries`Finally, we will use `DataFrame` method to create pandas [dataframe](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) and use `to_csv` method to store required data in CSV format.<jupyter_code>def scrape_yahoo_news(url, path=None):
    """Get the yahoo finance market news and write them to CSV file """
    if path is None:
        path = 'stock-market-news.csv'
        
    print('Requesting html page')
    doc = get_page(url)

    print('Extracting news tags')
    news_list = get_news_tags(doc)

    print('Parsing news tags')
    news_data = [parse_news(news_tag) for news_tag in news_list]

    print('Save the data to a CSV')
    news_df = pd.DataFrame(news_data)
    news_df.to_csv(path, index=None)
    
    #This return statement is optional, we are doing this just analyze the final output 
    return news_df<jupyter_output><empty_output><jupyter_text>Scraping the news using `scrape_yahoo_news` function<jupyter_code>YAHOO_NEWS_URL = BASE_URL+'/topic/stock-market-news/'
news_df = scrape_yahoo_news(YAHOO_NEWS_URL)<jupyter_output>Requesting html page
Extracting news tags
Parsing news tags
Save the data to a CSV<jupyter_text>The "stock-market-news.csv" should be available in File $\rightarrow$ Open Menu. You can download the file or directly open it on browser. Please verify the file content and compare it with the actual information available on the webpage.You can also check the data by grabbing a few rows from the data frame returned by the `scrape_yahoo_news` function<jupyter_code>news_df.head()<jupyter_output><empty_output><jupyter_text>**Summary** : Hopefully I was able to explain this simple but very powerful Python technique to scrape the Yahoo! finance market news. These steps can be used to scrape any web page. You just have to do a little research to identify the required `` and use relevant python methods to collect the data.   2. Web Scraping CryptocurrenciesIn phase One we were able to scrape the [yahoo market news](https://finance.yahoo.com/topic/stock-market-news/) web page. However, if you've noticed, as we scroll down the webpage more news will appear at the bottom of the page. This is called dynamic page loading. The previous technique is a basic Python method useful to scrape static data, To scrape the dynamically loading data will use a different method called web scraping using **Selenium**. Let's move ahead with this topic. The goal of this section is to extract top listing [Crypto currencies](https://finance.yahoo.com/cryptocurrencies) from Yahoo! finance. ![](https://i.imgur.com/sF6k0Pk.jpg)Here's an outline of the steps we'll follow**2.1 Introduction of selenium****2.2 Downloads & Installation****2.3 Install & Import libraries****2.4 Create Web Driver****2.5 Exploring and locating Elements****2.6 Extract & Compile the information into a python list****2.7 Save the extracted information to a CSV file**  2.1 Introduction of selenium**[Selenium](https://www.selenium.dev/)** is an open-source web-based automation tool. Python language and other languages are used with Selenium for testing as well as web scraping. Here we will use Chrome browser, but you can try on any browser.**Why you should use Selenium?**- Clicking on buttons- Filling forms- Scrolling- Taking a screen-shot- Refreshing the pageYou can find proper documentation on selenium [here](https://selenium-python.readthedocs.io/)The following methods will help to find elements in a webpage (these methods will return a list):- `find_elements_by_name`- `find_elements_by_xpath`- `find_elements_by_link_text`- `find_elements_by_partial_link_text`- `find_elements_by_tag_name`- `find_elements_by_class_name`- `find_elements_by_css_selector`In this tutorial we will use only `find_elements_by_xpath` and `find_elements_by_tag_name` You can find complete documentation of these methods [here](https://selenium-python.readthedocs.io/locating-elements.html)  2.2 Downloads & Installation Unlike the previous section, here we'll have to do some prep work to implement this method. We will need to install Selenium & proper web browser driverIf you are using **Google Colab** platform then execute following code to perform Initial installation. This piece of code `'google.colab' in str(get_ipython())` is used to identify the Google Colab platform.<jupyter_code>if 'google.colab' in str(get_ipython()):
    print('Google CoLab Installation')
    !apt update --quiet
    !apt install chromium-chromedriver --quiet<jupyter_output><empty_output><jupyter_text>To run it on **Locally** you will need **Webdriver for Chrome** on your machine. You can download it from this link https://chromedriver.chromium.org/downloads and just copy the file in the folder where we will create the python file (No need of installation). But make sure that the driver‘s version matches the Chrome browser version installed on the local machine.![](https://i.imgur.com/FvQ586e.gif)![](https://i.imgur.com/wQbjRIU.png)  2.3 Install & Import librariesInstallation of the required libraries.<jupyter_code>!pip install selenium --quiet
!pip install pandas --quiet<jupyter_output><empty_output><jupyter_text>Once the Libraries installation is done, next step is to import all the required modules / libraries.<jupyter_code>print('Library Import')
if 'google.colab' not in str(get_ipython()):
    print('Not running on CoLab')
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.chrome.service import Service
    import os
else:
    print('Running on CoLab')
    
print('Common Library Import')
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd 
import time<jupyter_output><empty_output><jupyter_text>So all the necessary prep work is done. Let's  move ahead to implement this method.  2.4 Create Web DriverIn this step first we will create the instance of Chrome WebDriver using `webdriver.Chrome()` method. and then the `driver.get()` method will navigate to a page given by the URL. In this case also there is slight variation based on platform. Also we have used `options` parameters for e.g. `--headless` option will load the driver in background.<jupyter_code>if 'google.colab' in str(get_ipython()):
    print('Running on CoLab')
    def get_driver(url):
        """Return web driver"""
        colab_options = webdriver.ChromeOptions()
        colab_options.add_argument('--no-sandbox')
        colab_options.add_argument('--disable-dev-shm-usage')
        colab_options.add_argument('--headless')
        colab_options.add_argument('--start-maximized') 
        colab_options.add_argument('--start-fullscreen')
        colab_options.add_argument('--single-process')
        driver = webdriver.Chrome(options=colab_options)
        driver.get(url)
        return driver
else:
    print('Not running on CoLab')
    def get_driver(url):
        """Return web driver"""
        chrome_options = Options()
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-dev-shm-usage')
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--start-maximized') 
        chrome_options.add_argument('--start-fullscreen')
        chrome_options.add_argument('--single-process')
        serv = Service(os.getcwd()+'/chromedriver')
        driver = webdriver.Chrome(options=chrome_options, service=serv)
        driver.get(url)
        return driver<jupyter_output><empty_output><jupyter_text>Test run of `get_driver`<jupyter_code>driver = get_driver('https://finance.yahoo.com/cryptocurrencies')
print(driver.title)<jupyter_output><empty_output><jupyter_text>2.5 Exploring and locating ElementsThis is almost similar step that we have done in phase 1. We will try to identify relevant information like ``, `class` , `XPath` etc from the web page. Right-click and select the "Inspect" to do further analysis.As the webpage showing cryptocurrency information in the Table form. We can grab the table header by using tag ``, we will use find_elements by TAG to get the table headers. These headers can be used as columns for a CSV file.<jupyter_code>header = driver.find_elements(By.TAG_NAME, value= 'th')
print(header[0].text)
print(header[2].text)<jupyter_output><empty_output><jupyter_text>Creating a helper function to get first 10 columns from header, we have used List comprehension with conditions. You can also check out usage of `enumerate` method.<jupyter_code>def get_table_header(driver):
    """Return Table columns in list form """
    header = driver.find_elements(By.TAG_NAME, value= 'th')
    header_list = [item.text for index, item in enumerate(header) if index < 10]
    return header_list<jupyter_output><empty_output><jupyter_text>Next we find out number of rows available in a Page, you can see table rows are placed in `` tag, we can capture the `XPath` by selection `` tag the Right Click $\rightarrow$ Copy $\rightarrow$ Copy XPath.![](https://i.imgur.com/DVAYMzY.gif) So we get the  XPath value as `//*[@id="scr-res-table"]/div[1]/table/tbody/tr[1]`, Let's use this with `find_element()` & `By.XPATH`.<jupyter_code>txt=driver.find_element(By.XPATH, value='//*[@id="scr-res-table"]/div[1]/table/tbody/tr[1]').text
txt<jupyter_output><empty_output><jupyter_text>Above `XPath` points to first row, we can get rid of row number part from XPath and use it with `find_elements` to get hold of all the available rows. Let's implement this in a function.<jupyter_code>def get_table_rows(driver):
    """Get number of rows available on the page """
    tablerows = len(driver.find_elements(By.XPATH, value='//*[@id="scr-res-table"]/div[1]/table/tbody/tr'))
    return tablerows    
print(get_table_rows(driver))<jupyter_output><empty_output><jupyter_text>Similarly, we can take the XPath for any column value.![](https://i.imgur.com/aT3I3Ur.gif) This is the XPAth for a column `//*[@id="scr-res-table"]/div[1]/table/tbody/tr[1]/td[2]`.If you noticed the number after `tr` & `td` represents the `row_number` and `column_number`, we can check this with `find_element()` method<jupyter_code>driver.find_element(By.XPATH, value='//*[@id="scr-res-table"]/div[1]/table/tbody/tr[1]/td[2]').text<jupyter_output><empty_output><jupyter_text>So we can change the `row_number` & `column_number` in `XPath` and loop it through row count and column count to get all the available column values. Let's generalize this and put it in a function. We will get the data for one row at a time and return column values in the form of a dictionary<jupyter_code>def parse_table_rows(rownum, driver, header_list):
    """get the data for one row at a time and return column value in the form of dictionary"""
    row_dictionary = {}
    #time.sleep(1/3)
    for index , item in enumerate(header_list):
        time.sleep(1/20)
        column_xpath = '//*[@id="scr-res-table"]/div[1]/table/tbody/tr[{}]/td[{}]'.format(rownum, index+1)
        row_dictionary[item] = driver.find_element(By.XPATH, value=column_xpath).text
    return row_dictionary<jupyter_output><empty_output><jupyter_text>The Yahoo! Finance web page shows only 25 Cryptocurrencies per page and user will have to click `Next` button to load next set of crypto currencies. This is called **Pagination**. This is the main reason we are implementing selenium methods to handle events like pagination. you can perform multiple events like clicking, scrolling , refreshing etc. on a webpage using selenium methods.Now we will grab the `XPath` of `Next` button, find the element using `find_element` method, and after that we can perform click action using `.click()` method ![](https://i.imgur.com/tCxQKfR.gif)<jupyter_code>button_element = driver.find_element(By.XPATH, value = '//*[@id="scr-res-table"]/div[2]/button[3]')
button_element.click()
txt=driver.find_element(By.XPATH, value='//*[@id="scr-res-table"]/div[1]/table/tbody/tr[1]').text
txt<jupyter_output><empty_output><jupyter_text>Now I am trying to check the first row on the web page to verify if `.click()` really worked, and you will see first row has changed. Click action was successful.  In this section we have learned how to get required data points, and perform events on webpage.<jupyter_code>#terminating driver from test runs 
driver.close()
driver.quit()<jupyter_output><empty_output><jupyter_text>2.6 Extract & Compile the information into python listLet's put all the pieces in the puzzle, we will pass the integer `total_crypto` i.e. numbers of rows to be scraped (in this case 100 rows) in the function. Parse each row from the page and append the data in the `List` till the total parsed row count reach to `total_crypto`. In addition, we will perform `Next` button click if we are at the last row of the table. **Please Note** : Here to identify the `Next` button element we have used [WebDriverWait](https://www.selenium.dev/selenium/docs/api/java/org/openqa/selenium/support/ui/WebDriverWait.html) class instead of using `find_element()` method. In this technique we can pass some wait-time before grabbing the element. This type of implementation is done to avoid the [`StaleElementReferenceException`](https://stackoverflow.com/questions/27003423/staleelementreferenceexception-on-python-selenium).Code Sample:```element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="scr-res-table"]/div[2]/button[3]')))```<jupyter_code>def parse_multiple_pages(driver, total_crypto):
    """Loop through each row, perform Next button click at the end of page 
    return total_crypto numbers of rows 
    """
    table_data = []
    page_num = 1
    is_scraping = True
    header_list = get_table_header(driver)

    while is_scraping:
        table_rows = get_table_rows(driver)
        print('Found {} rows on Page : {}'.format(table_rows, page_num))
        print('Parsing Page : {}'.format(page_num))
        table_data += [parse_table_rows(i, driver, header_list) for i in range (1, table_rows + 1)]
        total_count = len(table_data)
        print('Total rows scraped : {}'.format(total_count))
        if total_count >= total_crypto:
            print('Done Parsing..')
            is_scraping = False
        else:    
            print('Clicking Next Button')
            element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="scr-res-table"]/div[2]/button[3]')))
            element.click() 
            page_num += 1
    return table_data<jupyter_output><empty_output><jupyter_text>2.7 Save the extracted information to a CSV file.This is the last step of this section, we are creating a last function which will be the placeholder for all helper functions and at the and we will save the data in CSV format using `pd.to_csv` method.<jupyter_code>def scrape_yahoo_crypto(url, total_crypto, path=None):
    """Get the list of yahoo finance crypto-currencies and write them to CSV file """
    if path is None:
        path = 'crypto-currencies.csv'
    print('Creating driver')
    driver = get_driver(url)    
    table_data = parse_multiple_pages(driver, total_crypto)
    driver.close()
    driver.quit()
    print('Save the data to a CSV')
    table_df = pd.DataFrame(table_data)
    table_df.to_csv(path, index=None)
    #This return statement is optional, we are doing this just analyze the final output 
    return table_df<jupyter_output><empty_output><jupyter_text>Time to scrape some cryptos!!! , we will scrape top 100 cryptos in Yahoo! Finance webpage by calling `scrape_yahoo_crypto`<jupyter_code>YAHOO_FINANCE_URL = BASE_URL+'/cryptocurrencies'
TOTAL_CRYPTO = 100
crypto_df = scrape_yahoo_crypto(YAHOO_FINANCE_URL, TOTAL_CRYPTO,'crypto-currencies.csv')<jupyter_output><empty_output><jupyter_text>The "crypto-currencies.csv" should be available in File $\rightarrow$ Open Menu. You can download the file or directly open it on browser. Please verify the file content and compare it with the actual information available on the webpage.You can also check the data by grabbing a few rows from the data frame returned by the `scrape_yahoo_crypto` function<jupyter_code>crypto_df.head()<jupyter_output><empty_output><jupyter_text>**Summary** : Hope you've enjoyed this tutorial. Selenium enables us to perform multiple actions on the web browser, which is really very handy for scraping different types of data from any webpage.  3. Web Scraping Market Events CalendarThis is the final segment of the tutorial in this section, we will learn how to extract embedded [JSON](https://www.w3schools.com/js/js_json_intro.asp) formatted data which can be easily converted to Python dictionary. Problem statement for section is to scrape date-wise market events from [Yahoo! finance](https://finance.yahoo.com/calendar). ![](https://i.imgur.com/bKQoAjs.png)Here's an outline of the steps we'll follow**3.1 Install & Import libraries****3.2 Download & Parse web page****3.3 Get Embedded Json data****3.4 Locating Json Keys****3.5 Pagination & Compiling the information into a python list****3.6 Save the extracted information to a CSV file**  3.1 Install & Import librariesFirst step to install and import Python Libraries<jupyter_code>!pip install requests --quiet
!pip install beautifulsoup4 --quiet
!pip install pandas --quiet
import re
import json
import requests
import pandas as pd
from io import StringIO
from bs4 import BeautifulSoup
from IPython.display import display<jupyter_output><empty_output><jupyter_text>3.2 Download & Parse web pageThis is exactly the same step that we've performed to download webpage in section 1.1, Here we have used [custom header](https://docs.python-requests.org/en/master/user/quickstart/custom-headers) in `requests.get()`Most of the things are explained in section 1.1, creating the helper function.<jupyter_code>def get_event_page(scraper_url):
    """Download a webpage and return a beautiful soup doc"""
    headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
                  "(KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
    }
    response = requests.get(scraper_url, headers=headers)
    if not response.ok:
        print('Status code:', response.status_code)
        raise Exception('Failed to fetch web page ' + scraper_url)
    # Construct a beautiful soup document
    doc = BeautifulSoup(response.text, 'html.parser')
    return doc
doc = get_event_page('https://finance.yahoo.com/calendar/earnings?from=2022-02-27&to=2022-03-05&day=2022-02-28')
print(doc.find('title'))<jupyter_output><title>Company Earnings Calendar - Yahoo Finance</title><jupyter_text>3.3 Get Embedded Json dataIn this step we will locate the Jason formatted data, Open the web page and do Right Click $\rightarrow$ View Page Source, If you scroll down to source page you will notice the [Json](https://www.w3schools.com/whatis/whatis_json.asp) formatted data. Apparently this information is `` tag which contains the following text `/* -- Data -- */`.![](https://i.imgur.com/2xlpNbw.gif) We will use [Regular expressions](https://docs.python.org/3/library/re.html) to get text  inside `` tag.<jupyter_code>pattern = re.compile(r'\s--\sData\s--\s')
#script_data = doc.find('script', text=pattern).text
script_data = doc.find('script', text=pattern).contents[0]<jupyter_output><empty_output><jupyter_text>Further, the `Json` formatted string has the first key as `context` and it ends at 12 characters from the end<jupyter_code>print(script_data[:150])
print(script_data[-150:])<jupyter_output>(function (root) {
/* -- Data -- */
root.App || (root.App = {});
root.App.now = 1650440790947;
root.App.main = {"context":{"dispatcher":{"stores":{"P
odal":{"strings":1},"tdv2-wafer-header":{"strings":1},"yahoodotcom-layout":{"strings":1}}},"options":{"defaultBundle":"td-app-finance"}}}};
}(this));<jupyter_text>We can grab the Json string using Python slicing.<jupyter_code>start  = script_data.find('context')-2
json_text  = script_data[start:-12]
print(json_text[:100])<jupyter_output>{"context":{"dispatcher":{"stores":{"PageStore":{"currentPageName":"calendar","currentEvent":{"event<jupyter_text>Using `json.loads()`method to convert Jason string into Python Dictionary<jupyter_code>parsed_dictionary = json.loads(json_text)
type(parsed_dictionary)<jupyter_output><empty_output><jupyter_text>Creating a function using above information.<jupyter_code>def get_json_dictionary(doc):
    """Get Json formated data in the form of Python Dictionary"""
    pattern = re.compile(r'\s--\sData\s--\s')
    script_data = doc.find('script', text=pattern).text
    script_data = doc.find('script', text=pattern).contents[0]
    
    start  = script_data.find('context')-2
    json_text  = script_data[start:-12]
    
    parsed_dictionary = json.loads(json_text)
    return parsed_dictionary<jupyter_output><empty_output><jupyter_text>3.4 Locating Json KeysSo basically the Json text is multi level nested dictionaries, and some keys are used to store all the meta data displayed on the webpage. In this section we will identify the keys for the data we are trying to scrape.We'll need some `Json Formatter` tool to navigate through multiple keys, I am using online tool https://jsonblob.com/. However, you can choose any tool.We will write the Json text into `my_json_file.json` file, then grab the file content and paste it to the left panel of https://jsonblob.com/. The JSON Blob it will do nice formatting. We can easily navigate through each Keys and search any item.<jupyter_code>with open('my_json_file.json', 'w', encoding="utf-8") as file:
    file.write(json_text)<jupyter_output><empty_output><jupyter_text>Next step is to find the Required Key location. Let's search the company name `3D Systems Corporation` displayed in the webpage in the [JSON Blob](https://jsonblob.com/) formatter.![](https://i.imgur.com/Iv8b7vl.png) ![](https://i.imgur.com/jpJYOCy.png) You can see the table data is stored in the `rows` key, and we can track down the parent keys as shown in the above screen, checkout the content of `row` key.<jupyter_code>parsed_dictionary['context']['dispatcher']['stores']['ScreenerResultsStore']['results']['rows'][:3]
print('Total Rows on the Current page :',len(parsed_dictionary['context']['dispatcher']['stores']['ScreenerResultsStore']['results']['rows']))<jupyter_output>Total Rows on the Current page : 100<jupyter_text>This sub-dictionary shows all the data displayed on the current page.You can do more research and exploration to get different information from the web page.<jupyter_code>print('Total Rows for the search criteria :',parsed_dictionary['context']['dispatcher']['stores']['ScreenerResultsStore']['results']['total'])
print("Columns")
parsed_dictionary['context']['dispatcher']['stores']['ScreenerResultsStore']['results']['columns']<jupyter_output>Columns<jupyter_text>Putting this in a function<jupyter_code>def get_total_rows(parsed_dictionary):
    '''Get the Total Rows for the search criteria & Columns detail''' 
    total_rows = parsed_dictionary['context']['dispatcher']['stores']['ScreenerResultsStore']['results']['total']
    return total_rows
def get_page_rows(parsed_dictionary):
    """Get the Content current page"""    
    data_dictionary = parsed_dictionary['context']['dispatcher']['stores']['ScreenerResultsStore']['results']['rows']
    return data_dictionary<jupyter_output><empty_output><jupyter_text>3.5 Pagination & Compiling the information into python listAs we saw in the previous section on how to handle `Pagination` using selenium methods, here we'll learn a new technique for accessing multiple pages.Most of the times webpage url gets changed at runtime depending on the user selection, e.g. In the below screen-shot, I selected the **Earnings** for **1-March-2022**. You can notice how that information is passed in the URL. ![](https://i.imgur.com/h5QU99h.png) Similarly, when i click next button, `offset`& `size` values gets changed in the url.![](https://i.imgur.com/jYa1vq5.png) So we can figure out the pattern & structure of the url and how it affects page navigation. In this case webpage url pattern is mentioned below:- The following values are used for calendar event types `event_types = ['splits','economic','ipo','earnings']`- Date passed in `yyyy-mm-dd` format- Page number is controlled by `offset` value (for first page `offset=0`)- Maximum number of rows in a page is assigned to `size`Based on the above information, we can build the URL at runtime and download the page, then extract the information. This is how we handle pagination.Putting all things together in a function. In this function we will pass `event_type` and `date`, then we will calculate the total rows for matching criteria using `get_total_rows` function. Maximum rows per page are constant (i.e. 100), so we can build iterating summation logic to calculate the total number of pages involved for the current criteria and extract each page data in the loop.<jupyter_code>def scrape_all_pages(event_type, date):
    """Loop through each row and return lists of data dictionary"""
    YAHOO_CAL_URL = BASE_URL+'/calendar/{}?day={}&offset={}&size={}'
    max_rows_per_page = '100' # this indicates max rows per page 
    page_number = 1
    final_data_dictionary = []
    
    while page_number > 0:
        print("Processing page # {}".format(page_number))
        page_url = str((page_number - 1 ) * int(max_rows_per_page))
        scrape_url = YAHOO_CAL_URL.format(event_type, date, page_url, max_rows_per_page)
        print("Scrape url for page {} is {}".format(page_number,scrape_url))
        page_doc = get_event_page(scrape_url)
        parse_dict = get_json_dictionary(page_doc)
        if page_number == 1:
            total_rows = get_total_rows(parse_dict)        
        final_data_dictionary += get_page_rows(parse_dict)
        if len(final_data_dictionary) >= total_rows:
            page_number = 0
            return final_data_dictionary
        page_number += 1<jupyter_output><empty_output><jupyter_text>creating one more variant of same function for the date range<jupyter_code>def scrape_all_pages_daterange(event_type, dates):
    """Loop through each row and return lists of data dictionary"""
    YAHOO_CAL_URL = BASE_URL+'/calendar/{}?day={}&offset={}&size={}'
    max_rows_per_page = '100' # this indicates max rows per page 
    data_dictionary_date = []
    for date in dates:
        print("Processing for date : {}".format(date))
        page_number = 1
        final_data_dictionary = []
    
        while page_number > 0:
            print("Processing page # {}".format(page_number))
            page_url = str((page_number - 1 ) * int(max_rows_per_page))
            scrape_url = YAHOO_CAL_URL.format(event_type, date, page_url, max_rows_per_page)
            print("Scrape url for page {} is {}".format(page_number,scrape_url))
            page_doc = get_event_page(scrape_url)
            parse_dict = get_json_dictionary(page_doc)
            if page_number == 1:
                total_rows = get_total_rows(parse_dict)        
            final_data_dictionary += get_page_rows(parse_dict)
            if len(final_data_dictionary) >= total_rows:
                page_number = 0
                data_dictionary_date += [{'primary-key':date, **item} for item in final_data_dictionary]
            else:    
                page_number += 1
        print("Processing done")    
    return data_dictionary_date<jupyter_output><empty_output><jupyter_text>3.6 Save the extracted information to a CSV file.In this last section, we will save the data to csv format using `pd.DataFrame()` & `to_csv()` and call everything in a single placeholder function.<jupyter_code>def scrape_yahoo_calendar(event_types, date_param):
    """Get the list of yahoo finance calendar and write them to CSV file """
    for event in event_types:
        data_dict = {}
        print('Web Scraping for ', event  )
        data_dict = scrape_all_pages(event, date_param)
        if len(data_dict) > 0:
            scraped_df = pd.DataFrame(data_dict)
            scraped_df.to_csv(event+'_'+date_param+'.csv',index=False)
            print("checking few rows.. for event : {} & date : {}".format(event, date_param))
            display(scraped_df.head())
        else:
            print("No data found for event : {} & date : {}".format(event, date_param))<jupyter_output><empty_output><jupyter_text>creating one more variant of same function for the date range<jupyter_code>def scrape_yahoo_calendar_daterange(event_types, date_param):
    """Get the list of yahoo finance calendar and write them to CSV file """
    for event in event_types:
        data_dict = {}
        print('Web Scraping for ', event  )
        data_dict = scrape_all_pages_daterange(event, date_param)
        if len(data_dict) > 0:
            scraped_df = pd.DataFrame(data_dict)
            scraped_df.to_csv(event+'.csv',index=False)
            print("checking few rows.. for event : {}".format(event))
            display(scraped_df.head())
        else:
            print("No data found for event : {} ".format(event))<jupyter_output><empty_output><jupyter_text>calling the final function `scrape_yahoo_calendar_daterange`<jupyter_code>BASE_URL = 'https://finance.yahoo.com' #Global Variable
date_param = ['2022-02-28','2022-03-01','2022-03-02']
event_types = ['splits','economic','ipo','earnings']
scrape_yahoo_calendar_daterange(event_types, date_param)<jupyter_output>Web Scraping for  splits
Processing for date : 2022-02-28
Processing page # 1
Scrape url for page 1 is https://finance.yahoo.com/calendar/splits?day=2022-02-28&offset=0&size=100
Processing done
Processing for date : 2022-03-01
Processing page # 1
Scrape url for page 1 is https://finance.yahoo.com/calendar/splits?day=2022-03-01&offset=0&size=100
Processing done
Processing for date : 2022-03-02
Processing page # 1
Scrape url for page 1 is https://finance.yahoo.com/calendar/splits?day=2022-03-02&offset=0&size=100
Processing done
checking few rows.. for event : splits<jupyter_text>calling the final function `scrape_yahoo_calendar`<jupyter_code>BASE_URL = 'https://finance.yahoo.com' #Global Variable 
#date_param = '2022-03-18' # no data condition
date_param = '2022-02-28'
event_types = ['splits','economic','ipo','earnings']
scrape_yahoo_calendar(event_types, date_param)<jupyter_output>Web Scraping for  splits
Processing page # 1
Scrape url for page 1 is https://finance.yahoo.com/calendar/splits?day=2022-02-28&offset=0&size=100
checking few rows.. for event : splits & date : 2022-02-28<jupyter_text>Total 4 csv files "event_type_yyyy-mm-dd.csv" should be available in File $\rightarrow$ Open Menu. You can download the file or directly open it in a browser. Please verify the file content and compare it with the actual information available on the webpage. **Summary** : This is a very useful technique which can be easily replicable. Without writing any customized code, we were able to extract the data from multiple types of web pages just by changing one variable (in this case `event_type`).   ReferencesReferences to some useful links.- https://htmldog.com/guides/html/- https://selenium-python.readthedocs.io/index.html- https://stackoverflow.com/questions/27003423/staleelementreferenceexception-on-python-selenium- https://www.w3schools.com/js/js_json_intro.asp- https://hhsm95.dev/blog/the-importance-of-using-user-agent-to-scraping-data/  Future WorkIdeas for future work- Automate this process using [AWS Lambda](https://aws.amazon.com/lambda/) to download daily market calendar, crypto-currencies & market news in CSV format.- Move the old files to an Archive folder append date-stamp to the file if required, also  delete the Archived files older than 2 weeks.- Process the raw data extracted from third technique using different methods of pandas   ConclusionIn this tutorial we implement the following web scraping techniques. - Use Requests, BeautifulSoup and HTML tags to extract web page. - Use Selenium to scrape data from dynamically loading websites. - Use embedded JSON data to scrape website.I hope I was able to teach you these webscraping methods and I hope you can use this knowledge to scrape any website.Thank you for reading. Happy coding!!!<jupyter_code>jovian.commit(project="yahoo-finance-web-scraper",files=['stock-market-news.csv','crypto-currencies.csv','splits_2022-02-28.csv','economic_2022-02-28.csv','ipo_2022-02-28.csv','earnings_2022-02-28.csv'])
jovian.commit(project="yahoo-finance-web-scraper",files=['yahooscraper.py','lambda_function.py'])
jovian.commit(project="yahoo-finance-web-scraper")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10841384774522966, 'fraction_numerical': 0.015361929704866079, 'mean_word_length': 4.848629167202986, 'pattern_counts': {'":': 18, '<': 227, '<?xml version=': 0, '>': 232, 'https://': 71, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '54', 'dir': 'jupyter-structured-clean-dedup', 'id': '138776', 'license': 'MIT', 'n_tokens_mistral': 14265, 'n_tokens_neox': 13360, 'n_words': 4509, 'path': 'yahoo-finance-web-scraper.ipynb', 'repo_name': 'vinodvidhole/yahoo-finance-scraper', 'size': '45437'}