<jupyter_start><jupyter_text>Scraping Rate.amThis notebook provides code for scraping rates from rate.am. The rates are provided inside an HTML table, thus **pandas.read_html()** function is probably the most user friendly method of extrating infromation from rate.am. However, as one may be interested in extracting information from similar websites with interactive components driven by JavaScript, we use Selenium here first to make some actions and get page soruce and then only use pandas for scraping and manipulation.Selenium functions and methods will be additionally posted in a separate document.Key points:- browser.page_source - provides the HTML source of the page loaded by Selenium,- browser.current_url - provides the URL of the page where Selenium has navigated (maybe different from the base URL has the programmer may aks Selenium to click buttons or follow links),- find_element_by_xpath() - Selenium method for finding HTML elements using Xpath approach- send_keys(Keys.PAGE_DOWN) - tells Selenium to "press" Page Down key on keyboard- browser.implicitly_wait(30) - tells Selenium to wait 30 seconds for some action to be completed.<jupyter_code>import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser = webdriver.Chrome()
url = "http://rate.am/en/armenian-dram-exchange-rates/banks/cash"
browser.get(url) #will wait until page is fully loaded
browser.find_element_by_xpath("//label[contains(text(),'Non-cash')]").click()
#browser.current_url
page = browser.page_source
browser.close()
all_tables = pd.read_html(page)
all_tables[2]
cols = [i for i in range(5,13)]
cols.append(1)
all_tables[2].iloc[2:19,cols]<jupyter_output><empty_output><jupyter_text>Starting from here we introduce several Selenium tricks for manipulating the page (such as clicking the Page Down key on the keyboard).<jupyter_code>browser = webdriver.Chrome()
browser.get(url)
button = browser.find_element_by_tag_name("html")
button.send_keys(Keys.PAGE_DOWN)<jupyter_output><empty_output><jupyter_text>```old=""new=" "while new>old:    old = browser.page_source    button.send_keys(Keys.END)    new = browser.page_source```<jupyter_code>browser.get("https://www.bloomberg.com/")
browser.implicitly_wait(30)
browser.find_element_by_partial_link_text("S&P")
#EC(presense_of_element_located())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08971646212441811, 'fraction_numerical': 0.006347862886161659, 'mean_word_length': 6.318885448916409, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '426372', 'license': 'Apache-2.0', 'n_tokens_mistral': 719, 'n_tokens_neox': 681, 'n_words': 233, 'path': '2018/rate.am.ipynb', 'repo_name': 'HrantDavtyan/Data_Scraping', 'size': '2363'}