<jupyter_start><jupyter_text>WebScraping with SeleniumThis notebook shows how use selenium to scrape data from an e-commerce web site.The scope is only to understand the capabilities of web scraping and prepare a dataset for academic purporse.Let’s begin writing our scraper!We will first install important modules and packages for our Notebook*   Selenium*   Chromium-chromedriver<jupyter_code>!pip install selenium
!apt-get update
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin<jupyter_output>Collecting selenium
[?25l  Downloading https://files.pythonhosted.org/packages/80/d6/4294f0b4bce4de0abf13e17190289f9d0613b0a44e5dd6a7f5ca98459853/selenium-3.141.0-py2.py3-none-any.whl (904kB)

[K     |▍                               | 10kB 13.6MB/s eta 0:00:01
[K     |▊                               | 20kB 18.7MB/s eta 0:00:01
[K     |█                               | 30kB 10.5MB/s eta 0:00:01
[K     |█▌                              | 40kB 9.7MB/s eta 0:00:01
[K     |█▉                              | 51kB 8.1MB/s eta 0:00:01
[K     |██▏                             | 61kB 9.4MB/s eta 0:00:01
[K     |██▌                             | 71kB 7.8MB/s eta 0:00:01
[K     |███                             | 81kB 8.1MB/s eta 0:00:01
[K     |███▎                            | 92kB 8.8MB/s eta 0:00:01
[K     |███▋                            | 102kB 9.1MB/s eta 0:00:01
[K     |████                            | 112kB 9.1MB/s eta 0:00:01
[K     |████▍                           | 122kB 9.1M[...]<jupyter_text>And now we will import some modules on our Notebook1.   sys: to setup the path of chrome driver2.   selenium: to emulate the user behaviour3.   pandas: to work with data4.   tqdm: to show a progress bar in our notebook5.   json: to work with json format<jupyter_code>import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
from tqdm import tqdm_notebook as tqdm
import pandas
import json
import pprint<jupyter_output><empty_output><jupyter_text>First we need to set the options for the ghost browser.The most important is `--headless` because we are in a "cloud" ntotebook. In our local notebook we can remove the `--headless` option.<jupyter_code>chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument("window-size=1900,800")
chrome_options.add_argument("--enable-javascript")
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36")<jupyter_output><empty_output><jupyter_text>`webdriver` is the most importa object in Selenium.With `webdriver` we can start the phantom browser, emulate the user navigation and scrape our data. Let’s now create a new instance of google chrome.We will navitage to page by a `get` request. With http we can do a get or a post (or others but they are not important for us): https://www.w3schools.com/tags/ref_httpmethods.asp ```wd.get("https://www.crunchbase.com/discover/organization.companies")```<jupyter_code>wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.get("http://startup.registroimprese.it/isin/search?0")<jupyter_output>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: DeprecationWarning: use options instead of chrome_options
  """Entry point for launching an IPython kernel.<jupyter_text>Take a look to the screenshot from our phantom browser.Note:- la language and the locale: It is en_US!- the screen ration: is it the same of our notebook?<jupyter_code>wd.save_screenshot('screenshot.png')

%pylab inline
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
img=mpimg.imread('/content/screenshot.png')
imgplot = plt.imshow(img)
plt.show()<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>Let's try to extract some information from the page`title` contains the title of our page.<jupyter_code>print(wd.title)<jupyter_output>startup and innovative SMEs<jupyter_text>How do we extract the values inside a page with CSS Selector?Selenium has a method called `find_elements_by_css_selector`.We will pass our CSS Selector into this method and get a list of selenium elements. Once we have the element, we can extract the text inside it using the `text` function.<jupyter_code>list_companies = wd.find_elements_by_css_selector("a.link")
print(len(list_companies))
print(list_companies[0].text)<jupyter_output>SAPORARTE S.R.L.<jupyter_text>Extract the start-up listLet's begin to download the list of companies.The list of companies is defined by the css selection rule`div.searchCompanyCard`<jupyter_code>list_companies = wd.find_elements_by_css_selector("div.searchCompanyCard")
print(len(list_companies))<jupyter_output>10<jupyter_text>Gread! We find 10 items!Now, we will extract the attribute from each single element.The scope is to create one dict for each item in the list with the attribute:1.   company name2.   link to the detail page3.   number of employees4.   vat code5.   economic sector6.   tags<jupyter_code>import pprint
import time
items = []
for item in list_companies:
    company_name = item.find_elements_by_css_selector("a.link")[0].text
    url = item.find_elements_by_css_selector("a")[0].get_attribute("href")
    vat_code = item.find_elements_by_css_selector("div.ten")[0].text
    number_of_employees = item.find_elements_by_css_selector("div.twelve")[3].text
    economic_sector = item.find_elements_by_css_selector("div.twelve")[1].text
    tags = item.find_elements_by_css_selector("div.sixteen")[2].text

    items.append({'company_name': company_name,
                  'url': url,
                  'vat_code': vat_code,
                  'number_of_employees': number_of_employees,
                  'tags': tags,
                  'economic_sector': economic_sector})


pprint.pprint(items[0:5])
len(items)
import pprint
import time
items = []
page = 1
end = False
wd.get("http://startup.registroimprese.it/isin/search?0")
while (not end and page < 5):
  print(f"Get data from {page} page...")
  time.sleep(1)
  list_companies = wd.find_elements_by_css_selector("div.searchCompanyCard")
  for item in list_companies:
      company_name = item.find_elements_by_css_selector("a.link")[0].text
      print(company_name)
      if (len(item.find_elements_by_css_selector("a")) > 1):
        url = item.find_elements_by_css_selector("a")[0].get_attribute("href")

      vat_code = item.find_elements_by_css_selector("div.ten")[0].text
      number_of_employees = item.find_elements_by_css_selector("div.twelve")[3].text
      economic_sector = item.find_elements_by_css_selector("div.twelve")[1].text
      tags = item.find_elements_by_css_selector("div.sixteen")[2].text

      items.append({'company_name': company_name,
                    'url': url,
                    'vat_code': vat_code,
                    'number_of_employees': number_of_employees,
                    'tags': tags,
                    'economic_sector': economic_sector})
  try:
    wd.find_elements_by_css_selector("i.right")[0].click()
    page = page + 1
  except:
    end = True
len(items)<jupyter_output><empty_output><jupyter_text>Pandas and data processing**Well!**We're starting to see another amazing library for working with data!It's **pandas**: *Python library for data analysis*.Basically, with pandas we can manipulate a data set or a historical series in Python.For now, we start to give a couple of concepts.Pandas is based on two types of data: **Series** and **DataFrame**:- `Series` represents a list of data- `DataFrame` represent a data set in tabular formatEach column of a `DataFrame` is a `Series`.We can create a `DataFrame` using the `pd.DataFrame` method by passing our dictionary as an input parameter.A `DataFrame` pandas can be easily exported in *CSV*, *Excel*, ...For more information about `Pandas` seehttps://pandas.pydata.org/getting_started.html<jupyter_code>import pandas as pd
ds_items = pd.DataFrame(items)
ds_items.set_index("vat_code")
ds_items.head()
ds_items.to_csv('ds_items.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10623864324651726, 'fraction_numerical': 0.029194427619624472, 'mean_word_length': 3.7694974003466206, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '445715', 'license': 'MIT', 'n_tokens_mistral': 2939, 'n_tokens_neox': 2675, 'n_words': 699, 'path': '1_Unioncamere.ipynb', 'repo_name': 'mauropelucchi/web-scraping-companies', 'size': '8255'}