<jupyter_start><jupyter_text>Collecting data from internet--- Web scraping consists of - obtaining raw data from source ( requests )- filtering out useful data from garbage ( beautifulsoup4 )- saving into appropriate format ( csv/json )`scrapy` is scrapping framework which extends above libraries<jupyter_code># requests is http library for python
# requests makes http easier than builtin urllib
import requests
url = 'http://www.mfd.gov.np'
req = requests.get(url)
req
dir(req)
req.headers
req.status_code
req.text
from bs4 import BeautifulSoup
soup = BeautifulSoup(req.text, "html.parser")<jupyter_output><empty_output><jupyter_text>*you can use `lxml` instead of `html.parser` which is must faster for large html content*<jupyter_code>soup
type(soup)
dir(soup)
tables = soup.find_all('table')
tables
len(tables)
tables[0]
tables[0].find_all('tr')
cities = []
headers = []
for row in tables[0].find_all('tr'):
    ths = row.find_all('th')
    if ths:
        headers = [th.text.strip() for th in ths]
    else:
        tds = row.find_all('td')
        data = {}
        if tds and len(tds) >= 4:
            data[headers[0]] = tds[0].text.strip()
            data[headers[1]] = tds[1].text.strip()
            data[headers[2]] = tds[2].text.strip()
            data[headers[3]] = tds[3].text.strip()
        cities.append(data)
print(cities)<jupyter_output>[{'Minimum Temp.(°C)': '18.1', '24 hrs Rainfall(mm)': '13.0', 'Maximum Temp.(°C)': '19.4', 'Station': 'Dadeldhura'}, {'Minimum Temp.(°C)': '25.2', '24 hrs Rainfall(mm)': '9.6', 'Maximum Temp.(°C)': '28.0', 'Station': 'Dipayal'}, {'Minimum Temp.(°C)': '24.0', '24 hrs Rainfall(mm)': '88.3', 'Maximum Temp.(°C)': '28.7', 'Station': 'Dhangadi'}, {'Minimum Temp.(°C)': '23.4', '24 hrs Rainfall(mm)': '29.3', 'Maximum Temp.(°C)': '27.2', 'Station': 'Birendranagar'}, {'Minimum Temp.(°C)': '25.5', '24 hrs Rainfall(mm)': '21.4', 'Maximum Temp.(°C)': '29.7', 'Station': 'Nepalgunj'}, {'Minimum Temp.(°C)': '16.5', '24 hrs Rainfall(mm)': '6.2', 'Maximum Temp.(°C)': '23.7', 'Station': 'Jumla'}, {'Minimum Temp.(°C)': '24.5', '24 hrs Rainfall(mm)': '5.4', 'Maximum Temp.(°C)': '29.2', 'Station': 'Dang'}, {'Minimum Temp.(°C)': '22.4', '24 hrs Rainfall(mm)': '24.5', 'Maximum Temp.(°C)': '25.7', 'Station': 'Pokhara'}, {'Minimum Temp.(°C)': '27.5', '24 hrs Rainfall(mm)': '73.5', 'Maximum Temp.(°C)': '27.6', '[...]<jupyter_text>**Alternative Method** *in case of multiple tables within webpage, we can use css selectors*<jupyter_code>div = soup.find('div', attrs={'class': 'weather-data-table'})
div
table = div.find('table')
# first_table = tables[0]
table.find_all('th', attrs={'class': 'center'})
data_set = []
for tr in table.find_all('tr'):
    _data = {}
    tds = tr.find_all('td')
    if tds and len(tds) > 3:
        # _data['Station'] = t
        # print(tds)
        _data['Station'] = tds[0].string
        _data['Maximum'] = tds[1].string
        _data['Minimum'] = tds[2].string
        _data['Rainfall'] = tds[3].string
        data_set.append(_data)
print(data_set)
data_set[0].keys()<jupyter_output><empty_output><jupyter_text>*writing to csv file*<jupyter_code>import csv
with open('dataset.csv', 'w') as csvfile:
    csvdoc = csv.DictWriter(csvfile, 
                            fieldnames=data_set[0].keys())
    csvdoc.writeheader()
    csvdoc.writerows(data_set)
data_set[0]
data_set[0].keys()<jupyter_output><empty_output><jupyter_text>*json output*<jupyter_code>import json
json.dump(data_set, open('dataset.json', 'w'))
json.dumps(data_set)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.2049619396673245, 'fraction_numerical': 0.03383140682266704, 'mean_word_length': 4.39209726443769, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '7695', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1407, 'n_tokens_neox': 1324, 'n_words': 301, 'path': 'References/D.Scraping Basics.ipynb', 'repo_name': 'rprasai/datascience', 'size': '3547'}