<jupyter_start><jupyter_text>**Web scraping 1**<jupyter_code>import pandas as pd
from bs4 import BeautifulSoup
import numpy as np
import requests
from time import sleep
from tqdm import tqdm
import numpy as np
from multiprocessing.dummy import Pool<jupyter_output><empty_output><jupyter_text>**Using API's** API´s (application programming interface) allow you to remotely excecute a given function. In this class we are going to see how to use API's to get real life data with Python, using the [Requests library](https://docs.python-requests.org/en/master/) - [Grate place for finding usefull API'S](https://rapidapi.com/hub)<jupyter_code>import datetime as dt<jupyter_output><empty_output><jupyter_text>- Lest´s see an example with [Bitcon](https://binance-docs.github.io/apidocs/spot/en/change-log) API to get data from ETHUSDT prices for every hour in in 2017. In this case we are don't have a suscription to the Bitcoin API. For this reason we have a limit on our requests responses. I strongly recomend getting a free subscrition if you are developing a project with Bitcoin data<jupyter_code>url = "https://api.binance.com/api/v3/klines"
startTime = str(int(dt.datetime(2017, 5, 1).timestamp() * 1000))
endTime = str(int(dt.datetime(2018, 5, 1).timestamp() * 1000))
limit = '1000'
req_params = {"symbol" : 'ETHUSDT', 'interval' : '1h', 'startTime' : startTime, 'endTime' : endTime, 'limit' : limit}
response=(requests.get(url, params = req_params))<jupyter_output><empty_output><jupyter_text>- The response of the request we made comes is delivered in form of a json. Let´s make a DataFrame out of this data.<jupyter_code>response.text
import json
df = pd.DataFrame(json.loads(response.text))
df
df = df.iloc[:, 0:6]
df.columns = ['datetime', 'open', 'high', 'low', 'close', 'volume']
df=df.set_index('datetime')
df.index=[dt.datetime.fromtimestamp(x / 1000.0) for x in df.index]<jupyter_output><empty_output><jupyter_text>- Can we make a function that requests data of any symbol for every specified time period?<jupyter_code>def extraer_data_Symbol(symbol, interval,start, end):
    req_params = {"symbol" : symbol, 'interval' : interval, 'startTime' : start, 'endTime' : end, 'limit' : limit}
    limit = '10000'
    response=(requests.get(url, params = req_params))
    df = pd.DataFrame(json.loads(response.text))
    df = df.iloc[:, 0:6]
    df.columns = ['datetime', 'open', 'high', 'low', 'close', 'volume']
    df.index = [dt.datetime.fromtimestamp(x / 1000.0) for x in df.datetime]
    return df

Sym='https://api.binance.com/api/v3/exchangeInfo'
Symbols=json.loads(requests.get(Sym).text)
activos=[]
for i in Symbols['symbols']:
    activos.append(i['symbol'])<jupyter_output><empty_output><jupyter_text>**Web scraping Beautifulsoup** [Documentación BeautifulSoup](https://beautiful-soup-4.readthedocs.io/en/latest/) - Generlay the first thing we want to do is to<jupyter_code>url='https://www.imdb.com/chart/moviemeter/?sort=ir,desc&mode=simple&page=1'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
datas=[]
for i in tqdm(range(0,len(soup.find_all('td',{'class':'titleColumn'})))):
    data={}
    try:
        data['titulo']=soup.find_all('td',{'class':'titleColumn'})[i].find('a').text
    except:pass
    try:
        data['link']='https://www.imdb.com/'+soup.find_all('td',{'class':'titleColumn'})[i].find('a').get('href')
    except:pass
    try:
        data['puntaje']=soup.find_all('td',{'class':'ratingColumn imdbRating'})[i].find('strong').text
    except:pass
    datas.append(data)
len(datas)
datas[0]
data=pd.DataFrame(datas)
links=data['link'].to_list()
infos=[]
for i in tqdm(links):
    soup=0
    page = requests.get(i)
    soup = BeautifulSoup(page.content, 'html.parser')
    info={}
    try:
        info['año_lanzamiento']=soup.find('ul',{'class':'ipc-inline-list ipc-inline-list--show-dividers TitleBlockMetaData__MetaDataList-sc-12ein40-0 dxizHm baseAlt'}).findAll('li')[0].text[:4]
    except:pass
    try:
        info['duracion_peli']=soup.find('ul',{'class':'ipc-inline-list ipc-inline-list--show-dividers TitleBlockMetaData__MetaDataList-sc-12ein40-0 dxizHm baseAlt'}).findAll('li')[2].text
    except:pass
        
    
    infos.append(info)<jupyter_output>3%|▎         | 3/100 [00:08<04:39,  2.88s/it]<jupyter_text>**Paralelizar scraping**<jupyter_code>a=int(len(links)/10)
infos=[]
def scrap(x):
    global infos
    for i in tqdm(links[x:x+10]):
        soup=0
        page = requests.get(i)
        soup = BeautifulSoup(page.content, 'html.parser')
        info={}
        
        try:
            info['año_lanzamiento']=soup.find('ul',{'class':'ipc-inline-list ipc-inline-list--show-dividers TitleBlockMetaData__MetaDataList-sc-12ein40-0 dxizHm baseAlt'}).findAll('li')[0].text[:4]
        except:pass
        try:
            info['duracion_peli']=soup.find('ul',{'class':'ipc-inline-list ipc-inline-list--show-dividers TitleBlockMetaData__MetaDataList-sc-12ein40-0 dxizHm baseAlt'}).findAll('li')[2].text
        except:pass
        
        infos.append(info)
j=0
k=[]
for i in range(0,10):
    k.append(j)
    j+=10
pool=Pool(10)
re=pool.map(scrap, k)
pd.DataFrame(infos)<jupyter_output><empty_output><jupyter_text>- **Using headers**: Sometimes pages won´t respond to your request if your request is not done as a human would do it. One way you can try to make your requet more "human like" is by adding headers.<jupyter_code>headers={
    'authority': 'www.imdb.com',
'method': 'GET',
'path': '/chart/moviemeter/?sort=ir,desc&mode=simple&page=1',
'scheme': 'https',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-encoding': 'gzip, deflate, br',
'accept-language':'es-ES,es;q=0.9',
'cache-control': 'max-age=0',
'cookie': '<KEY>; session-id=136-8467877-9681758; adblk=adblk_no; ubid-main=133-3750919-3113945; session-id-time=2082787201l; session-token=<KEY>SBpG+fidsVTBP0wmmQqV; csm-hit=tb:s-PGX2A0861DEA1XENQNVS|1629483407253&t:1629483408943&adb:adblk_no',
'sec-ch-ua': '"Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"',
'sec-ch-ua-mobile': '?0',
'sec-fetch-dest': 'document',
'sec-fetch-mode': 'navigate',
'sec-fetch-site':'none',
'sec-fetch-user': '?1',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36'
}
url='https://www.imdb.com/chart/moviemeter/?sort=ir,desc&mode=simple&page=1'
page = requests.get(url,headers=headers)
soup = BeautifulSoup(page.content, 'html.parser')<jupyter_output><empty_output><jupyter_text>- [Grate place for finding usefull API'S](https://rapidapi.com/hub) - Scraping an image<jupyter_code>###nordvpn
###headers<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16943423952975753, 'fraction_numerical': 0.03512123438648053, 'mean_word_length': 5.15370705244123, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 38, 'https://': 10, 'lorem ipsum': 0, 'www.': 4, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '500218', 'license': 'MIT', 'n_tokens_mistral': 2658, 'n_tokens_neox': 2510, 'n_words': 494, 'path': 'e-ta4_webscraping_basics/e-ta4_webscraping_basics.ipynb', 'repo_name': 'npenag/e-TA', 'size': '7070'}