<jupyter_start><jupyter_text>Com o novo sistema do ICB, temos agora os arquivos .ics disponibilizados pelo instituto. Yay, sucesso.Para raspar isso de forma simples, vamos usar o Selenium para rodar o Python headless<jupyter_code>from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("https://ww3.icb.usp.br/calendario-de-eventos/mes/")


event_month = driver.find_elements_by_xpath('//*[@id="tribe-bar-date"]')[0]
event_month.clear()
event_month.send_keys("04/2020")
event_button = driver.find_elements_by_xpath('/html/body/div[2]/div/div[2]/div/div[2]/div[2]/form/div/div/div/div[3]/input')[0]
event_button.click()
download_button = driver.find_elements_by_xpath('/html/body/div[2]/div/div[2]/div/div[2]/div[3]/a')[0]
download_button.click()<jupyter_output><empty_output><jupyter_text>Hm, o site tem um bug que só baixa os eventos de janeiro. Fiz o report em 30/01/2020. Veremos.  Eles irão trabalhar no bug. Enquanto isso, vou tentar outra abordagem: pegar todos os links do calendário e raspar as páginas.<jupyter_code>from bs4 import BeautifulSoup
import requests
import re

html_page = requests.get("https://ww3.icb.usp.br/calendario-de-eventos/mes/")
get_all_links(html_page)


%run "/home/lubianat/Documents/Projects/side/seminUSP/src/cal_extract.py"



def get_event_ical(event_url):
    event_page = requests.get(event_url)
    event_soup = BeautifulSoup(event_page.text)

    event_title = event_soup.find('h1',attrs={"class":"tribe-events-single-event-title"}, text = True)
    event_title = event_title.get_text()
    print(event_title)
    infos = event_soup.find(class_="tribe-events-single-event-description tribe-events-content")
    info_text = infos.get_text()
    magic_event = magic_extract_event(info_text)
    magic_event.add('summary', event_title )
    return(magic_event)



from icalendar import Calendar, Event
import datetime as dt


cal = Calendar()

soup = BeautifulSoup(html_page.text)
tags =  soup.find_all('a')
for tag in tags:
    link = tag.get('href')
    if '/eventos/' in link:
            try:
                event_ical = get_event_ical(link)
                cal.add_component(event_ical)
            except Exception as ex:
                print(ex)

                import tempfile, os

directory = tempfile.mkdtemp()
f = open(os.path.join(directory, 'example.ics'), 'wb')
f.write(cal.to_ical())
f.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12643207855973815, 'fraction_numerical': 0.012274959083469721, 'mean_word_length': 4.3618421052631575, 'pattern_counts': {'":': 1, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '529854', 'license': 'MIT', 'n_tokens_mistral': 906, 'n_tokens_neox': 866, 'n_words': 169, 'path': 'src/raspando_icb_usp_com_selenium.ipynb', 'repo_name': 'lubianat/seminUSP', 'size': '2444'}