<jupyter_start><jupyter_text>Part_1 -  Data Collection 1.1. Get the list of books Scrap the links in a page containing list of booksThanks to this function, for each page that arrive in input as href, we are able to scrap the book's Urls in that. N.B: each page has 100 books, so we have to find 100 links for page<jupyter_code>def scrap_url(href, driver , nlp):
    driver.get(href)       
    #time.sleep(5)
    page_soup = BeautifulSoup(driver.page_source, features="lxml")
    
    links = page_soup.find_all('a' , itemprop="url")
    
    lista_links = []
    
    i = 2
    for link in links:
        link_full = link.get('href')
        if (i % 2) == 0 :
            string1 = 'https://www.goodreads.com/en'
            link_full = string1 + link_full
            lista_links.append(link_full)
        
        i = i+1
        
    
    
    
    return lista_links<jupyter_output><empty_output><jupyter_text>Take the book's linksThanks to this script we are able to read the web-pages that contain the list of best books on the web-site "https://www.goodreads.com" and, for each page, take the Url of the book we are interested in, saving them in a file called "lista_url.txt". In each page, we will scrap the links of 100 books**lista_url.txt** : This file will contain 30k rows, each row is the link of a book<jupyter_code>#chromedriver = r"C:\Users\thoma\Desktop\HW3_ADM\chromedriver_win32"
driver = webdriver.Chrome(chromedriver)

for i in range (1, 301) :
    href = "https://www.goodreads.com/list/show/1.Best_Books_Ever?page="
    
    stringa2 = i
    stringa2 = str(stringa2)
    href = href + stringa2
    
    nlp = spacy.load('en_core_web_sm')
    
    #Let's use the fuction able to find the urls in the page.
    urls = scrap_url(href,driver,nlp)
    
    with open('lista_url.txt', 'a') as f:
        for item in urls:
            f.write("%s\n" % item)<jupyter_output><empty_output><jupyter_text>1.2. Crawl books Download the HTML pagesWith this function we are able to save in a folder "html_folder" all the books as .html page.It takes the page, as link from "lista_url.txt", and download it savingig as .html file in the setted folder/filepath.After this step, we have all the book's html-pages downloaded, and we are ready to scrap them.<jupyter_code>#filepath = r'C:\Users\thoma\Desktop\HW3_ADM\html_folder'

#Da lanciare come sta settato ora per prendere nuovi file

with open("lista_url.txt") as file_in:
    
    line_count = 1    # article_i at i-th row of "lista_url.txt"
    
    for link in islice(file_in, 1, 30000):
        
        page = requests.get(link, allow_redirects=True)
        contenuto = page.text
        soup = BeautifulSoup (contenuto , features = 'lxml')
        
        
        if line_count == 30002 :
            break
        string1 = "article_"
        string2 = str(line_count)
        string3 = ".html"
        title = string1 + string2 + string3   
        # We automatically assign to each book the name "article_i.html"
        
        
        
        with open (os.path.join(filepath, title), "w",encoding='utf-8' ) as f2:
            f2.write(str(soup))
        line_count += 1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09402515723270441, 'fraction_numerical': 0.018238993710691823, 'mean_word_length': 2.681712962962963, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '103113', 'license': 'MIT', 'n_tokens_mistral': 1085, 'n_tokens_neox': 1046, 'n_words': 334, 'path': 'collector.ipynb', 'repo_name': 'TommasoBer/ADM_HW3', 'size': '3180'}