<jupyter_start><jupyter_text>1.3 Parse downloaded pagesThis function **scrap_book** is the main core of our scraping part.Using the "BeautifulSoup" library, *scrap_book* takes as input the html page in soup_format and scrapes the different data that we need to build our dataset.As others input, there is **nlp** that is a tool for the Natural Language Processing and **n_link** that is useful to retrive the book's Url from the list previopus created<jupyter_code>def scrap_book(page_soup , nlp , n_link):
    
    n_link = int(n_link)
    n_link = n_link-1
    
    #Scrap the book's title
    book_title = page_soup.find_all('h1', id ="bookTitle")[0].contents[0].replace('\n', '').strip()
    
    print(book_title)
    
    # Scrap the book's serie
    series = page_soup.find_all('h2', id="bookSeries")[0].contents[0].replace('\n', '').strip() 
    
    #Scrap the book's author
    author = page_soup.find_all('span', itemprop='name')[0].contents[0].replace('\n', '').strip()
    
    #Scrap the rating value
    rating_value = page_soup.find_all('span', itemprop='ratingValue')[0].contents[0].replace('\n', '').strip()
    
    #Scrap the Rating_Count and Review_Count
    ratings = page_soup.find_all('a', href="#other_reviews")
    rating_count = -1
    rating = -1
    for raiting in ratings:
        if raiting.find_all('meta', itemprop="ratingCount"):
            rating_count = raiting.text.replace('\n', '').strip().split(' ')[0]
        elif raiting.find_all('meta', itemprop="reviewCount"):
            review_count = raiting.text.replace('\n', '').strip().split(' ')[0]
    
    
    #Scrap the Plot
    description = ' '.join([c for c in page_soup.find_all('span', id = re.compile(r'freeText\d'))[0].contents \
                           if isinstance(c, str)])
     
    
    doc = nlp(description)
    token_list_plot = [token for token in doc if not token.is_stop and not token.is_punct]
    
    
    #Scrap number of pages
    n_pages = page_soup.find_all('span', itemprop='numberOfPages')[0].contents[0].strip().split(' ')[0]
    
    #Scrap Publishing date of the book
    pub_date = page_soup.find_all('div', {"class": "row"})[1].contents[0].strip().split(' ')
    pub_date = " ".join(pub_date)
    pub_date = pub_date.replace('\n', '').strip()
    pub_date = pub_date.replace(" ","")
    pub_date = pub_date.split('d')[1]
    
    
    
    #Scrap Characters
   
    Characters = []
    lenght_c = page_soup.find_all('a', {'href': re.compile(r'/characters/')})
    l = len(lenght_c)
    for i in range (0, l):
        character = page_soup.find_all('a', {'href': re.compile(r'/characters/')})[i].contents[0]
        Characters.append(character)
        
        
    #Scrap Settings
    Settings = []
    lenght_s = page_soup.find_all('a', {'href': re.compile(r'/places/')})
    l2 = len(lenght_s)
    for i in range (0, l2):
        setting = page_soup.find_all('a', {'href': re.compile(r'/places/')})[i].contents[0]
        Settings.append(setting)    
    Settings = " ".join(Settings)
    
    
    #scrap original book's link
    f=open('lista_url.txt')
    lines=f.readlines()
    URL = lines[n_link]
    

    return book_title,series,author,rating_value,rating_count, \
            review_count,token_list_plot,n_pages,pub_date,Characters,Settings,URL<jupyter_output><empty_output><jupyter_text>---------------------------------------------------------------------------------------This is the script that we launch in order to extract information from the html pages and store them in the file "articles.tsv". **filepath** : It is the forder in which there are the .html pages previous downloaded**articles.tsv** : It 's the file that we build. Using the function *scrap_book* we extract info about each page/book and put them in this file, in order to build our dataset.<jupyter_code>#filepath = r"C:\Users\thoma\Desktop\HW3_ADM\html_folder3\articoli_7"
nlp = spacy.load('en_core_web_sm')
with open('articles.tsv', 'a', encoding="utf-8") as out_file:
    tsv_writer = csv.writer(out_file, delimiter='\t')
    #tsv_writer.writerow(['bookTitle', 'bookSeries','bookAuthors','ratingValue','ratingCount','reviewCount','Plot','NumberofPages','Published','characters','Settings','URL'])

    for i in range (24001,30001):
        try :
            print(i)

            string1 = "article_"
            string2 = str(i)
            string3 = ".html"
            title = string1 + string2 + string3

            soup = BeautifulSoup(open(os.path.join(filepath, title),encoding='utf-8' ), features="lxml")


            bookTitle,bookSeries,bookAuthors,ratingValue,ratingCount, reviewCount, Plot, \
            NumberofPages,Published, characters,Settings,URL = scrap_book(soup, nlp,string2)

            riga_info = (bookTitle,bookSeries,bookAuthors,ratingValue,ratingCount, reviewCount, \
                         Plot, NumberofPages,Published,characters,Settings,URL)


            tsv_writer.writerow([riga_info[0],riga_info[1],riga_info[2],riga_info[3],riga_info[4],riga_info[5], \
                                 riga_info[6],riga_info[7],riga_info[8],riga_info[9],riga_info[10],riga_info[11]])
        
        #For some reason related to the differents pages of the website could be an error in scrping process: we handle it.
        except IndexError as e :
            print(" lost a book because of scraping error")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1543175487465181, 'fraction_numerical': 0.012256267409470752, 'mean_word_length': 3.0958174904942966, 'pattern_counts': {'":': 1, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '64256', 'license': 'MIT', 'n_tokens_mistral': 1788, 'n_tokens_neox': 1743, 'n_words': 393, 'path': 'parser.ipynb', 'repo_name': 'TommasoBer/ADM_HW3', 'size': '5385'}