<jupyter_start><jupyter_text>Useful libraries:<jupyter_code>import requests
import pandas as pd
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>**1.1** **Get the list of animes**With the for loop I get the url of all the pages (50 per page except the last one which has less,383 pag).We 'send' a request to the page, to collect the ursl.We read the html and take the href attributes of the  tag<jupyter_code>urls=[]
for i in range(0,19500,50):
  url='https://myanimelist.net/topanime.php?limit='+str(i)
  soup= BeautifulSoup(requests.get(url).text,'html.parser')

  for tag in soup.find_all('tr'):
    links=tag.find_all('a')
    for l in links:
      if type(l.get('id'))== str and len(l.contents[0]) >1:
        urls.append(l.get('href'))


#links= soup.find_all('a', {"class":"hoverinfo_trigger fl-l ml12 mr8"})  
  #for x in links:
  # urls.append(x['href'])

len(urls) #19122 giusto<jupyter_output><empty_output><jupyter_text>We need to save it as text<jupyter_code>with open('urls.txt', 'w') as f: #open the file con write, chiude da solo in automatico, lo trovo in file sample data
    for line in urls:
        f.write(line)
        f.write('\n')
import os.path

#!ls pages/3pag & just a check<jupyter_output><empty_output><jupyter_text>**1.2** **Crawl animes**Create the folders. Put all of them under "pages". Each folder has a name that refers to the number of the page from which the links it contains come from.<jupyter_code>if not os.path.isdir('/content/pages/'): os.mkdir('/content/pages/')
for i in range(383):
  path= '/content/pages/'+ str(i)+'pag/'
  if not os.path.isdir(path):
    os.mkdir(path)<jupyter_output><empty_output><jupyter_text><jupyter_code>!rm -rf file_hw3.zip to delete folders that are not needed.<jupyter_output><empty_output><jupyter_text>We read our document and we save the html of each anime.The path is: \content\pages\ipag\article_i.html<jupyter_code>with open('urls.txt', 'r',encoding='utf-8') as f:
  for i,line in enumerate(f ,start=1):
    if i==19122:
      break
    
    page=requests.get(line)
    code= page.text
    name= 'article_'+str(i)
    path= '/content/pages/'+ str(i//50)+'pag/'
    Name= os.path.join(path,name + '.html')
    with open(Name,'w',encoding='utf-8') as page:
      page.write(code)<jupyter_output><empty_output><jupyter_text>Zip our file to to be able to download it, could be useful.<jupyter_code>!zip -r /content/file_hw3.zip /content/pages<jupyter_output><empty_output><jupyter_text>**1.3 Parse downloaded pages**<jupyter_code>with open('/content/pages/3pag/article_150.html', encoding='utf-8') as f:
   soup=BeautifulSoup(f, 'html.parser')

soup<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14440298507462687, 'fraction_numerical': 0.019029850746268655, 'mean_word_length': 4.790496760259179, 'pattern_counts': {'":': 1, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '361549', 'license': 'MIT', 'n_tokens_mistral': 1006, 'n_tokens_neox': 976, 'n_words': 256, 'path': 'HW3_ADM.ipynb', 'repo_name': 'Gcorsetti/ADM-HMW3_AnimeList', 'size': '2680'}