<jupyter_start><jupyter_text>Start with a base URL, in this case just www.darkhorse.com, and a start_url, in this case where the browse section starts on page 1.  Since different comics have different contributers (artists,writers, inkers etc) we need to generate a base list of attribute we will collect, so we loop through the first page and create a list of attributes.<jupyter_code>spec_url = 'https://www.darkhorse.com/Comics/Browse/January+1986-December+2020---0-Z/P5wfwkt8?page=1'
html_titles = requests.get(spec_url).text
titles = BeautifulSoup(html_titles, 'lxml')
prod_list = titles.find_all('div',{"class":"list_item"})
page_links = [prod_list[i].select('a')[1]['href'] for i in range(len(prod_list))]

base_specs = []
for link in page_links:
        request = requests.get(base_url + link).text
        soup = BeautifulSoup(request, 'lxml')
        for i in soup.find_all('dt'):
            if i.text[:-1] not in base_specs:
                base_specs.append(i.text[:-1])<jupyter_output><empty_output><jupyter_text>Now that we have a base list of specifications/attributes for a comic, we need to loop through each book on each page of the site the get the info we want.<jupyter_code>page_n = 1
books = []
while True:
    html_titles = requests.get(start_url+str(page_n)).text #get page 'i'
    titles = BeautifulSoup(html_titles, 'lxml') #soupify
    prod_list = titles.find_all('div',{"class":"list_item"}) #get list of titles
    if not prod_list:
        print('No more results: Exiting...') #check to see if results exits, other wise break
        break
    else:
        print('Getting page:{}'.format(page_n))
        
        page_links = [prod_list[i].select('a')[1]['href'] for i in range(len(prod_list))] #list of links
        for link in page_links: #iterate through links)
            data = []
            request = requests.get(base_url + link).text
            soup = BeautifulSoup(request, 'lxml')            
            data.append(link.split('/')[2]) #append job id
            data.append(soup.find_all('h2',{'class':'title'})[0].text) #append title
            data.append(soup.find_all('div',{'class':'product-description'})[0].text) #append description
            spec = [i.text[:-1] for i in soup.find_all('dt')] #generate list of specs
            res = [i.text for i in soup.find_all('dd')] #generate spec details
            d = {i:j for i,j in zip(spec,res)}
            for i in base_specs: #iterate through base_specs; if present add to data, else append 'Na'
                if i in d.keys():
                    data.append(d[i])
                else:
                    data.append('Na')
            
            books.append(data) #append book data to list of books       
    page_n += 1 #go to next page<jupyter_output>Getting page:1
Getting page:2
Getting page:3
Getting page:4
Getting page:5
Getting page:6
Getting page:7
Getting page:8
Getting page:9
Getting page:10
Getting page:11
Getting page:12
Getting page:13
Getting page:14
Getting page:15
Getting page:16
Getting page:17
Getting page:18
Getting page:19
Getting page:20
Getting page:21
Getting page:22
Getting page:23
Getting page:24
Getting page:25
Getting page:26
Getting page:27
Getting page:28
Getting page:29
Getting page:30
Getting page:31
Getting page:32
Getting page:33
Getting page:34
Getting page:35
Getting page:36
Getting page:37
Getting page:38
Getting page:39
Getting page:40
Getting page:41
Getting page:42
Getting page:43
Getting page:44
Getting page:45
Getting page:46
Getting page:47
Getting page:48
Getting page:49
Getting page:50
Getting page:51
Getting page:52
Getting page:53
Getting page:54
Getting page:55
Getting page:56
Getting page:57
Getting page:58
Getting page:59
Getting page:60
Getting page:61
Getting page:62
Getting page:63
G[...]<jupyter_text>'books' is a list of lists, each list containing the desired information. We can use pandas to house the info in a DataFrame.<jupyter_code>books = pd.DataFrame.from_records(books)
cols = ['Job#','Title','Description'] + base_specs
books.columns = cols
books['Description'] = [i.replace('\n','') for i in books['Description']]
books

books.to_csv('DH_archive.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11283767630886923, 'fraction_numerical': 0.033707865168539325, 'mean_word_length': 3.727683615819209, 'pattern_counts': {'":': 2, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '225847', 'license': 'MIT', 'n_tokens_mistral': 1461, 'n_tokens_neox': 1314, 'n_words': 418, 'path': 'DH_scrape.ipynb', 'repo_name': 'ExtraLime/DH_archive', 'size': '4183'}