<jupyter_start><jupyter_text>Selecting a fiction canon> with an additive approach- toc: true - badges: true- comments: true- categories: [canon] Still working on the Invisible Canon. Today I want to find a superset of canons for fiction: not genre fiction, but "literature". Of course, the Stanford Literary Lab has already done work on this. In their *Pamphlet 8: Between Canon and Corpus* they demonstrate that overlaying different "top 100 books of the 20th century" lists leads to significant overlap, and can be used to triangulate a sort of "most voted for" list. They use six different lists and end up with roughly 400 unique books. I can use this "found canon" technique to craft a data feature, something like "number of times this work has been canonized". That's useful in its own right, but it could also be the target for a collaborative filtering system.I need to extract that information from a bunch of HTML tables and PDF reports, because academics. And then map it to the features from my previous distillation. One problem that will arise is that goodreads data is sorted by edition, while the canon lists are sorted by work. So the first problem is to reshape the data. Pandas provides MultiIndex, a way of stacking data in multiple dimensions. let's try that<jupyter_code>import pandas as pd

pd.set_option("display.max_columns", None)

total_df = pd.read_csv('../../records/cleaned_goodreads_books.csv')
total_df = total_df.drop(['Unnamed: 0', 'Unnamed: 0.1', 'Unnamed: 0.1.1'], axis=1)
total_df.tail()
total_df = total_df.set_index(['work_id'])
total_df
total_df.loc[total_df.index.duplicated() == True]
len(total_df.index.unique())<jupyter_output><empty_output><jupyter_text>It worked! Now I have a list of 927,891 individual works. Time to import the different canons, and associate their data to the books!  Gather the canonsI can build here off the great work of others. As mentioned above, the Stanford Literary Lab published [six canons](file:///home/mage/Downloads/LiteraryLabPamphlet8.pdf)(PDF) that they concatenated into their 20th Century Fiction corpus. I've also found the incredible [The Greatest Books](https://thegreatestbooks.org) project by <NAME>, which does a similar thing but with 196 lists! Unfortunately he only distributes the rankings post-weighting, whereas I would like each list to be a separate column and compare how many times each book gets mentioned. I have sent him a request for the lists but i may have to scrape this website myself...For now, I should gather the lists that *didn't* make it onto Mr. Sherman's list. Those would be the bestsellers and the reader rankings, for the most part. He explicitly prefers prestige over popularity. So there are three lists from the LitLab pamphlet that I suspect didn't make it into the Greatest Books list, and a few from other places around the web, especially [Penguin's 100 must-read classic books, as chosen by our readers](https://www.penguin.co.uk/articles/2018/100-must-read-classic-books.html). I err on the side of popularity, myself, but the goodreads ratings should do plenty to balance that effect.<jupyter_code>%ls ../../records
bestsellers = pd.read_csv('../../records/wikipedia-bestselling-books.csv')
library = pd.read_csv('../../records/library-journal.csv')
penguin = pd.read_csv('../../records/penguin-readers.csv')
ml_readers = pd.read_csv('../../records/modern-library-readers-list.csv')
pw_readers = pd.read_csv('../../records/pub-weekly.csv') 
psa = pd.read_csv('../../records/postcolonial-studies.csv') 
for i in [bestsellers, library, penguin, ml_readers, pw_readers, psa]:
    print(len(i))<jupyter_output>167
150
100
100
83
100<jupyter_text>See how these lists are all different lengths? That's why we can't just average out the different rankings. We need each book to have a one-hot encoding of the list, either 0 if it's not on or 1 if it is. So we have to add columns to our dataframe, one for each list. That really means I need a bunch of small lists, not one megalist like The Greatest Books provides. I'll have to scrape them.<jupyter_code>import requests
from bs4 import BeautifulSoup
url = "https://thegreatestbooks.org/lists/28"
r = requests.get(url)
htm = BeautifulSoup(r.text, 'html.parser')
h4s = htm.find_all('h4')
[a.get_text() for a in h4s[0].findAll('a')]
def get_list_from_htm(htm):
    title = htm.find_all('h2')[0].get_text()
    h4s = htm.find_all('h4')
    books = [[a.get_text() for a in o.findAll('a')][:2] for o in h4s if len(o) > 1]
    return(title, books)
get_list_from_htm(htm)<jupyter_output><empty_output><jupyter_text>Hey, that worked! The website has good semantic HTML, so it will be easy to extrapolate the same process to the rest of the I want to make a dataframe of these lists, organized by title. That way I can sort it into the larger Goodreads dataset, or extract information from that one to this one, either way.I don't care about relative rankings within a given list, just the cumulative amount of rankings *across* lists. So I can use a one-hot encoding: 0 if a book is not on a list, 1 if it is. There's probably a nifty method for this, but I've never done it before, so I'm just going to implement it manually right now.First I'll try it on the data I already have. Then I'll start scraping the website.<jupyter_code>local_lists = [bestsellers, library, penguin, ml_readers, pw_readers, psa]
for i in local_lists:
    print(i.keys())<jupyter_output>Index(['Book', 'Author(s)', 'Original language', 'First published',
       'Approximate sales', 'Genre', 'Author'],
      dtype='object')
Index(['\nK', 'L', 'M', 'R', 'App.', 'Points', 'Rank', 'Author', 'Title',
       'Date'],
      dtype='object')
Index(['Title', ' Author', ' Year', 'Author'], dtype='object')
Index(['Book', ' Author', ' Date', ' Rank', 'Author'], dtype='object')
Index(['Book', ' Author', ' Date', 'Author'], dtype='object')
Index(['Title', ' Author', ' Date', ' Rank', 'Author'], dtype='object')<jupyter_text>Don't care about date or genre or any of that, I can reconstruct that later. Either 'Book' or 'Title' exist in each one, and 'Author' or in one case 'Author(s)', and that's all I can get from the Greatest Books website, so that's what i will work with here.I didn't pair names with these lists, so I'll just zip up a little list of titles real quick:<jupyter_code>list_names = ['Wikipedia Bestselling', 'Library Journal', 'Penguin Readers', 'Modern Library Readers', 'PW Bestsellers', 'Postcolonial Studies']
titles_df = pd.DataFrame()
for i in range(6):
    name = list_names[i]
    print(name)
    df = local_lists[i]
    if 'Book' in df.keys():
        bk = 'Book'
    else: bk = 'Title'
    au = 'Author(s)' if 'Author(s)' in df.keys() else 'Author' if 'Author' in df.keys() else ' Author'
    df['Author'] = df[au]
    new_df = df[[bk, 'Author']].set_index(bk)
    new_df[name] = 1
    titles_df = titles_df.append(new_df)

titles_df
onehot = titles_df.fillna(0).sum(level=0)

onehot<jupyter_output><empty_output><jupyter_text>Brilliant! That's a one-hot encoding for each list, sorted by item. Now I just encapsulate that logic into a function and run it across each new list as I scrape them from the internet!<jupyter_code>full_titles_df = titles_df
def books_df_from_list(list_from_htm):
    name, books = list_from_htm
    new_df = pd.DataFrame(data=books, columns=['Title','Author'])
    new_df[name] = 1
    return(new_df)<jupyter_output><empty_output><jupyter_text>Need a list of all the fiction lists, not the nonfiction ones. Can scrape URLs from the homepage<jupyter_code>url = "https://thegreatestbooks.org/lists/details"
r = requests.get(url)
htm = BeautifulSoup(r.text, 'html.parser')
hrefs = [o.get('href') for o in htm.find_all('a') if o.get('href') is not None]
links = [o for o in hrefs if '/lists/' in o]
links = [o for o in links if 'http' not in o]
links[:5]<jupyter_output><empty_output><jupyter_text>Now we put it all together, pull each list and append it to the df, then merge the duplicates for a great big one-hot-encoded best Books list...<jupyter_code>from tqdm import tqdm
for link in tqdm(links):
    try:
        url = f"https://thegreatestbooks.org{link}"
        r = requests.get(url)
        htm = BeautifulSoup(r.text, 'html.parser')    
        bks = get_list_from_htm(htm)
        new_df = books_df_from_list(bks)
        full_titles_df = full_titles_df.append(new_df.set_index('Title'))
    except IndexError as e:
        print(e, page)
        pass

full_titles_df
full_onehot = full_titles_df.fillna(0).sum(level=0)

full_onehot
full_onehot.to_csv('../../records/greatest-book-lists-onehot.csv')<jupyter_output><empty_output><jupyter_text>And there we have it! 136 book lists, 11,225 votes, and 4,414 unique titles. Now we can do all kinds of math on these features, including recommender systems, networkgraphs, etc. But for now: a quick ranking by count, then sweet sleep...<jupyter_code>counts = zip(full_onehot.index, full_onehot.sum(axis=1))
counts_df = pd.DataFrame(sorted(counts, key=lambda x:x[1], reverse=True))
counts_df.columns = ['Title', 'Listed count']
counts_df.index = counts_df.index + 1
counts_df<jupyter_output><empty_output><jupyter_text>That looks like the classics, all right... :shrug:I'll save this too, and upload to my website so you can peruse it yourself. Find it [here](/assets/2021-07-27-found-canon.csv) (CSV)<jupyter_code>counts_df.to_csv('../assets/2021-07-27-found-canon.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11080537623028892, 'fraction_numerical': 0.011853106148798814, 'mean_word_length': 4.658682634730539, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 42, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '568998', 'license': 'Apache-2.0', 'n_tokens_mistral': 3105, 'n_tokens_neox': 2936, 'n_words': 1142, 'path': '_notebooks/2021-07-26-fiction-canon.ipynb', 'repo_name': 'deepfates/fastpages', 'size': '9456'}