I am writing a program to analyze books from the Gutenberg archive. The program takes the title and URL and finds the text and downloads it. Then it goes through the text and tokenizes it. Here is the code:from urllib import requestimport nltkimport os.pathcanon_titles = [    'Moby Dick',    'Great Expectations']canon_urls = [    'http://www.gutenberg.org/files/2701/2701-0.txt',  # Moby Dick    'http://www.gutenberg.org/files/1400/1400-0.txt']# Canon_beginnings is the location of where the book actually begins in# the text file (skips headers, publisher info, etc.)canon_beginnings = [    28876,    886]# canon endings exists just to grab a small amount of text for prototypingcanon_endings = [x + 500 for x in canon_beginnings]canon_raw = [None] * len(canon_titles)canon_tokens = [None] * len(canon_titles)canon_words = [None] * len(canon_titles)canon_words2tokens = [None] * len(canon_titles)canon_pos = [None] * len(canon_titles)# Now I combine all these together into a dictionarycanon_dict = {z[0]: list(z[1:]) for z in zip(canon_titles, canon_urls, canon_beginnings, canon_endings, canon_raw, canon_tokens,                                             canon_words, canon_words2tokens, canon_pos)}# Now I go through each title in the dict and see if I already have the text (I rerun this in Jupyter Notebook sometimes)# And if not I grab it from onlinefor x in canon_dict:    print(Working on {}.format(x))    if canon_dict[x][3] == None:        print({} does not already exist, grabbing the text.format(x))        url = canon_dict[x][0]        response = request.urlopen(url)        canon_text_draft = response.read().decode('utf8')        canon_dict[x][3] = canon_text_draft[canon_dict[x][1]:canon_dict[x][2]]    else:        print(Already have this text, skipping)# OK, now we'll tokenize, do parts of speech, etc.def tokinze_text(raw_text):    tokens = nltk.word_tokenize(raw_text)    return tokens# Now let's find the tokensfor x in canon_dict:    print(canon_dict[x][3])    canon_dict[x][4] = tokinze_text(canon_dict[x][3])I would like to go much further with this - extracting parts of speech, rare words, etc., but I'm worried that my basic data structures are wrong. The python dictionary is feeling unwieldy as it gets larger. Should this entire thing be done in a pandas dataframe instead? Also, should the raw text be held in a separate structure from the rest? If I want to run a bunch of analysis on some of the results (like the tokenized text), will having the massive raw text slow everything down?
Tokenizing texts from Gutenberg archive for analysis
python;strings;python 3.x;http;natural language processing
This is the ideal place for a class. Each book is its own object with its own method of returning its tokens. I would make a method tokens, which I would make a property that fills itself on the first call to it.Something like this:from urllib import requestimport nltkclass Book(object):    def __init__(self, title, url, start=0, end=-1):        self.title = title        self.url = url        self.start = start        self.end = end        self.raw_ = None        self.tokens_ = None        # self.words = None        # self.words2tokens = None        # self.pos = None    def __str__(self):        return self.title    @property    def raw(self):        if self.raw_ is None:            response = request.urlopen(self.url)            draft = response.read().decode('utf8')            self.raw_ = draft[self.start:self.end]        return self.raw_    @property    def tokens(self):        if self.tokens_ is None:            self.tokens_ = nltk.word_tokenize(self.raw)        return self.tokens_if __name__ == __main__:    books = [Book('Moby Dick', 'http://www.gutenberg.org/files/2701/2701-0.txt', 28876, 28876 + 500),             Book('Great Expectations', 'http://www.gutenberg.org/files/1400/1400-0.txt', 886, 886 + 500)]    for book in books:        print book        print book.tokensI commented out the words, words2tokens and pos attributes as they are not currently needed.Alternatively, if you don't want to insist on the delayed getting of the values, you can do it all already in the constructor:class Book(object):    def __init__(self, title, url, start=0, end=-1):        self.title = title        self.url = url        self.start = start        self.end = end        self.raw = self.get_raw(url)        self.tokens = nltk.word_tokenize(self.raw)        # self.words = None        # self.words2tokens = None        # self.pos = None    def __str__(self):        return self.title    def get_raw(self, url):        response = request.urlopen(url)        draft = response.read().decode('utf8')        return draft[self.start:self.end]Which you can use exactly the same way.