<jupyter_start><jupyter_text>Dividing your text into equal unitsWhen working with an unstructured text, we might want to divide into different units to support our analysis. We got into this a little bit in the section on [working with plain text](plain_text.ipynb), when we discussed how we might break a text apart into structural components that may or not be repersented in the text itself in easy to grab ways. We discussed:* breaking the text into lines* breaking the text into chapters* breaking a poem into stanzasThose structural units come from the text in some way, but there are others way to divide up texts that might be helpful for certain types of text analysis. In particular, it's often advantageous to know how to quickly divide a single long work into smaller units. In this section we'll talk briefly about how and why you might do so.In a common example, we might want to get a more nuanced sense of how particular modes of analysis might change over the course of a text. To do that, the first necessary action is to divide the text into smaller equal portions that can be individually analyzed. Below we divide the text of Jacob's Room into 100 even pieces.<jupyter_code>import math

# Read in the text
filename = 'corpus/woolf/1922_jacobs_room.txt'
with open(filename, 'r') as fin:
    text = fin.read()

# get the length of the text
text_length = len(text)
text_chunks = []
# specify the number of chunks
number_of_chunks = 100

# loop over the length of text, divide it into pieces, and make a new list of the pieces
for i in range(number_of_chunks):
    chunk_size = text_length/number_of_chunks
    chunk_start = math.floor(chunk_size * i)
    chunk_end = math.floor(chunk_size * (i +1))
    text_chunks.append(text[chunk_start:chunk_end])


print('number of chunks in this text: ' + str(len(text_chunks)))
print('length of chunk 1: ' + str(len(text_chunks[0])))
print('length of chunk 2: ' + str(len(text_chunks[1])))
print('length of chunk 3: ' + str(len(text_chunks[2])))<jupyter_output>number of chunks in this text: 100
length of chunk 1: 3138
length of chunk 2: 3138
length of chunk 3: 3139<jupyter_text>Dividing up the text in this way provides us with a series of small texts, each of which can be subjected to analysis. We can then string together the analysis of these smaller pieces to make arguments about trends in the overal piece. Below we take the same bit of code, wrap it into a function, and then use it to track changes in the use of the word Jacob over the course of the novel. This will allow us to more easily drop it in where we need it.<jupyter_code>import nltk
import matplotlib.pyplot as plt

def get_chunks(text, num_chunks):
    # takes in a text and a specified number of chunks. divides that text into those equal units
    text_length = len(text)
    text_chunks = []
    number_of_chunks = num_chunks
    for i in range(number_of_chunks):
        chunk_size = text_length/number_of_chunks
        chunk_start = math.floor(chunk_size * i)
        chunk_end = math.floor(chunk_size * (i +1))
        text_chunks.append(text[chunk_start:chunk_end])
    return text_chunks

# reads in the text
filename = 'corpus/woolf/1922_jacobs_room.txt'
with open(filename, 'r') as fin:
    raw_text = fin.read()

    
# creates a divide version of the text
chunked_text = get_chunks(raw_text, 100)

# tokenizes each divided piece of the text
tokenized_text = [nltk.word_tokenize(chunk) for chunk in chunked_text]

# counts the number of occurrences of 'Jacob' in any given unit
jacob_counts = [nltk.FreqDist(tokenized_chunk)['Jacob'] for tokenized_chunk in tokenized_text]

# prints the results
print(jacob_counts)<jupyter_output>[2, 6, 4, 0, 1, 1, 0, 3, 3, 7, 2, 0, 0, 4, 2, 2, 1, 2, 8, 2, 0, 1, 4, 5, 6, 3, 12, 2, 0, 0, 4, 1, 3, 13, 1, 1, 1, 6, 4, 4, 4, 7, 6, 6, 5, 0, 1, 2, 7, 6, 2, 3, 0, 3, 6, 5, 6, 6, 4, 6, 4, 5, 2, 2, 2, 8, 0, 3, 10, 3, 7, 9, 9, 5, 1, 2, 5, 6, 7, 0, 4, 9, 3, 1, 6, 6, 3, 3, 8, 7, 4, 1, 4, 6, 7, 6, 4, 0, 7, 4]<jupyter_text>Remember - we have divided the text up into 100 (roughly) equal units. Using the FreqDist() module in the NLTK package we get a quick count of the word 'Jacob' in the text. The result is that we use our divided version of the text to create a new list of word counts for 'Jacob' in each piece. We can then take this information and plot it to get a sense of the change in usage over the course of the novel.<jupyter_code>plt.style.use('seaborn-whitegrid')
plt.plot(jacob_counts)
plt.show()<jupyter_output><empty_output><jupyter_text>This visualization might be a little difficult to read. It looks pretty noisy. Fortunately we already have code that can help us parse things a little differently. Rather than slotting things into 100 equal parts, we can use the same function to shift to ten equal chunks for the whole novel.<jupyter_code>chunked_text = get_chunks(raw_text, 10)
tokenized_text = [nltk.word_tokenize(chunk) for chunk in chunked_text]
jacob_counts = [nltk.FreqDist(tokenized_chunk)['Jacob'] for tokenized_chunk in tokenized_text]
print(jacob_counts)
plt.style.use('seaborn-whitegrid')
plt.plot(jacob_counts)
plt.show()<jupyter_output>[27, 23, 33, 38, 44, 41, 39, 52, 50, 43]
{'fraction_non_alphanumeric': 0.08201466615206483, 'fraction_numerical': 0.033577769201080666, 'mean_word_length': 4.432914046121593, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '172375', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1835, 'n_tokens_neox': 1636, 'n_words': 757, 'path': 'dividing.ipynb', 'repo_name': 'walshbr/humanists-nlp-cookbook', 'size': '5182'}