<jupyter_start><jupyter_text>Homework: _State Your Assumptions_ > POLONIUS\> _What do you read, my lord?_> > HAMLET\> _Words, words, words_> >  -- _Hamlet_, Act 2, Scene 2This homework deals with the assumptions made when taking text from its original "raw" form into something more _computable_.- Assumptions about the _shape_ of text (e.g. how to break a corpus into documents)- Assumptions about what makes a _token_, an _entity_, etc. - Assumptions about what interesting or important _content_ looks like, and how that informs our analyses.There are three parts: 1. Splitting Lines from Shakespeare2. Tokenizing and Aligning lines into plays3. Assessing and comparing characters from within each play**NB**\This file is merely a _template_, with instructions; do not feel constrained to using it directly if you do not wish to.   Get the DataSince the class uses `dvc`, it is possible to get this dataset either using the command line (e.g. `dvc import https://github.com/TLP-COI/text-data-course resources/data/shakespeare/shakespeare.txt`), or using the python api (if you wish to use python)<jupyter_code>from dvc.api import read,get_url
import pandas as pd

txt = read('resources/data/shakespeare/shakespeare.txt', 
           repo='https://github.com/TLP-COI/text-data-course')

print(txt[:250])<jupyter_output>First Citizen:
Before we proceed any further, hear me speak.

All:
Speak, speak.

First Citizen:
You are all resolved rather to die than to famish?

All:
Resolved. resolved.

First Citizen:
First, you know <NAME> is chief enemy to the people.
{'fraction_non_alphanumeric': 0.07869481765834933, 'fraction_numerical': 0.005118362124120281, 'mean_word_length': 4.585714285714285, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 12, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '179726', 'license': 'MIT', 'n_tokens_mistral': 469, 'n_tokens_neox': 441, 'n_words': 204, 'path': 'homework/hw1-assumptions/readme.ipynb', 'repo_name': 'tbsexton/text-data-spr22', 'size': '1570'}