<jupyter_start><jupyter_text>Getting startedThis is only quick overview for getting started. Corpus loading, text preprocessing, etc. are explained in depth in the respective chapters. Loading a built-in text corpusOnce you have installed tmtoolkit, you can start by loading a built-in dataset. Let's import the [Corpus](api.rsttmtoolkit-corpus) class first and have a look which datasets are available:<jupyter_code>from tmtoolkit.corpus import Corpus

Corpus.builtin_corpora()<jupyter_output><empty_output><jupyter_text>Let's load one of these corpora, the [News Articles dataset from Harvard Dataverse](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/GMFCTR):<jupyter_code>corpus = Corpus.from_builtin_corpus('english-NewsArticles')
corpus<jupyter_output><empty_output><jupyter_text>We can have a look which documents were loaded (showing only the first ten document labels):<jupyter_code>corpus.doc_labels[:10]<jupyter_output><empty_output><jupyter_text>The first 100 characters from the the document `NewsArticles-1`:<jupyter_code>corpus['NewsArticles-1'][:100]<jupyter_output><empty_output><jupyter_text>The [Corpus](api.rsttmtoolkit-corpus) class is for loading and managing *plain text* corpora, i.e. a set of documents with a label and their content as text strings. It resembles a [Python dictionary](https://docs.python.org/3/tutorial/datastructures.htmldictionaries). See [working with text corpora](text_corpora.ipynb) for more information. Tokenizing a corpusFor quantitative text analysis, you usually work with words in documents as units of interest. This means the plain text strings in the corpus' documents need to be split up into individual *tokens* (words, punctuation, etc.). For a quick starter, we can do so by using [tokenize](api.rsttmtoolkit.preprocess.tokenize):<jupyter_code>from tmtoolkit.preprocess import tokenize

doc_labels = corpus.doc_labels   # save the document labels as list for later use

docs = tokenize(corpus.values())<jupyter_output><empty_output><jupyter_text>The function `tokenize()` takes a sequence of text strings, tokenizes them and returns a list of tokenized documents:<jupyter_code>type(docs)<jupyter_output><empty_output><jupyter_text>Each document in `docs` in turn is a list of token strings (words, punctuation). Let's peek into the first document (index 0) and return the first ten tokens from it:<jupyter_code>docs[0][:10]<jupyter_output><empty_output><jupyter_text>`docs` and `doc_labels` are aligned, i.e. the first element in `doc_labels` is the label of the first tokenized document in `docs`:<jupyter_code>doc_labels[0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08781226343679031, 'fraction_numerical': 0.008327024981074944, 'mean_word_length': 7.182662538699691, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '269268', 'license': 'Apache-2.0', 'n_tokens_mistral': 835, 'n_tokens_neox': 795, 'n_words': 262, 'path': 'doc/source/getting_started.ipynb', 'repo_name': 'shalevy1/tmtoolk', 'size': '2642'}