<jupyter_start><jupyter_text>Created by [<NAME>](http://nkelber.com) and Ted Lawless for [JSTOR Labs](https://labs.jstor.org/) under [Creative Commons CC BY License](https://creativecommons.org/licenses/by/4.0/)For questions/comments/improvements, email <EMAIL>.____ Tokenizing Text Files**Description:**You may have text files and metadata that you want to tokenize into ngrams with Python. This notebook takes as input:* Plain text files (.txt) in a folder * A metadata CSV file called 'metadata.csv'and outputs a single JSON-L file containing the unigrams, bigrams, trigrams, full-text, and metadata. **Use Case:** For Researchers (Mostly code without explanation, not ideal for learners)**Difficulty:** Intermediate**Completion time:** 10-15 minutes**Knowledge Required:** * Python Basics ([Start Python Basics I](./python-basics-1.ipynb))**Knowledge Recommended:*** [Working with Dataset Files](./working-with-dataset-files.ipynb)**Data Format:** .txt, .csv, .jsonl**Libraries Used:*** json* collections* pandas* constellate client**Research Pipeline:**1. Scan documents2. OCR files3. Clean up texts4. **Tokenize text files** (this notebook)___  Import Libraries<jupyter_code>from collections import Counter
import gzip
import json
import os

import pandas as pd<jupyter_output><empty_output><jupyter_text>Download and inspect sample filesFor purposes of this tutorial, we will download a set of sample files from Project Gutenberg using a helper function from the `constellate` client.<jupyter_code>from constellate import download_gutenberg_sample

text_file_directory = download_gutenberg_sample()<jupyter_output><empty_output><jupyter_text>You now have sample text files and a CSV of metadata in your data directory. You can list the contents of this directory with this command.<jupyter_code>!ls -lt ~/data/gutenberg-sample<jupyter_output><empty_output><jupyter_text>You can see the first 20 lines of a sample file with this command.<jupyter_code>!head -n 20 ~/data/gutenberg-sample/205-0.txt<jupyter_output><empty_output><jupyter_text>Define a tokenizing function<jupyter_code>def constellate_ngrams(text, n=1):
    # Define a Counter object to hold our ngrams.
    c = Counter()
    # Replace line breaks in the text.
    t = text.replace("\r", " ").replace("\n", "")
    # Convert the text to a list of words.
    words = t.split()
    # Slice the words into ngrams.
    for grams in zip(*[words[i:] for i in range(n)]):
        g = " ".join(grams)
        c[g] += 1
    return c<jupyter_output><empty_output><jupyter_text>Tokenize a textLet's tokenize one of the sample files using our function.<jupyter_code># Read in one of the texts. See note about file paths.
with open(f"{text_file_directory}{os.sep}205-0.txt") as input_file:
    text = input_file.read()
    
unigrams = constellate_ngrams(text)
unigrams.most_common(10)<jupyter_output><empty_output><jupyter_text>You can create bigrams or trigrams (or n grams) by changing the `n` keyword argument passed to the function.<jupyter_code>bigrams = constellate_ngrams(text, n=2)
bigrams.most_common(10)<jupyter_output><empty_output><jupyter_text>Creating a Constellate JSONL fileFor your analysis, you may want to create files that conform to the same data specification as the files provided by Constellate. The following steps show you how to load metadata and the raw text, create ngrams and output a JSONL (JSON lines) file that matches, in format, what you download from the Constellate web application.<jupyter_code>df = pd.read_csv(text_file_directory + os.sep + "metadata.csv")

df.head()<jupyter_output><empty_output><jupyter_text>Loop through the dataframe and print out some of the metadata.<jupyter_code>for item in df.itertuples():
    print(item.title, item.author, item.url)<jupyter_output><empty_output><jupyter_text>Now convert the metadata to the Constellate schema as defined [here](https://docs.constellate.org/what-format-are-jstor-portico-datasets/) by mapping the column names from the source csv to the corresponding Constellate schema attributes.<jupyter_code># Create a list to hold our documents.
documents = []

for item in df.itertuples():
    document = {
        "id": item.url,
        "title": item.title,
        "creator": [item.author],
        "docType": "book",
        "publicationYear": item.published,
        "url": item.url,
        "language": [item.language]
    }
    documents.append(document)<jupyter_output><empty_output><jupyter_text>Now that we have our metadata stored in a list, let's revise our function to capture the full text of the documents and generate ngrams.<jupyter_code># Create a list to hold our documents.
documents = []

for item in df.itertuples():
    document = {
        "id": item.url,
        "title": item.title,
        # A document can have authors/creators, so map as a list.
        "creator": [item.author],
        "docType": "book",
        "publicationYear": item.published,
        "url": item.url,
        # A document can have multiple languages, so map as a list.
        "language": [item.language]
    }
    # Read in the full text
    with open(text_file_directory + "/" + item.file) as text_file:
        text = text_file.read()
    
    # Split the text into pages. See note below.
    document["fullText"] = text.split("\n\n\n")
    # Generate ngrams
    document["unigramCount"] = constellate_ngrams(text, n=1)
    document["bigramCount"] = constellate_ngrams(text, n=2)
    document["trigramCount"] = constellate_ngrams(text, n=3)
    # Add our document to the list of documents
    documents.append(document)
    print(f"{item.title} procesed")<jupyter_output><empty_output><jupyter_text>Inspect the first document and print some of the metadata and content.<jupyter_code>first_doc = documents[0]
print(first_doc["title"], first_doc["publicationYear"])<jupyter_output><empty_output><jupyter_text>Print the twenty five most common trigrams.<jupyter_code>for term, count in first_doc["trigramCount"].most_common(25):
    print(term, count)<jupyter_output><empty_output><jupyter_text>Print the first 500 characters of "page" 20.<jupyter_code>print(first_doc["fullText"][20][:500])<jupyter_output><empty_output><jupyter_text>Generate a Constellate gzip fileYou may now want to create a gzip file so that it matches what you have downloaded from Constellate. You could also then use the `gzip_reader` that's part of the `constellate` client to read it.<jupyter_code>output_file = text_file_directory + os.sep + "sample_gutenberg_dataset.json.gzip"

with gzip.open(output_file, "wb") as handle:
    for doc in documents:
        # Convert the document to a string and add the line separator
        raw = json.dumps(doc) + "\n"
        handle.write(raw.encode())<jupyter_output><empty_output><jupyter_text>Now use the dataset reader to read your file back in and verify it is what we expect.<jupyter_code>from constellate import dataset_reader

for doc in dataset_reader(output_file):
    print(doc["title"], doc["creator"], doc["publicationYear"])
    # See note about assert
    assert(doc["unigramCount"] is not None)
    assert(doc["fullText"] is not None)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10479921645445642, 'fraction_numerical': 0.006436266965160207, 'mean_word_length': 4.830342577487765, 'pattern_counts': {'":': 14, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '236276', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2273, 'n_tokens_neox': 2156, 'n_words': 698, 'path': 'tokenizing-text-files.ipynb', 'repo_name': 'LorenzoBabini/intro', 'size': '7161'}