<jupyter_start><jupyter_text>Assignment 1 - Spatial Analytics <NAME>  Task Using the corpus called 100-english-novels found on the cds-language GitHub repo, write a Python programme which does the following:Calculate the total word count for each novel.Calculate the total number of unique words for each novel.Save result as a single file consisting of three columns: filename, total_words, unique_words Importing packages<jupyter_code>import os
from pathlib import Path
import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>creating data path<jupyter_code>data_path = os.path.join("..","data","100_english_novels", "corpus")

# Count the length of the path to isolate filename in the loop
path_len = len(data_path)<jupyter_output><empty_output><jupyter_text>Defining a function that reads the files from the folder, calculates the wordcount and number of unique words. Lastly it appends the results to a data frame and saves it as a data frame<jupyter_code># making an empty dataframe using pandas
word_count_df = pd.DataFrame(columns = ['filename', 'total_words', 'unique_words'])

# for loop to split and count words
for filename in Path(data_path).glob("*.txt"):
    with open(filename, "r", encoding="utf-8") as file:
        loaded_text = file.read()
        # split on whitespaces
        split_text = loaded_text.split()
        # save word count
        total_words = len(split_text)
        # count unique words with numpy
        values, unique = np.unique(split_text, return_counts=True)
        unique_words = len(unique)
        # saving the filename, total_words and unique_words in a temporary dataframe (this will be overwritten by new iteration)
        temp_df = ({'filename': filename, 'total_words': total_words, 'unique_words': unique_words})
        # appending the temporary dataframe to the empty dataframe
        word_count_df = word_count_df.append(temp_df, ignore_index=True)<jupyter_output><empty_output><jupyter_text>Saving the results<jupyter_code># defining the path for results
path2results = os.path.join("..", "Assignment1", "word_count_df.csv")
# saving the dataframe as csv file
word_count_df.to_csv(path2results)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08190045248868778, 'fraction_numerical': 0.004977375565610859, 'mean_word_length': 4.943548387096774, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '468347', 'license': 'MIT', 'n_tokens_mistral': 656, 'n_tokens_neox': 637, 'n_words': 223, 'path': 'Assignment1/word_counts.ipynb', 'repo_name': 'haniamatera/language_analytics', 'size': '2216'}