<jupyter_start><jupyter_text>Notes:In this iteration, I want to experiment with creating datasets like the one described [here](https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus) from the PDFs that Robert's already sent over. In the initial hypothesis of what V2 would entail, I figured that the process to create the first dataset from an initial PDF would be a bit painful.  Some Concerns:I wonder if it might not just be more prudent to search out a pretrained dataset for this exact circumstance. It seems like there's more available than I thought, and they may have been created by trained doctors instead of my kind of random floundering. I also haven't done a deep dive into Google's own NLP resources, which I abandoned because I thought they wouldn't suit the task at hand. Current Strategy I'm planning on using python to automatically write the csv for me, because the process looks like it's done in a repetitive fashion as long as I can write the rules well enough. I think it can write all the rules so they apply to a Pandas dataframe, and then once I have the frame created, I can just write it to a csv.  Pitfalls/Need to Do'sVery uncertain on how a PD actually handles slicing - the original tutorial on Treehouse wasn't very helpful. I know it involves two properties of the dataframe, `.loc` is one of them, but they seemed tempermental when the dude was going over them, so I'm not 100% sure how to use them.<jupyter_code>import os
import csv
import nltk
import numpy as np
import pandas as pd
import re<jupyter_output><empty_output><jupyter_text>ColumnsIndex no. = sentence number lemma = lemma of a token in sentence next-lemma = lemma of next token in sentence next-next-lemma = Lemma of token at +2nd position to the current token in sentence<jupyter_code># list of strings
myColumns = ['no.','lemma','next-lemma','next-next-lemma']
# list of integers
myIndex = [1, 2, 3]
# numpy array (if you try and run a single calc repetitively, you need .T to transpose it)
data = np.array([np.arange(3)]*4).T

ann_corpus = pd.DataFrame(data, index=myIndex, columns=myColumns)
ann_corpus<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.058768656716417914, 'fraction_numerical': 0.005597014925373134, 'mean_word_length': 4.813008130081301, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '587227', 'license': 'Apache-2.0', 'n_tokens_mistral': 603, 'n_tokens_neox': 569, 'n_words': 322, 'path': 'notes/NER-v2.ipynb', 'repo_name': 'ncaldwell17/Mercury', 'size': '2144'}