<jupyter_start><jupyter_text>Using an already annotated corpus This notebook assumes that you are familiar with the basic usage of PyLigher. If not you can read the README.md or check the Simple_usage notebook. In most cases, you want to use an already annotated corpus, or the one you just annotated. Hopefully, PyLighter provides an easy solution for this.  Loading the corpus and its labels In this example, we will use the file _annotation.csv_, but you can use any data you want as long as it is in IOB2 format with annotation charachter by charachter.<jupyter_code>import pandas as pd
df = pd.read_csv("annotation.csv", sep=";")
import ast
corpus = df.document.to_list()
labels = df.labels.apply(ast.literal_eval).to_list()<jupyter_output><empty_output><jupyter_text>Note the usage of _ast.literal_eval_ to transform a representation of list as string to a list.Moreover in most cases, you don't want to start back from 0. So you can use the argument _start_index_ to start back where you left. In this example, we will assume that you left at the second document.<jupyter_code>start_index = 1<jupyter_output><empty_output><jupyter_text>Start annotating<jupyter_code>from pylighter import Annotation
annotation = Annotation(corpus, start_index=start_index, labels=labels)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.061208875286916604, 'fraction_numerical': 0.0022953328232593728, 'mean_word_length': 5.88421052631579, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '463600', 'license': 'MIT', 'n_tokens_mistral': 389, 'n_tokens_neox': 368, 'n_words': 164, 'path': 'demo/Annotated_corpus.ipynb', 'repo_name': 'infinex/PyLighter', 'size': '1307'}