<jupyter_start><jupyter_text>I'm experimenting here with a process to turn annotations created within PDF files stored as part of a Zotero library into metadata contents and structured annotations for the bibliographic record. This is essentially for cases where there is no good citation metadata already in existence somewhere on the web (e.g., for certain types of government reports) and we need to extract that content from within PDFs. It's also for cases where built-in structured PDF metadata is no good, which is the case for anything other than professionally built PDFs (e.g., just exporting a PDF from your word processor does not build a good PDF). This technique also holds promise for setting up training data for building various kinds of entity recognition models to auto-extract particular concepts from full texts processed with NLP.I used the ZotFile plugins for Zotero, inspired by [this video](https://www.youtube.com/watch?v=_Fjhad-Z61o&t=1251s). In Zotero, the process includes storing the PDF file as an attachment so that Zotero is "managing" it, annotating the file using some type of PDF markup tool (I used Preview on Mac), and then running the ZotFile tool to extract annotations from the PDF which creates a rich text (html) note for the item in Zotero. The notes are synced to the group library with Zotero online where they can be picked up for processing.For annotation, I used a combination of highlighting particular text and then tagging that text with a keyword corresponding to a target part of the citation metadata I'm trying to identify (e.g., title, authors, etc.). I should then be able to pull these two pieces out of the generated markdown into a data structure that I can feed back into the corresponding record via the Zotero API.For the Python processing workflow part of this, I used the pyzotero package to connect to the Zotero group library, read items with notes, and then processed those items to generate structured data from the HTML notes that can be reinjected back into the items.<jupyter_code>!pip install pyzotero
from bs4 import BeautifulSoup
import re
from pyzotero import zotero
from getpass import getpass<jupyter_output><empty_output><jupyter_text>This function handles the process of working a given Zotero child item's note content to detect if it was generated using the ZotFile process and then extracting the annotations into a usable data structure (list of dictionaries).<jupyter_code>annotation_property_map = [
    {
        "property_name": "title",
        "property_type": "single"
    },
    {
        "property_name": "date",
        "property_type": "single"
    },
    {
        "property_name": "institution",
        "property_type": "single"
    },
    {
        "property_name": "author",
        "property_type": "multi"
    },
    {
        "property_name": "project",
        "property_type": "tag"
    },
    {
        "property_name": "place",
        "property_type": "tag"
    },
    {
        "property_name": "commodity",
        "property_type": "tag"
    },
]

def structured_annotations(item_key, annotation_html, property_map=annotation_property_map):
    extract_keywords = [i["property_name"] for i in property_map]

    annotations_soup = BeautifulSoup(annotation_html, 'html.parser')
    pattern = '\"(.*?)\"'

    paragraphs = annotations_soup.find_all("p")

    if "Extracted Annotations" not in paragraphs[0].text:
        return None

    annotation_texts = list()
    for index,p in enumerate(paragraphs):
        em_in_p = p.find("em")
        if em_in_p:
            p_text = em_in_p.text
        else:
            p_text = p.text

        p_text_parts = p_text.split()

        if len(p_text_parts) > 0 and p_text_parts[0] in extract_keywords:
            prop = p_text_parts[0]
            annotation_text = re.search(pattern, paragraphs[index - 1].text)
            if annotation_text is not None:
                annotation_texts.append({
                    "item_key": item_key,
                    "text": annotation_text.group(1),
                    "property": prop
                })
    return annotation_texts

def update_zotero_item(
    item_key, 
    annotations_list, 
    zotero_api, 
    commit_update=True,
    property_map=annotation_property_map):
    update_item = zotero_api.item(item_key)
    if not update_item:
        return

    available_updates = [i for i in annotations_list if i["item_key"] == item_key]
    if available_updates:
        single_value_props = [i["property_name"] for i in property_map if i["property_type"] == "single"]
        tag_props = [i["property_name"] for i in property_map if i["property_type"] == "tag"]

        for prop in single_value_props:
            update_value = next((i["text"] for i in available_updates if i["property"] == prop), None)
            if update_value is not None:
                update_item["data"][prop] = update_value

        update_item["data"]["place"] = ",".join([i["text"] for i in available_updates if i["property"] == "place"])
        update_item["data"]["creators"] = [{'creatorType': 'author', 'name': i["text"]} for i in available_updates if i["property"] == "author"]
        update_item["data"]["tags"] = [{'tag': i["text"]} for i in available_updates if i["property"] in tag_props]

    if commit_update:
        zotero_api.update_item(update_item)

    return update_item<jupyter_output><empty_output><jupyter_text>To interface with the Zotero API, you need to provide a library ID and an API key. This should work for anyone with a group library that has followed the same process I outlined above.<jupyter_code>zot = zotero.Zotero(input("Library ID "), "group", getpass(prompt="API Key "))<jupyter_output>Library ID 4373054
API Key ··········<jupyter_text>This process would need to be worked out further in production practice, but we essentially walk the items in a given library looking for notes. There might be some more efficient way to zero in on these, but I haven't figured it out yet with pyzotero. Here, we make a pass through every item, look for items with children, get the children, and then get any that have notes. I assume that this might be additive where all note items that can be processed will be yielding annotations for the given parent item. I send the note html along with the associated item key to the function to return available structured annotations for further processing. Since every annotation will essentially contain a key/value pair (property and text content), we can simply build out an array of these with their item keys for further processing.<jupyter_code>item_annotations = list()
for item in zot.all_top():
    if item["meta"]["numChildren"] > 0:
        note_children = [i for i in zot.children(item["key"]) if i["data"]["itemType"] == "note"]
        if note_children:
            for note_child in note_children:
                extracted_annotations = structured_annotations(item["key"], note_child["data"]["note"])
                if extracted_annotations:
                    item_annotations.extend(extracted_annotations)<jupyter_output><empty_output><jupyter_text>In this case, we got the one item where I've annotated and then extracted text snippets corresponding to specific metadata elements I'm identifying and wanting to work with.<jupyter_code>item_annotations<jupyter_output><empty_output><jupyter_text>I added an additional function to work through the annotations gathered and commit them back to the respective Zotero items cataloging the annotated files. This basically roundtrips the process, letting us propose a workflow concentrated on marking up "messy" PDF files using meta keywords and then leveraging Zotero and ZotFile to build out the catalog records from the annotation markup.<jupyter_code>for item_key in list(set([i["item_key"] for i in item_annotations])):
    update_zotero_item(item_key, item_annotations, zot, commit_update=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07865309712275412, 'fraction_numerical': 0.002638522427440633, 'mean_word_length': 3.8067632850241546, 'pattern_counts': {'":': 17, '<': 24, '<?xml version=': 0, '>': 26, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '510798', 'license': 'Unlicense', 'n_tokens_mistral': 2209, 'n_tokens_neox': 2119, 'n_words': 890, 'path': 'Extracted_PDF_Annotation_via_Zotero.ipynb', 'repo_name': 'skybristol/experiments', 'size': '7959'}