<jupyter_start><jupyter_text>Downloading subject headings from the library of congress LoC don't make it easy to get a complete list of subject headings. The neatest route seems to be the [downloads section of the LoC website](https://id.loc.gov/download/), where they make a [skos](https://www.w3.org/2004/02/skos/), [ndjson](http://ndjson.org/) version of the headings available.In this notebook we'll download the file, and parse out the useful information from it. In the next notebook we'll focus on getting meaningful results from the intersection of this dataset and the queries sent through [wellcomecollection.org/collections](https://wellcomecollection.org/collections).<jupyter_code>import zipfile
from pathlib import Path

import httpx
import orjson
from tqdm.notebook import tqdm
url = "https://lds-downloads.s3.amazonaws.com/lcsh.skos.ndjson.zip"
filename = Path(url).name
data_dir = Path("../data/lcsh")

if not data_dir.exists():
    data_dir.mkdir()<jupyter_output><empty_output><jupyter_text>We've defined where we want to fetch the file from, and where we want to save it - now we just need to download it. It's a fairly large file so I've added a progress bar<jupyter_code>file_path = data_dir / filename
if not file_path.exists():
    with open(file_path, "wb") as download_file:
        with httpx.stream("GET", url) as response:
            total = int(response.headers["Content-Length"])
            with tqdm(
                total=total, unit_scale=True, unit_divisor=1024, unit="B", desc=filename
            ) as progress:
                num_bytes_downloaded = response.num_bytes_downloaded
                for chunk in response.iter_bytes():
                    download_file.write(chunk)
                    progress.update(
                        response.num_bytes_downloaded - num_bytes_downloaded
                    )
                    num_bytes_downloaded = response.num_bytes_downloaded<jupyter_output><empty_output><jupyter_text>We also need to unzip the file.<jupyter_code>with zipfile.ZipFile(file_path, "r") as zip_ref:
    zip_ref.extractall(data_dir)<jupyter_output><empty_output><jupyter_text>extract the useful dataThis is a pretty big file - let's find out how many lines (ie records) it contains<jupyter_code>! wc -l ../data/lcsh/lcsh.skos.ndjson<jupyter_output><empty_output><jupyter_text>450645 lines is probably big enough to be worth iterating through it gradually, rather than reading it all at once. Let's set up a function to yield lines one-by-one<jupyter_code>def load_records(file_path):
    with open(file_path) as f:
        while line := f.readline():
            yield orjson.loads(line)<jupyter_output><empty_output><jupyter_text>all that's left to do now is work through each of those records and extract the LCSH ID and the heading (`prefLabel`) for each record<jupyter_code>data = {}

generator = load_records(data_dir / Path(url).stem)

for record in tqdm(generator, total=450645):
    lcsh_id = Path(record["@context"]["about"]).name
    for item in record["@graph"]:
        if item["@id"] == record["@context"]["about"]:
            try:
                data[lcsh_id] = item["skos:prefLabel"]["@value"]
            except KeyError:
                # have inspected these lines. it looks like they're
                # all duplicate/deleted records
                pass<jupyter_output><empty_output><jupyter_text>Now we can save our cleaned records to the `/data` directory for use in future notebooks<jupyter_code>with open(data_dir / "lcsh_ids_and_labels.json", "wb") as f:
    f.write(orjson.dumps(data))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09697977279024661, 'fraction_numerical': 0.006650041562759767, 'mean_word_length': 3.628205128205128, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '377814', 'license': 'MIT', 'n_tokens_mistral': 1117, 'n_tokens_neox': 1078, 'n_words': 325, 'path': 'notebooks/concepts/notebooks/01 - fetching LCSH labels.ipynb', 'repo_name': 'wellcomecollection/data-science', 'size': '3609'}