<commit_before>

Table of contents
-----------------
* [Overview of ead_cpf_utils](#overview-of-ead_cpf_utils)
* [What you might need to get started](#what-you-might-need-to-get-started)
* [Getting the code](#getting-the-code)
* [Quickly run the code](#quickly-run-the-code)
* [Creating or obtaining the XML data files](#creating-or-obtaining-the-XML-data-files)
* [Building your own list of WorldCat agency codes](#building-your-own-list-of-worldcat-agency-codes)
* [Common error messages](#common-error-messages)
* [Overview for large input files](#overview-for-large-input-files)
* [Build WorldCat agency codes from a very large input file](#build-WorldCat-agency-codes-from-a-very-large-input-file)
* [What are the other files?](#what-are-the-other-files)
* [Handling large numbers of input records](#handling-large-numbers-of-input-records)
* [Command line params of oclc_marc2cpf.xsl](#command-line-params-of-oclc_marc2cpfxsl)
* [How do I get a block of records from the MARC input?](#how-do-i-get-a-block-of-records-from-the-marc-input)
* [Example config file](#example-config-file)
* [QA notes](#qa-notes)


Overview of ead_cpf_utils
-------------------------

These are XSLT scripts that convert MARC into EAD-CPF (Corporations, Persons, and Families). Some sample data
is included. There are also Perl scripts which are primarily used when the number of input records to be
processed will exceed the memory of the computer.


What you might need to get started
----------------------------------

You will need all the files from the GitHub repository ead_cpf_utils, Saxon 9he (xslt processor), Java (to run
Saxon). While you won't necessarily use every file from the eac_cpf_utils repository, many of the files cross
reference each other.

You will have to run some commands in the terminal window. For Linux I like xterm, although any terminal or
console should be fine. Mac users should use Terminal: Applications -> Utilities -> Terminal.  If you are
using Microsoft Windows, it may be possible to use the command prompt or PowerShell, but I recommend that you
install and use cygwin.
<commit_msg>checkpoint<commit_after>

Table of contents
-----------------
* [Overview of ead_cpf_utils](#overview-of-ead_cpf_utils)
* [What you might need to get started](#what-you-might-need-to-get-started)
* [Getting the code](#getting-the-code)
* [Quickly run the code](#quickly-run-the-code)
* [Creating or obtaining the XML data files](#creating-or-obtaining-the-xml-data-files)
* [Building your own list of WorldCat agency codes](#building-your-own-list-of-worldcat-agency-codes)
* [Common error messages](#common-error-messages)
* [Overview for large input files](#overview-for-large-input-files)
* [Build WorldCat agency codes from a very large input file](#build-worldcat-agency-codes-from-a-very-large-input-file)
* [What are the other files?](#what-are-the-other-files)
* [Handling large numbers of input records](#handling-large-numbers-of-input-records)
* [Command line params of oclc_marc2cpf.xsl](#command-line-params-of-oclc_marc2cpfxsl)
* [How do I get a block of records from the MARC input?](#how-do-i-get-a-block-of-records-from-the-marc-input)
* [Example config file](#example-config-file)
* [QA notes](#qa-notes)


Overview of ead_cpf_utils
-------------------------

These are XSLT scripts that convert MARC into EAD-CPF (Corporations, Persons, and Families). Some sample data
is included. There are also Perl scripts which are primarily used when the number of input records to be
processed will exceed the memory of the computer.


What you might need to get started
----------------------------------

You will need all the files from the GitHub repository ead_cpf_utils, Saxon 9he (xslt processor), Java (to run
Saxon). While you won't necessarily use every file from the eac_cpf_utils repository, many of the files cross
reference each other.

You will have to run some commands in the terminal window. For Linux I like xterm, although any terminal or
console should be fine. Mac users should use Terminal: Applications -> Utilities -> Terminal.  If you are
using Microsoft Windows, it may be possible to use the command prompt or PowerShell, but I recommend that you
install and use cygwin.

{'fraction_non_alphanumeric': 0.12799241526428062, 'fraction_numerical': 0.001422137947380896, 'mean_word_length': 3.9940828402366866, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '698082', 'n_tokens_mistral': 1296, 'n_tokens_neox': 1199, 'n_words': 472}