<commit_before>= Digitization Locker

Digitization Locker is a demonstration project for streamlining digitization
workflows using a microservices approach. 

== The Story

The basic idea is that there is a space where TIFs can be dropped
from scanning workstations. This directory is called the locker. When a file
has been stable in the locker for a number of checks, then it is moved into
the holding area. Every night the files in the holding area are processed.

== Processing the Holding Tank

Processing of files currently involves:

1. Moving the master into archival storage in a pairtree.
2. Creating a stub metadata record.
3. If the files are {mead}[https://github.com/jronallo/mead] identifiers 
   and the EAD XML is accessible, then stub descriptive metadata in the form
   of a a title is added.
4. Creation of a JPG access derivative which is placed in a separate
   pairtree from access copies.
  

== Configuration
  
Many of the options are configurable. See config/app_config.yml.
  
<commit_msg>correct typo<commit_after>= Digitization Locker

Digitization Locker is a demonstration project for streamlining digitization
workflows using a microservices approach. 

== The Story

The basic idea is that there is a space where TIFs can be dropped
from scanning workstations. This directory is called the locker. When a file
has been stable in the locker for a number of checks, then it is moved into
the holding area. Every night the files in the holding area are processed.

== Processing the Holding Tank

Processing of files currently involves:

1. Moving the master into archival storage in a pairtree.
2. Creating a stub metadata record.
3. If the files are {mead}[https://github.com/jronallo/mead] identifiers 
   and the EAD XML is accessible, then stub descriptive metadata in the form
   of a title is added.
4. Creation of a JPG access derivative which is placed in a separate
   pairtree from access copies.
  

== Configuration
  
Many of the options are configurable. See config/app_config.yml.
  

{'fraction_non_alphanumeric': 0.03938946331856229, 'fraction_numerical': 0.003938946331856229, 'mean_word_length': 3.9926289926289926, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1813688', 'n_tokens_mistral': 550, 'n_tokens_neox': 515, 'n_words': 302}