<commit_before># MM/LTOpers Database Features

## Introduction
The MM database supports the storing and querying of a variety of information created by both Media Microservices and LTOpers. This document will provide a breakdown of the structure of the database, as well as examples of the scripts currently used to interact with it.

## Database structure
The Database is divided into six tables with the following structure:

***

#### object:
__Contains identifying information for objects in the database.__

|objectIdentifierValueID|objectIdentifierValue|objectDB_Insertion|object_LastTouched|
|---|---|---|---|
|Auto-incrementing number for object records|Name of object|Date/Time Object added to DB|Date/Time Object was last updated|

***

#### event:
__Tracks occurences of events affecting Objects contained in database.__

|eventIdentifierValueID|objectIdentifierValue|eventType|eventDateTime|eventDetail|eventOutcome|eventDetailOPT|eventDetailCOMPNAME|linkingAgentIdentifierValue|
|---|---|---|---|---|---|---|---|---|
|Auto-incrementing number for event records|Name of target object for event|PREMIS type for event|Date/Time of event|Name of script|Outcome of event|Options used for script|Computer used for event|User who initiated event|

***

#### fixity
__Captures fixity information generated by fixity events.__

|fixityIdentifierValueID|eventIdentifierValue|objectIdentifierValue|eventDateTime|eventDetail|messageDigestAlgorithm|messageDigestSOURCE|messageDigestPATH|messageDigestFILENAME|messageDigestHASH|
|---|---|---|---|---|---|---|---|---|---|
|Auto-incrementing number for fixity records|ID of corresponding event|Name of corresponding object|Date/Time of fixity event|Script name|Hashing algorithm|Source of hash|Location of hash|Name of file hashed|Hash|

***

#### objectCharacteristics
__Captures information from sidecar files generated during microservices ingest.__

|objectCharacteristicValueID|mediaInfo|captureLog|
|---|---|---|
|Auto-incrementing number for object characteristic records|Contents of associated media info xml file|Contents of associated capture log|

***

#### ltoSchema
__Captures contents of LTO schema files generated through LTOpers.__

|ltoSchemaValueID|ltoID|fileName|filePath|fileSize|modifyTime|
|---|---|---|---|---|---|
|Auto-incrementing number for object characteristic records|ID for corresponding tape|file name|File path|File size|Date/Time of last file modification|

***

#### fingerprints
__Stores fingerprints (perceptual hash information).__

|hashNumber|objectIdentifierValue|startframe|endframe|hash1|hash2|hash3|hash4|hash5|
|---|---|---|---|---|---|---|---|---|
|Auto-incrementing number for hashes|Name of corresponding item|Start frame for hash|End frame for hash|First component of hash|Second component of hash|Third component of hash|Fourth component of hash|Fifth component of hash|
<commit_msg>add script information<commit_after># MM/LTOpers Database Features

## Introduction
The MM database supports the storing and querying of a variety of information created by both Media Microservices and LTOpers. This document will provide a breakdown of the structure of the database, as well as examples of the scripts currently used to interact with it.

## Database structure
The Database is divided into six tables with the following structure:

***

#### object:
__Contains identifying information for objects in the database.__

|objectIdentifierValueID|objectIdentifierValue|objectDB_Insertion|object_LastTouched|
|---|---|---|---|
|Auto-incrementing number for object records|Name of object|Date/Time Object added to DB|Date/Time Object was last updated|

***

#### event:
__Tracks occurences of events affecting Objects contained in database.__

|eventIdentifierValueID|objectIdentifierValue|eventType|eventDateTime|eventDetail|eventOutcome|eventDetailOPT|eventDetailCOMPNAME|linkingAgentIdentifierValue|
|---|---|---|---|---|---|---|---|---|
|Auto-incrementing number for event records|Name of target object for event|PREMIS type for event|Date/Time of event|Name of script|Outcome of event|Options used for script|Computer used for event|User who initiated event|

***

#### fixity
__Captures fixity information generated by fixity events.__

|fixityIdentifierValueID|eventIdentifierValue|objectIdentifierValue|eventDateTime|eventDetail|messageDigestAlgorithm|messageDigestSOURCE|messageDigestPATH|messageDigestFILENAME|messageDigestHASH|
|---|---|---|---|---|---|---|---|---|---|
|Auto-incrementing number for fixity records|ID of corresponding event|Name of corresponding object|Date/Time of fixity event|Script name|Hashing algorithm|Source of hash|Location of hash|Name of file hashed|Hash|

***

#### objectCharacteristics
__Captures information from sidecar files generated during microservices ingest.__

|objectCharacteristicValueID|mediaInfo|captureLog|
|---|---|---|
|Auto-incrementing number for object characteristic records|Contents of associated media info xml file|Contents of associated capture log|

***

#### ltoSchema
__Captures contents of LTO schema files generated through LTOpers.__

|ltoSchemaValueID|ltoID|fileName|filePath|fileSize|modifyTime|
|---|---|---|---|---|---|
|Auto-incrementing number for object characteristic records|ID for corresponding tape|file name|File path|File size|Date/Time of last file modification|

***

#### fingerprints
__Stores fingerprints (perceptual hash information).__

|hashNumber|objectIdentifierValue|startframe|endframe|hash1|hash2|hash3|hash4|hash5|
|---|---|---|---|---|---|---|---|---|
|Auto-incrementing number for hashes|Name of corresponding item|Start frame for hash|End frame for hash|First component of hash|Second component of hash|Third component of hash|Fourth component of hash|Fifth component of hash|

## Database Search Scripts

#### searchlto

Searches the LTO schema information stored in the database. Input for standard search can consist either of a Tape ID or a Media ID. If the `-c` flag is used, input is a file to be compared against information stored in the DB. Input file will be compared for modification time, file size and file name.

Usage: searchlto [-h] (help) [-c] (compares an input file to file information stored in DB) [-f] (shows full schema) [Input]



#### searchfingerprint

Generates fingerprints (perceptual hashes) from whole or specified portion of an input video file and compares them against fingerprints stored in the mm database. Outputs any detected matches in 500 frame segments both in the terminal and in an optional preview window. Preview window will attempt to locate the portions of input video for which matches were found.

Usage: "searchfingerprint" [ -h ] (help) [ -i ] (set in time) [ -o ] (set out time) [ -t ] (text only-don't display video preview) [Input]

{'fraction_non_alphanumeric': 0.11050457362053703, 'fraction_numerical': 0.0019179699026261434, 'mean_word_length': 5.401322001888574, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1413495', 'n_tokens_mistral': 1735, 'n_tokens_neox': 1690, 'n_words': 656}