<commit_before># Ingest

Ingestion is the process of processing a `.gvcf` file to index information for later processing.<commit_msg>Update ingest.md<commit_after># Ingest

Ingestion is the process of processing a `.gvcf` file to index information for later processing.

The goal here is to scale with the number of 'sequencers'.  The assumption is that single sample VCF files will be imported into the envoronment.   First, in a relatively raw form that can be accessed in Spark/Map-Reduce, but more importantly, also in a 'pre-joined' way inside HBase so that data can be rapidly extracted or cross queried for the clinical oncology use case.  (e.g. I have ~20 variants / genes, show me all samples that share the same variant as my current patient)

The scaling experement verifies that adding additional inputs from sequencers scales linearly with adding additional hardware on the cluster.

'Spark may be a great place to do the injest' -<NAME> 2016
{'fraction_non_alphanumeric': 0.04502617801047121, 'fraction_numerical': 0.0062827225130890054, 'mean_word_length': 4.526011560693641, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '564559', 'n_tokens_mistral': 253, 'n_tokens_neox': 244, 'n_words': 147}