<commit_before># Lightning Documentation : Master

Lightning Concept
---

Lightning is a system that allows efficient access to
large scale population genomic data with a focus
on clinical and research use.
Lightning's primary focus is human genomic
data, but this could be extended to encompass other organisms.
Genomes are stored in a compressed format that compromises
between size and accessibility.
Additional data sources, such as phenotype data from
the Harvard Personal Genome Project and variant data from
the ClinVar database, are added for practical use.

The Lightning system is a combination of a conceptual way
to think about genomes (genomic tiling),
the internal representation of genomes for efficient access
(the compact genome format and auxiliary data), and
software that manages access to the data.

This document will be focusing on some of the concepts
motivating the rest of the architecture and data formats.
Please refer to [Lightning Architecture](Lightning-Architecture.md)
for a description on the Lightning architecture.
and
[Lightning Data](Lightning-Data.md) for references to the
data structures used.

Lightning Data
===

Overview
---

Lightning allows efficient access to large scale population genomic data with a focus
on clinical and research use.
Lightning is for populations human genomic
data but this could be extended to encompass other organisms.
Genomes are stored in a compressed format that is a compromise
between size and accessibility.
Additional data data sources, such as phenotype data from
the Harvard Personal Genome Project and variant data from
the ClinVar database, are added for practical use.

The Lightning system is a combination of a conceptual way
to think about genomes (genomic tiling),
the internal representation of genomes for efficient access
(the compact genome format and auxiliary data) and the
software that manages access to the data.

This document will be focusing on the data files and formats
used in the Lightning system.
Please refer to [Lightning Concepts](Lightning-Concepts.md)
for the description of the tiling system and
[Lightning Architecture](Lightning-Architecture.md) for references to the
engineering architecture.

Data Overview
---

Here is a list of the main data components:
<commit_msg>15115 - Editing Master Lightning Doc<commit_after># Lightning Documentation : Master

Lightning Concept
===

Lightning is a system that allows efficient access to
large scale population genomic data with a focus
on clinical and research use.
Lightning's primary focus is on human genomic
data, but this could be extended to encompass other organisms.
Genomes are stored in a compressed format that compromises
between size and accessibility.
Additional data sources, such as phenotype data from
the Harvard Personal Genome Project, and variant data from
the ClinVar database, are added for practical use.

The Lightning system is a combination of a conceptual way
to think about genomes (genomic tiling),
the internal representation of genomes for efficient access
(the compact genome format and auxiliary data), and
software that manages access to the data.

This document will be focusing on some of the concepts
motivating the rest of the architecture and data formats.
Please refer to [Lightning Architecture](Lightning-Architecture.md)
for a description on the Lightning architecture
and
[Lightning Data](Lightning-Data.md) for references to the
data structures used.

Lightning Data
===

Data Overview
---

Here is a list of the main data components:

{'fraction_non_alphanumeric': 0.03389830508474576, 'fraction_numerical': 0.0014124293785310734, 'mean_word_length': 5.063356164383562, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7191453', 'n_tokens_mistral': 856, 'n_tokens_neox': 780, 'n_words': 508}