<jupyter_start><jupyter_text>Goal:Outline the process of producing shared QC metric schema that delegates to picard names when they are adequately descriptive of what they measure. The following two workflows were used to extract metrics, and files were downloaded to `picard_metric_dir` and `optimus_metric_dir`: ```https://job-manager.mint-dev.broadinstitute.org/jobs/a39b92db-bed0-40d4-83de-3ca0505dc5a8   10x v2https://job-manager.mint-dev.broadinstitute.org/jobs/b9ff68b4-2434-4909-8275-850cb84ebb13   ss2```<jupyter_code>import os
from crimson import picard<jupyter_output><empty_output><jupyter_text>Examine SS2 pipeline metrics outputsListed below are the file names of metrics files emitted by a smart-seq2 workflow<jupyter_code>picard_metric_dir = os.path.expanduser('~/Desktop/picard')
!ls $picard_metric_dir<jupyter_output>SRR1294925_qc.alignment_summary_metrics.txt
SRR1294925_qc.bait_bias_detail_metrics.txt
SRR1294925_qc.bait_bias_summary_metrics.txt
SRR1294925_qc.base_distribution_by_cycle_metrics.txt
SRR1294925_qc.error_summary_metrics.txt
SRR1294925_qc.gc_bias.detail_metrics.txt
SRR1294925_qc.gc_bias.summary_metrics.txt
SRR1294925_qc.pre_adapter_detail_metrics.txt
SRR1294925_qc.quality_by_cycle_metrics.txt
SRR1294925_qc.quality_distribution_metrics.txt<jupyter_text>This method parses a few of the files that are in a consistent format<jupyter_code>metric_files = [os.path.join(picard_metric_dir, f) for f in os.listdir(picard_metric_dir)]

def parse_picard(metric_file):
    with open(metric_file, 'r') as f:
        json_data = picard.parse(f)
        metric_class_name = json_data['metrics']['class']
        metrics = {}
        for d in json_data['metrics']['contents']:
            for k, v in d.items():
                metrics[k] = type(v)
        del metrics['SAMPLE_ALIAS'], metrics['LIBRARY']
    return metric_class_name, metrics<jupyter_output><empty_output><jupyter_text>This is a map between the metric class and the names of metrics calculated by each class, mapped to the output type. Caveat: 5 of the files don't decode. Those are printed in full below.<jupyter_code>all_metrics_and_names = {}
for m in metric_files[:-2]:
    try:
        all_metrics_and_names.__setitem__(*parse_picard(m))
    except:
        print(m)
all_metrics_and_names<jupyter_output><empty_output><jupyter_text>Below, files that didn't convert are just printed to console to get a sense of their metric names<jupyter_code>!cat $picard_metric_dir/SRR1294925_qc.base_distribution_by_cycle_metrics.txt
!cat $picard_metric_dir/SRR1294925_qc.gc_bias.summary_metrics.txt
!cat $picard_metric_dir/SRR1294925_qc.gc_bias.detail_metrics.txt
!cat $picard_metric_dir/SRR1294925_qc.error_summary_metrics.txt
!cat $picard_metric_dir/SRR1294925_qc.quality_by_cycle_metrics.txt
!cat $picard_metric_dir/SRR1294925_qc.alignment_summary_metrics.txt<jupyter_output>## htsjdk.samtools.metrics.StringHeader
# CollectMultipleMetrics INPUT=/cromwell_root/broad-dsde-mint-dev-cromwell-execution/cromwell-executions/TestSmartSeq2SingleCellPR/dbec853f-f908-44d5-abf8-c2b3e9a1c1dd/call-target_workflow/SmartSeq2SingleCell/efca6617-3b23-4620-8227-dd9484b9547f/call-HISAT2PairedEnd/SRR1294925_qc.bam ASSUME_SORTED=true OUTPUT=SRR1294925_qc METRIC_ACCUMULATION_LEVEL=[ALL_READS] FILE_EXTENSION=.txt PROGRAM=[CollectAlignmentSummaryMetrics, CollectInsertSizeMetrics, CollectGcBiasMetrics, CollectBaseDistributionByCycle, QualityScoreDistribution, MeanQualityByCycle, CollectSequencingArtifactMetrics, CollectQualityYieldMetrics] VALIDATION_STRINGENCY=SILENT REFERENCE_SEQUENCE=/cromwell_root/hca-dcp-mint-test-data/reference/GRCh38_Gencode/GRCh38.primary_assembly.genome.fa    STOP_AFTER=0 INCLUDE_UNPAIRED=false VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_[...]<jupyter_text>Optimus MetricsNow, do the same for Optimus metrics. Optimus has all of the metrics in one file, although may not have the depth of analysis that the picard ones do. We could use picard + user research to identify missing metrics and expand our complement as recommended.<jupyter_code>import pandas as pd

optimus_metric_dir = os.path.expanduser('~/Desktop/optimus')

print('cell metrics\n')
for c in pd.read_csv(os.path.join(optimus_metric_dir, 'merged-cell-metrics.csv.gz')).columns[1:]:
    print(c)

print('\ngene metrics\n')
for c in pd.read_csv(os.path.join(optimus_metric_dir, 'merged-gene-metrics.csv.gz')).columns[1:]:
    print(c)<jupyter_output>cell metrics

n_reads
noise_reads
perfect_molecule_barcodes
reads_mapped_exonic
reads_mapped_intronic
reads_mapped_utr
reads_mapped_uniquely
reads_mapped_multiple
duplicate_reads
spliced_reads
antisense_reads
molecule_barcode_fraction_bases_above_30_mean
molecule_barcode_fraction_bases_above_30_variance
genomic_reads_fraction_bases_quality_above_30_mean
genomic_reads_fraction_bases_quality_above_30_variance
genomic_read_quality_mean
genomic_read_quality_variance
n_molecules
n_fragments
reads_per_molecule
reads_per_fragment
fragments_per_molecule
fragments_with_single_read_evidence
molecules_with_single_read_evidence
perfect_cell_barcodes
reads_mapped_intergenic
reads_unmapped
reads_mapped_too_many_loci
cell_barcode_fraction_bases_above_30_variance
cell_barcode_fraction_bases_above_30_mean
n_genes
genes_detected_multiple_observations

gene metrics

n_reads
noise_reads
perfect_molecule_barcodes
reads_mapped_exonic
reads_mapped_intronic
reads_mapped_utr
reads_mapped_uniquely
reads_mapped_[...]
{'fraction_non_alphanumeric': 0.07172463506938187, 'fraction_numerical': 0.04379167417552712, 'mean_word_length': 7.712715855572998, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '505699', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2206, 'n_tokens_neox': 1958, 'n_words': 333, 'path': 'analyses/metrics/determine_qc_metric_schema.ipynb', 'repo_name': 'HumanCellAtlas/skylab-analysis', 'size': '5549'}