<jupyter_start><jupyter_text>Which Illumina replicates have the most reads<jupyter_code>log = """Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P201 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 512952 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P202 ...
[E::hts_open_format] Failed to open file "c8903173-fa62-40ef-938d-b2f3bc2ded7a.bam" : No such file or directory
samtools bam2fq: Cannot read file "c8903173-fa62-40ef-938d-b2f3bc2ded7a.bam": No such file or directory
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P203 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 1506850 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P204 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7628062 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P205 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 4865832 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/PCtrl-1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 3704058 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P206 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 843978 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P207 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8686918 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P208 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7044004 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P209 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 5375586 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P210 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 5541282 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/PCtrl-2 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 5995356 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P211 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6204742 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P212 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 5392074 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P213 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 335906 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P214 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 5249570 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P215 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 4562012 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/PCtrl-3 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 5729924 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P216 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 4261300 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P217 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 4754716 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P218 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 4127046 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P219 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6320304 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P220 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7913642 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/PCtrl-4 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 3508394 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P221 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 5506098 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P222 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 3602582 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P223 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 3868806 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P224 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 16259172 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P225 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 358508 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/PCtrl-5 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 4893646 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P226 ...
[E::hts_open_format] Failed to open file "fad80bbf-50e2-403f-beed-d40fafc5202f.bam" : No such file or directory
samtools bam2fq: Cannot read file "fad80bbf-50e2-403f-beed-d40fafc5202f.bam": No such file or directory
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P227 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6836632 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P228 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 3947488 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P229 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6165866 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P230 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8675034 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/PCtrl-6 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 3537368 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P202 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7637918 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P203 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7369748 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P204 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8931860 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P205 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6973944 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P209 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6467238 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P210 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8136382 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P211 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6738760 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P212 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8501868 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P214 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7568100 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P215 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7355188 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P216 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6860014 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P217 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8785700 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P218 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7132708 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P221 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8635980 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P222 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7432384 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P223 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 10314772 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P224 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6996926 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P225 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7302992 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P226 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7413090 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P227 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6214300 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P228 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7635494 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P229 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8224008 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P230 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8433420 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P206_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 976840 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P219_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7823882 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P220_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8345412 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P222_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 3144502 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P225_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 363184 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P227_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7525428 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P228_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 4407484 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P229_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6651428 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P228_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7566494 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P229_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8276894 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/culture/P230_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 7385096 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P201_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 643760 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P202_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 4579114 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P203_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 1590352 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P204_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 9164788 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P205_rep1 ...
[E::hts_open_format] Failed to open file "02b01033-8310-4187-8007-92992893079f.bam" : No such file or directory
samtools bam2fq: Cannot read file "02b01033-8310-4187-8007-92992893079f.bam": No such file or directory
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P210_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6656830 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P211_rep1 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 6124074 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P227_rep2 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 8051014 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P228_rep2 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 4381138 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P229_rep2 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 5148456 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P206_rep2 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 851262 reads
Processing /nfs/research/zi/zi/projects/tb/mada-mumbai-sputum/illumina/sputum/P222_rep2 ...
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 4016770 reads"""
d = dict()
for line in log.splitlines():
    if line.startswith("Proce"):
        sample = Path(line.split()[1]).name
    elif "No such" in line:
        continue
    elif "processed " in line:
        n_reads = int(line.split()[2])
        d[sample] = n_reads
sorted((k, v) for k, v in d.items())
# sputum fastqs
sputum_fqs = """P201_R1.fq.gz       P206_rep1_R1.fq.gz  P213_R1.fq.gz       P221_R1.fq.gz       P227_rep2_R1.fq.gz  PCtrl-3_R1.fq.gz  Test_23_R1.fq.gz  Test_43_R1.fq.gz  Test_55_R1.fq.gz
P201_R2.fq.gz       P206_rep1_R2.fq.gz  P213_R2.fq.gz       P221_R2.fq.gz       P227_rep2_R2.fq.gz  PCtrl-3_R2.fq.gz  Test_23_R2.fq.gz  Test_43_R2.fq.gz  Test_55_R2.fq.gz
P201_rep1_R1.fq.gz  P206_rep2_R1.fq.gz  P214_R1.fq.gz       P222_R1.fq.gz       P228_R1.fq.gz       PCtrl-4_R1.fq.gz  Test_28_R1.fq.gz  Test_45_R1.fq.gz  Test_56_R1.fq.gz
P201_rep1_R2.fq.gz  P206_rep2_R2.fq.gz  P214_R2.fq.gz       P222_R2.fq.gz       P228_R2.fq.gz       PCtrl-4_R2.fq.gz  Test_28_R2.fq.gz  Test_45_R2.fq.gz  Test_56_R2.fq.gz
P202_R1.fq.gz       P207_R1.fq.gz       P215_R1.fq.gz       P222_rep1_R1.fq.gz  P228_rep1_R1.fq.gz  PCtrl-5_R1.fq.gz  Test_32_R1.fq.gz  Test_46_R1.fq.gz  Test_60_R1.fq.gz
P202_R2.fq.gz       P207_R2.fq.gz       P215_R2.fq.gz       P222_rep1_R2.fq.gz  P228_rep1_R2.fq.gz  PCtrl-5_R2.fq.gz  Test_32_R2.fq.gz  Test_46_R2.fq.gz  Test_60_R2.fq.gz
P202_rep1_R1.fq.gz  P208_R1.fq.gz       P216_R1.fq.gz       P222_rep2_R1.fq.gz  P228_rep2_R1.fq.gz  PCtrl-6_R1.fq.gz  Test_34_R1.fq.gz  Test_47_R1.fq.gz  Test_65_R1.fq.gz
P202_rep1_R2.fq.gz  P208_R2.fq.gz       P216_R2.fq.gz       P222_rep2_R2.fq.gz  P228_rep2_R2.fq.gz  PCtrl-6_R2.fq.gz  Test_34_R2.fq.gz  Test_47_R2.fq.gz  Test_65_R2.fq.gz
P203_R1.fq.gz       P209_R1.fq.gz       P217_R1.fq.gz       P223_R1.fq.gz       P229_R1.fq.gz       Test_15_R1.fq.gz  Test_36_R1.fq.gz  Test_48_R1.fq.gz  Test_NC1_R1.fq.gz
P203_R2.fq.gz       P209_R2.fq.gz       P217_R2.fq.gz       P223_R2.fq.gz       P229_R2.fq.gz       Test_15_R2.fq.gz  Test_36_R2.fq.gz  Test_48_R2.fq.gz  Test_NC1_R2.fq.gz
P203_rep1_R1.fq.gz  P210_R1.fq.gz       P218_R1.fq.gz       P224_R1.fq.gz       P229_rep1_R1.fq.gz  Test_16_R1.fq.gz  Test_37_R1.fq.gz  Test_50_R1.fq.gz  Test_NC2_R1.fq.gz
P203_rep1_R2.fq.gz  P210_R2.fq.gz       P218_R2.fq.gz       P224_R2.fq.gz       P229_rep1_R2.fq.gz  Test_16_R2.fq.gz  Test_37_R2.fq.gz  Test_50_R2.fq.gz  Test_NC2_R2.fq.gz
P204_R1.fq.gz       P210_rep1_R1.fq.gz  P219_R1.fq.gz       P225_R1.fq.gz       P229_rep2_R1.fq.gz  Test_18_R1.fq.gz  Test_38_R1.fq.gz  Test_51_R1.fq.gz  Test_NC3_R1.fq.gz
P204_R2.fq.gz       P210_rep1_R2.fq.gz  P219_R2.fq.gz       P225_R2.fq.gz       P229_rep2_R2.fq.gz  Test_18_R2.fq.gz  Test_38_R2.fq.gz  Test_51_R2.fq.gz  Test_NC3_R2.fq.gz
P204_rep1_R1.fq.gz  P211_R1.fq.gz       P219_rep1_R1.fq.gz  P225_rep1_R1.fq.gz  P230_R1.fq.gz       Test_19_R1.fq.gz  Test_40_R1.fq.gz  Test_52_R1.fq.gz  Test_NC4_R1.fq.gz
P204_rep1_R2.fq.gz  P211_R2.fq.gz       P219_rep1_R2.fq.gz  P225_rep1_R2.fq.gz  P230_R2.fq.gz       Test_19_R2.fq.gz  Test_40_R2.fq.gz  Test_52_R2.fq.gz  Test_NC4_R2.fq.gz
P205_R1.fq.gz       P211_rep1_R1.fq.gz  P220_R1.fq.gz       P227_R1.fq.gz       PCtrl-1_R1.fq.gz    Test_20_R1.fq.gz  Test_41_R1.fq.gz  Test_53_R1.fq.gz  Test_NC5_R1.fq.gz
P205_R2.fq.gz       P211_rep1_R2.fq.gz  P220_R2.fq.gz       P227_R2.fq.gz       PCtrl-1_R2.fq.gz    Test_20_R2.fq.gz  Test_41_R2.fq.gz  Test_53_R2.fq.gz  Test_NC5_R2.fq.gz
P206_R1.fq.gz       P212_R1.fq.gz       P220_rep1_R1.fq.gz  P227_rep1_R1.fq.gz  PCtrl-2_R1.fq.gz    Test_22_R1.fq.gz  Test_42_R1.fq.gz  Test_54_R1.fq.gz  Test_NC6_R1.fq.gz
P206_R2.fq.gz       P212_R2.fq.gz       P220_rep1_R2.fq.gz  P227_rep1_R2.fq.gz  PCtrl-2_R2.fq.gz    Test_22_R2.fq.gz  Test_42_R2.fq.gz  Test_54_R2.fq.gz  Test_NC6_R2.fq.gz""".split()
len(sputum_fqs)
# culture fastqs
culture_fqs = """P202_R1.fq.gz  P210_R1.fq.gz  P216_R1.fq.gz  P223_R1.fq.gz  P228_R1.fq.gz       P230_rep1_R1.fq.gz  Test_22_R1.fq.gz  Test_36_R1.fq.gz  Test_42_R1.fq.gz  Test_48_R1.fq.gz  Test_54_R1.fq.gz
P202_R2.fq.gz  P210_R2.fq.gz  P216_R2.fq.gz  P223_R2.fq.gz  P228_R2.fq.gz       P230_rep1_R2.fq.gz  Test_22_R2.fq.gz  Test_36_R2.fq.gz  Test_42_R2.fq.gz  Test_48_R2.fq.gz  Test_54_R2.fq.gz
P203_R1.fq.gz  P211_R1.fq.gz  P217_R1.fq.gz  P224_R1.fq.gz  P228_rep1_R1.fq.gz  Test_15_R1.fq.gz    Test_23_R1.fq.gz  Test_37_R1.fq.gz  Test_43_R1.fq.gz  Test_50_R1.fq.gz  Test_55_R1.fq.gz
P203_R2.fq.gz  P211_R2.fq.gz  P217_R2.fq.gz  P224_R2.fq.gz  P228_rep1_R2.fq.gz  Test_15_R2.fq.gz    Test_23_R2.fq.gz  Test_37_R2.fq.gz  Test_43_R2.fq.gz  Test_50_R2.fq.gz  Test_55_R2.fq.gz
P204_R1.fq.gz  P212_R1.fq.gz  P218_R1.fq.gz  P225_R1.fq.gz  P229_R1.fq.gz       Test_16_R1.fq.gz    Test_28_R1.fq.gz  Test_38_R1.fq.gz  Test_45_R1.fq.gz  Test_51_R1.fq.gz  Test_56_R1.fq.gz
P204_R2.fq.gz  P212_R2.fq.gz  P218_R2.fq.gz  P225_R2.fq.gz  P229_R2.fq.gz       Test_16_R2.fq.gz    Test_28_R2.fq.gz  Test_38_R2.fq.gz  Test_45_R2.fq.gz  Test_51_R2.fq.gz  Test_56_R2.fq.gz
P205_R1.fq.gz  P214_R1.fq.gz  P221_R1.fq.gz  P226_R1.fq.gz  P229_rep1_R1.fq.gz  Test_19_R1.fq.gz    Test_32_R1.fq.gz  Test_40_R1.fq.gz  Test_46_R1.fq.gz  Test_52_R1.fq.gz  Test_60_R1.fq.gz
P205_R2.fq.gz  P214_R2.fq.gz  P221_R2.fq.gz  P226_R2.fq.gz  P229_rep1_R2.fq.gz  Test_19_R2.fq.gz    Test_32_R2.fq.gz  Test_40_R2.fq.gz  Test_46_R2.fq.gz  Test_52_R2.fq.gz  Test_60_R2.fq.gz
P209_R1.fq.gz  P215_R1.fq.gz  P222_R1.fq.gz  P227_R1.fq.gz  P230_R1.fq.gz       Test_20_R1.fq.gz    Test_34_R1.fq.gz  Test_41_R1.fq.gz  Test_47_R1.fq.gz  Test_53_R1.fq.gz  Test_65_R1.fq.gz
P209_R2.fq.gz  P215_R2.fq.gz  P222_R2.fq.gz  P227_R2.fq.gz  P230_R2.fq.gz       Test_20_R2.fq.gz    Test_34_R2.fq.gz  Test_41_R2.fq.gz  Test_47_R2.fq.gz  Test_53_R2.fq.gz  Test_65_R2.fq.gz""".split()
len(culture_fqs)
data = []
src = "sputum"
seen = defaultdict(list)
for f in sputum_fqs:
    m = re.search(r"^(?P<sample>\S+)_(?P<num>R\d).fq", f)
    if not m:
        raise ValueError(f)
    sample = m.group("sample")
    num = m.group("num")
    seen[sample].append(num)
    try:
        guuid = odf.at[(sample, src), "guuid"]
    except KeyError:
        guuid = None
    
    data.append((sample, src, guuid))
    
for k, v in seen.items():
    assert sorted(v) == sorted(["R1", "R2"]), k
    
src = "culture"
seen = defaultdict(list)
for f in culture_fqs:
    m = re.search(r"^(?P<sample>\S+)_(?P<num>R\d).fq", f)
    if not m:
        raise ValueError(f)
    sample = m.group("sample")
    num = m.group("num")
    seen[sample].append(num)
    try:
        guuid = odf.at[(sample, src), "guuid"]
    except KeyError:
        guuid = None
    data.append((sample, src, guuid))
    
for k, v in seen.items():
    assert sorted(v) == sorted(["R1", "R2"]), k
illumina_df = pd.DataFrame(set(data), columns=["sample", "source", "guuid"])
illumina_df.set_index(["sample", "source"], inplace=True)
illumina_df.sort_index()
illumina_df.to_csv("illumina_samples.csv")<jupyter_output><empty_output><jupyter_text>Adding the smear results to the existing sample CSV files<jupyter_code>illumina_df = pd.read_csv("illumina_samples.csv")
ont_df = pd.read_csv("ont_samples.csv")
# mada smear results
s="""P201	  +
P202	  +
P203	  +
P204	  +
P205	  +
PCtrl-1	 -
P206	  +
P207	  +
P208	  +
P209	  +
P210	  +
PCtrl-2	 -
P211	  ++
P212	  ++
P213	  ++
P214	  ++
P215	  ++
PCtrl-3	 -
P216	  ++
P217	  ++
P218	  ++
P219	  ++
P220	  ++
PCtrl-4	 -
P221	  +++
P222	  +++
P223	  +++
P224	  +++
P225	  +++
PCtrl-5	 -
P226	  +++
P227	  +++
P228	  +++
P229	  +++
P230	  +++
PCtrl-6	 -"""
smear_results = dict()
for row in s.splitlines():
    sample, result = row.split()
    assert sample not in smear_results
    smear_results[sample] = result
# fmr smear results
s="""15	2+
16	3+
18	3+
19	1+
20	1+
22	1+
23	2+
28	2+
32	1+
34	1+
36	3+
37	3+
38	2+
40	2+
41	2+
42	1+
43	3+
45	3+
46	2+
47	3+
48	2+
50	1+
51	3+
52	1+
53	2+
54	1+
55	2+
56	3+
60	1+
65	3+
NC1	-
NC2	-
NC3	-
NC4	-
NC5	-
NC6	-
36	3+
37	3+
43	3+
45	3+
47	3+
NC3	-
15	2+
28	2+
40	2+
48	2+
55	2+
NC2	-
42	1+
54	1+
50	1+
52	1+
60	1+
NC1	-"""
for row in s.splitlines():
    sample, result = row.split()
    sample = f"Test_{sample}"
    if "+" in result:
        result = "+" * int(result[0])
    if sample in smear_results:
        assert smear_results[sample] == result, f"Smear results for {sample} don't match"
    else:
        smear_results[sample] = result
smear_results
ont_samples = set(ont_df["sample"])
illumina_samples = set(illumina_df["sample"])
samples = ont_samples.union(illumina_samples)
samples = {s.replace("-singleplex", "").replace("-multiplex", "").replace("_rep1", "").replace("_rep2", "") for s in samples}
len(samples)
len(smear_results)
smear_results.keys() - samples
smear_df = pd.DataFrame(smear_results.items(), columns=["sample", "smear_result"])
smear_df
smear_df.to_csv("smear_results.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1314632614611681, 'fraction_numerical': 0.10278417416788779, 'mean_word_length': 4.255445544554456, 'pattern_counts': {'":': 3, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '167571', 'license': 'MIT', 'n_tokens_mistral': 12730, 'n_tokens_neox': 10933, 'n_words': 1524, 'path': 'docs/data_cleaning.ipynb', 'repo_name': 'mbhall88/basecall-tb-sputum-project', 'size': '23885'}