<jupyter_start><jupyter_text>Note: HEK 293 didn't work because the cell line has a space so the filename has a space :(. Need to rerun using the sanitized cell line name. Should skip the ones that are already done since we don't use `--force`.<jupyter_code>! load-into-counting.py --help
for cell_line, df in metadata.groupby('cell_line'):
    print(cell_line, df.shape,)
    read1 = df.acc + "_pass_1.fastq.gz"
    read2 = df.acc + "_pass_2.fastq.gz"
    reads = ' '.join(read1) + ' ' + ' '.join(read2)
    
    # Sanitize the cell line name
    cell_line = cell_line.replace(' ', '_')
    output_folder = f"/mnt/data/khmer_k32/{cell_line}"
    output_file = f'{output_folder}/khmer_countgraph'
#     print(output_file)
    ! mkdir -p $output_folder
    # ksize of 32 is the default
    ! load-into-counting.py --summary-info tsv --max-memory-usage 60G --ksize 32 --threads 16  $output_file $reads
    
range(2)
ls -lh /mnt/data/khmer_k32/*/*
import numpy as np
np.power(4., -16)
1e8*np.log(1-np.power(4., -16))
np.exp(1e8*np.log(1-np.power(4., -16)))
1- np.exp(1e8*np.log(1-np.power(4., -16)))
desired_probability = 0.01
genome_size = 1e8

np.log(genome_size*(1-desired_probability)/desired_probability)
desired_probability = 0.001
genome_size = 1e8

np.log(genome_size*(1-desired_probability)/desired_probability)
desired_probability = 1e-6
genome_size = 1e8

np.log(genome_size*(1-desired_probability)/desired_probability)
! extract-paired-reads.py --help<jupyter_output>|| This is the script extract-paired-reads.py in khmer.
|| You are running khmer version 2.1.1
|| You are also using screed version 1.0
||
|| If you use this script in a publication, please cite EACH of the following:
||
||   * MR Crusoe et al., 2015. http://dx.doi.org/10.12688/f1000research.6924.1
||
|| Please see http://khmer.readthedocs.io/en/latest/citations.html for details.

usage: extract-paired-reads.py [--version] [--info] [-h] [-d OUTPUT_DIR]
                               [-p filename] [-s filename] [-f]
                               [--gzip | --bzip]
                               [infile]

Take a mixture of reads and split into pairs and orphans.

positional arguments:
  infile

optional arguments:
  --version             show program's version number and exit
  --info                print citation information
  -h, --help            show this help message and exit
  -d OUTPUT_DIR, --output-dir OUTPUT_DIR
                        Output split rea[...]
{'fraction_non_alphanumeric': 0.129136400322841, 'fraction_numerical': 0.035108958837772396, 'mean_word_length': 2.7222222222222223, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '102163', 'license': 'MIT', 'n_tokens_mistral': 936, 'n_tokens_neox': 853, 'n_words': 226, 'path': 'notebooks/003_count_kmers.ipynb', 'repo_name': 'czbiohub/sequence-localization', 'size': '2478'}