<jupyter_start><jupyter_text>If you would like more explanation to Data preparation part, please go to cDNA_subwords_vocab64 notebook.  Setup<jupyter_code>!pip show fastai
!pip show biopython
from Bio import SeqIO
from fastai.text.all import *<jupyter_output><empty_output><jupyter_text>Data preparation<jupyter_code>!wget -nc http://ftp.ensembl.org/pub/release-103/fasta/homo_sapiens/cdna/Homo_sapiens.GRCh38.cdna.abinitio.fa.gz
!yes n | gunzip Homo_sapiens.GRCh38.cdna.abinitio.fa.gz<jupyter_output>File ‘Homo_sapiens.GRCh38.cdna.abinitio.fa.gz’ already there; not retrieving.

gzip: Homo_sapiens.GRCh38.cdna.abinitio.fa already exists;	not overwritten
yes: standard output: Broken pipe<jupyter_text>Token preparation<jupyter_code>with open("Homo_sapiens.GRCh38.cdna.abinitio.fa", "rt") as handle:
  txts = L(str(record.seq).lower() for record in SeqIO.parse(handle, "fasta"))
txt = txts[0]
txts = txts[1:10001]
SPECIAL_TOKENS = 2
ALPHABET = 4
VOCAB_SIZE = 2048 + SPECIAL_TOKENS + ALPHABET
tokenizer = SubwordTokenizer(vocab_sz=VOCAB_SIZE, special_toks=[], cache_dir='tmp/vocab2048', lang='dna')
tokenizer.setup(txts)
toks = first(tokenizer([txt]))
print(coll_repr(toks, 30))
txt[:100]
tkn = Tokenizer(tokenizer, rules=[], sep='')
print(coll_repr(tkn(txt), 30))
toks_all = txts.map(tkn)<jupyter_output><empty_output><jupyter_text>Tokens analysis<jupyter_code>from operator import add

tokens = reduce(add, toks_all)<jupyter_output><empty_output><jupyter_text>Top 10 most common tokens<jupyter_code>import collections

elements_count = collections.Counter(tokens)
print(elements_count.most_common(10))<jupyter_output>[('tga', 16335), ('tag', 9722), ('taa', 9384), ('ctga', 8428), ('atga', 7842), ('ag', 7732), ('ttga', 6836), ('tgtga', 6109), ('cgg', 5413), ('cccag', 5359)]<jupyter_text>Eight most common tokens correnspond to stop codons.  Distribution of occurences<jupyter_code>counts = []
for count in elements_count:
    counts.append(elements_count[count])
from matplotlib import pyplot

pyplot.figure(figsize=(10,4))
pyplot.hist(counts, bins=range(min(counts), max(counts) + 500, 500))
pyplot.xlabel("occurence count")
pyplot.ylabel("distinct subword count")
pyplot.show()<jupyter_output><empty_output><jupyter_text>Distribution of lengths of subwords<jupyter_code>lengths = []
vocab = set(tokens)
for token in vocab:
    lengths.append(len(token))
from matplotlib import pyplot

pyplot.figure(figsize=(4,6))
pyplot.hist(lengths, bins=range(min(lengths), max(lengths) + 1, 1))
pyplot.xlabel("length")
pyplot.ylabel("count")
pyplot.show()<jupyter_output><empty_output><jupyter_text>Longest words in vocabulary<jupyter_code>vocab = list(vocab)
vocab.sort(key=len)
print(vocab[-1])
print(vocab[-2])
print(vocab[-3])<jupyter_output>gaagaagaagaagaag
aatggaatcgaatgga
aatggaatggaatgga<jupyter_text>Longest subwords are repeats. And two of them differ only in one letter.  Least freaquent subwords in training data<jupyter_code>print(elements_count.most_common()[-1])
print(elements_count.most_common()[-2])
print(elements_count.most_common()[-3])
print(elements_count.most_common()[-4])<jupyter_output>('t', 86)
('c', 116)
('a', 142)
('g', 148)
{'fraction_non_alphanumeric': 0.1436288528757547, 'fraction_numerical': 0.036542739116619004, 'mean_word_length': 8.341246290801188, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '457559', 'license': 'Apache-2.0', 'n_tokens_mistral': 1323, 'n_tokens_neox': 1235, 'n_words': 222, 'path': 'experiments/subwords_tokenization/cDNA_subwords_vocab2048.ipynb', 'repo_name': 'katarinagresova/genomic_ML_playground', 'size': '3147'}