<jupyter_start><jupyter_text>Check that it's correct with the `head` of the initial data<jupyter_code>target_df.loc[nucleus_cytoplasm_single_ensg.ensg_id.head()]<jupyter_output><empty_output><jupyter_text>Download and filter sequence data<jupyter_code>cd /mnt/data
mkdir -p genome/hg38/ensembl/v92
cd genome/hg38/ensembl/v92/<jupyter_output>/src/myhome/code/sequence-localization/notebooks/genome/hg38/ensembl/v92<jupyter_text>Download ENSEMBL fasta files. I don't understand the difference between `cds` and `cdna` so I downloaded them both. `pep` is the protein sequence<jupyter_code># %%bash

# wget ftp://ftp.ensembl.org/pub/release-92/fasta/homo_sapiens/cds/Homo_sapiens.GRCh38.cds.all.fa.gz
# wget ftp://ftp.ensembl.org/pub/release-92/fasta/homo_sapiens/cdna/Homo_sapiens.GRCh38.cdna.all.fa.gz
# wget ftp://ftp.ensembl.org/pub/release-92/fasta/homo_sapiens/pep/Homo_sapiens.GRCh38.pep.all.fa.gz
ls<jupyter_output><empty_output><jupyter_text>Unzip the files<jupyter_code># ! gunzip --keep *.gz
ls -lha
! head *.fa<jupyter_output>head: cannot open '*.fa' for reading: No such file or directory<jupyter_text>Good, the fasta descriptione includes the ENSG ID<jupyter_code>fastas = ['Homo_sapiens.GRCh38.cdna.all.fa', 'Homo_sapiens.GRCh38.cds.all.fa', 'Homo_sapiens.GRCh38.pep.all.fa']
%%time

from Bio import SeqIO



for fasta in fastas:
    print(fasta)
#     input_fasta = os.path.join('/mnt/data/', fasta)
    output_fasta = os.path.join(data_folder, f"{fasta}.nuclear_or_cytoplasmic")
    ensgs = []
    lengths = pd.Series()

    nuclear_or_cytoplasmic = []
    for record in SeqIO.parse(fasta, "fasta"):
        for ensg_id in nucleus_cytoplasm_single_ensg['ensg_id']:
            if ensg_id in record.description:
                nuclear_or_cytoplasmic.append(record)
                ensgs.append(ensg_id)
                lengths[record.id] = len(record)
    print(f'\t{len(nuclear_or_cytoplasmic)} sequences found for {len(set(ensgs))} ENSEMBL ids') 
    csv = os.path.join(data_folder, f"{fasta}.nuclear_or_cytoplasmic.target.tsv")
    target_df.loc[ensgs].to_csv(csv, index=False, header=False, sep='\t')
    SeqIO.write(nuclear_or_cytoplasmic, output_fasta, "fasta")


target_df.head()
ls -lha /mnt/data
! echo $data_folder
! cp /mnt/data/*nuclear_or_cytoplasmic $data_folder
! ls -lha $data_folder
! head $data_folder/*<jupyter_output>==> /home/ubuntu/code/sequence-localization/data/021_get_nucleus_cytoplasm_sequences/Homo_sapiens.GRCh38.cdna.all.fa.nuclear_or_cytoplasmic <==
>ENST00000419783.2 cdna chromosome:GRCh38:3:49357171:49358600:-1 gene:ENSG00000233276.4 gene_biotype:polymorphic_pseudogene transcript_biotype:protein_coding gene_symbol:GPX1 description:glutathione peroxidase 1 [Source:HGNC Symbol;Acc:HGNC:4553]
GAGCCCTCGAGGGCCCCAGCCCTTGGAAGGGTAACCTGGACCGCTGCCGCCTGGTTGCCT
GGGCCAGACCAGACATGCCTGCTGCTCCTTCCGGCTTAGGAGGAGCACGCGTCCCGCTCG
GGCGCACTCTCCAGCCTTTTCCTGGCTGAGGAGGGGCCGAGCCCTCCGGGTAGGGCGGGG
GCCGGATGAGGCGGGACCCTCAGGCCCGGAAAACTGCCTGTGCCACGTGACCCGCCGCCG
GCCAGTTAAAAGGAGGCGCCTGCTGGCCTCCCCTTACAGTGCTTGTTCGGGGCGCTCCGC
TGGCTTCTTGGACAATTGCGCCATGTGTGCTGCTCGGCTAGCGGCGGCGGCGGCGGCGGC
CCAGTCGGTGTATGCCTTCTCGGCGCGCCCGCTGGCCGGCGGGGAGCCTGTGAGCCTGGG
CTCCCTGCGGGGCAAGGTACTACTTATCGAGAATGTGGCGTCCCTCTGAGGCACCACGGT
CCGGGACTACACCCAGATGAACGAGCTGCAGCGGCGCCTCGGACCCCGGGGCCTGGTGGT

==> /home/ubuntu/code/sequence-localization/dat[...]
{'fraction_non_alphanumeric': 0.10730253353204174, 'fraction_numerical': 0.02533532041728763, 'mean_word_length': 6.050420168067227, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '592392', 'license': 'MIT', 'n_tokens_mistral': 1556, 'n_tokens_neox': 1394, 'n_words': 172, 'path': 'notebooks/021_get_nucleus_cytoplasm_sequences.ipynb', 'repo_name': 'czbiohub/sequence-localization', 'size': '3355'}