<jupyter_start><jupyter_text>**Importing General Dependencies**<jupyter_code>import numpy as np
import pandas as pd
import glob
import subprocess
import os
import time
import re
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
sns.set()<jupyter_output><empty_output><jupyter_text>Notebook 6 - MASH Analysis for Marine Cyanobacterial GenomesThis notebook uses MASH to perform a whole genome comparison between our 24 marine cyanbacteria and the other published cyanobacteria from NCBI **Installing MASH**<jupyter_code>if os.path.exists("./mash-OSX64-v2.2/mash"):
    print("MASH already downloaded")
else:
    !wget https://github.com/marbl/Mash/releases/download/v2.2/mash-OSX64-v2.2.tar
    !tar xopf mash-OSX64-v2.2.tar; rm mash-OSX64-v2.2.tar<jupyter_output>MASH already downloaded<jupyter_text>**Sketching Finished Cyanobiome and NCBI Datasets** This step is going to create the minhashes from the references to be probed in the soon to be queried genomes<jupyter_code>start_time = time.time()

if os.path.exists("./mash-OSX64-v2.2/ncbi_genomes.msh"):
    print("Sketch already exists")
else:
    !./mash-OSX64-v2.2/mash sketch -o ./mash-OSX64-v2.2/ncbi_genomes ./ncbi_genomes/*.fasta
    
print('\n' + "--- %s seconds ---" %(time.time()-start_time))<jupyter_output>Sketch already exists

--- 0.00020003318786621094 seconds ---<jupyter_text>**Subsetting the 24 Marine Cyanobacterial Genomes**<jupyter_code>query_genomes = ['JAAHII01','JAAHIH01','JAAHGF01','JAAHHC01','JAAHHD01','JAAHFN01','JAAHGM01','JAAHGB01','JAAHFW01','JAAHFZ01','JAAHFP01','JAAHGH01','JAAHGC01','JAAHGA01','JAAHHJ01','JAAHGO01','JAAHGU01','JAAHGW01','JAAHGS01','JAAHHM01','JAAHHP01','JAAHHN01','JAAHGL01','JAAHFU01']

!mkdir ./query_genomes/
glob_list = glob.glob('./ncbi_genomes/*fasta')

for item in glob_list:
    name = os.path.basename(item).split('.')[0]
    if name in query_genomes:
        cmd = 'cp %s ./query_genomes/'%item
        subprocess.call(cmd,shell=True)<jupyter_output><empty_output><jupyter_text>**Running MASH and Creating the MASH Dataframe with Similarity Scores**<jupyter_code>def get_mash_df(inputs_folder,sketch):
    start_time = time.time()
    input_list = glob.glob("%s*.fasta"%inputs_folder)
    final_df = pd.DataFrame()
    for index,item in enumerate(input_list):
        scores = []
        ncbi_item = './ncbi_genomes/%s'%(os.path.basename(item))
        print(item,ncbi_item)
        cmd = "./mash-OSX64-v2.2/mash dist %s %s > ./mash-OSX64-v2.2/temp.txt"%(sketch,item)
        subprocess.call(cmd,shell=True)
        df = pd.read_csv("./mash-OSX64-v2.2/temp.txt",sep="\t",names=["reference","query","distance","pvalue","matching-hashes"])
        df = df[df.reference != ncbi_item]
        row = df[df.distance == df.distance.min()]
        final_df = final_df.append(row[:1], ignore_index = True)
    print('\n' + "--- %s seconds ---" %(time.time()-start_time))
    return final_df
        
mash_df = get_mash_df("./query_genomes/",
                      "./mash-OSX64-v2.2/ncbi_genomes.msh")
mash_df
mash_df.sort_values(by='distance')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15367268041237114, 'fraction_numerical': 0.039304123711340205, 'mean_word_length': 4.554561717352415, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '162266', 'license': 'MIT', 'n_tokens_mistral': 1278, 'n_tokens_neox': 1184, 'n_words': 201, 'path': 'notebook6-mash_analysis.ipynb', 'repo_name': 'tiagolbiotech/cyanobiome', 'size': '3104'}