<jupyter_start><jupyter_text>Detecting horizontal gene transfer (HGT)This sheet shows how we flag genes as potential HGT events.<jupyter_code>import pandas as pd
from matplotlib import pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Number of taxa inferred to have been lost for each gene by each algorithm<jupyter_code>lossTaxa = pd.read_csv("lossTaxa_HUMAN.csv",index_col=0)
lossTaxa.head()<jupyter_output><empty_output><jupyter_text>Get algorithms that have been removed as false positives<jupyter_code># Map genes to trimmed algorithms

outlierD = {}
with open("lossStats_HUMAN.csv") as f:
    f.readline() # skip header
    for line in f:
        line = line.strip().split(",")
        if line[3] == '':
            continue
        outlierD[line[0]] = line[3].split()
        
outlierD[outlierD.keys()[0]]<jupyter_output><empty_output><jupyter_text>Calculate average number of lossTaxa for each gene and add this as a new column to lossTaxa<jupyter_code>dbsTrimmed = 0
avgs = pd.Series()
for index,row in lossTaxa.iterrows():
    if index in outlierD:
        dbsTrimmed += len(outlierD[index])
        dbs = [i for i in lossTaxa.columns if i not in outlierD[index]]
    else:
        dbs = lossTaxa.columns
    avgs[index] = row[dbs].mean() # only include algorithm that have not been trimmed
    
dbsTrimmed
lossTaxa["Avg"] = avgs
lossTaxa.head()<jupyter_output><empty_output><jupyter_text>The average has a fat-tailed distributionWe will flag all the genes in the 95th percentile<jupyter_code>def floatRange(start,stop,step):
    i = start
    while i <= stop:
        yield i
        i += step
        
quantile_steps = [i for i in floatRange(0,1,.05)]
    
quantiles = lossTaxa["Avg"].quantile(quantile_steps)
quantiles
quantiles.iloc[-1]
lossTaxa["Avg"].hist(bins=50,color='grey')
bline = plt.axvline(24.92,color='black',label="95th percentile")
plt.legend()

#plt.savefig("AvgLossTaxa_distribution.svg")<jupyter_output><empty_output><jupyter_text>Flag genes<jupyter_code>lossTaxa["HGT_flag"] = lossTaxa["Avg"] >= 24.92
lossTaxa.head()
lossTaxa.to_csv("HGTFlag_HUMAN.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11484593837535013, 'fraction_numerical': 0.011671335200746966, 'mean_word_length': 4.8712328767123285, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '493864', 'license': 'MIT', 'n_tokens_mistral': 788, 'n_tokens_neox': 751, 'n_words': 176, 'path': 'Notebooks/HGT_flagging.ipynb', 'repo_name': 'bliebeskind/Gene-Ages', 'size': '2142'}