<jupyter_start><jupyter_text>Pre-process networks<jupyter_code>import pandas as pd
import os
import networkx as nx
import numpy as np
import pickle as pkl
PATH = "D:/Projects/cTaG2.0"
DATAPATH = "D:/Projects/data/cTaG2.0"

# PATH = "/data/malvika/cTaG2.0"
# DATAPATH = "/data/malvika/data/cTaG2.0"<jupyter_output><empty_output><jupyter_text>The cut-off taken as 700, which corresponds to 0.7 confidence score. String db suggests 0.9 for very high cut-off and 0.7 high confidence.<jupyter_code>%%time
# Load data
os.chdir(DATAPATH + "/network")
fname="9606.protein.links.detailed.v11.5.txt.gz.inproc"
data_string = pd.read_csv(fname, sep=" ", header=0)
data_string = data_string[(data_string.experimental > 700) | (data_string.database >700)]
data_string.shape
%%time
# load network
os.chdir(DATAPATH + "/network")
fname = "9606.protein.aliases.v11.5.txt"
data_map = pd.read_csv(fname, sep="\t", header=0)
data_map = data_map[data_map.source == "BioMart_HUGO"]
data_map.drop_duplicates(subset=["#string_protein_id"], keep=False, inplace=True)
data_map.index = data_map["#string_protein_id"]

temp = [data_map.loc[x, "alias"] if x in data_map.index else np.nan for x in data_string.protein1]
data_string["gene1"] = temp
temp = [data_map.loc[x, "alias"] if x in data_map.index else np.nan for x in data_string.protein2]
data_string["gene2"] = temp
data_string = data_string.dropna(axis=0)<jupyter_output>Wall time: 15.3 s<jupyter_text>Define graph for PPI network<jupyter_code>G = nx.Graph()
G = nx.from_pandas_edgelist(data_string, 'gene1', 'gene2',
                            edge_attr=['experimental', 'database', 'combined_score'])<jupyter_output><empty_output><jupyter_text>Degree of nodes<jupyter_code>%%time
deg = G.degree()
os.chdir(PATH + "/data/network")
fname = "string_degree.pkl"
with open(fname, 'wb') as f:
    pkl.dump(deg, f)<jupyter_output>CPU times: user 193 ms, sys: 12.1 ms, total: 205 ms
Wall time: 200 ms<jupyter_text>Closeness centrality<jupyter_code>%%time
cc = nx.closeness_centrality(G)
os.chdir(PATH + "/data/network")
fname = "string_cc.pkl"
with open(fname, 'wb') as f:
    pkl.dump(cc, f)<jupyter_output>CPU times: user 4min 16s, sys: 182 ms, total: 4min 16s
Wall time: 4min 16s<jupyter_text>Betweeness centrality<jupyter_code>%%time
bc = nx.betweenness_centrality(G, k=int(G.number_of_nodes()/2), weight='combined_score')
os.chdir(PATH + "/data/network")
fname = "string_bc.pkl"
with open(fname, 'wb') as f:
    pkl.dump(bc, f)<jupyter_output>CPU times: user 31min 10s, sys: 1.64 s, total: 31min 12s
Wall time: 31min 12s<jupyter_text>Neighbours of genes<jupyter_code>def getneighbors(gene, G, n = 1):
    """
    

    Parameters
    ----------
    gene : TYPE
        DESCRIPTION.
    G : TYPE
        DESCRIPTION.
    n : TYPE, optional
        DESCRIPTION. The default is 1.

    Returns
    -------
    None.

    """
    neighbors = [None] * n
    for i in range(n):
        if i == 0:
            neighbors[i] = list(G.neighbors(gene))
        else:           
            for g in neighbors[i - 1]:
                if neighbors[i] is None:
                    neighbors[i] =  list(G.neighbors(g))
                else:
                    neighbors[i] =  neighbors[i] + list(G.neighbors(g))
    return [n for neigh in neighbors for n in neigh]<jupyter_output><empty_output><jupyter_text>While the networkx package can be used to find immediate neighbours of a gene, for generation of RNA features we require all the neighbours at 1 to n steps. The function defined above helps find all neighbours at >=n steps.Given below is a code for generating a dictionary for nodes in graph and their neighbours. the value of n can be changed as desired. The file is uploaded for generating RNA features.<jupyter_code>%%time
n = 2
neighbors = {}
for gene in G.nodes:
    neighbors[gene] = getneighbors(gene, G, n = n)
os.chdir(PATH + "/data/network")
fname = "string_neigh_{}.pkl".format(n)
with open(fname, 'wb') as f:
    pkl.dump(neighbors, f)<jupyter_output>Wall time: 7.18 s<jupyter_text>Save network<jupyter_code>os.chdir(PATH + "/data/network")
fname = "string_graph.pkl"
with open(fname, 'wb') as f:
    pkl.dump(G, f)
data_string = data_string.sample(frac=0.2).reset_index(drop=True)
data_string.shape
data_string.experimental.min()
data_string.experimental.max()
data_string.shape
data_string[data_string.experimental >700].shape
data_string[data_string.database >700].shape
98130+193660
data_string[(data_string.experimental > 700) | (data_string.database >700)].shape
data_string.columns
data_string[(data_string.experimental > 700) | (data_string.database >700)].gene1.unique().shape
data_string[(data_string.experimental > 700) | (data_string.database > 700)].gene2.unique().shape
len(set(data_string[(data_string.experimental > 700) | (data_string.database >700)].gene1.unique()).union(data_string[(data_string.experimental > 700) | (data_string.database >700)].gene2.unique()))
G = nx.Graph()

data_temp = data_string[(data_string.experimental > 0.7) & (data_string.database > 0.7)]
data_temp.shape
data_string.shape
data_temp = data_string[(data_string.experimental > 900) | (data_string.database > 900)]
data_temp.shape
import matplotlib.pyplot as plt
import seaborn as sns
plt.hist(data_string.experimental, bins=100)
plt.hist(data_string.database, bins=100)
plt.boxplot((data_string.experimental - data_string.experimental.min()) /data_string.experimental.max())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13334559499724113, 'fraction_numerical': 0.03071546808901968, 'mean_word_length': 4.543323139653415, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '323595', 'license': 'CC0-1.0', 'n_tokens_mistral': 2137, 'n_tokens_neox': 1942, 'n_words': 447, 'path': 'code/preprocess_networks.ipynb', 'repo_name': 'RamanLab/PIVOT', 'size': '5437'}