<jupyter_start><jupyter_text>Load APID PPI data (Dr. <NAME>)Ref: __[APID interactomes: providing proteome-based interactomes with controlled quality for multiple species and derived networks](https://academic.oup.com/nar/article/44/W1/W529/2499348)__APID server: http://apid.dep.usal.es/Last downloaded on 2018/04(Apr)/05 Level 2: proven by 2 experiments or more:- 133 993 interactions- 17 324 interacting proteins Raw colums:- InteractionID	- UniprotID_A	- UniprotName_A	- GeneName_A	- UniprotID_B	- UniprotName_B	- GeneName_B	- Experiments: number of experiments- Methods: number of methods- Publications: number of publications- 3DStructures: number of 3D structures (PDBs)- CurationEvents: number of curation events (including source databases)<jupyter_code>df_apid = pd.read_csv(data_folder + "9606_Q2.txt", sep="\t")
print("interacting proteins number:", df_apid.shape[0])

# keep only interactions between Human-Human proteins
df_apid_HH = df_apid[df_apid.UniprotName_A.str.contains("_HUMAN") & 
                     df_apid.UniprotName_B.str.contains("_HUMAN") == True]
print("interacting Human-Human proteins number:", df_apid_HH.shape[0])

print("{} interspecific interactions removed ({}%)"
      .format(df_apid.shape[0]-df_apid_HH.shape[0],
             round((1-df_apid_HH.shape[0]/df_apid.shape[0])*100,1)))

df_apid_HH.head()
# keep only GeneSymbol ID and UniProt ID
# then change column names
df_apid_HH = df_apid_HH[["GeneName_A", "UniprotID_A", "GeneName_B", "UniprotID_B"]]
df_apid_HH.columns = ["GeneSymbol_1", "UniProt_1", "GeneSymbol_2", "UniProt_2"]
df_apid_HH.head()<jupyter_output><empty_output><jupyter_text>There are some Gene Symbols not found<jupyter_code># APID's UniProt
apid_unip_1 = df_apid_HH.UniProt_1.drop_duplicates()
apid_unip_2 = df_apid_HH.UniProt_2.drop_duplicates()
apid_unip_rows_1 = list(apid_unip_1.index)
apid_unip_rows_2 = list(apid_unip_2.index)
apid_unip = apid_unip_1.append(apid_unip_2).drop_duplicates()

# APID's Gene Symbol
apid_geneSymb = df_apid_HH.GeneSymbol_1.append(df_apid_HH.GeneSymbol_2)
apid_geneSymb = apid_geneSymb.drop_duplicates()

print("unique UniProt number:", len(apid_unip))
print("unique Gene Symbol number:", len(apid_geneSymb))
diff = len(apid_unip)-len(apid_geneSymb)
print("there are {} fewer Gene Symbols than UniProt ID ({}%)"
      .format(diff, round((diff/len(apid_unip))*100, 1)))
# get index of NaN values in GeneSymbol_1 & 2
null_columns = df_apid_HH.columns[df_apid_HH.isnull().any()]
print("----- NaN values in Gene Symbols -----")
print(df_apid_HH[null_columns].isnull().sum())

df_apid_HH[df_apid_HH.isnull().any(axis=1)][null_columns]
null_idx = list(df_apid_HH[df_apid_HH.isnull().any(axis=1)][null_columns].index)

df_apid_HH.loc[null_idx].head()<jupyter_output>----- NaN values in Gene Symbols -----
GeneSymbol_1    126
GeneSymbol_2    135
dtype: int64<jupyter_text>Sum of NaN values in GeneSymbol_1&2 (258+209=467)  => One Gene Symbol can have several UniProt ID   Load UniProtKB dataComplete human gene data set downloaded from __[UniProtKB](ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/idmapping/)__"HUMAN_9606_idmapping_selected.tab.gz" (2018/01/31) Raw colums:1. UniProtKB-AC2. UniProtKB-ID3. GeneID (EntrezGene)4. RefSeq5. GI6. PDB7. GO8. UniRef1009. UniRef9010. UniRef5011. UniParc12. PIR13. NCBI-taxon14. MIM15. UniGene16. PubMed17. EMBL18. EMBL-CDS19. Ensembl20. Ensembl_TRS21. Ensembl_PRO22. Additional PubMed<jupyter_code># only get UniProtKB-AC and GeneID (EntrezGene)
df_unip = pd.read_csv(data_folder + "HUMAN_9606_idmapping_selected.tab", sep="\t",
                      index_col=False, usecols=[0,2], names=["UniProt_AC", "EntrezGene"])
df_unip.head()<jupyter_output><empty_output><jupyter_text>Load HGNC dataComplete HGNC dataset downloaded from __[HGNC (HUGO Gene Nomenclature Committee)](https://www.genenames.org/cgi-bin/statistics)__ "hgnc_complete_set.txt" (2018/02(Feb)/09) Raw colums:1. hgnc_id	2. symbol	3. name	4. locus_group	5. locus_type	6. status	7. location	8. location_sortable	9. alias_symbol	10. alias_name	11. prev_symbol	12. prev_name	13. gene_family	14. gene_family_id	15. date_approved_reserved	16. date_symbol_changed	17. date_name_changed	18. date_modified	19. entrez_id	20. ensembl_gene_id	21. vega_id	22. ucsc_id	23. ena	24. refseq_accession	25. ccds_id	26. uniprot_ids	27. pubmed_id	28. mgd_id	29. rgd_id	30. lsdb	31. cosmic	32. omim_id	33. mirbase	34. homeodb	35. snornabase	36. bioparadigms_slc	37. orphanet	38. pseudogene.org	39. horde_id	40. merops	41. imgt	42. iuphar	43. kznf_gene_catalog	44. mamit-trnadb	45. cd	46. lncrnadb	47. enzyme_id	48. intermediate_filament_db	49. rna_central_ids<jupyter_code># only get symbol (Gene Symbol), entrez_id (EntrezGene) and uniprot_ids
df_hgnc = pd.read_csv(data_folder + "hgnc_complete_set.txt", sep="\t",
                      index_col=False, usecols=[1,18,25])

print("----- NaN values in raw data ({} rows) -----".format(df_hgnc.symbol.size))
df_hgnc.isnull().sum()<jupyter_output>----- NaN values in raw data (42471 rows) -----<jupyter_text>ID mapping with APID HGNC: from Gene Symbol to EntrezGene ID Overlapping analysis between Gene Symobl and EntrezGene<jupyter_code># Gene Symbols in APID
print("all unique Gene Symbols in APID:", len(apid_geneSymb))

# Gene Symbols in HGNC 
print("all Gene Symbols in HGNC:", df_hgnc.symbol.size)
intersection_geneSymb = list((set(apid_geneSymb)).intersection(set(df_hgnc.symbol)))
print("intersection of Gene Symbols:", len(intersection_geneSymb))

# print("{} of APID's UnitProt ID are not found in EntrezGene ID "
#       .format(len(apid_unip)-len(intersection_prot)))
# merge on APID's GeneSymbol (1 then 2) to get corresponding EntrezGene ID
# keeping the index from the 1st frame
joined1 = df_apid_HH.reset_index().merge(
    df_hgnc, how='left', left_on='GeneSymbol_1', right_on='symbol').set_index('index')
joined2 = joined1.reset_index().merge(
    df_hgnc, how='left', left_on='GeneSymbol_2', right_on='symbol').set_index('index')

# remove redundant columns
df = joined2.drop(['symbol_x', 'uniprot_ids_x', 'symbol_y', 'uniprot_ids_y'], axis=1)

# rename columns
df = df.rename(columns={'entrez_id_x': 'EntrezGene_1',
                        'entrez_id_y': 'EntrezGene_2'})
# reorder columns
cols = list(df)
cols.insert(0, cols.pop(cols.index('EntrezGene_1')))
cols.insert(3, cols.pop(cols.index('EntrezGene_2')))
df = df.loc[:, cols]

df.head()
df.isnull().sum().GeneSymbol_1
# Get Gene Symbols with which EntrezGene are NaN
df_geneSymb_entrezNan_1 = df[df.EntrezGene_1.isnull()].GeneSymbol_1
df_geneSymb_entrezNan_2 = df[df.EntrezGene_2.isnull()].GeneSymbol_2

# remove duplicates
df_geneSymb_entrezNan_1 = df_geneSymb_entrezNan_1.drop_duplicates()
df_geneSymb_entrezNan_2 = df_geneSymb_entrezNan_2.drop_duplicates()

df_geneSymb_entrezNan = df_geneSymb_entrezNan_1.append(df_geneSymb_entrezNan_2).drop_duplicates()

print("There are {} ({}%) missing EntrezGene ID via Gene Symbol ID mapping"
     .format(len(df_geneSymb_entrezNan),
             round((len(df_geneSymb_entrezNan)/len(apid_geneSymb))*100, 1)))<jupyter_output>There are 159 (1.1%) missing EntrezGene ID via Gene Symbol ID mapping<jupyter_text>UniProtKB-AC: from UniProt ID to EntrezGene ID For Gene Symbols not previously found in APID<jupyter_code># UniProtKB
unip_entrez = df_unip.UniProt_AC.drop_duplicates()
print('all UniProtKB ID:', len(unip_entrez))

# ADIP
apid_unip = df_apid_HH.UniProt_1.append(df_apid_HH.UniProt_2)
apid_unip = apid_unip.drop_duplicates()

overlap_uniprot_adip = (len(apid_unip)/len(unip_entrez))*100
print("{} interacting proteins in ADIP ({}%)"
      .format(len(apid_unip), round(overlap_uniprot_adip, 1)))
# data with NaN in GeneSymbol_1 or 2
df_apid_geneSymb_nan1 = df[df.GeneSymbol_1.isnull()]
df_apid_geneSymb_nan2 = df[df.GeneSymbol_2.isnull()]

# merge on APID's UniProt_1 and 2 SEPARATELY to get corresponding EntrezGene ID
# keeping the index from the 1st frame
joined_1 = df_apid_geneSymb_nan1.reset_index().merge(
    df_unip, how='left', left_on='UniProt_1', right_on='UniProt_AC').set_index('index')
joined_2 = df_apid_geneSymb_nan2.reset_index().merge(
    df_unip, how='left', left_on='UniProt_2', right_on='UniProt_AC').set_index('index')

# replace new EntrezGene values found in UniProtKB
df.EntrezGene_1[joined_1.index] = joined_1.EntrezGene
df.EntrezGene_2[joined_2.index] = joined_2.EntrezGene
# rows with both GeneSymbol_1&2 are NaN
df.loc[joined_1.index.intersection(joined_2.index)]
nan_before = df.isnull().sum().GeneSymbol_1 + df.isnull().sum().GeneSymbol_2
nan_after = joined_1.EntrezGene.isnull().sum() + joined_2.EntrezGene.isnull().sum() 
print("NaN Entrez Gene before mapping with UniProtKB:", nan_before)
print("NaN Entrez Gene after mapping with UniProtKB:", nan_after)
print("newly found: {} ({}%)"
      .format(nan_before-nan_after, 100-round(nan_after/nan_before, 1)*100))
df_entrez = df[df['EntrezGene_1'].notnull()]
df_entrez = df_entrez[df_entrez['EntrezGene_2'].notnull()]
df_entrez.EntrezGene_1 = df_entrez.EntrezGene_1.astype(int)
df_entrez.EntrezGene_2 = df_entrez.EntrezGene_2.astype(int)

print("removed PPI number: {} ({}%)".format(df.shape[0]-df_entrez.shape[0],
                                          round(100-df_entrez.shape[0]/df.shape[0]*100,1)))

# unique number
df_entrez_unique_unip = df_entrez.UniProt_1.append(df_entrez.UniProt_2)
df_entrez_unique_unip = df_entrez_unique_unip.drop_duplicates()

df_entrez_unique_entrez = df_entrez.EntrezGene_1.append(df_entrez.EntrezGene_2)
df_entrez_unique_entrez = df_entrez_unique_entrez.drop_duplicates()

df_entrez_unique_geneSymb = df_entrez.GeneSymbol_1.append(df_entrez.GeneSymbol_2)
df_entrez_unique_geneSymb = df_entrez_unique_geneSymb.drop_duplicates()

print("unique UniProt:", len(df_entrez_unique_unip))
print("unique Entrez Gene:", len(df_entrez_unique_entrez))
print("unique Gene Symbol:", len(df_entrez_unique_geneSymb))
print("\nall interactions (PPI):", df_entrez.shape[0])
# save
filename = (data_folder + 'PPI_APID_{}.csv'.
            format(datetime.datetime.now().strftime("%Y_%m_%d")))
df_entrez.to_csv(filename, sep='\t', index=False)<jupyter_output><empty_output><jupyter_text>PPI network adjacency matrixaround 30-40 secondes to generate a new adjacency matrix<jupyter_code>import glob
from scipy.io import loadmat, savemat
gene_id_ppi, network = load_data.load_PPI_network(data_folder, ppi_data="APID")
network<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12425695110258869, 'fraction_numerical': 0.03566634707574305, 'mean_word_length': 5.972593582887701, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '614439', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4408, 'n_tokens_neox': 3945, 'n_words': 771, 'path': 'data_formatting/PPI_APID.ipynb', 'repo_name': 'ntraut/StratiPy', 'size': '10443'}