<jupyter_start><jupyter_text>Pivot Merged NCBITaxon KG Edges This notebook pivots the data from the NCBITaxon Merged KG Edges tsv.   Output is saved to the file `data/pivot_merged-kg_edges_NCBITaxon.tsv.gz`<jupyter_code>import pandas as pds<jupyter_output><empty_output><jupyter_text>Load file `data/merged-kg_edges_NCBITaxon_clean.tsv.gz` into dataframe<jupyter_code>kgEdgesDf = pds.read_csv('data/merged-kg_edges_NCBITaxon_clean.tsv.gz', sep='\t')
len(kgEdgesDf)<jupyter_output><empty_output><jupyter_text>Create a unique list of columns from the objects and get a unique list of subjects. The subjects are sorted in order to create a more efficient index.<jupyter_code>cols = ['subject'] +list(kgEdgesDf['object'].unique()) # note: prepend a subject col
subjects = kgEdgesDf['subject'].unique()
subjects.sort()
print('num cols:', len(cols))
print('num subjects:', len(subjects))<jupyter_output>num cols: 328
num subjects: 31835<jupyter_text>Create an empty pivot "template" with subjects as index and each uinque object as a column The NaNs are filled with 0 in order to make filling the template easier.<jupyter_code>pivotDf = pds.DataFrame(columns=cols)
pivotDf['subject'] = subjects
pivotDf.fillna(0, inplace=True)
pivotDf.set_index('subject', inplace=True)
# pivotDf.head() # peek at data<jupyter_output><empty_output><jupyter_text>fill in pivot template<jupyter_code>for idx, subj, obj, in kgEdgesDf[['subject', 'object']].itertuples():
    pivotDf.loc[subj, obj] = pivotDf.loc[subj, obj] + 1
# pivotDf.head() # peek at data<jupyter_output><empty_output><jupyter_text>Save pivoted data<jupyter_code>pivotDf.to_csv('data/pivot_merged-kg_edges_NCBITaxon.tsv.gz', sep='\t', compression='gzip')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11863425925925926, 'fraction_numerical': 0.00636574074074074, 'mean_word_length': 7.776649746192893, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '554034', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 661, 'n_tokens_neox': 632, 'n_words': 144, 'path': 'notebooks/pivot-kg-edges/pivot-merged-kg-edges-NCBITaxon.ipynb', 'repo_name': 'realmarcin/kg-microbe', 'size': '1728'}