<jupyter_start><jupyter_text>PyGNA Workflow The workflow involves the following three steps1. Generate GMT files from CSV files in case GMT file isn't available2. Generate matrices3. Perform analysis for single or multiple genesets. Get the results in the form of pdf or png  Data Loading  Generating GMT files from  a table This is when you have a table data from csv or Deseq. The following utlity can be used to generate gmt files from table data.<jupyter_code>$ pygna geneset-from-table <filename>.csv <setname> <filename>.gmt --name-colum <gene_names_column> --filter-column <filter-col> <'less'> --threshold <th> --descriptor <descriptor string>
$ pygna geneset-from-table <deseq>.csv diff_exp <deseq>.gmt --descriptor deseq#for table from deseq<jupyter_output><empty_output><jupyter_text>Merging different Genesets It is also possible to merge different setnames in a single gmt file through the function generate-group-gmt. You can override the default parameters, to match the columns in your table.*generate-group-gmt* generates a GMT file of multiple setnames. From the table file, it groups the names in the group_col (the column you want to use to group them) and prints the genes in the name_col. Set the descriptor according to your needs. OR you could simply concatenate all the files.   Computing rwr and sp matrices<jupyter_code>$ pygna build-distance-matrix <network> <network_sp>.hdf5
$ pygna build-rwr-diffusion <network> --output-file <network_rwr>.hdf5<jupyter_output><empty_output><jupyter_text>Topology Tests<jupyter_code>$ pygna test-topology-module <network> <geneset> <table_results_test>_topology_module.csv --number-of-permutations 100 --cores 4
$ pygna test-topology-rwr <network> <geneset> <network_rwr>.hdf5 <table_results_test>_topology_rwr.csv --number-of-permutations 100 --cores 4
$ pygna test-topology-internal-degree <network> <geneset> <table_results_test>_topology_internal_degree.csv --number-of-permutations 100 --cores 4
$ pygna test-topology-sp <network> <geneset> <network_sp>.hdf5 <table_results_test>_topology_sp.csv --number-of-permutations 100 --cores 4
$ pygna test-topology-total-degree <network> <geneset> <table_results_test>_topology_total_degree.csv --number-of-permutations 100 --cores 4<jupyter_output><empty_output><jupyter_text>Association tests If only A_geneset_file is passed the analysis is run on all the pair of sets in the file, if both A_geneset_file and B_geneset_file are passed, one can specify the setnames for both, if there is only one geneset in the file, setname_X can be omitted, if both sets are in the same file, B_geneset_file can be not specified, but setnames are needed<jupyter_code>pygna test-association-rwr [-h] [--setname-a SETNAME_A] [--file-geneset-b FILE_GENESET_B] [--setname-b SETNAME_B] [--size-cut SIZE_CUT] [-k] [-c CORES] [-i]
                                [--number-of-permutations NUMBER_OF_PERMUTATIONS] [--n-bins N_BINS] [--results-figure RESULTS_FIGURE]
                                network-file file-geneset-a rwr-matrix-filename output-table

    Performs comparison of network location analysis.

    It computes a p-value for the shortest path distance
    between two genesets being smaller than expected by chance.

    If only A_geneset_file is passed the analysis is run on all the pair of sets in the file, if both
    A_geneset_file and B_geneset_file are passed, one can specify the setnames for both, if there is only one
    geneset in the file, setname_X can be omitted, if both sets are in the same file, B_geneset_file can be not
    specified, but setnames are needed.


positional arguments:
network-file          network file
file-geneset-a        GMT geneset file
rwr-matrix-filename   .hdf5 file with the RWR matrix obtained by pygna
output-table          output results table, use .csv extension

optional arguments:
-h, --help            show this help message and exit
--setname-a SETNAME_A
                        Geneset A to analyse (default: -)
--file-geneset-b FILE_GENESET_B
                        GMT geneset file (default: -)
--setname-b SETNAME_B
                        Geneset B to analyse (default: -)
--size-cut SIZE_CUT   removes all genesets with a mapped length < size_cut (default: 20)
-k, --keep            if true, keeps the geneset B unpermuted (default: False)
-c CORES, --cores CORES
                        Number of cores for the multiprocessing (default: 1)
-i, --in-memory       set if you want the large matrix to be read in memory (default: False)
--number-of-permutations NUMBER_OF_PERMUTATIONS
                        number of permutations for computing the empirical pvalue (default: 500)
--n-bins N_BINS       if >1 applies degree correction by binning the node degrees and sampling according to geneset distribution (default: 1)
--results-figure RESULTS_FIGURE
                        heatmap of results (default: -)
$ pygna test-association-sp <network> <geneset> <network_sp>.hdf5 <table_results_test>_association_sp.csv -B <geneset_pathways> --keep --number-of-permutations 100 --cores 4
$ pygna test-association-rwr <network> <geneset> <network_rwr>.hdf5 <table_results_test>_association_rwr.csv -B <geneset_pathways> --keep --number-of-permutations 100 --cores 4<jupyter_output><empty_output><jupyter_text>Visualisation<jupyter_code>Usage: pygna paint-datasets-stats [-h] [-a ALTERNATIVE] table-filename output-file #GNT barplot
Usage: pygna paint-summary-gnt [-h] [-s SETNAME] [-t THRESHOLD] [-c COLUMN_FILTER] [--larger] [--less-tests LESS_TESTS] output-figure [input_tables [input_tables ...]]#GNT Summary
Usage: pygna paint-comparison-matrix [-h] [-r] [-s] [-a] table-filename output-file#heatmap
Usage: pygna paint-volcano-plot [-h] [-r] [-i ID_COL] [--threshold-x THRESHOLD_X] [--threshold-y THRESHOLD_Y] [-a] table-filename output-file#volcanoplot<jupyter_output><empty_output><jupyter_text>Snakemake Workflow 1) Install Snakemake2) Make changes to the config file and rules files accordingly(changing the path/parameters etc)3) Run the analysisAll the steps from above are boiled down to one or two steps.<jupyter_code>snakemake --use-conda -n#dry run

snakemake --snakefile Snakefile_paper --configfile config_paper --use-conda --cores $N#to replicate the results of the paper<jupyter_output><empty_output><jupyter_text>To obtain all the results for the single geneset (avoid the first step to have the full regeneration of all files):<jupyter_code>snakemake snakemake --snakefile Snakefile_paper single_all --configfile config_paper_single.yaml -t 
 
 snakemake --snakefile Snakefile_paper single_all --configfile config_paper_single.yaml --use-conda<jupyter_output><empty_output><jupyter_text>To obtain the results for the multi geneset<jupyter_code>snakemake snakemake --snakefile Snakefile_paper multi_all --configfile config_paper_multi.yaml -t 
 
 snakemake --snakefile Snakefile_paper multi_all --configfile config_paper_multi.yaml<jupyter_output><empty_output><jupyter_text>Paper Use Case  Using Commandline Since the distance matrices are already built and the merged geneset(gmt) already obtained, topology and association analysis can be carried out directly.  **Topology Analysis**<jupyter_code>#file names: biogrid_3.168_filtered.tsv merged.gmt goslim.gmt interactome_RWR.hdf5 interactome_SP.hdf5
cd /home/gee3/Documents/PyGNA/data_tcga_workflow/external/
! pygna test-topology-module biogrid_3.168_filtered.tsv merged.gmt table_topology_module3.csv --number-of-permutations 100 --cores 2




! pygna test-topology-rwr biogrid_3.168_filtered.tsv merged.gmt interactome_RWR.hdf5 tableresults_topology_rwr.csv --number-of-permutations 10 --cores 3
! pygna test-topology-internal-degree biogrid_3.168_filtered.tsv merged.gmt table_topology_internal_degree.csv --number-of-permutations 10 --cores 3
! pygna test-topology-sp biogrid_3.168_filtered.tsv merged.gmt interactome_SP.hdf5 table_topology_sp.csv --number-of-permutations 10 --cores 2

! pygna test-topology-total-degree biogrid_3.168_filtered.tsv merged.gmt table_topology_total_degree.csv --number-of-permutations 100 --cores 4<jupyter_output><empty_output><jupyter_text>**Association Tests** In a GNA two genesets are tested for their association. When testing a signle geneset against many pathways it is recommended the –keep flag is used. This way, while resampling only the geneset a will be randomly permuted and the geneset b is going to be kept as it is. This strategy is more conservative and is helpful in testing whether the tested geneset is more strongly connected to the pathway (or any other geneset of interest) than expected by chance.<jupyter_code>! pygna test-association-rwr biogrid_3.168_filtered.tsv merged.gmt interactome_RWR.hdf5 table_association_rwr.csv --file-geneset-b goslim_entrez.gmt --keep --number-of-permutations 100 --cores 4<jupyter_output><empty_output><jupyter_text>If you don't include the --results-figure flag at the comparison step, plot the matrix as follows<jupyter_code>! pygna paint-comparison-matrix table_association_rwr.csv heatmap_association_rwr.png --rwr --annotate<jupyter_output><empty_output><jupyter_text>If setname B is not passed, the analysis is run between each couple of setnames in the geneset as follows(The only difference between single geneset and multiple genests. No within comprison in multi):<jupyter_code>! pygna test-association-rwr biogrid_3.168_filtered.tsv merged.gmt interactome_RWR.hdf5 table_within_comparison_rwr.csv --number-of-permutations 100 --cores 2

! pygna paint-comparison-matrix table_within_comparison_rwr.csv heatmap_within_comparison_rwr.png --rwr --single-geneset
! pygna test-association-sp biogrid_3.168_filtered.tsv merged.gmt interactome_SP.hdf5 table_association_SP.csv --file-geneset-b goslim_entrez.gmt --keep --number-of-permutations 2 --cores 1 
! pygna paint-comparison-matrix table_association_sp.csv heatmap_association_sp.png --rwr --annotate#default heatmap 
! pygna test-association-sp biogrid_3.168_filtered.tsv merged.gmt interactome_RWR.hdf5 table_within_comparison_sp.csv --number-of-permutations 2 --cores 2
! pygna paint-comparison-matrix table_within_comparison_sp.csv heatmap_within_comparison_rwr.png --rwr --single-geneset<jupyter_output><empty_output><jupyter_text>**Diagnostic** Distribution plotWhen running a statistical test, one might want to visually assess the null distribution. By passing -d \ through command line, a distribution plot of the empirical null is shown for each test.<jupyter_code>! pygna test-topology-total-degree biogrid_3.168_filtered.tsv merged.gmt diagnstic_total_degree.csv -d "diagnostic/" --number-of-permutations 2 --cores 2<jupyter_output><empty_output><jupyter_text>**Visualisation** There are four main types of figures currently implemented in PyGNA, namely bar plots, point plots, heatmaps and volcano plots, to visualize to GNT and GNA results.Barplots are used to plot the GNT results for a single statistic. For each geneset a red bar represents the observed statistic, whereas a blue one represents the average of the empirical null distribution. Conversely, a dot plot can be used to summarize multiple tests for the same geneset. In order to show all the results in the same figure, the observed values are transformed in absolute normalized z-scores, such that all significant tests have z-score >0 and are marked with a red dot. GNA results can instead be visualised on heatmaps, with the color gradients used to report the strength of association between two genesets. When an all-vs-all test is conducted, a lower triangular matrix is shown, with stars denoting significance. If, instead, a M-vs-N test was conducted, a complete heatmap would be included in the plot.Alternatively, volcano plots can be used to visualize one-vs-many GNA results, for testing a geneset against a large number of datasets (e.g. gene ontologies). The plot shows the normalized z-score on the x-axis and the −log10 of the p-value adjusted to control the False Discovery Rate (FDR) on the y-axis. Significant results are shown with red crosses, whereas not significant associations are represented by blue dots.Can be annotated to fid out the top 5 terms.<jupyter_code>! pygna paint-datasets-stats table_topology_module.csv gnt_tm.png #GNT barplot
! pygna paint-summary-gnt dotplt.png #GNT Summary
! pygna paint-comparison-matrix table_association_sp.csv withncomp_sp.pdf #heatmap
! pygna paint-volcano-plot table_association_sp.csv volcno_sp.png #volcanoplot<jupyter_output><empty_output><jupyter_text>**Benchmarking**  GNT and GNA benchmarking using SBM<jupyter_code>! pygna generate-gnt-sbm "benchmarking/gnt_sbm.tsv" 'benchmarking/gnt_sbm.gmt'
! pygna generate-gna-sbm "benchmarking/gna_sbm.tsv" 'benchmarking/gna_sbm.gmt'<jupyter_output>[50, 50, 50, 50, 50, 50, 700]
INFO:root:Network written on benchmarking/gnt_sbm.tsv
[50, 50, 50, 50, 50, 50, 50, 50, 600]
INFO:root:Network written on benchmarking/gna_sbm.tsv
Generatedbenchmarking/gna_sbm.tsv<jupyter_text>GNT and GNA benchmarking using SBM<jupyter_code>! pygna generate-hdn-network benchmarking/ hdn_network<jupyter_output>INFO:root:Reject=True
INFO:root:Reject=True
INFO:root:Nodes: 1000, in LCC: 999
INFO:root:Reject=True
INFO:root:Nodes: 1000, in LCC: 999
INFO:root:Reject=True
INFO:root:Nodes: 1000, in LCC: 999
INFO:root:Reject=False
INFO:root:Nodes: 1000, in LCC: 1000
INFO:root:Network written on benchmarking/hdn_network_s_0_network.tsv
INFO:root:Reject=True
INFO:root:Reject=False
INFO:root:Nodes: 1000, in LCC: 1000
INFO:root:Network written on benchmarking/hdn_network_s_1_network.tsv
INFO:root:Reject=True
INFO:root:Reject=False
INFO:root:Nodes: 1000, in LCC: 1000
INFO:root:Network written on benchmarking/hdn_network_s_2_network.tsv
INFO:root:Reject=True
INFO:root:Reject=True
INFO:root:Nodes: 1000, in LCC: 999
INFO:root:Reject=False
INFO:root:Nodes: 1000, in LCC: 1000
INFO:root:Network written on benchmarking/hdn_network_s_3_network.tsv
INFO:root:Reject=True
INFO:root:Reject=False
INFO:root:Nodes: 1000, in LCC: 1000
INFO:root:Network written on benchmarking/hdn_network_s_4_network.tsv<jupyter_text>Given the generated network and node list of HDNs, novel genesets made of mixtures of the two can be generate.The original network with a number of HDNs, then the partial, extended, and branching genesets can be generated.  **Adding Extended Genesets** Creates new genesets from the vip list, number of genesets and portion of genes    can be specified by input. The final new geneset is going to be formed by:    percentage ev * HDN_total + ratio* percentage ev*vips total.<jupyter_code>!pygna hdn-add-extended input-geneset-file#Genesets are input to identify<jupyter_output><empty_output><jupyter_text>**Adding Partial Genesets**Creates new genesets from the vip list, number of genesets and portion of    genes can be specified by input.<jupyter_code>!pygna hdn-add-partial input-geneset-file<jupyter_output><empty_output><jupyter_text>**Adding Branching Genesets**  Creates new genesets from the vip list, new genesets are created adding 1 step    nodes to vips. The new genes are created as branches.<jupyter_code>!pygna hdn-add-branching input-geneset-file<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08032181482458454, 'fraction_numerical': 0.01622263255077816, 'mean_word_length': 4.968122786304605, 'pattern_counts': {'":': 0, '<': 117, '<?xml version=': 0, '>': 118, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '96852', 'license': 'Apache-2.0', 'n_tokens_mistral': 5127, 'n_tokens_neox': 4731, 'n_words': 1580, 'path': 'Pathway Analysis/Case Studies/code/PyGNA workflow.ipynb', 'repo_name': 'shravyagupta/BoltBio', 'size': '15164'}