<jupyter_start><jupyter_text>Microbiome experiment step-by-step analysisThis is a jupyter notebook example of how to load, process and plot data from a microbiome experiment using Calour.  Setup  Import the calour module<jupyter_code>import calour as ca<jupyter_output><empty_output><jupyter_text>(optional) Set the level of feedback messages from calourcan use:* 1 for debug (lots of feedback on each command)* 11 for info (useful information from some commands)* 21 for warning (just warning messages)The Calour default is warning (21)<jupyter_code>ca.set_log_level(11)<jupyter_output><empty_output><jupyter_text>Also enable interactive plots inside the jupyter notebook<jupyter_code>%matplotlib notebook<jupyter_output><empty_output><jupyter_text>Loading the dataFor an amplicon experiment we use **ca.read_amplicon()**First parameter is the location+name of the biom table file (can be hdf5/json/txt biom table - see here for details)Second (optional) parameter is the sample mapping file locaion+name. First column should be the sample id (identical to the sample ids in the biom table). Rest of the column are information fields about each sample.normalize=XXX : tells calour to rescale each sample to XXX reads (by dividing each feature frequency by the total number of reads in the sample and multiplying by XXX). Alternatively, can use normalize=None to skip normalization (i.e. in the case the biom table is already rarified)min_reads=XXX : throw away samples with less than min_reads total (before normalization). Useful to get rid of samples with small number of reads. Can use min_reads=None to keep all samples.We will use the data from:<NAME>., <NAME>., <NAME>., <NAME>., <NAME>. and <NAME>., 2016.Reduced diversity and altered composition of the gut microbiome in individuals with myalgic encephalomyelitis/chronic fatigue syndrome.Microbiome, 4(1), p.30.<jupyter_code>dat=ca.read_amplicon('data/chronic-fatigue-syndrome.biom',
                     'data/chronic-fatigue-syndrome.sample.txt',
                     normalize=10000,min_reads=1000)
print(dat)<jupyter_output>AmpliconExperiment with 87 samples, 2129 features<jupyter_text>Process the data  Get rid of the features (bacteria) with small amount of readsWe throw away all features with total reads (over all samples) < 10 (after each sample was normalized to 10k reads/sample). So a bacteria present (with 1 read) in 10 samples will be kept, as well as a bacteria present in only one sample, with 10 reads in this sample.Note alternatively we could filter based on mean reads/sample or fraction of samples where the feature is present. Each method filters away slightly different bacteria. See **filtering** notebook for details on the filtering functions.<jupyter_code>dat=dat.filter_sum_abundance(10)<jupyter_output>2020-09-11 19:34:53 INFO After filtering, 1100 remain.<jupyter_text>Cluster (reorder) the features so similarly behaving bacteria are close to each otherFeatures are clustered (hierarchical clustering) based on euaclidian distance between features (over all samples) following normalizing each feature to mean 0 std 1. For more details and examples, see **sorting** notebook or **cluster_features documentation*** Note that if we have a lot of features, clustering is slow, so it is recommended to first filter away the non-interesting features.<jupyter_code>datc=dat.cluster_features()<jupyter_output>2020-09-11 19:34:54 INFO After filtering, 1100 remain.<jupyter_text>Sort the samples according to physical functioning and Disease stateNote that order within each group of similar value is maintained. We first sort by physical functioning, then sort by the disease state. So within each disease state, samples will still be sorted by physical functioning.<jupyter_code>datc=datc.sort_samples('Physical_functioning')
datc=datc.sort_samples('Subject')<jupyter_output><empty_output><jupyter_text>Plotting the dataColumns (x-axis) are the samples, rows (y-axis) are the features. We will show on the x-axis the host-individual field of each sample.we will use the jupyter notebook GUI so we will see the interactive plot in the notebook. Alternatively we could use the qt5 GUI to see the plot in a separate standalone window.A few cool things we can do with the interactive plot:* Click with the mouse on the heatmap to see details about the feature/sample selected (including information from **dbBact**).* use SHIFT+UP or SHIFT+DOWN to zoom in/out on the features* use UP/DOWN to scroll up/down on the features* use SHIFT+RIGHT or SHIFT+LEFT to zoom in/out on the samples* use RIGHT/LEFT to scroll left/right on the samplesSee **here** for more details<jupyter_code>datc.plot(sample_field='Subject', gui='jupyter')<jupyter_output><empty_output><jupyter_text>Adding a field to the top barNow let's add the values of the "Sex" field into the xbar on topFirst we'll also sort by sex, so values will be continuous (note we then sort by the disease state to get the two groups separated).<jupyter_code>datc=datc.sort_samples('Sex')
datc=datc.sort_samples('Subject')
datc.plot(sample_field='Subject', gui='jupyter',barx_fields=['Sex'])<jupyter_output><empty_output><jupyter_text>Differential abundance testingLet's look for bacteria separating sick from healthyWe ask it to find all bacteria significantly different between samples with 'Control' and 'Patient' in the 'Subject' field.By default calour uses the mean of the ranks of each feature (over all samples), with dsFDR multiple hypothesis correction.For more information, see **notebook** and **function doc**<jupyter_code>dd=datc.diff_abundance(field='Subject',val1='Control',val2='Patient', random_seed=2018)<jupyter_output>2020-09-11 19:35:18 INFO 87 samples with both values
2020-09-11 19:35:18 INFO After filtering, 1100 remain.
2020-09-11 19:35:18 INFO 39 samples with value 1 (['Control'])
2020-09-11 19:35:19 INFO number of higher in Control: 36. number of higher in Patient : 17. total 53<jupyter_text>Plotting the differentially abundant featuresLet's plot to see the behavior of these bacteria.The output of diff_abundance is an Experiment with only the significant bacteria, which are sorted by the effect size. On the bottom is the bacteria with the largest effect size (higher in Control compared to Patient).<jupyter_code>dd.plot(sample_field='Subject', gui='jupyter')<jupyter_output><empty_output><jupyter_text>dbBact term enrichmentWe can ask what is special in the bacteria significanly higher in the Control vs. the Patient group and vice versa.We supply the parameter `ignore_exp=[12]` to ignore annotations regarding this experiment (expid=12) since it is already in the dbBact database.* Note since we need to get the per-feature annotations from dbBact, we need a live internet connection to run this command.<jupyter_code>ax, enriched=dd.plot_diff_abundance_enrichment(term_type='combined',ignore_exp=[12])
ax, enriched=dd.plot_diff_abundance_enrichment(term_type='combined',ignore_exp=[12])<jupyter_output><empty_output><jupyter_text>The enriched terms are in a calour experiment class (terms are features, bacteria are samples), so we can see thelist of enriched terms with the p-value (pval) and effect size (odif)<jupyter_code>enriched.feature_metadata<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07125137816979052, 'fraction_numerical': 0.023153252480705624, 'mean_word_length': 6.045631067961165, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '21192', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2184, 'n_tokens_neox': 1961, 'n_words': 871, 'path': 'doc/source/notebooks/microbiome_step_by_step.ipynb', 'repo_name': 'sadaf86/calour', 'size': '7287'}