<jupyter_start><jupyter_text>Microbiome machine learning analysis  Setup  Import the calour module<jupyter_code>from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.model_selection import RepeatedStratifiedKFold
from calour.training import plot_scatter, plot_roc, plot_cm
import calour as ca
%matplotlib notebook<jupyter_output><empty_output><jupyter_text>Regression   Loading the dataWe will use the data from Qitta study 103 (https://qiita.ucsd.edu/study/description/103)<jupyter_code>dat=ca.read_amplicon('data/88-soil.biom',
                     'data/88-soil.sample.txt',
                     normalize=100,min_reads=10)
print(dat)<jupyter_output>AmpliconExperiment ("88-soil.biom") with 88 samples, 7396 features<jupyter_text>Process the data  Get rid of the features (bacteria) with small amount of readsWe throw away all features with total reads (over all samples) < 1 (after each sample was normalized to 100 reads/sample). Note alternatively we could filter based on mean reads/sample or fraction of samples where the feature is present. Each method filters away slightly different bacteria. See **filtering** notebook for details on the filtering functions.<jupyter_code>dat=dat.filter_abundance(1)
dat<jupyter_output><empty_output><jupyter_text>Use soil microbiome to predict its pH Let's look at the distribution of pH for all the samples:<jupyter_code>dat.sample_metadata['ph'].hist()
dat.sort_samples('ph').sort_centroid(n=0.001).plot(sample_field='ph', gui='jupyter')<jupyter_output><empty_output><jupyter_text>We can then run regression analysis:<jupyter_code>it = dat.regress('ph', RandomForestRegressor(random_state=0), cv=5, params=[{'n_estimators':3}, {'n_estimators': 500}])<jupyter_output><empty_output><jupyter_text>This function returns a generator, which yields the prediction result for each parameter set specified in `params`. Here we would like to see how the number of trees (named `n_estimators`) in the model impact its performance. The result with `n_estimators = 10` is:<jupyter_code>res1 = next(it)
res1.head()<jupyter_output><empty_output><jupyter_text>We can plot out the result as following. Each dot is a sample with observed and predicted pH, colored by the fold of cross validation the sample is from. The diagonal line indicates perfect predition. The correlation coefficient and its p-value between the prediction and observation are also annotated around the top of the plot.<jupyter_code>plot_scatter(res1, cv=True)<jupyter_output><empty_output><jupyter_text>Let's look at the result for `n_estimators = 500`:<jupyter_code>res2 = next(it)
res2.head()
plot_scatter(res2, cv=True)<jupyter_output><empty_output><jupyter_text>From the plot, you can see, with more trees in the Random Forest model, the prediction is much better with a higher correlation coefficient.  Classification We can do analysis similarly for classification. Let's show it with another data set that was introduced in previous notebook.<jupyter_code>dat=ca.read_amplicon('data/chronic-fatigue-syndrome.biom',
                     'data/chronic-fatigue-syndrome.sample.txt',
                     normalize=10000,min_reads=1000)
print(dat)<jupyter_output>AmpliconExperiment ("chronic-fatigue-syndrome.biom") with 87 samples, 2129 features<jupyter_text>Let's see if we can distinguish patient samples from control samples with classification:<jupyter_code>dat.sample_metadata['Subject'].value_counts()
it = dat.classify('Subject', RandomForestClassifier(random_state=0), cv=RepeatedStratifiedKFold(5, 3), params=[{'n_estimators':3}, {'n_estimators': 500}])
res1 = next(it)
res1.head()<jupyter_output><empty_output><jupyter_text>We can plot out the result as ROC curve or confusion matrix.<jupyter_code>plot_roc(res1, classes=['Patient'])<jupyter_output><empty_output><jupyter_text>You can also plot confustion matrix:<jupyter_code>plot_cm(res1)<jupyter_output><empty_output><jupyter_text>Let's look at the result for `n_estimators = 500`:<jupyter_code>res2 = next(it)
res2.head()
plot_roc(res2, classes=['Patient'])<jupyter_output><empty_output><jupyter_text>You can also plot confustion matrix:<jupyter_code>plot_cm(res2, normalize=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09487666034155598, 'fraction_numerical': 0.019449715370018977, 'mean_word_length': 6.941619585687382, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 55, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '75295', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1406, 'n_tokens_neox': 1308, 'n_words': 366, 'path': 'doc/source/notebooks/microbiome_machine_learning.ipynb', 'repo_name': 'pennyneve/calour', 'size': '4216'}