<jupyter_start><jupyter_text>$\textit{Note:}$ In order to run the following notebook, one should install QIIME 2. The installation instructions can be found [here](https://docs.qiime2.org/2021.4/install/native/)[QIIME 2](https://docs.qiime2.org/2021.4/) is a microbiome bioinformatics platform to analyze amplicon sequence data.<jupyter_code>import qiime2
import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Metadata of pH values in soil samples Read metadata about the samples, We need only pH<jupyter_code>metadata = pd.read_table('../../data/soil/original/88soils_modified_metadata.txt', index_col=0)
ph = metadata["ph"].to_csv('../../data/soil/processed/ph.csv', index=True)<jupyter_output><empty_output><jupyter_text>Import count OTUs data<jupyter_code>%%bash
qiime tools import \
    --input-path ../../data/soil/original/238_otu_table.biom \
    --output-path ../../data/soil/original/88soils.biom.qza \
    --type FeatureTable[Frequency]<jupyter_output><empty_output><jupyter_text>Load the table with 88 soil samples<jupyter_code>table_art = qiime2.Artifact.load('../../data/soil/original/88soils.biom.qza')
all_samples = table_art.view(pd.DataFrame)<jupyter_output><empty_output><jupyter_text>Preprocessing Select only those OTUs occuring at least $100$ times<jupyter_code>%%bash
qiime feature-table filter-features \
    --i-table ../../data/soil/original/88soils.biom.qza \
    --o-filtered-table ../../data/soil/original/88soils_filt100.biom.qza \
    --p-min-frequency 100<jupyter_output><empty_output><jupyter_text>[Replace](https://docs.qiime2.org/2018.6/plugins/available/composition/add-pseudocount/) zeros with pseudo $1$ in all samples for a smooth CLR transformation in the later analysis.<jupyter_code>%%bash
qiime composition add-pseudocount \
    --i-table ../../data/soil/original/88soils_filt100.biom.qza \
    --p-pseudocount 1 \
    --o-composition-table ../../data/soil/original/88soils_composition.biom.qza<jupyter_output><empty_output><jupyter_text>Load the table of filtered and zero-replaced data<jupyter_code>table_1 = qiime2.Artifact.load('../../data/soil/original/88soils_composition.biom.qza')
df = table_1.view(pd.DataFrame)<jupyter_output><empty_output><jupyter_text>Select 116 OTUs which are has been filtered out by [Morton et al](https://www.nature.com/articles/s41467-019-10656-5)<jupyter_code>morton = pd.read_excel('../../data/soil/original/ph_morton.xlsx', engine='openpyxl')
morton_otus = np.array(morton['#OTU_ID'])
our_otus = list(map(int, np.array(df.columns)))
diff = np.setdiff1d(our_otus, morton_otus)
diff = ["".join(item) for item in diff.astype(str)]<jupyter_output><empty_output><jupyter_text>Select OTUs described by Morton et al.<jupyter_code>final = df.loc[:, ~df.columns.isin(diff)]
final.shape
final.to_csv('../../data/soil/processed/soil_116.csv', index=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14994793474488025, 'fraction_numerical': 0.02846233946546338, 'mean_word_length': 5.224622030237581, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '20683', 'license': 'MIT', 'n_tokens_mistral': 1122, 'n_tokens_neox': 1014, 'n_words': 173, 'path': 'examples/soil/prepare_soil_data.ipynb', 'repo_name': 'bio-datascience/GGLasso', 'size': '2881'}