<jupyter_start><jupyter_text>Filter the table to only include the normalization features, make sure to remove the discardable features.<jupyter_code>table['spike-collapsed'] = normalization.sum(axis=1)
table.drop(discardable.columns, axis=1, inplace=True)
table.drop(normalization.columns, axis=1, inplace=True)
def normalize_by_sequence(row, reference=None):
    ref = row[reference].sum()
    return ((row * 10000) / ref).astype(np.int)

norm = table.apply(normalize_by_sequence, axis=1, reference='spike-collapsed')<jupyter_output><empty_output><jupyter_text>After the normalization, the reference sequence is uniform through all the samples, hence we should see 10,000 return from the following operation.<jupyter_code>norm['spike-collapsed'].sum() / len(norm)<jupyter_output><empty_output><jupyter_text>Remove the spike altogether.<jupyter_code>norm.drop(['spike-collapsed'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Save as a new artifact and rarefy the table.<jupyter_code>normalized_feature_table = q2.Artifact.import_data('FeatureTable[Frequency]', norm)
normalized_feature_table.save('feature-table.normalized.qza')
normalized_feature_table.view(pd.DataFrame).sum(axis=1).sort_values(ascending=False)
res = FT.methods.rarefy(normalized_feature_table, 18813)
res.rarefied_table.save('feature-table.even.18813.normalized.qza')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12247838616714697, 'fraction_numerical': 0.019452449567723344, 'mean_word_length': 9.213235294117647, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '655881', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 481, 'n_tokens_neox': 454, 'n_words': 79, 'path': 'notebooks/0.1.3-acinetobacter-normalization.ipynb', 'repo_name': 'y1zhong/anaerobic-digesters', 'size': '1388'}