<jupyter_start><jupyter_text>Candidate classification: Multilabel and multiclass method evaluation Problem specificationPredict biosynthetic class (classes) of BGC candidates. Methods Multilabel approachAssign a set of classes (labels) to each BGC candidateModel output is a binary vector:<jupyter_code>activity = pd.read_csv("../data/mibig/mibig_compounds.csv").set_index('contig_id')
activity = activity[activity['any_known_activity']]
activity.head()
all_labels = activity[[c for c in activity.columns if c.startswith('any_') and c != 'any_known_activity']]
all_labels = all_labels.rename(columns={c: c.replace('any_','') for c in all_labels.columns})
label_counts = all_labels.sum().sort_values()
label_counts.plot.barh()
label_counts[::-1]
labels = all_labels[['antibacterial','cytotoxic','inhibitor','antifungal']]
labels.head()<jupyter_output><empty_output><jupyter_text>Input data  Domain setBinary vector of top 2000 pfam IDs, 1 = pfam is present in sequence, 0 = pfam is not present<jupyter_code>get_samples('domain_set').head(3)<jupyter_output><empty_output><jupyter_text>Classification methods- K Nearest Neighbors- Random Forest Validation methods- Train/test split validation- N-Fold Crossvalidation  Results<jupyter_code>def get_sample_responses(vectors, responses):
    samples = get_samples(vectors)
    responses = responses.loc[samples.index].dropna()
    samples = samples.loc[responses.index]
    return samples, responses<jupyter_output><empty_output><jupyter_text>Multilabel classification<jupyter_code>ml_samples, ml_responses = get_sample_responses('domain_set', labels)
len(ml_samples)
major_bgc_ids = pd.Series(ml_samples.index).apply(lambda contig_id: contig_id.split('.')[0])
major_to_minor_ids = pd.Series(ml_samples.index, index=major_bgc_ids)
major_to_minor_ids.head()
unique_major_ids = major_bgc_ids.unique()
len(unique_major_ids)<jupyter_output><empty_output><jupyter_text>Cross-validation<jupyter_code>def create_model():
    return RandomForestClassifier(n_estimators=100)
splitter = KFold(n_splits=5, shuffle=True, random_state=0)

scores = []
accuracies = []
for id_train_idx, id_test_idx in splitter.split(unique_major_ids):
    train_major_ids, test_major_ids = unique_major_ids[id_train_idx], unique_major_ids[id_test_idx]
    train_minor_ids, test_minor_ids = major_to_minor_ids.loc[train_major_ids], major_to_minor_ids.loc[test_major_ids]
    X_train, X_test = ml_samples.loc[train_minor_ids].values, ml_samples.loc[test_minor_ids].values
    y_train, y_test = ml_responses.loc[train_minor_ids].values, ml_responses.loc[test_minor_ids].values
    print('Train:', len(X_train), 'Test:', len(X_test))
    model = create_model()
    # fitting the model
    model.fit(X_train, y_train)
    # predict the response
    pred = (model.predict(X_test) >= 0.5).astype(np.int)
    aucs = metrics.roc_auc_score(y_test, pred, average=None)
    precisions = metrics.precision_score(y_test, pred, average=None)
    recalls = metrics.recall_score(y_test, pred, average=None)
    accuracy = metrics.accuracy_score(y_test, pred)
    accuracies.append(accuracy)
    scores += [{'Precision': p, 'Recall': r, 'AUC': a, 'Class': c, 'Samples': int(sum(ml_responses[c]))} for p,r,a,c in zip(precisions, recalls, aucs, ml_responses.columns)]

scores = pd.DataFrame(scores).groupby('Class').mean().sort_values('Samples', ascending=False)
print('Accuracy (exact match)', np.mean(accuracies))
print(scores.mean())
scores
merged_scores = scores[['Samples','AUC','Precision','Recall']]
merged_scores = merged_scores.sort_values(by='Samples', ascending=False)
merged_scores.index = [v.capitalize() for v in merged_scores.index]
merged_scores = merged_scores.append(pd.Series(merged_scores.mean(), name='Average'))
merged_scores<jupyter_output><empty_output><jupyter_text>Candidate activity prediction<jupyter_code>model = RandomForestClassifier(n_estimators=100)
model.fit(ml_samples.values, ml_responses.values)
columns = np.array([l.replace('any_','').capitalize() for l in ml_responses.columns])
def read_vectors_to_predict(path, train_columns):
    vectors = pd.read_pickle(path)
    for col in set(train_columns).difference(vectors.columns):
        vectors[col] = 0
    return vectors[train_columns.values]
model_path = '128lstm-100pfamdim-8pfamiter-posweighted-neg-10k-fpr2'
candidates = read_vectors_to_predict("../data/bacteria/candidates/{}/domain_set.pkl".format(model_path), ml_samples.columns)
candidates.head()
results = np.array([r[:,1] for r in model.predict_proba(candidates.values)]).transpose()
predictions = pd.DataFrame(results, index=candidates.index, columns=columns)
predictions['classes'] = [';'.join(columns[x >= 0.5]) for x in results]
predictions.head()
predictions.to_csv('../data/bacteria/candidates/{}/classes/activity-domain_set-rf/classes.csv'.format(model_path))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13060549700351312, 'fraction_numerical': 0.006819590824550527, 'mean_word_length': 7.5968028419182945, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '552872', 'license': 'MIT', 'n_tokens_mistral': 1734, 'n_tokens_neox': 1631, 'n_words': 276, 'path': 'notebooks/CandidateActivityClassification.ipynb', 'repo_name': 'prihoda/bgc-pipeline-1', 'size': '4839'}