<jupyter_start><jupyter_text>Read previously trained model<jupyter_code>DF_test = pd.read_csv('DF_test.csv', index_col=0)  # filtered in sastry_preprocessing.ipynb
aa_model = load_model('aa_model.h5')  # trained in ndac_workflow.ipynb<jupyter_output>WARNING:tensorflow:From /Users/joshsmith/Git/NovoNordisk_Capstone/.env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py:497: calling conv1d (from tensorflow.python.ops.nn_ops) with data_format=NHWC is deprecated and will be removed in a future version.
Instructions for updating:
`NHWC` for data_format is deprecated, use `NWC` instead<jupyter_text>predict expression for unseen PrESTs<jupyter_code>X_test, y_test = encode_sequence(DF_test['aa_seq'], DF_test['expressed'],
                                 max_length=200)
pred = aa_model.predict_proba(X_test)
DF_test.loc[:, 'pred'] = pred<jupyter_output><empty_output><jupyter_text>We now have a predicted expression probability for each PrEST. Our knowledge of it's true expression ('expressed') will be used for a retrospective analysis to estimate experimental savings.<jupyter_code>DF_test[DF_test.uniprot_id == 'O43506'][['uniprot_id', 'aa_seq', 'pred', 'expressed']].sort_values('pred', ascending=False)<jupyter_output><empty_output><jupyter_text>Original Data Statistics<jupyter_code>print('Original total number of experiments:',len(DF_test))
print('Original total number of proteins:',len(DF_test.uniprot_id.unique()))
print('Original Number of passed experiments:',len(DF_test[DF_test.expressed == 1.0]))
print('Original Pass rate: %.2f%%'%(np.true_divide(len(DF_test[DF_test.expressed == 1.0]),len(DF_test))*100))
print('Number of proteins with >1 high expression:',len(DF_test[DF_test.expressed == 1.0].uniprot_id.unique()))<jupyter_output>Original Number of passed experiments: 5684
Original Pass rate: 49.16%
Number of proteins with >1 high expression: 3587<jupyter_text>Grab only the top predicted prESTs for each protein<jupyter_code>n = 4
np.random.seed(0)

output_df = pd.DataFrame(columns=['prest_id','uniprot_id','expressed','pred','aa_seq'])
remaining_df = DF_test.copy()

for i in range(n):
    print('Iteration',i)
    new_output_df = remaining_df.sort_values(['uniprot_id','pred'],ascending=[1,0]).drop_duplicates('uniprot_id')
    output_df = pd.concat([output_df,new_output_df])
   
    pred_pos_proteins = set(output_df[output_df.pred > 0.5].uniprot_id)
    true_pos_proteins = set(output_df[output_df.expressed == 1.0].uniprot_id)
    print('Total number of proposed experiments:',len(output_df))
    print('Total number of expressed proteins:',len(true_pos_proteins))
    print('Overall pass rate:',np.true_divide(len(true_pos_proteins),len(output_df)))

    # Prepare for next iteration
    remaining_df = remaining_df.drop(new_output_df.index)
    remaining_df = remaining_df[remaining_df.uniprot_id.isin(true_pos_proteins)==False]

    print
print('Percent saved experiments:',(1 - np.true_divide(len(output_df),len(DF_test)))*100,'%')<jupyter_output>Iteration 0
Total number of proposed experiments: 4759
Total number of expressed proteins: 3079
Overall pass rate: 0.6469846606429922
Iteration 1
Total number of proposed experiments: 6439
Total number of expressed proteins: 3543
Overall pass rate: 0.5502407206087901
Iteration 2
Total number of proposed experiments: 6816
Total number of expressed proteins: 3586
Overall pass rate: 0.5261150234741784
Iteration 3
Total number of proposed experiments: 6896
Total number of expressed proteins: 3587
Overall pass rate: 0.5201566125290024
Percent saved experiments: 40.3563397336101 %
{'fraction_non_alphanumeric': 0.11390284757118928, 'fraction_numerical': 0.04773869346733668, 'mean_word_length': 6.979955456570156, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '659139', 'license': 'MIT', 'n_tokens_mistral': 1323, 'n_tokens_neox': 1182, 'n_words': 264, 'path': 'examples/retrospective_analysis.ipynb', 'repo_name': 'anotherjoshsmith/NovoNordisk_Capstone', 'size': '3582'}