<jupyter_start><jupyter_text>Exploratory Data Analysis<jupyter_code>train_variants_df = pd.read_csv("training_variants")
test_variants_df = pd.read_csv("test_variants")
train_text_df = pd.read_csv("training_text", sep="\|\|", engine='python', header=None, skiprows=1, names=["ID","Text"])
test_text_df = pd.read_csv("test_text", sep="\|\|", engine='python', header=None, skiprows=1, names=["ID","Text"])

print("Train and Test variants shape : ",train_variants_df.shape, test_variants_df.shape)
print("Train and Test text shape : ",train_text_df.shape, test_text_df.shape)
train_variants_df.head()
test_variants_df.head()
train_text_df.head()
test_text_df.head()
train_text_df['Text'][3]
train_variants_df.Class.value_counts()<jupyter_output><empty_output><jupyter_text>Looks like class 7 is dominant. Data is unevenly distributed so it is fair to call this unbalanced classification in Training dataset. Classes 3, 9, and 8 have very low distribution. Plotting the value count for each class:<jupyter_code>train_variants_df['Class'].value_counts(sort=False).plot(kind="bar", rot=0)
plt.ylabel('Frequency', fontsize=12)
plt.xlabel('Class number', fontsize=12)
plt.title("Frequency of Classes", fontsize=15)
geneCounts = train_variants_df.Gene.value_counts()
geneCounts.nlargest(20)<jupyter_output><empty_output><jupyter_text>Would it be fair to assume that genes are classes are correlated?<jupyter_code>f,axs = plt.subplots(2,2, figsize = (10,10))
f.suptitle("Distribution of frequency occuring genes into various classes", fontsize=15)

train_variants_df.loc[train_variants_df['Gene'] =='BRCA1'].Class.value_counts().plot(kind='bar',ax=axs[0,0])
axs[0,0].set_title('BRCA1')
train_variants_df.loc[train_variants_df['Gene'] =='TP53'].Class.value_counts().plot(kind='bar',ax=axs[0,1])
axs[0,1].set_title('TP53')
train_variants_df.loc[train_variants_df['Gene'] =='EGFR'].Class.value_counts().plot(kind='bar',ax=axs[1,0])
axs[1,0].set_title('EGFR')
train_variants_df.loc[train_variants_df['Gene'] =='PTEN'].Class.value_counts().plot(kind='bar',ax=axs[1,1])
axs[1,1].set_title('PTEN')
plt.xlabel('Class', fontsize=12)
plt.ylabel('Count', fontsize=12)
plt.show()<jupyter_output><empty_output><jupyter_text>BRCA1: most distribution in class 5TP53: most distribution in class 4...Can genes' mutation and classes have any correlation?<jupyter_code>variationCounts = train_variants_df.Variation.value_counts()
variationCounts.nlargest(20)<jupyter_output><empty_output><jupyter_text>Five types of mutations have most distribution. Low distribution variations are all amino acid swaps at a particular location in the protein.<jupyter_code># replace X123X variation types with "AminoAcid swap" and look at distribution of classes for each variation

mutationTypes = {'Truncating Mutations','Deletion','Amplification','Fusions','Overexpression','AminoAcidSwap'}

# copy the variation column
train_variants_df['Variation_Alt'] = train_variants_df['Variation']

# replace values
train_variants_df['Variation_Alt'] = np.where(train_variants_df['Variation_Alt'].isin(mutationTypes), train_variants_df['Variation_Alt'], 'AminoAcidSwap')

train_variants_df['Variation_Alt']
g = sns.catplot(x='Class',data=train_variants_df[['Variation_Alt','Class']],col='Variation_Alt',kind='count',col_wrap=3,
           height = 10,sharey=False,sharex=False)
g.set_yticklabels(size = 18)
g.set_xticklabels(size = 18)
g.set_ylabels(size = 18)
g.set_xlabels(size = 18)
g.set_titles(size = 25)

plt.show()<jupyter_output>C:\Users\shashi\Documents\Anaconda\lib\site-packages\seaborn\categorical.py:3793: UserWarning: Setting `sharex=False` with `color=None` may cause different levels of the `x` variable to share colors. This will change in a future version.
  warnings.warn(msg.format("sharex", "x"), UserWarning)<jupyter_text>Some types of variations are heavily associated with certain class. The most prevalent variation AminoAcidSwap is widely distributed among classes.  The type of variation and the gene itself have some predictive power for the mutation to be classified in a certain class.  Let's count number of words per text entry.<jupyter_code># add information about word count and character count
train_text_df["num_words"] = train_text_df["Text"].apply(lambda x: len(str(x).split()) )
train_text_df["num_chars"] = train_text_df["Text"].apply(lambda x: len(str(x)) )

test_text_df["num_words"] = test_text_df["Text"].apply(lambda x: len(str(x).split()) )
test_text_df["num_chars"] = test_text_df["Text"].apply(lambda x: len(str(x)) )
plt.figure(figsize=(12, 8))
sns.distplot(train_text_df.num_words.values, bins=50, kde=False, color='red')
plt.xlabel('Number of words in text', fontsize=12)
plt.ylabel('Count', fontsize=12)
plt.title("Frequency of number of words", fontsize=15)
plt.show()<jupyter_output>C:\Users\shashi\Documents\Anaconda\lib\site-packages\seaborn\distributions.py:2551: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>Although many text entries are fairly long, it would be difficult to say if they play a role in predicting the class of a particular variation. But the lengthy text entries can have more than one gene variation that will need to be classified separately. Checking for empty text entries would be helpful here.<jupyter_code>train_text_df[train_text_df["num_words"]<5]<jupyter_output><empty_output><jupyter_text>Before we remove these empty text entries, we need to merge training variant and test variant dataframe with the respective text dataframes.  Combining variation data with text data<jupyter_code>train_df = pd.merge(train_variants_df, train_text_df, how='left', on='ID').fillna('')
test_df = pd.merge(test_variants_df, test_text_df, how='left', on='ID').fillna('')<jupyter_output><empty_output><jupyter_text>Training Data<jupyter_code>train_df.head()
train_df[train_df["num_words"]<200]
train_df.iloc[693].Text<jupyter_output><empty_output><jupyter_text>Now let's remove the empty rows from combined dataframes.<jupyter_code># remove rows with NaN
train_df = train_df[train_df['Text'].notna()]
test_df = test_df[test_df['Text'].notna()]

train_df[train_df["num_words"]<200]<jupyter_output><empty_output><jupyter_text>ID 344, 346, and 348 associated with CDH1 gene comes from the same text and has same class but has three different AminoAcidSwap.ID 2900, 2906, and 2908 associated with NF2 gene also comes from same text, has three different AminoAcidSwap but one of them belongs to a different class.  Test Data<jupyter_code>test_df.head()<jupyter_output><empty_output><jupyter_text>Let's prepare solution set and test set first<jupyter_code>solution = pd.read_csv('stage1_solution_filtered.csv')
solution2 = pd.read_csv('stage_2_private_solution.csv')
solution.head()
solution2.head()
print('Stage-1 solution has', len(solution), 'entries')
print('Stage-2 solution has', len(solution2), 'entries')
print('The testing set has', len(test_df), 'entries')<jupyter_output>Stage-1 solution has 368 entries
Stage-2 solution has 125 entries
The testing set has 5668 entries<jupyter_text>Since the testing set has more than 5000 entries and in order to align stage1 + stage 2 solution to the testing set, we first need to merge the test set and solution set. Let's first combine solution sets<jupyter_code>#combine solution sets
stacked_solution = solution.append(solution2)
stacked_solution.shape
merge_test_solution = pd.merge(test_df,stacked_solution,on='ID')
merge_test_solution.head()
solution_df_truncated = merge_test_solution.iloc[:,6:6+9]
solution_df_truncated.index = merge_test_solution.ID
print('size of combined & truncated solution is', solution_df_truncated.shape)
solution_df_truncated.head()
test_df_truncated = merge_test_solution.iloc[:,0:6]
print('size of truncated test set is', test_df_truncated.shape)
test_df_truncated.head()<jupyter_output>size of truncated test set is (493, 6)<jupyter_text>test_df_truncated and solution_df_truncated are variables to be used An alternate way<jupyter_code>IDs_to_drop = set(test_df.ID) - set(solution.ID)
test_df_stage1 = test_df.drop(labels=list(IDs_to_drop))
print('The solution has', len(solution), 'entries')
print('The testing set has', len(test_df), 'entries')
print('The truncated testing set has', len(test_df_stage1), 'entries')<jupyter_output>The solution has 368 entries
The testing set has 5668 entries
The truncated testing set has 368 entries<jupyter_text>Let's clean the text data<jupyter_code>import nltk
nltk.download('stopwords')
import string
import re #regular expressions
from nltk.corpus import stopwords
# note: to use stopwords, first do nltk.download() to download the various corpi in nltk.

# TO DO:
# we should remove the references from the text field!! e.g., if there are references like (1,2) or [5][6]
# use regex expression to clean these up!

def textClean(text):
    text = re.sub(r"[^A-Za-z0-9^,!.\/'+-=)(]", " ", text)
    text = text.lower().split()
    stops = set(stopwords.words("english"))
    text = [w for w in text if not w in stops]    
    text = " ".join(text)
    return(text)

def cleanup(text):
    text = textClean(text)
    text= text.translate(str.maketrans("","", string.punctuation))
    return text
allText_train = train_df['Text'].apply(cleanup)
allText_test = test_df_truncated['Text'].apply(cleanup)
# before
train_df['Text'][20][1500:2500]
# after
allText_train[20][1500:2500]<jupyter_output><empty_output><jupyter_text>Let's train and test our models<jupyter_code>from sklearn.model_selection import cross_val_predict
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss, accuracy_score
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import SVC
from sklearn.decomposition import TruncatedSVD
from sklearn.ensemble import RandomForestClassifier
import scikitplot.plotters as skplt
# put everything together into a function
from sklearn.metrics import roc_curve, auc

def plot_roc(y_true,y_pred,title='ROC title'):
    # calculate ROC in a one-vs-all manner
    # y_true needs to be binary (one-hot encoded)
    # y_pred needs to be probabilities
    fpr = dict()
    tpr = dict()
    roc_auc = dict()
    num_classes = y_true.shape[1]

    for i in range(num_classes):
        fpr[i], tpr[i], _ = roc_curve(y_true[:, i],y_pred[:, i])
        roc_auc[i] = auc(fpr[i], tpr[i])

    # Plot of a ROC curve for a specific class
    plt.figure()
    for i in range(num_classes):
        plt.plot(fpr[i], tpr[i], label='Class %i (area = %0.2f)' %(i+1,roc_auc[i]))
    plt.plot([0, 1], [0, 1], 'k--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title(title)
    plt.legend(loc="lower right")
    plt.show()

    return roc_auc
def evaluate_features(X, y, X_test, clf=None):
    if clf is None:
        clf = LogisticRegression()
    
    probas = cross_val_predict(clf, X, y, cv=StratifiedKFold(random_state=8), 
                              n_jobs=-1, method='predict_proba', verbose=2)
    clf.fit(X,y)
    y_pred = clf.predict_proba(X_test)
    pred_indices = np.argmax(probas, axis=1)
    classes = np.unique(y)
    preds = classes[pred_indices]
    print('Log loss: {}'.format(log_loss(y, probas)))
    print('Accuracy: {}'.format(accuracy_score(y, preds)))
    skplt.plot_confusion_matrix(y, preds)
    print('y_true shape is', y_true.shape)
    print('y_pred shape is', y_pred.shape)
    plot_roc(y_true, y_pred)
count_vectorizer = CountVectorizer(
    analyzer="word", tokenizer=nltk.word_tokenize,
    preprocessor=None, stop_words='english', max_features=None)
bag_of_words = count_vectorizer.fit_transform(allText_train)
len(count_vectorizer.get_feature_names())
svd = TruncatedSVD(n_components=25, n_iter=25, random_state=12)
truncated_train = svd.fit_transform(bag_of_words)
# one hot -> series
sol_size = len(solution_df_truncated)
desired_class = np.zeros((sol_size,1))
desired_ID = np.zeros((sol_size,1))

for i in range(sol_size):
    desired_class[i] = np.where(solution_df_truncated.iloc[i] == 1)[0][0]+1
    desired_ID[i] =  solution_df_truncated.index[i]
desired_class.shape


count_vectorizer = CountVectorizer(
    analyzer="word", tokenizer=nltk.word_tokenize,
    preprocessor=None, stop_words='english', max_features=None)

bag_of_words = count_vectorizer.fit_transform(allText_test)

len(count_vectorizer.get_feature_names())
svd = TruncatedSVD(n_components=25, n_iter=25, random_state=12)
truncated_test = svd.fit_transform(bag_of_words)

y_true = solution_df_truncated.values
X_train_all = truncated_train
y_train_all = train_df['Class'].values.ravel()
X_test = truncated_test
evaluate_features(truncated_train, train_df['Class'].values.ravel(), X_test)
evaluate_features(truncated_train, train_df['Class'].values.ravel(), X_test,
                  RandomForestClassifier(n_estimators=1000, max_depth=5, verbose=1))
count_vectorizer = TfidfVectorizer(
    analyzer="word", tokenizer=nltk.word_tokenize,
    preprocessor=None, stop_words='english', max_features=None)    

tfidf = count_vectorizer.fit_transform(allText_train)

len(count_vectorizer.get_feature_names())
svd = TruncatedSVD(n_components=25, n_iter=25, random_state=12)
truncated_train = svd.fit_transform(tfidf)
evaluate_features(truncated_train, train_df['Class'].values.ravel(), X_test)
evaluate_features(truncated_train, train_df['Class'].values.ravel(), X_test, 
                  RandomForestClassifier(n_estimators=1000, max_depth=5, verbose=1))<jupyter_output>C:\Users\shashi\Documents\Anaconda\lib\site-packages\sklearn\model_selection\_split.py:293: FutureWarning: Setting a random_state has no effect since shuffle is False. This will raise an error in 0.24. You should leave random_state to its default (None), or set shuffle=True.
  warnings.warn(
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
[Parallel(n_jobs=-1)]: Done   2 out of   5 | elapsed:   16.2s remaining:   24.4s
[Parallel(n_jobs=-1)]: Done   5 out of   5 | elapsed:   16.4s remaining:    0.0s
[Parallel(n_jobs=-1)]: Done   5 out of   5 | elapsed:   16.4s finished
[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[Parallel(n_jobs=1)]: Done 1000 out of 1000 | elapsed:   12.4s finished
[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[Parallel(n_jobs=1)]: Done 1000 out of 1000 | elapsed:    0.2s finished
C:\Users\shashi\Documents\Anaconda\lib\site-packages\sklearn\utils\deprecation.py:86: Future[...]<jupyter_text>Let's abbreviate the data through undersampling. Will that improve our confusion matrix? ROC Curve?<jupyter_code>train_df<jupyter_output><empty_output><jupyter_text>We have unbalanced classes.Let's slice the classes with higher distribution.Let's trim the classes 1, 2, 4, and, 7 to the size of classes 5 and 6 to have even distribution of data.<jupyter_code>train_df1 = train_df[train_df["Class"]== 1]
train_df1 = train_df1.head(250)

train_df2 = train_df[train_df["Class"]== 2]
train_df2 = train_df2.head(250)
train_df4 = train_df[train_df["Class"]== 4]
train_df4 = train_df4.head(250)

train_df5 = train_df[train_df["Class"]== 5]
train_df6 = train_df[train_df["Class"]== 6]
train_df7 = train_df[train_df["Class"]== 7]
train_df7 = train_df7.head(250)<jupyter_output><empty_output><jupyter_text>Let's concatenate all the sliced classes to form a new training dataframe.<jupyter_code>new_train_df = pd. concat([train_df1, train_df2, train_df4, train_df5, train_df6, train_df7], axis=0, ignore_index=True)
new_train_df
newText_train = new_train_df['Text'].apply(cleanup)
new_train_df['Text'][4][1500:2000]
newText_train[4][1500:2000]
merge_test_solution1 = merge_test_solution[merge_test_solution['class1']==1]
merge_test_solution2 = merge_test_solution[merge_test_solution['class2']==1]
merge_test_solution4 = merge_test_solution[merge_test_solution['class4']==1]
#merge_test_solution389 =pd. concat([merge_test_solution3, merge_test_solution8, merge_test_solution9])
#merge_test_solution389
#merge_test_solution389.shape
#IDs_to_keep = set(merge_test_solution.ID) - set(merge_test_solution3.ID)
merge_test_solution5 = merge_test_solution[merge_test_solution['class5']==1]
merge_test_solution6 = merge_test_solution[merge_test_solution['class6']==1]
merge_test_solution7 = merge_test_solution[merge_test_solution['class7']==1]
merge_test_solution_ab =pd. concat([merge_test_solution1, merge_test_solution2, merge_test_solution4, merge_test_solution5, merge_test_solution6, merge_test_solution7])
merge_test_solution_ab
merge_test_solution_ab = merge_test_solution_ab.drop(['class3', 'class8', 'class9'],axis=1)
merge_test_solution_ab
solution_df_truncated_ab = merge_test_solution_ab.iloc[:,6:6+6]
solution_df_truncated_ab.index = merge_test_solution_ab.ID
print('size of truncated & abreviated solution is', solution_df_truncated_ab.shape)
solution_df_truncated_ab
test_df_truncated_ab = merge_test_solution_ab.iloc[:,0:6]
print('size of truncated test set is', test_df_truncated_ab.shape)
test_df_truncated.head()
allText_test_ab = test_df_truncated_ab['Text'].apply(cleanup)
count_vectorizer = CountVectorizer(
    analyzer="word", tokenizer=nltk.word_tokenize,
    preprocessor=None, stop_words='english', max_features=None)
bag_of_words = count_vectorizer.fit_transform(newText_train)
len(count_vectorizer.get_feature_names())
svd = TruncatedSVD(n_components=25, n_iter=25, random_state=12)
truncated_train_ab = svd.fit_transform(bag_of_words)
count_vectorizer = CountVectorizer(
    analyzer="word", tokenizer=nltk.word_tokenize,
    preprocessor=None, stop_words='english', max_features=None)

bag_of_words = count_vectorizer.fit_transform(allText_test_ab)

len(count_vectorizer.get_feature_names())
svd = TruncatedSVD(n_components=25, n_iter=25, random_state=12)
truncated_test_ab = svd.fit_transform(bag_of_words)
X_train_ab = truncated_train_ab
y_true = solution_df_truncated_ab.values
y_train_ab = new_train_df['Class'].values.ravel()
X_test = truncated_test_ab
evaluate_features(truncated_train_ab, new_train_df['Class'].values.ravel(),X_test)
evaluate_features(truncated_train_ab, new_train_df['Class'].values.ravel(), X_test
                  RandomForestClassifier(n_estimators=1000, max_depth=5, verbose=1))
count_vectorizer = TfidfVectorizer(
    analyzer="word", tokenizer=nltk.word_tokenize,
    preprocessor=None, stop_words='english', max_features=None)    

tfidf = count_vectorizer.fit_transform(newText_train)

len(count_vectorizer.get_feature_names())
svd = TruncatedSVD(n_components=25, n_iter=25, random_state=12)
truncated_train_ab = svd.fit_transform(tfidf)
evaluate_features(truncated_train_ab, new_train_df['Class'].values.ravel(),X_test)
evaluate_features(truncated_train_ab, new_train_df['Class'].values.ravel(),X_test, 
                  RandomForestClassifier(n_estimators=1000, max_depth=5, verbose=1))<jupyter_output><empty_output><jupyter_text>Run our test data on our trained model<jupyter_code>evaluate_features(truncated_test, desired_class.ravel(),X_test)
evaluate_features(truncated_test, desired_class.ravel(),X_test, 
                  RandomForestClassifier(n_estimators=1000, max_depth=5, verbose=1))
count_vectorizer = TfidfVectorizer(
    analyzer="word", tokenizer=nltk.word_tokenize,
    preprocessor=None, stop_words='english', max_features=None)    

tfidf = count_vectorizer.fit_transform(allText_test)

len(count_vectorizer.get_feature_names())
svd = TruncatedSVD(n_components=25, n_iter=25, random_state=12)
truncated_test = svd.fit_transform(tfidf)
evaluate_features(truncated_test, desired_class.ravel(), X_test)
evaluate_features(truncated_test, desired_class.ravel(), X_test, 
                  RandomForestClassifier(n_estimators=1000, max_depth=5, verbose=1))<jupyter_output>C:\Users\shashi\Documents\Anaconda\lib\site-packages\sklearn\model_selection\_split.py:293: FutureWarning: Setting a random_state has no effect since shuffle is False. This will raise an error in 0.24. You should leave random_state to its default (None), or set shuffle=True.
  warnings.warn(
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
[Parallel(n_jobs=-1)]: Done   2 out of   5 | elapsed:    5.1s remaining:    7.7s
[Parallel(n_jobs=-1)]: Done   5 out of   5 | elapsed:    6.5s remaining:    0.0s
[Parallel(n_jobs=-1)]: Done   5 out of   5 | elapsed:    6.5s finished
[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[Parallel(n_jobs=1)]: Done 1000 out of 1000 | elapsed:    3.5s finished
[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[Parallel(n_jobs=1)]: Done 1000 out of 1000 | elapsed:    0.2s finished
C:\Users\shashi\Documents\Anaconda\lib\site-packages\sklearn\utils\deprecation.py:86: Future[...]
{'fraction_non_alphanumeric': 0.11779745277212253, 'fraction_numerical': 0.024951470100847497, 'mean_word_length': 6.937617437053739, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 75, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '390738', 'license': 'MIT', 'n_tokens_mistral': 7774, 'n_tokens_neox': 7241, 'n_words': 1549, 'path': 'Classical_machine_learning_algo.ipynb', 'repo_name': 'cr718c3ramire/PersonalizedMed_erdos_bootcamp2021team', 'size': '21121'}