<jupyter_start><jupyter_text>**Question 1: Consensus clustering** **a) What is consensus clustering? Mention 2 applications of consensus clustering.** *   Consensus Clustering (Also known as clustering aggregation or clustering ensembles)  is an algorithmic technique in which a number of different (input) clusterings have been obtained for a particular dataset and the objective is to find a single (consensus) clustering which is a better fit as compared to the existing clusterings. It is thus the problem of reconsiling clustering information about the same dataset coming from different sources or from different runs of the same algorithm.It yields a stable and robust final clustering that is in aggrement with the multiple clusterings. *   Let us have a given set of n data points as X = {x1, x2,.....,xn}, and a set of k clusterings C = {C1, C2,....Ck} of the data points in X. Each clustering Ci is a mapping from X to {1,...,TCi}, TCi is the number of points in cluster Ci. The problem of consensus clustering is to find a new cluster C* of the data X such that it best summarizes the clustering ensemble C. **Applications of consensus clustering are as follows:**1. Consensus clustering is used in scenario where there is a very high dimensional data. It thus helps in reducing the number of features from the high-dimensional dataset by pre-filtering the features and then applying the consensus algorithm to select the best features appropriately. 2. Since it uses the clustering results from various clusterings, it helps to mine the pattern that is statistically stable across the various runs of the clustering algorithms even if the sample varies slighly in each run. In other words, it discovers the robustness of the discovered clusters to sampling variability.  **b) Define consensus matrix in your own words with the help of the formula.** Consensus matrix is a matrix based on the cooccurrence of vertices in clusters of the input partitions.It is an (N × N ) matrix that stores, for each pair of data items, the ratio of the number of times for which both the items got clustered together to the number of times both the items were selected in the sampling, for all the runs. It is then averaged over the connectivity matrices of every perturbed dataset to create the consensus matrix.It provides for a similarity measure that can be used in conjunction with an agglomerative hierarchical tree construction algorithm to yield a dendogram of item adjacencies.The mathematical representation of the consensus matrix is as follows:Let us assume that D(1) , D(2) , . . . , D(H) be the list of H perturbed datasets obtained by resampling the originaldataset D. Also, let M(h) denote the (N × N ) connectivity matrix corresponding to datasetD(h). The entries of this matrix are defined as follows:M(h)(i,j)  = 1, if items i and j belong to the same cluster.           = 0, if they belong to different clusters.Let I(h) be the (N × N ) indicator matrix such that its (i, j)-th entry is equal to 1 if both items i and j are present in the dataset D (h) , and 0 otherwise.The consensus matrix M can then be defined as a properly normalized sum of theconnectivity matrices of all the perturbed datasets {D (h) : h = 1, 2, . . . , H }:Mat(i, j) = (M (h) (i, j) for all h) / (I(h) (i, j) for all h) **c) Mention pros and cons of consensus clustering.** **Pros of Consensus Clustering:***   It helps to identify the statistically stable clusters ingoring the slight data variability.*   It acts as a good measure to prescribe the number of clusters to be taken for various clustering schemes that require taking prior number of clusters.*   It helps in identifying the outliers and the representatives clusters for further clustering analysis of a high-dimensional dataset.**Cons of Consensus Clustering:***   It involves a good amount of computational overhead as compared to the traditional clustering schemes because of its consensus technique.*   Since it is distance-based, it requires efficient feature engineering in case of a high-dimensional dataset. **d) Fill in the gaps for the pseudocode of the consensus clustering. Pseudocode provided in the assignment.** The blanks are filled up as follows*  **belonging to all the items of K***   M is the set of connectivity maatrices which is initially empty.*   D is the generic dataset and D(h) is the dataset obtained by resampling D *   **K** clusters*   **Mat(K)** which is the consensus matrix for a specified k*   **D , Mat(K)***    **Mat(K)** **e) Perform (k-means) consensus clustering of samples for the given gene expression data . Take minimum clusters as 2, maximum clusters as 6, resampling proportion as 80% and number of iterations as 10. Find:**<jupyter_code># Importing the necessary packages
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
np.random.seed(0)<jupyter_output><empty_output><jupyter_text>Load the gene expression data<jupyter_code># Mounting the Google Drive for accessing the Cleveland dataset
from google.colab import drive
drive.mount('/content/drive')

dataUrl = "/content/drive/My Drive/Colab Notebooks/Data Files/allData.csv"
geneExpData = pd.read_csv(dataUrl)
print(geneExpData.head(5))
geneExpData = geneExpData.drop(['Unnamed: 0'], axis=1)
# Converting the dataset into 2D array
geneExpData = geneExpData.to_numpy()
!cp "/content/drive/My Drive/Colab Notebooks/consensusClustering.py" .
# Importing the necessary packages for consensus clustering
import consensusClustering
from consensusClustering import ConsensusCluster
# Performing consensus k-means clustering with the given parameters
cc = ConsensusCluster(KMeans,2,6,10,resample_proportion=0.8)
# Fitting the data
c = cc.fit(geneExpData)<jupyter_output><empty_output><jupyter_text>**i) Best number of clusters**<jupyter_code>print("The best number of clusters is " + str(c[1]))<jupyter_output>The best number of clusters is 2<jupyter_text>**ii) Change in area under CDF**<jupyter_code>print("Change in area under CDF is" + str(c[0]))<jupyter_output>Change in area under CDF is[ 9.91589355e+00 -1.52611628e-03  3.77181738e-03]<jupyter_text>**iii)Best cluster from the consensus matrix for each sample**<jupyter_code>cc.predict()<jupyter_output><empty_output><jupyter_text>**Question 2. Read about bi-clustering and explain it in your own words. Give an example of an algorithm based on bi-clustering.** Bi-clustering is a method in machine learning that allows selecting the area of interest in the Big data by selecting similar rows and columns which have similar features and cluster them together. The algorithm produces biclusters between rows and columns which produce the same behavior together. For example, finding the genes that are co-regulated in subset conditions  subgroups of genes and subgroups of conditions, where the genes exhibit highly correlated activities for every condition or we want to discover which of our patients have similar diseases.Bi-clustering is one of the NP's complete problems; it is not an easy algorithm to solve as it is computationally expensive so there are several algorithms used as **Robust Biclustering Algorithm (RoBA**) which made up of three steps parts. The first is data conditioning  which clean data to get rid of the noise and to solve the problem of missing values. The second step is to decompose the data matrix A into elementary matrix . Lastly extracting any type of biclusters defined by the user as b-iclustering with constant values. **Question 3. NMF Clustering** **a) Explain Non-negative matrix factorization (NMF) in your own words.**  NMF (Nonnegative Matrix Factorization)  is a matrix factorization method where we constrain the matrices to be nonnegative. If I have a matrix m*n which does not have any negative values in it and we want to factorize this matrix into two non-negative matrices W and H where W is N*K and H is M*K so to do that we must solve the minimization problem, we need to minimize the distance between X and W-H arg: min(w, H) cost (X-WH)subject to W, H > 0 where we want to minimize divergence between X and WH, but this problem is NP-Hard problem as it is convex in W or H but not in both so in this problem we cannot find the global minimum however finding a good local minimum can be satisfying and update done by Kim & Park, 2007 where the additionally enforce sparsity in W and H matrices where allow set of certain features specific to each other. NMF is distinguished from PCA and VQ by its use of nonnegativity constraints. **b) Mention pros and cons of NMF** **Pros of NMF:***  It represent data with non-negative features well ( it has non-negative constraint).*  NMF doesn’t use the distance based measures.*  Joint feature selection and clustering.**Cons of NMF:***   Can only detect linearly separable clusters.*   It is computationally taxing.*   Different NMF algorithms find different factorizations and same NMF algorithm with different parameters may lead to different factorization*   One requirement of NMF is that the values of data should be non-negative, while in many real-world problems the non-negative constraints cannot be satisfied. **c) Mention an application of NMF** *   Applications in image processing, text processing and mining, music transcription, video analysis, bioinformatics, chemistry. Image mining applications include data compression, finding similar images, cluster images.*   In Text Mining application document-term matrix is constructed with the weights of various terms from a set of documents. This matrix is factorized into a term-feature and a feature-document matrix. The features are derived from the contents of the documents, and the feature-document matrix describes data clusters of related documents. Text mining applications include data compression,finding similar terms,finding similar documents,clustering documents,topic detection and tracking. **d) Use the nimfa package for NMF clustering on gene expression data. Use the parameters (10 ranks, 50 maximum iterations and 25 runs) to compute the following:****i) From the average connectivity matrix across multiple runs compute consensus matrix**<jupyter_code>!pip install nimfa
# Importing relevant packages
import numpy as np
import nimfa
from matplotlib import pyplot as plt
import matplotlib.gridspec as gridspec
import scipy.cluster.hierarchy as ch
import urllib
dataUrlAll = "/content/drive/My Drive/Colab Notebooks/Data Files/allData.csv"
allData = pd.read_csv(dataUrlAll)
print(allData.head(5))
allData = allData.drop(['Unnamed: 0'], axis=1)
# Converting the dataset into 2D array
allData = allData.to_numpy()
# Performing the NMF clustering
nmf_clustering = nimfa.Nmf(allData,rank=10, max_iter=50, n_run=25)
# Fitting the data
n = nmf_clustering()<jupyter_output><empty_output><jupyter_text>**ii) Produce a heatmap with a dendogram from the clustering results you obtained**<jupyter_code>fig = plt.figure(figsize=(15, 10))
heatmapGS = gridspec.GridSpec(1, 2)

C = 1 - n.fit.consensus()
Y = ch.linkage(C, method='ward')

denAX = fig.add_subplot(heatmapGS[0,0])
denD = ch.dendrogram(Y, orientation='right', link_color_func=lambda k: 'black')


heatmapAX = fig.add_subplot(heatmapGS[0,1])
D = C[denD['leaves'], :][:, denD['leaves']]
axi = heatmapAX.imshow(D, interpolation='nearest', aspect='equal', origin='lower', cmap='RdBu') 

cb = fig.colorbar(axi) 
cb.set_label('Distance', fontsize=20)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06465067778936393, 'fraction_numerical': 0.009210983663538407, 'mean_word_length': 5.047819232790331, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '632862', 'license': 'MIT', 'n_tokens_mistral': 3127, 'n_tokens_neox': 2891, 'n_words': 1596, 'path': 'Exercise_8.ipynb', 'repo_name': 'itsroops/Biomedical_Data_Science_and_AI', 'size': '11508'}