I have a set of documents as given in the example below.doc1 = {'Science': 0.7, 'History': 0.05, 'Politics': 0.15, 'Sports': 0.1}doc2 = {'Science': 0.3, 'History': 0.5, 'Politics': 0.1, 'Sports': 0.1}I want to cluster the documents and identify the most prominent document within the cluster.e.g, cluster 1 includes = {doc1, doc4, doc5. doc8} and I want to get the most prominent document that represents this cluster (e.g., doc8). (or to identify the main theme of the cluster)Please let me know a suitable approach to achieve this :)
Cluster documents and identify the prominent document in the cluster?
machine learning;data mining;clustering
A very simple approach would be to find some kind of centroid for each cluster (e.g. averaging the distributions of the documents belonging to each cluster respectively) and then calculating the cosine distance of each document within the cluster from the corresponding centroid. The document with the shorter distance will be the closest to the centroid, hence the most representative.Continuing from the previous example:import pandas as pdimport numpy as npfrom sklearn.metrics import pairwise_distancesfrom scipy.spatial.distance import cosinefrom sklearn.cluster import DBSCANfrom sklearn.preprocessing import StandardScaler# Initialize some documentsdoc1 = {'Science':0.8, 'History':0.05, 'Politics':0.15, 'Sports':0.1}doc2 = {'News':0.2, 'Art':0.8, 'Politics':0.1, 'Sports':0.1}doc3 = {'Science':0.8, 'History':0.1, 'Politics':0.05, 'News':0.1}doc4 = {'Science':0.1, 'Weather':0.2, 'Art':0.7, 'Sports':0.1}collection = [doc1, doc2, doc3, doc4]df = pd.DataFrame(collection)# Fill missing values with zerosdf.fillna(0, inplace=True)# Get Feature Vectorsfeature_matrix = df.as_matrix()# Fit DBSCANdb = DBSCAN(min_samples=1, metric='precomputed').fit(pairwise_distances(feature_matrix, metric='cosine'))labels = db.labels_n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)print('Estimated number of clusters: %d' % n_clusters_)# Find the representativesrepresentatives = {}for label in set(labels):    # Find indices of documents belonging to the same cluster    ind = np.argwhere(labels==label).reshape(-1,)    # Select these specific documetns    cluster_samples = feature_matrix[ind,:]    # Calculate their centroid as an average    centroid = np.average(cluster_samples, axis=0)    # Find the distance of each document from the centroid    distances = [cosine(sample_doc, centroid) for sample_doc in cluster_samples]    # Keep the document closest to the centroid as the representative    representatives[label] = cluster_samples[np.argsort(distances),:][0]for label, doc in representatives.iteritems():    print(Label : %d -- Representative : %s % (label, str(doc)))