I provide an offline library of Music to my users.My goal is to understand what my users are looking for, which means translate raw user searches to: Music Artists, Songs, Albums and then add music to the company library.What are the suggested clustering algorithms to group common short sentences into a single entity. Example:Taylor Swift Shake it OffTaylor Swift Shaek it Off Shake it off Twylor Swift Shake it OffI tried this example and works fine for a specific number of clusters (K) where K < N | K <= N. But since searches are unpredictable need to find a way to automate the number of clusters: my goal is to cluster 2 or more similar items and let alone single searches in independent clusters, example:Cluster 1:Taylor Swift Shake it OffTaylor Swift Shaek it Off Shake it off Twylor Swift Shake it OffCluster 2AvicciAvicii IbizaAvicci Electro HouseCluster 3Juan GabrielCluster 4AdeleCluster 5Britney SpearsBritney Spears VMA 2016
Music corpus sentence level clustering
clustering;nlp;text mining
I would suggest hierarchical clustering. It's unsupervised, and you don't need to predefine number of clusters. How it works (for the bottom up version) is each sentence (or object) is initialized as its own cluster. At each iteration of the algorithm, the two clusters of the smallest intra-cluster distance are joined, all the way until there's a 'root' where everything is one cluster. The result of this is a big dendrogram, and you can cut the dendrogram at whatever point you want to define clusters. Or you can just inspect it. Here's a nice numerical example of the alg in action.It should detect the same types of clusters you're already finding, and you wouldn't need to redefine a distance metric. And, I anticipate you'll get the same amount of class separation you seem to already be getting; in fact, the nice thing about the dendrogram from hierarchical clustering is that it illustrates class separation nicely. it's 'hclust' in R, not sure for Python.