I am currently selecting features of products by using LDA to group 6000 keywords of product into topics. Here is the sample of my dataset after being organized into list of keywords for each product id.I consider each id as a document and each keywords as the word in a document for the case of LDA.It didn't work out as I expected as each topic have many identical keywords with different weight. I removed 100 most common keywords but there are still some identical keywords in the topics. Here is the sample output:How can I deal with the identical keywords in my topics? and also how to deal with the 100 most common keywords I removed?Here is the graph showing the frequency of the words in all documentsEach word only presents once in each document, but may present in different documents. I updated a new graph of the frequency of the wordsThank you so much. Any suggestion is appreciated.
Need help with LDA for selecting features
machine learning;bigdata;topic model;lda