Q: How they utilize LDA and Gibbs sampling to evaluate ISWC and WWW publications?
Text: Introduction
Today, Internet is one of the widest available media worldwide. It has essentially become a huge hit of data that has the potential to serve many information centric applications in our life. Recommendation system takes an essential part of many internet services and online applications, including applications like social-networking and recommendation of products (films, music, articles,..i.e.). Recommendation techniques have been used by the most known companies such as Amazon, Netflix and eBay to recommend releated items or products by estimating the probable preferences of customers. These techniques are profitable to both service provider and user. According to pervious works, two popular approaches for building recommendation systems can be categorized as content-based (CB), collaborative filtering (CF).
Content-based (CB) recommending is adopted for recommendation systems model widely, which takes advantage of the property of items to create features and characteristics to coordinate user profiles. It can be relied on the properties of the items that each user likes to discover what else the user may like. One major issue of CB filtering method is that the recommendation system is required to gain an understanding of user preferences for some sorts of items and deploy these for other sorts of items.
Nevertheless CF has two widely known problems which are sparsity and cold start (CS). In the rating matrix, The percentage of elements which get values is small. Even it is possible that CF considers only a few rating for popular items. For instance, upon a considerable Netflix rating dataset which is provided for Netflix Prize competition, there are about 100 milion ratings for about 18,000 movies that are given by 480,000 users. The percentage of rating matrix elements which are received ratings is 1. With a sparse ranking matrix it is very challenging topic to make an effective recommendation, depending on estimation of the relationship between items and users. CS problem is another widely known issue for CF approach, which can occur on new users or items. In terms of achieving an effective recommendation, CF approach requires either ratings on an item or a large number of ratings from a user.
Recently, researchers proposed various methods based on probabilistic topic modeling methods BIBREF0. LDA is a generative probabilistic model broadly used in the information retrieval field. Researchers have used topic modeling methods based on LDA for building recommendation systems in various subjects, including app recommendation BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, event recommendationBIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, hashtag recommendation BIBREF16, BIBREF17, BIBREF18, BIBREF19, BIBREF20, BIBREF21, BIBREF22, BIBREF23, BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF28, BIBREF29, BIBREF30, BIBREF31, BIBREF32, BIBREF33, BIBREF34, BIBREF35, BIBREF36, social networks and media BIBREF37, BIBREF38, BIBREF6, BIBREF9, BIBREF39, BIBREF40, BIBREF41. In this paper, We present a taxonomy of recommendation systems applications based on topic modeling (LDA) of the recent research and evaluate ISWC and WWW publications in computer science between 2013 to 2017 from DBLP dataset.
Natural language Processing and LDA topic model
Topic models are a powerful and practical tool for analyzing huge text documents in Natural language processing. Topic models can automatically cluster words into topics and discover relationship between documents from a dataset. For example; we can assume a three-topic model of a News Dataset, including "sport", "money" and “politic”. The most common words in the sport topic (Topic 1) might be “gym”, “football”, and “tennis” and in addition, for politic topic (Topic 2) might be “senator”, “president”, and “election”; while the money topic (Topic 3) can be made up of words such as “dollar”, “currency”, and “euro”. Figure 1; show a simple example for understanding a topic discovery from group of words. LDA is a popular technique to semantic analysis in topic modeling and text mining. LDA can apply in a diversity of text-information to evaluate topic trends over time and analyze large numbers of documents.
In-process detail for LDA, defined a corpus (text) as $D={d_{1},d_{2},d_{3},....,d_{M}}$ where M is number of text documents and $d_{M}$ is a number of text documents in the corpus. A document is a series of N words denoted by $W={w_{1},w_{2},w_{3},....,w_{N}}$ , where $w_{N}$ is the $Nth$ word in the sequence of text document. In addition, z is a latent variable representing the hidden topic associated with each showed word. The generative procedure for LDA, formally defined as:
For topic index $k\in \left\lbrace 1,2,...,k\right\rbrace $ i. Selected a word distribution $\beta _{k}~Dir\left(\eta \right)$
For text document $ D\in \left\lbrace 1,2,...,d\right\rbrace $ i.Selected a topic distribution $\_{k}~Dir\left(\eta \right)$ ii.For $n\in \left\lbrace 1,....,N_{d}\right\rbrace $ word a.Selected a topic assignment $z_{dn}~Mult\left(\theta _{d}\right)$ b.Selected a word $w_{dn}~Mult\left(\beta _{z_{d,n}}\right)$
Mult() is a multinomial distribution, and $Dir()$ is a Dirichlet distribution which is a prior distribution of Mult(),$\alpha $ and $\theta $ are hyperparameters.
Natural language Processing and LDA topic model ::: Gibbs sampling and Learning LDA
As previously mentioned, Topic modeling can find a collection of distributions over words for each topic and the relationship of topics with each document. To perform approximate inference and learning LDA, there are many inference methods for LDA topic model such as Gibbs sampling, collapsed Variational Bayes, Expectation Maximization. Gibbs sampling is a popular technique because of its simplicity and low latency. However, for large numbers of topics, Gibbs sampling can become unwieldy. In this paper, we use Gibbs Sampling in our experiment in section 5.
Topic model based on recommendation systems: Recent research
In this section, we considered six recommendation systems based on LDA which includes: scientific paper recommendation BIBREF42, BIBREF43, BIBREF44, BIBREF45, BIBREF46, BIBREF47, BIBREF48, BIBREF49, music and video recommendation BIBREF50, BIBREF51, BIBREF52, BIBREF53, BIBREF54, BIBREF55, BIBREF56, BIBREF57, BIBREF58, BIBREF59, BIBREF60, location recommendationBIBREF61, BIBREF37, BIBREF62, BIBREF63, BIBREF64, BIBREF65, BIBREF66, travel and tour recommendationBIBREF67, BIBREF38, BIBREF68, BIBREF69, app recommendationBIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, friend recommendation BIBREF41, BIBREF70, BIBREF71, as shown in Figure 2.
Topic model based on recommendation systems: Recent research ::: Topic model based on scientific paper recommendation
In recent years a considerable amount of research has addressed the task of defining models and systems for scientific papers recommendation; this trend has emerged as a natural consequence of the increasing growth of the number of scientific publications. For example, Youn and et al, proposed an approach to scientific articles'recommendation of user's interests based on a topic modeling framework. The authors, used a LDA model in order to extract the topics of the followees'tweets (followed Twitterers) and the paper titlesBIBREF49. They apply the Twitter-LDA algorithm simultaneously on the followees' tweets and the paper titles with the number of topics set to 200, they utilized the intersection of topics found in both paper titles and followees's tweets. Each followee of a user is ranked as follows:
where $T_f$denotes all tweets by a followee,
${Topics}_p,\ $denotes the set of topics defining the titles of scientific articles,
${Topics}_f,\ $denotes the set of topics defining the tweets of a followee and $n\left(t,T_f\right)\ $the number of times a particular topic `t' from within occurs among the tweets of a followee. Based on the ranking scores of all followees of a particular user,and obtained top-k researchers followed by a target user. For evaluation approach, considered DBLP database as a large academic bibliographic network.
Also, some researchers, introduced a combined model based on traditional collaborative filtering and topic modeling and designed a novel algorithm to scientific articles recommendation for users from an online community, called CTR model. They considered LDA to initialize the CTR model, Infact they combined the matrix factorization and the LDA model, and is shown their approach better than the recommendations based on matrix factorization. For evaluation and test, Used a large dataset from a bibliography sharing service (CiteULike) BIBREF47. Table 1, shown some impressive work based on LDA for paper recommendation.
Topic model based on recommendation systems: Recent research ::: Topic model based on music and video recommendation
Video and music recommendation has become an essential way for helping people explore the video world and discover the ones that may be of interest to them. Recently, analyze user interests and a good video or music recommendation in internet society is a big challenge. Hariri and et al. proposed a combined approach based on content and collaborative filtering methods from the sequence of songs listened to generate music recommendation. They applied a LDA model to reduce the dimensionality of the feature and obtain the hidden relationships between songs and tags. They collected 218,261 distinct songs from "Art of the Mix" website for evaluation their approach BIBREF60. Yan and et al, focused on the efficiency of users' information content on the online social network and provided a solution as a personalized video recommendation with considering users' cross-network social and content data. They applied a topic model based on LDA for each user, that user as document and user's hashtags as word, with considering user's information from Twitter BIBREF52. They derived Twitter user topic distributions and observed user-video interactions on YouTube, and presented a solution for user preference transfer:
$u^{T_t}_i$, Twitter user tweet topic distribution matrix $u^{T_t}$ = $\lbrace $$u^{T_t}_i$ ;...;$\ u^{T_t}_{|U|}$ $\rbrace $, $u^{T_f}_i,$ Twitter user social topic distribution matrix $u^{T_f}$= $\lbrace $$u^{T_f}_i$ ;...;$\ u^{T_f}_{|U|}$ $\rbrace $, $W_1$and $W_2,\ $with observations of the overlapped user's Twitter and YouTube behaviors, $\mathrm {\Omega }$ as the collection of all the observed user-video pairs, $\eta $ is a trade-off parameter to balance the contribution of different types of user's behaviors on Twitter, where $v_j$ is the $j^{th}$row of $V$ , $L_j$ is the $j^{th}$column of L, $L_{jj}$ is the entry located in the $j^{th}$column and $j^{th}$ row of L. Based on, updated $v_j$ and $W_1$, $W_2$ iteratively until convergence or maximum iteration. The update rules are:
where $\gamma $ denotes the learning rate.
With the derived transfer matrices $W_1$, $W_2$ and video latent factor representations V, given a test user $u_i$ with his/her tweeting activity, friend collection, and the corresponding Twitter topical distributions $u^{T_t}_i$ , $u^{f_i}_i$ , we can estimate $u_i$ preferences on YouTube videos as:
For test and experiment, utilized their approach on YouTube-Twitter dataset that include 9,253,729 tweeting behaviors and 1,097,982 video-related behavior and showed combining auxiliary network information and utilizing a cross-network Collaborative can lead to generating novel recommendations and increasing satisfaction for users.
In addition, some researchers; proposed an approach based on Collaborative Filter (CF) Algorithm and utilized the application of session variety and temporal context. They applied a LDA model to temporal properties extraction of sessions and that considering sessions as documents and songs as words. For evaluation of this approach, they used Last.fm dataset(log) that includes 19,150,868 entries from 992 users. Results showed that the approach with using temporal information can increase the accuracy of music recommendations BIBREF54.
In addition, other researchers used a dynamic framework based on four aspects of user's preference (collaborative aspect, content aspect, popularity aspect, and randomization aspect) for movie recommendation. The authors applied a linear combination model to generate the final recommendation list BIBREF51. Hu et al, proposed a novel topic modeling to audio retrieval, called GaussianLDA. In general, in this approach it was assumed that each audio document includes various latent topics and each topic considered as a Gaussian distribution. They prepared 1214 audio documents (length: between 0.82 s to 1 min), that each audio document is related with a category in different subject that includes: bell, river, rain, laugh, dog, gun and so on. Their results showed that the GaussianLDA model significantly outperform the standard LDA topic modelBIBREF56. Table 2, shown some impressive work based on LDA for music and video recommendation.
Topic model based on recommendation systems: Recent research ::: Topic model based on location recommendation
Recommendation systems based on location can suggest a set of places that users may be interested in, based on their history and behavior analysis. LDA can also be used for location recommendation. Kurashima et al, proposed a novel topic model for recommending new locations to visit, called Geo Topic Model. This model can predict user's interest and the user's spatial area based on features of visited locations. They used Tabelog-based (tablelog.com) and Flickr-based real-location log data for evaluation of their approach. They found that this model can discover latent topics related to art, great views, nature, atmosphere, and construction and other from logs of visited places BIBREF63. Table 3, shown some impressive work based on LDA for location recommendation.
Liu and et al, investigated the POI recommendation issue in LBSNs by mining textual information and proposed a 'Topic and Location-aware' probabilistic matrix factorization (TL-PMF) method for Point-of-Interest recommendation to discover personalized recommendations from favorite places BIBREF37. The distribution over the observed ratings as well as the textual information is:
Where, ${r}_{ij}$ be the rating of user ${u}_{i}$ for ${POI}_{{c}_{j}}{C}_{j}$,
${U}_{i}$and ${C}_{i}$ are the user and POI latent feature space vector respectively,
$\mathcal {N}\left(\mathrm {\textrm {-}}\right|,{\sigma }^{2})$ is a Gaussian distribution with mean and variance ${\sigma }^{2}$,
${I}_{ij}$ is the indicator function,
Function $f(U_i,C_{j,{TL}_{ij}})$ is to approximate the rating of user $u_i$ for ${POI}_{cj}$.
they analyzed the topic characteristics from POIs across various geographical areas. The experiments were conducted on a large real-world LBSN dataset; they analyzed the topic characteristics from POIs across various geographical areas.
Topic model based on recommendation systems: Recent research ::: Topic model based on friend recommendation
Friend recommendation is a popular method to help users to make new friends and discover interesting information. Friend recommendation is a relative challenging issue contrasted with group or item recommendations in online social networksBIBREF70, BIBREF41, BIBREF71. To address this challenge issue, a recent work in [79] proposed a friend recommendation based on LDA, which contains two stages: first step, they applied tag-user information to produce a possible friend list and then they created a topic model to demonstrate the relationship between user's friend making behaviour and image features. They applied experiments on the Flickr as a standard dataset and showed that their recommends friends more quickly than traditional methods.
Topic model based on recommendation systems: Recent research ::: Topic model Based on travel and tour recommendation
Definitely, recommendation systems can have a significant impact to build a smart travel recommendation. Many different techniques have recently been developed to support travel recommendation based on different kinds of data. For example, in BIBREF67, the authors proposed a novel generative probabilistic model named socoLDA with heterogeneous social influence to better capture users's travel interests. They introduced the framework of travel-package recommendation named socoTraveler, which applys socoLDA to show a user's travel interests in topic space and find similar users to produce recommendations with user-based collaborative filtering. In addition, in BIBREF68, the authors proposed a framework to suggest top-k tours with highest marks for a user by using the photos shared by other users in an online social network. Table 4, shown some impressive work based on LDA for travel and tour recommendation.
Topic model based on recommendation systems: Recent research ::: Topic model based on app recommendation
Currently, a wide range of recommendation approaches have been proposed and applied to recommend mobile apps. For example, In BIBREF3, proposed a novel probabilistic model, named Goal-oriented Exploratory Model (GEM), to combine the identification of exploratory behavior and mobile app recommendations into a unified framework. The authors, employed the idea of LDA to design a topic model to cluster items into goals and identify the personal distribution of goals for each user and developed an effective and efficient algorithm, which integrates Expectation-Maximization (EM) algorithm with collapsed Gibbs sampling for model learning. They collected a mobile app dataset from Qihoo 360 Mobile Assistant, an open mobile app platform in China for Android users. Lin et al, investigated the cold-start issue with using the social information for App recommendation in Twitter and used a LDA model to discovering latent group from "Twitter personalities" to recommendations discovery. This approach is based on a simple "averaging" method where the probability of how likely the target user will like the app is the expectation of how the Twitter-followers like the app. Given a set of Twitter-followers $T$, the probability that user $u$ likes app a is defined as follows:
where T(a) is the set of possible Twitter-followers following app a, in which assume that:
(i) Twitter-followers are examined once at a time to make a decision about whether an app is liked or disliked.
(ii) when the Twitter follower is known, the judgement does not depend on the app any more, i.e.,
(iii) the fact that given a user and an app, there is no judgement involved, i.e.,
(iv) the fact that an app has a given Twitter follower is independent from the user, i.e.,
Equation (DISPLAY_FORM24) is then reduced to the estimation of two quantities:
1. The probability that user u likes app a given that app a has Twitter-follower t, i.e., $p(+|t,\ u)$, and
2. The probability of considering Twitter-follower t given app a, i.e., $p(t|a)$
$p(+|t,\ u)$ is straightforward to estimate as it can be rewritten as:
where $p(+,\ t|u)$ and $p(-,\ t|u)$ are derived from LDA, which is the probability that Twitter-follower t occurs in an app that is liked (or disliked) by user u.
For test and experiment, they considered Apple's iTunes App Store and Twitter as dataset. Experimental results show, their approach significantly better than other state-of-the-art recommendation techniques BIBREF74.
In BIBREF4, the authors proposed an allocation-based probabilistic mechanism that considers multiple user-app factors to help users with app recommendations. This framework that can capture geographical influences on usage behaviors and effectively model user mobility patterns, which in turn affect app usage patterns. They used Gibbs sampling to approximately estimate and infer the parameters of LDA. The authors measure the similarity of two location blocks by the similarity of app usage pattern, which is calculated by Pearson's correlation similarity BIBREF75:
Specifically,
defined $c({\mu }_i,{\alpha }_j,l_x)$=1, if user ${\mu }_i$has launched app ${\alpha }_j$at location block $l_x$.
Otherwise, $c({\mu }_i,{\alpha }_j,l_x)$=0. Therefore, the number of users who have launched ${\alpha }_j$ at location block $l_x$ is $I_{aj,l_x}=\sum _{{\mu }_i\in U}{c({\mu }_i,{\alpha }_j,l_x)}$.
where $\overline{I_{l_x}}$ denotes average mobile app influences at location block $l_x$.
To determine whether a location block $l_z$ belongs to a location $L$,
They define the app usage pattern coefficient as $r_{l_x}$for geographical block $l_x$. The initial collaborative filter coefficient is the average mobile app influence. To decide whether location block $l_z$ belongs to geographical region $L$. Also, they calculate its collaborative filter coefficient as follows :
where ${s}_{L}$ is the number of location blocks in geographical region $L$. If the value of $r_{l_z}$$>$$\ r_{th}$, where $r_{th}$ is a predefined threshold value, location block $l_z$will be included in $L$.
Experiment: Semantic Discovery and researchers behavior analysis ::: Dataset
We extracted ISWC and WWW conferences publications from DBLP website by only considering conferences for which data was available for years 2013-2017. In total, It should be noted that in these experiments, we considered abstracts and titles from each article. In this paper, we used MALLET (http://mallet.cs.umass.edu/) to implement the inference and obtain the topic models. In addition, our full dataset is available at https://github.com/JeloH/Dataset_DBLP. The most important goal of this experiment is discover the trends of the topics and find relationship between LDA topics and paper features and generate trust tags.
Experiment: Semantic Discovery and researchers behavior analysis ::: Parameter Settings
In this paper, all experiments were carried out on a machine running Windows 7 with CoreI3 and 4 GB memory. We learn a LDA model with 100 topics; $\alpha =0.01$, $\beta = 0.01$ and using Gibbs sampling as a parameter estimation. Related words for a topic are quite intuitive and comprehensive in the sense of supplying a semantic short of a specific research field.
Experiment: Semantic Discovery and researchers behavior analysis ::: Semantic analysis and generate tags
In this section, we provide the results and discovered topics of 100-topics for ISWC and WWW conferences.
According to Table 6 , the following observations can be made:
In ISWC conference, Topic 25 sounds considerably more generic and is consistent with 'Concept Representation' in general, and marked by representation, structure, investigate, detection, values, and concept. Also, we can see that from 20 generated words in Topic 20, some words are very related to each other in means such as documents, scientific, networks, web, health, kg and we found that this topic covers papers that propose models in 'Documents Analysis in health research'.
In WWW conference, Topic 7, this is our question; the word of 'ad'! Is for 'Advertisement' or 'Ad Hoc Network'? As we can see, the word 'ad' can be related to 'Advertisement' or also to 'ad hoc network'. To answer this question, it is very easy to see that topic 7 reveals social, modeling, ranking, question, ad, browser. If only we consider the words 'browser', news, social, we can predict that this topic can be related to 'Advertisement' and this topic covers papers that propose methods in 'Question Answering and Social Media'.
Discussion, Open Issues and Future Directions
In this study, we focused on the LDA approaches to recommendation systems and given the importance of research, we have studied recent impressive articles on this subject and presented a taxonomy of recommendation systems based on LDA of the recent research. we evaluated ISWC and WWW conferences articles from DBLP website and used the Gibbs sampling algorithm as an evaluation parameter. We succeeded in discovering the relationship between LDA topics and paper features and also obtained the researchers' interest in research field. According to our studies, some issues require further research, which can be very effective and attractive for the future.
Discussion, Open Issues and Future Directions ::: Topic modeling methods and traditional methods in recommendation systems
There are differences between recommendation systems based on LDA and traditional collaborative filtering (CF), we discus about the issue of 'Cold-start', 'latent user interest' and Sparsity in the field of recommender systems. It should be noted that to overcome the major weaknesses of CF-based recommendation systems, many models have been proposed, such as BIBREF47.
Recommendation systems based on LDA in cold-start, the cold start problem occurs when a new item or user has just logged into that system; it is difficult to find similar ones because there is not enough information. LDA can be effective and useful to deal with cold-start in recommendation systems. There are approaches based on LDA to deal with this problem that some of the methods are combined with CF methods, for example, Lin et al, investigated the cold-start issue with use the social information for app recommendation in Twitter and used a LDA model to discovering latent group from 'Twitter personalities' to recommendations discovery and shown that their approach overcomes the difficulty of cold-start app recommendation. Also Some researchers investigated the cold-start problem in tag recommendation, for example; In BIBREF60 , presented a system recommendation based on LDA for Cold-Start in Music Recommendation and showed that their approach can be useful in handling the cold start problem where a new song hasn't occurred in the training data. Other researchers also analyze the cold problem in the Video recommendation, for example; In BIBREF52 introduced a unified YouTube video recommendation solution via cross-network collaboration and LDA to address the typical cold-start and data sparsity problems in recommender systems and show that this approach can be effective in terms of precision and improving the diversity of the recommended videos.
Recommendation Systems based on LDA for 'Sparsity problem', the data sparsity challenge happens when the ratio is too small to supply enough information for effective predictions in CF systems, and hence the access matrix is very sparse. Recommendation systems based on LDA with ratings data can provide significant advantage and addition can be useful for exploratory data analysis and dimensionality reduction in huge content-text. This dimensionality reduction can also help to alleviate the sparseness problem which is inherent to many traditional collaborative-filtering systems. There are approaches based on LDA to deal with this Scarcity problem such as BIBREF76
Recommendation Systems based on LDA for 'user latent interests', latent interest refers to long-term interest in a specific topic, in fact; A latent interest can be viewed as one specific characteristic of the users and the items who have this latent interest will prefer the items with this characteristic. Finding a community with a latent interest in another could help in recommending interesting new communities for a user. However, for CF systems, it is hard to identify the user latent interests, since the only information available is the user interaction information with the system. While topic models can be utilized to simulate the user latent interests and showed the way of extracting these interests from the latent Dirichlet allocation (LDA) model by the Gibbs sampling method in our experiment. We should note that, in terms of items, a latent interest can be viewed as one specific characteristic of the items and the users who have this latent interest will prefer the items with this characteristic. Fortunately, as a simulation tool, the topic model (e.g., LDA) can be utilized to learn the meaning, significance, characteristics and attributes of items in a data-driven, i.e., from given rating records, possibly without further content or prior knowledge of these itemsBIBREF77.
Conclusion
In this paper, we presented a taxonomy of recommendation systems and applications based on LDA of the recent research, including app, travel, friend, location, scientific paper, and music recommendation. Furthermore, we applied LDA algorithm and Gibbs sampling on ISWC and WWW conference's publications from 2013-2017. Generally, recommendation systems can be an impressive interface between online users and websites in the Internet communities. Our study suggest that NLP methods based on LDA can discover hidden aspects to better understanding of the behaviors of the people to build smart recommendation systems in online communities.
discover the trends of the topics and find relationship between LDA topics and paper features and generate trust tags,  learn a LDA model with 100 topics; $\alpha =0.01$, $\beta = 0.01$ and using Gibbs sampling as a parameter estimation