<commit_before>---
layout: single
excerpt: "Visualizing LDA Topic Correlation Network"
title:  "Using R to detect communities of correlated topics"
categories: [text mining]
tags: [visualization, lda, community detection, social network]
---

![](/images/unnamed-chunk-5-1.png)

Creating a topic network
------------------------

For [Project Mosaic](http://projectmosaic.uncc.edu), I'm researching UNCC publications in social science and computing & informatics by analyzing the abstract text and the co-authorship social network.

For text mining, I'm running topic modeling ([Latent Dirichlet Allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) or LDA for short) on five years of peer-reviewed publication abstracts to identify key research themes by university researchers. (If you're not familiar with LDA, a good start is [<NAME>'s "Probabilistic Topic Models"](https://www.cs.princeton.edu/~blei/papers/Blei2012.pdf).)

One problem I came across was: how to measure the relationships (correlations) between topics? In particular, I want to creating a network visualization that connects similar topics and helps users easier navigate a large collection of topics (in this case 100 topics).

In this tutorial, I accomplish this by combining code from two awesome resources:

*   [<NAME>'s Topic Modeling Repo](https://github.com/trinker/topicmodels_learning) (see "Network of the Word Distributions Over Topics") 
*   [<NAME>'s Network Visualization Tutorial](http://kateto.net/network-visualization).

Data preparation
----------------

Our first step is to load our topic matrices that are outputs of LDA. There are two outputs to LDA: a word-topic matrix and a document-topic matrix. For this tutorial, I uploaded previously saved LDA results that were saved as flat (csv) files.

As an alternative to loading flat files, you can use the output of the `topicmodels` package `lda` function to create any word-topic and document-topic matrices. Take the output of your `lda` function and run the `posterior` function on the output.

``` r<commit_msg>Revise topic network post<commit_after>---
layout: single
excerpt: "Visualizing LDA Topic Correlation Network"
title:  "Using R to Detect Communities of Correlated Topics"
categories: [text mining]
tags: [visualization, lda, community detection, social network]
---

![](/images/unnamed-chunk-5-1.png)

Creating a topic network
------------------------

For [Project Mosaic](http://projectmosaic.uncc.edu), I'm researching UNCC publications in social science and computing & informatics by analyzing the abstract text and the co-authorship social network.

For text mining, I'm running topic modeling ([Latent Dirichlet Allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) or LDA for short) on five years of peer-reviewed publication abstracts to identify key research themes by university researchers. (If you're not familiar with LDA, a good start is [<NAME>'s "Probabilistic Topic Models"](https://www.cs.princeton.edu/~blei/papers/Blei2012.pdf).)

One problem I came across was: how to measure the relationships (correlations) between topics? In particular, I want to creating a network visualization that connects similar topics and helps users easier navigate a large collection of topics (in this case 100 topics).

In this tutorial, I accomplish this by combining code from two awesome resources:

*   [<NAME>'s Topic Modeling Repo](https://github.com/trinker/topicmodels_learning) (see "Network of the Word Distributions Over Topics") 
*   [<NAME>'s Network Visualization Tutorial](http://kateto.net/network-visualization).

Data preparation
----------------

Our first step is to load our topic matrices that are outputs of LDA. There are two outputs to LDA: a word-topic matrix and a document-topic matrix. For this tutorial, I uploaded previously saved LDA results that were saved as flat (csv) files.

As an alternative to loading flat files, you can use the output of the `topicmodels` package `lda` function to create any word-topic and document-topic matrices. Take the output of your `lda` function and run the `posterior` function on the output.

``` r
{'fraction_non_alphanumeric': 0.09606147934678194, 'fraction_numerical': 0.004322766570605188, 'mean_word_length': 4.5092592592592595, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '910838', 'n_tokens_mistral': 1123, 'n_tokens_neox': 1043, 'n_words': 514}