<commit_before># About PubMed Topic Modeler

The goal for the PubMed Topic Modeler is to query PubMed, use various natural language processing methods (NLP) to cleanse the text, and generate topics that are contained within the texts queried.

The idea is that a user can use this tool to determine the different domains of research in a given area of research. Specifically, many Alzheimer's disease articles may deal with bioinformatics and genomics. You might be interested in potential subtopics that are an intersection of these two domains. It is my hope that the topic modeler can provide a better, finer grained, method to categorize abstracts. If not, it's still an intersting little exercise in using PubMed abstracts as a source of information for different NLP and machine learning techniques.

Specifically, these scripts model text using latent semantic analysis (LSA). There's a really good detailed explanation of [LSA on Wikipedia](https://en.wikipedia.org/wiki/Latent_semantic_analysis).

This is current a work in progress, and is part of the Insight Computer Science mini-specialization program.

## General Schema of the project:

![Alt Text](https://github.com/mjtat/PubMed-Topic-Modeler/blob/master/images/schema.png?raw=true)


**REQUIRED MODULES**
You will need to download `nltk`, `gensim`, and `pandas` to run these scripts. To do that, you can type the following in terminal / console:

`pip install nltk` or `conda install nltk` (if you have Anaconda)

`pip install gensim` or `conda install gensim` (if you have Anaconda)

`pip install gensim` or `conda install gensim` (if you have Anaconda)

`pip install pandas` (pandas is already included in Anaconda)

* pubmed_query.py
* preprocess.py
* topic_modeling.py
* figure_plotting.py
* execute.py

pubmed_query, preprocess, and topic_modeling are placed within the scripts folder. It is not necessary for you to run them unless you choose to. You can simply download the repository, edit the path argument in the execute.py file (if necessary), then run the execute.py script.

1. *pubmed_query.py* makes use of the PubMed open-source API, and retrieves abstracts. It also retrieves other information, such as date of publication, and Pubmed IDs (PMID).

* The query process can be invoked with the following syntax: ```pubmed_query(email, terms, author, number_of_articles).return_abstracts()``` It will return a dictionary with all abstracts.

2. *preprocess.py* primarily does data cleaning. It removes low information words, tokenizes, then lemmatizes each word.

* The cleaning process can be invoked with the following syntax: ``` cleaned_text(dict_here).clean() ```. It will return a dictionary of cleaned text, which can be queried as follows: ``` dictionary[key][3] ```

3. *topic_modeling.py* makes use of the `gensim` package to covert text into a large corpora, vectorize the text using term-frequency inverse-document frequency (TF-IDF), and extract topics using latent semantic analysis (also known as latent semantic indexing).


4. *execute.py* is a wrapper that runs all three functions sequentially, such that you do not have to inspect the scripts individually. `execute.py` can also be run on the command line.

**The modeling process can be invoked with the following syntax, specifically from the execute.py file:** ```topic, scores = run_modeler(email, keyword args, number of abstracts to return, number of topics to model)```. Note that email and keywords are strings, and number of abstracts / topics are ints.
<commit_msg>new readme<commit_after># About PubMed Topic Modeler

The goal for the PubMed Topic Modeler is to query PubMed, use various natural language processing methods (NLP) to cleanse the text, and generate topics that are contained within the texts queried.

The idea is that a user can use this tool to determine the different domains of research in a given area of research. Specifically, many Alzheimer's disease articles may deal with bioinformatics and genomics. You might be interested in potential subtopics that are an intersection of these two domains. It is my hope that the topic modeler can provide a better, finer grained, method to categorize abstracts. If not, it's still an intersting little exercise in using PubMed abstracts as a source of information for different NLP and machine learning techniques.

Specifically, these scripts model text using latent semantic analysis (LSA). There's a really good detailed explanation of [LSA on Wikipedia](https://en.wikipedia.org/wiki/Latent_semantic_analysis).

This is current a work in progress, and is part of the Insight Computer Science mini-specialization program.

## General Schema of the project:

![Alt Text](https://github.com/mjtat/PubMed-Topic-Modeler/blob/master/images/schema.png?raw=true)


**REQUIRED MODULES**
You will need to download `nltk`, `gensim`, and `pandas` to run these scripts. To do that, you can type the following in terminal / console:

`pip install nltk` or `conda install nltk` (if you have Anaconda)

`pip install gensim` or `conda install gensim` (if you have Anaconda)

`pip install gensim` or `conda install gensim` (if you have Anaconda)

`pip install pandas` (pandas is already included in Anaconda)

* pubmed_query.py
* preprocess.py
* topic_modeling.py
* figure_plotting.py
* execute.py

pubmed_query, preprocess, and topic_modeling are placed within the folder. It is not necessary for you to run them unless you choose to. All you need to run is execute.py script.

1. *pubmed_query.py* makes use of the PubMed open-source API, and retrieves abstracts. It also retrieves other information, such as date of publication, and Pubmed IDs (PMID).

* The query process can be invoked with the following syntax: ```pubmed_query(email, terms, author, number_of_articles).return_abstracts()``` It will return a dictionary with all abstracts.

2. *preprocess.py* primarily does data cleaning. It removes low information words, tokenizes, then lemmatizes each word.

* The cleaning process can be invoked with the following syntax: ``` cleaned_text(dict_here).clean() ```. It will return a dictionary of cleaned text, which can be queried as follows: ``` dictionary[key][3] ```

3. *topic_modeling.py* makes use of the `gensim` package to covert text into a large corpora, vectorize the text using term-frequency inverse-document frequency (TF-IDF), and extract topics using latent semantic analysis (also known as latent semantic indexing).


4. *execute.py* is a wrapper that runs all three functions sequentially, such that you do not have to inspect the scripts individually. `execute.py` can also be run on the command line.

**The modeling process can be invoked with the following syntax, specifically from the execute.py file:** ```topic, scores = run_modeler(email, keyword args, number of abstracts to return, number of topics to model)```. Note that email and keywords are strings, and number of abstracts / topics are ints.

{'fraction_non_alphanumeric': 0.06689684937419077, 'fraction_numerical': 0.0014386419220256078, 'mean_word_length': 4.906542056074766, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2779979', 'n_tokens_mistral': 1860, 'n_tokens_neox': 1803, 'n_words': 967}