Hugging Face's logo

MaartenGr
/
BERTopic_ArXiv

Text Classification
BERTopic
Model card Files Community
BERTopic_ArXiv / README.md
MaartenGr's picture
MaartenGr
Update README.md
43dd6fb
preview code
|
raw
history blame contribute delete
19.5 kB
metadata
tags:
  - bertopic
library_name: bertopic
pipeline_tag: text-classification

BERTopic_ArXiv

This is a BERTopic model. BERTopic is a flexible and modular topic modeling framework that allows for the generation of easily interpretable topics from large datasets.

This pre-trained model demonstrates the use of several representation models that can be used within BERTopic. This model was trained on ~30000 ArXiv abstracts with the following topic representation methods (bertopic.representation):

  • POS
  • KeyBERTInspired
  • MaximalMarginalRelevance
  • KeyBERT + MaximalMarginalRelevance
  • ChatGPT labels
  • ChatGPT summaries

An example of the default c-TF-IDF representations:

"multiaspect.png"

An example of labels generated by ChatGPT (gpt-3.5-turbo):

"multiaspect.png"

To generate these images, you can follow along with this tutorial: Open In Colab

Usage

To use this model, please install BERTopic:

pip install -U bertopic
pip install -U safetensors

You can use the model as follows:

from bertopic import BERTopic
topic_model = BERTopic.load("MaartenGr/BERTopic_ArXiv")

topic_model.get_topic_info()

To view all different topic representations (keywords, labels, summary, etc.) you can run the following:

>>> topic_model.get_topic(0, full=True)
{'Main': [['dialogue', 0.02704485163341523],
  ['dialog', 0.01677038224466311],
  ['response', 0.011692640237477233],
  ['responses', 0.01002788412923778],
  ['intent', 0.00990720856306287],
  ['oriented', 0.009217253131615378],
  ['slot', 0.009177118721490055],
  ['conversational', 0.009129311385144046],
  ['systems', 0.009101146153425574],
  ['conversation', 0.008845392252307181]],
 'POS': [['dialogue', 0.02704485163341523],
  ['dialog', 0.01677038224466311],
  ['response', 0.011692640237477233],
  ['responses', 0.01002788412923778],
  ['intent', 0.00990720856306287],
  ['slot', 0.009177118721490055],
  ['conversational', 0.009129311385144046],
  ['systems', 0.009101146153425574],
  ['conversation', 0.008845392252307181],
  ['user', 0.008753551043296965]],
 'KeyBERTInspired': [['task oriented dialogue', 0.6559894680976868],
  ['dialogue systems', 0.6249060034751892],
  ['oriented dialogue', 0.5788208246231079],
  ['dialog systems', 0.530449628829956],
  ['dialogue state', 0.5167528390884399],
  ['response generation', 0.5143576860427856],
  ['spoken language understanding', 0.46739083528518677],
  ['oriented dialog', 0.4600704610347748],
  ['dialog', 0.4534587264060974],
  ['dialogues', 0.44082391262054443]],
 'MMR': [['dialogue', 0.02704485163341523],
  ['dialog', 0.01677038224466311],
  ['response', 0.011692640237477233],
  ['responses', 0.01002788412923778],
  ['intent', 0.00990720856306287],
  ['oriented', 0.009217253131615378],
  ['slot', 0.009177118721490055],
  ['conversational', 0.009129311385144046],
  ['systems', 0.009101146153425574],
  ['conversation', 0.008845392252307181]],
 'KeyBERT + MMR': [['task oriented dialogue', 0.6559894680976868],
  ['dialogue systems', 0.6249060034751892],
  ['oriented dialogue', 0.5788208246231079],
  ['dialog systems', 0.530449628829956],
  ['dialogue state', 0.5167528390884399],
  ['response generation', 0.5143576860427856],
  ['spoken language understanding', 0.46739083528518677],
  ['oriented dialog', 0.4600704610347748],
  ['dialog', 0.4534587264060974],
  ['dialogues', 0.44082391262054443]],
 'OpenAI_Label': [['Challenges and Approaches in Developing Task-oriented Dialogue Systems',
   1]],
 'OpenAI_Summary': [['Task-oriented dialogue systems and their components, such as dialogue policy, natural language understanding, dialogue state tracking, response generation, and end-to-end training using neural networks. These components are crucial in assisting users to complete various activities such as booking tickets and restaurant reservations through spoken language understanding dialogue. The challenge lies in tracking dialogue states of multiple domains and obtaining annotations for training. Effective SLU is achieved by utilizing context from the prior dialogue history.',
   1]]}

Topic overview

  • Number of topics: 107
  • Number of training documents: 33189
Click here for an overview of all topics.
Topic ID Topic Keywords Topic Frequency Label
-1 language - models - model - data - based 20 -1_language_models_model_data
0 dialogue - dialog - response - responses - intent 14247 0_dialogue_dialog_response_responses
1 speech - asr - speech recognition - recognition - end 1833 1_speech_asr_speech recognition_recognition
2 tuning - tasks - prompt - models - language 1369 2_tuning_tasks_prompt_models
3 summarization - summaries - summary - abstractive - document 1109 3_summarization_summaries_summary_abstractive
4 question - answer - qa - answering - question answering 893 4_question_answer_qa_answering
5 sentiment - sentiment analysis - aspect - analysis - opinion 837 5_sentiment_sentiment analysis_aspect_analysis
6 clinical - medical - biomedical - notes - patient 691 6_clinical_medical_biomedical_notes
7 translation - nmt - machine translation - neural machine - neural machine translation 586 7_translation_nmt_machine translation_neural machine
8 generation - text generation - text - language generation - nlg 558 8_generation_text generation_text_language generation
9 hate - hate speech - offensive - speech - detection 484 9_hate_hate speech_offensive_speech
10 news - fake - fake news - stance - fact 455 10_news_fake_fake news_stance
11 relation - relation extraction - extraction - relations - entity 450 11_relation_relation extraction_extraction_relations
12 ner - named - named entity - entity - named entity recognition 376 12_ner_named_named entity_entity
13 parsing - parser - dependency - treebank - parsers 370 13_parsing_parser_dependency_treebank
14 event - temporal - events - event extraction - extraction 314 14_event_temporal_events_event extraction
15 emotion - emotions - multimodal - emotion recognition - emotional 300 15_emotion_emotions_multimodal_emotion recognition
16 word - embeddings - word embeddings - embedding - words 292 16_word_embeddings_word embeddings_embedding
17 explanations - explanation - rationales - rationale - interpretability 212 17_explanations_explanation_rationales_rationale
18 morphological - arabic - morphology - languages - inflection 204 18_morphological_arabic_morphology_languages
19 topic - topics - topic models - lda - topic modeling 200 19_topic_topics_topic models_lda
20 bias - gender - biases - gender bias - debiasing 195 20_bias_gender_biases_gender bias
21 law - frequency - zipf - words - length 185 21_law_frequency_zipf_words
22 legal - court - law - legal domain - case 182 22_legal_court_law_legal domain
23 adversarial - attacks - attack - adversarial examples - robustness 181 23_adversarial_attacks_attack_adversarial examples
24 commonsense - commonsense knowledge - reasoning - knowledge - commonsense reasoning 180 24_commonsense_commonsense knowledge_reasoning_knowledge
25 quantum - semantics - calculus - compositional - meaning 171 25_quantum_semantics_calculus_compositional
26 correction - error - error correction - grammatical - grammatical error 161 26_correction_error_error correction_grammatical
27 argument - arguments - argumentation - argumentative - mining 160 27_argument_arguments_argumentation_argumentative
28 sarcasm - humor - sarcastic - detection - humorous 157 28_sarcasm_humor_sarcastic_detection
29 coreference - resolution - coreference resolution - mentions - mention 156 29_coreference_resolution_coreference resolution_mentions
30 sense - word sense - wsd - word - disambiguation 153 30_sense_word sense_wsd_word
31 knowledge - knowledge graph - graph - link prediction - entities 149 31_knowledge_knowledge graph_graph_link prediction
32 parsing - semantic parsing - amr - semantic - parser 146 32_parsing_semantic parsing_amr_semantic
33 cross lingual - lingual - cross - transfer - languages 146 33_cross lingual_lingual_cross_transfer
34 mt - translation - qe - quality - machine translation 139 34_mt_translation_qe_quality
35 sql - text sql - queries - spider - schema 138 35_sql_text sql_queries_spider
36 classification - text classification - label - text - labels 136 36_classification_text classification_label_text
37 style - style transfer - transfer - text style - text style transfer 136 37_style_style transfer_transfer_text style
38 question - question generation - questions - answer - generation 129 38_question_question generation_questions_answer
39 authorship - authorship attribution - attribution - author - authors 127 39_authorship_authorship attribution_attribution_author
40 sentence - sentence embeddings - similarity - sts - sentence embedding 123 40_sentence_sentence embeddings_similarity_sts
41 code - identification - switching - cs - code switching 121 41_code_identification_switching_cs
42 story - stories - story generation - generation - storytelling 118 42_story_stories_story generation_generation
43 discourse - discourse relation - discourse relations - rst - discourse parsing 117 43_discourse_discourse relation_discourse relations_rst
44 code - programming - source code - code generation - programming languages 117 44_code_programming_source code_code generation
45 paraphrase - paraphrases - paraphrase generation - paraphrasing - generation 114 45_paraphrase_paraphrases_paraphrase generation_paraphrasing
46 agent - games - environment - instructions - agents 111 46_agent_games_environment_instructions
47 covid - covid 19 - 19 - tweets - pandemic 108 47_covid_covid 19_19_tweets
48 linking - entity linking - entity - el - entities 107 48_linking_entity linking_entity_el
49 poetry - poems - lyrics - poem - music 103 49_poetry_poems_lyrics_poem
50 image - captioning - captions - visual - caption 100 50_image_captioning_captions_visual
51 nli - entailment - inference - natural language inference - language inference 96 51_nli_entailment_inference_natural language inference
52 keyphrase - keyphrases - extraction - document - phrases 95 52_keyphrase_keyphrases_extraction_document
53 simplification - text simplification - ts - sentence - simplified 95 53_simplification_text simplification_ts_sentence
54 empathetic - emotion - emotional - empathy - emotions 95 54_empathetic_emotion_emotional_empathy
55 depression - mental - health - mental health - social media 93 55_depression_mental_health_mental health
56 segmentation - word segmentation - chinese - chinese word segmentation - chinese word 93 56_segmentation_word segmentation_chinese_chinese word segmentation
57 citation - scientific - papers - citations - scholarly 85 57_citation_scientific_papers_citations
58 agreement - syntactic - verb - grammatical - subject verb 85 58_agreement_syntactic_verb_grammatical
59 metaphor - literal - figurative - metaphors - idiomatic 83 59_metaphor_literal_figurative_metaphors
60 srl - semantic role - role labeling - semantic role labeling - role 82 60_srl_semantic role_role labeling_semantic role labeling
61 privacy - private - federated - privacy preserving - federated learning 82 61_privacy_private_federated_privacy preserving
62 change - semantic change - time - semantic - lexical semantic 82 62_change_semantic change_time_semantic
63 bilingual - lingual - cross lingual - cross - embeddings 80 63_bilingual_lingual_cross lingual_cross
64 political - media - news - bias - articles 77 64_political_media_news_bias
65 medical - qa - question - questions - clinical 75 65_medical_qa_question_questions
66 math - mathematical - math word - word problems - problems 73 66_math_mathematical_math word_word problems
67 financial - stock - market - price - news 69 67_financial_stock_market_price
68 table - tables - tabular - reasoning - qa 69 68_table_tables_tabular_reasoning
69 readability - complexity - assessment - features - reading 65 69_readability_complexity_assessment_features
70 layout - document - documents - document understanding - extraction 64 70_layout_document_documents_document understanding
71 brain - cognitive - reading - syntactic - language 62 71_brain_cognitive_reading_syntactic
72 sign - gloss - language - signed - language translation 61 72_sign_gloss_language_signed
73 vqa - visual - visual question - visual question answering - question 59 73_vqa_visual_visual question_visual question answering
74 biased - biases - spurious - nlp - debiasing 57 74_biased_biases_spurious_nlp
75 visual - dialogue - multimodal - image - dialog 55 75_visual_dialogue_multimodal_image
76 translation - machine translation - machine - smt - statistical 54 76_translation_machine translation_machine_smt
77 multimodal - visual - image - translation - machine translation 52 77_multimodal_visual_image_translation
78 geographic - location - geolocation - geo - locations 51 78_geographic_location_geolocation_geo
79 reasoning - prompting - llms - chain thought - chain 48 79_reasoning_prompting_llms_chain thought
80 essay - scoring - aes - essay scoring - essays 45 80_essay_scoring_aes_essay scoring
81 crisis - disaster - traffic - tweets - disasters 45 81_crisis_disaster_traffic_tweets
82 graph - text classification - text - gcn - classification 44 82_graph_text classification_text_gcn
83 annotation - tools - linguistic - resources - xml 43 83_annotation_tools_linguistic_resources
84 entity alignment - alignment - kgs - entity - ea 43 84_entity alignment_alignment_kgs_entity
85 personality - traits - personality traits - evaluative - text 42 85_personality_traits_personality traits_evaluative
86 ad - alzheimer - alzheimer disease - disease - speech 40 86_ad_alzheimer_alzheimer disease_disease
87 taxonomy - hypernymy - taxonomies - hypernym - hypernyms 39 87_taxonomy_hypernymy_taxonomies_hypernym
88 active learning - active - al - learning - uncertainty 37 88_active learning_active_al_learning
89 reviews - summaries - summarization - review - opinion 36 89_reviews_summaries_summarization_review
90 emoji - emojis - sentiment - message - anonymous 35 90_emoji_emojis_sentiment_message
91 table - table text - tables - table text generation - text generation 35 91_table_table text_tables_table text generation
92 domain - domain adaptation - adaptation - domains - source 35 92_domain_domain adaptation_adaptation_domains
93 alignment - word alignment - parallel - pairs - alignments 34 93_alignment_word alignment_parallel_pairs
94 indo - languages - indo european - names - family 34 94_indo_languages_indo european_names
95 patent - claim - claim generation - chemical - technical 32 95_patent_claim_claim generation_chemical
96 agents - emergent - communication - referential - games 32 96_agents_emergent_communication_referential
97 graph - amr - graph text - graphs - text generation 31 97_graph_amr_graph text_graphs
98 moral - ethical - norms - values - social 29 98_moral_ethical_norms_values
99 acronym - acronyms - abbreviations - abbreviation - disambiguation 27 99_acronym_acronyms_abbreviations_abbreviation
100 typing - entity typing - entity - type - types 27 100_typing_entity typing_entity_type
101 coherence - discourse - discourse coherence - coherence modeling - text 26 101_coherence_discourse_discourse coherence_coherence modeling
102 pos - taggers - tagging - tagger - pos tagging 25 102_pos_taggers_tagging_tagger
103 drug - social - social media - media - health 25 103_drug_social_social media_media
104 gender - translation - bias - gender bias - mt 24 104_gender_translation_bias_gender bias
105 job - resume - skills - skill - soft 21 105_job_resume_skills_skill

Training Procedure

The model was trained as follows:

from cuml.manifold import UMAP
from cuml.cluster import HDBSCAN
from bertopic import BERTopic
from sklearn.feature_extraction.text import CountVectorizer
from bertopic.representation import PartOfSpeech, KeyBERTInspired, MaximalMarginalRelevance, OpenAI

# Prepare sub-models
embedding_model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
umap_model = UMAP(n_components=5, n_neighbors=50, random_state=42, metric="cosine", verbose=True)
hdbscan_model = HDBSCAN(min_samples=20, gen_min_span_tree=True, prediction_data=False, min_cluster_size=20, verbose=True)
vectorizer_model = CountVectorizer(stop_words="english", ngram_range=(1, 3), min_df=5)

# Summarization with ChatGPT
summarization_prompt = """
I have a topic that is described by the following keywords: [KEYWORDS]
In this topic, the following documents are a small but representative subset of all documents in the topic:
[DOCUMENTS]

Based on the information above, please give a description of this topic in the following format:
topic: <description>
"""
summarization_model = OpenAI(model="gpt-3.5-turbo", chat=True, prompt=summarization_prompt, nr_docs=5, exponential_backoff=True, diversity=0.1)

# Representation models
representation_models = {
    "POS": PartOfSpeech("en_core_web_lg"),
    "KeyBERTInspired": KeyBERTInspired(),
    "MMR": MaximalMarginalRelevance(diversity=0.3),
    "KeyBERT + MMR": [KeyBERTInspired(), MaximalMarginalRelevance(diversity=0.3)],
    "OpenAI_Label": OpenAI(model="gpt-3.5-turbo", exponential_backoff=True, chat=True, diversity=0.1),
    "OpenAI_Summary": [KeyBERTInspired(), summarization_model],
}

# Fit BERTopic
topic_model= BERTopic(
        embedding_model=embedding_model,
        umap_model=umap_model,
        hdbscan_model=hdbscan_model,
        vectorizer_model=vectorizer_model,
        representation_model=representation_models,
        verbose=True
).fit(docs)

Training hyperparameters

  • calculate_probabilities: False
  • language: None
  • low_memory: False
  • min_topic_size: 10
  • n_gram_range: (1, 1)
  • nr_topics: None
  • seed_topic_list: None
  • top_n_words: 10
  • verbose: True

Framework versions

  • Numpy: 1.22.4
  • HDBSCAN: 0.8.29
  • UMAP: 0.5.3
  • Pandas: 1.5.3
  • Scikit-Learn: 1.2.2
  • Sentence-transformers: 2.2.2
  • Transformers: 4.29.2
  • Numba: 0.56.4
  • Plotly: 5.13.1
  • Python: 3.10.11