Introducing BERTopic Integration with the Hugging Face Hub
Maarten Grootendorst, Daniel van Strien
We are thrilled to announce a significant update to the BERTopic Python library, expanding its capabilities and further streamlining the workflow for topic modelling enthusiasts and practitioners. BERTopic now supports pushing and pulling trained topic models directly to and from the Hugging Face Hub. This new integration opens up exciting possibilities for leveraging the power of BERTopic in production use cases with ease.What is Topic Modelling?Topic modelling is a method that can help uncover hidden themes or "topics" within a group of documents. By analyzing the words in the documents, we can find patterns and connections that reveal these underlying topics. For example, a document about machine learning is more likely to use words like "gradient" and "embedding" compared to a document about baking bread. Each document usually covers multiple topics in different proportions. By examining the word statistics, we can identify clusters of related words that represent these topics. This allows us to analyze a set of documents and determine the topics they discuss, as well as the balance of topics within each document. More recently, new approaches to topic modelling have moved beyond using words to using more rich representations such as those offered through Transformer based models. What is BERTopic?BERTopic is a state-of-the-art Python library that simplifies the topic modelling process using various embedding techniques and c-TF-IDF to create dense clusters allowing for easily interpretable topics whilst keeping important words in the topic descriptions.An overview of the BERTopic libraryWhilst BERTopic is easy to get started with, it supports a range of advanced approaches to topic modelling including guided, supervised, semi-supervised and manual topic modelling. More recently BERTopic has added support for multi-modal topic models. BERTopic also have a rich set of tools for producing visualizations. BERTopic provides a powerful tool for users to uncover significant topics within text collections, thereby gaining valuable insights. With BERTopic, users can analyze customer reviews, explore research papers, or categorize news articles with ease, making it an essential tool for anyone looking to extract meaningful information from their text data.BERTopic Model Management with the Hugging Face HubWith the latest integration, BERTopic users can seamlessly push and pull their trained topic models to and from the Hugging Face Hub. This integration marks a significant milestone in simplifying the deployment and management of BERTopic models across different environments.The process of training and pushing a BERTopic model to the Hub can be done in a few linesfrom bertopic import BERTopictopic_model = BERTopic("english")topics, probs = topic_model.fit_transform(docs)topic_model.push_to_hf_hub('davanstrien/transformers_issues_topics')You can then load this model in two lines and use it to predict against new data.from bertopic import BERTopictopic_model = BERTopic.load("davanstrien/transformers_issues_topics")By leveraging the power of the Hugging Face Hub, BERTopic users can effortlessly share, version, and collaborate on their topic models. The Hub acts as a central repository, allowing users to store and organize their models, making it easier to deploy models in production, share them with colleagues, or even showcase them to the broader NLP community. You can use the libraries filter on the hub to find BERTopic models. Once you have found a BERTopic model you are interested in you can use the Hub inference widget to try out the model and see if it might be a good fit for your use case.Once you have a trained topic model, you can push it to the Hugging Face Hub in one line. Pushing your model to the Hub will automatically create an initial model card for your model, including an overview of the topics created. Below you can see an example of the topics resulting from a model trained on ArXiv data. Click here for an overview of all topics.Topic IDTopic KeywordsTopic FrequencyLabel-1language - models - model - data - based20-1_language_models_model_data0dialogue - dialog - response - responses - intent142470_dialogue_dialog_response_responses1speech - asr - speech recognition - recognition - end18331_speech_asr_speech recognition_recognition2tuning - tasks - prompt - models - language13692_tuning_tasks_prompt_models3summarization - summaries - summary - abstractive - document11093_summarization_summaries_summary_abstractive4question - answer - qa - answering - question answering8934_question_answer_qa_answering5sentiment - sentiment analysis - aspect - analysis - opinion8375_sentiment_sentiment analysis_aspect_analysis6clinical - medical - biomedical - notes - patient6916_clinical_medical_biomedical_notes7translation - nmt - machine translation - neural machine - neural machine translation5867_translation_nmt_machine translation_neural machine8generation - text generation - text - language generation - nlg5588_generation_text generation_text_language generation9hate - hate speech - offensive - speech - detection4849_hate_hate speech_offensive_speech10news - fake - fake news - stance - fact45510_news_fake_fake news_stance11relation - relation extraction - extraction - relations - entity45011_relation_relation extraction_extraction_relations12ner - named - named entity - entity - named entity recognition37612_ner_named_named entity_entity13parsing - parser - dependency - treebank - parsers37013_parsing_parser_dependency_treebank14event - temporal - events - event extraction - extraction31414_event_temporal_events_event extraction15emotion - emotions - multimodal - emotion recognition - emotional30015_emotion_emotions_multimodal_emotion recognition16word - embeddings - word embeddings - embedding - words29216_word_embeddings_word embeddings_embedding17explanations - explanation - rationales - rationale - interpretability21217_explanations_explanation_rationales_rationale18morphological - arabic - morphology - languages - inflection20418_morphological_arabic_morphology_languages19topic - topics - topic models - lda - topic modeling20019_topic_topics_topic models_lda20bias - gender - biases - gender bias - debiasing19520_bias_gender_biases_gender bias21law - frequency - zipf - words - length18521_law_frequency_zipf_words22legal - court - law - legal domain - case18222_legal_court_law_legal domain23adversarial - attacks - attack - adversarial examples - robustness18123_adversarial_attacks_attack_adversarial examples24commonsense - commonsense knowledge - reasoning - knowledge - commonsense reasoning18024_commonsense_commonsense knowledge_reasoning_knowledge25quantum - semantics - calculus - compositional - meaning17125_quantum_semantics_calculus_compositional26correction - error - error correction - grammatical - grammatical error16126_correction_error_error correction_grammatical27argument - arguments - argumentation - argumentative - mining16027_argument_arguments_argumentation_argumentative28sarcasm - humor - sarcastic - detection - humorous15728_sarcasm_humor_sarcastic_detection29coreference - resolution - coreference resolution - mentions - mention15629_coreference_resolution_coreference resolution_mentions30sense - word sense - wsd - word - disambiguation15330_sense_word sense_wsd_word31knowledge - knowledge graph - graph - link prediction - entities14931_knowledge_knowledge graph_graph_link prediction32parsing - semantic parsing - amr - semantic - parser14632_parsing_semantic parsing_amr_semantic33cross lingual - lingual - cross - transfer - languages14633_cross lingual_lingual_cross_transfer34mt - translation - qe - quality - machine translation13934_mt_translation_qe_quality35sql - text sql - queries - spider - schema13835_sql_text sql_queries_spider36classification - text classification - label - text - labels13636_classification_text classification_label_text37style - style transfer - transfer - text style - text style transfer13637_style_style transfer_transfer_text style38question - question generation - questions - answer - generation12938_question_question generation_questions_answer39authorship - authorship attribution - attribution - author - authors12739_authorship_authorship attribution_attribution_author40sentence - sentence embeddings - similarity - sts - sentence embedding12340_sentence_sentence embeddings_similarity_sts41code - identification - switching - cs - code switching12141_code_identification_switching_cs42story - stories - story generation - generation - storytelling11842_story_stories_story generation_generation43discourse - discourse relation - discourse relations - rst - discourse parsing11743_discourse_discourse relation_discourse relations_rst44code - programming - source code - code generation - programming languages11744_code_programming_source code_code generation45paraphrase - paraphrases - paraphrase generation - paraphrasing - generation11445_paraphrase_paraphrases_paraphrase generation_paraphrasing46agent - games - environment - instructions - agents11146_agent_games_environment_instructions47covid - covid 19 - 19 - tweets - pandemic10847_covid_covid 19_19_tweets48linking - entity linking - entity - el - entities10748_linking_entity linking_entity_el49poetry - poems - lyrics - poem - music10349_poetry_poems_lyrics_poem50image - captioning - captions - visual - caption10050_image_captioning_captions_visual51nli - entailment - inference - natural language inference - language inference9651_nli_entailment_inference_natural language inference52keyphrase - keyphrases - extraction - document - phrases9552_keyphrase_keyphrases_extraction_document53simplification - text simplification - ts - sentence - simplified9553_simplification_text simplification_ts_sentence54empathetic - emotion - emotional - empathy - emotions9554_empathetic_emotion_emotional_empathy55depression - mental - health - mental health - social media9355_depression_mental_health_mental health56segmentation - word segmentation - chinese - chinese word segmentation - chinese word9356_segmentation_word segmentation_chinese_chinese word segmentation57citation - scientific - papers - citations - scholarly8557_citation_scientific_papers_citations58agreement - syntactic - verb - grammatical - subject verb8558_agreement_syntactic_verb_grammatical59metaphor - literal - figurative - metaphors - idiomatic8359_metaphor_literal_figurative_metaphors60srl - semantic role - role labeling - semantic role labeling - role8260_srl_semantic role_role labeling_semantic role labeling61privacy - private - federated - privacy preserving - federated learning8261_privacy_private_federated_privacy preserving62change - semantic change - time - semantic - lexical semantic8262_change_semantic change_time_semantic63bilingual - lingual - cross lingual - cross - embeddings8063_bilingual_lingual_cross lingual_cross64political - media - news - bias - articles7764_political_media_news_bias65medical - qa - question - questions - clinical7565_medical_qa_question_questions66math - mathematical - math word - word problems - problems7366_math_mathematical_math word_word problems67financial - stock - market - price - news6967_financial_stock_market_price68table - tables - tabular - reasoning - qa6968_table_tables_tabular_reasoning69readability - complexity - assessment - features - reading6569_readability_complexity_assessment_features70layout - document - documents - document understanding - extraction6470_layout_document_documents_document understanding71brain - cognitive - reading - syntactic - language6271_brain_cognitive_reading_syntactic72sign - gloss - language - signed - language translation6172_sign_gloss_language_signed73vqa - visual - visual question - visual question answering - question5973_vqa_visual_visual question_visual question answering74biased - biases - spurious - nlp - debiasing5774_biased_biases_spurious_nlp75visual - dialogue - multimodal - image - dialog5575_visual_dialogue_multimodal_image76translation - machine translation - machine - smt - statistical5476_translation_machine translation_machine_smt77multimodal - visual - image - translation - machine translation5277_multimodal_visual_image_translation78geographic - location - geolocation - geo - locations5178_geographic_location_geolocation_geo79reasoning - prompting - llms - chain thought - chain4879_reasoning_prompting_llms_chain thought80essay - scoring - aes - essay scoring - essays4580_essay_scoring_aes_essay scoring81crisis - disaster - traffic - tweets - disasters4581_crisis_disaster_traffic_tweets82graph - text classification - text - gcn - classification4482_graph_text classification_text_gcn83annotation - tools - linguistic - resources - xml4383_annotation_tools_linguistic_resources84entity alignment - alignment - kgs - entity - ea4384_entity alignment_alignment_kgs_entity85personality - traits - personality traits - evaluative - text4285_personality_traits_personality traits_evaluative86ad - alzheimer - alzheimer disease - disease - speech4086_ad_alzheimer_alzheimer disease_disease87taxonomy - hypernymy - taxonomies - hypernym - hypernyms3987_taxonomy_hypernymy_taxonomies_hypernym88active learning - active - al - learning - uncertainty3788_active learning_active_al_learning89reviews - summaries - summarization - review - opinion3689_reviews_summaries_summarization_review90emoji - emojis - sentiment - message - anonymous3590_emoji_emojis_sentiment_message91table - table text - tables - table text generation - text generation3591_table_table text_tables_table text generation92domain - domain adaptation - adaptation - domains - source3592_domain_domain adaptation_adaptation_domains93alignment - word alignment - parallel - pairs - alignments3493_alignment_word alignment_parallel_pairs94indo - languages - indo european - names - family3494_indo_languages_indo european_names95patent - claim - claim generation - chemical - technical3295_patent_claim_claim generation_chemical96agents - emergent - communication - referential - games3296_agents_emergent_communication_referential97graph - amr - graph text - graphs - text generation3197_graph_amr_graph text_graphs98moral - ethical - norms - values - social2998_moral_ethical_norms_values99acronym - acronyms - abbreviations - abbreviation - disambiguation2799_acronym_acronyms_abbreviations_abbreviation100typing - entity typing - entity - type - types27100_typing_entity typing_entity_type101coherence - discourse - discourse coherence - coherence modeling - text26101_coherence_discourse_discourse coherence_coherence modeling102pos - taggers - tagging - tagger - pos tagging25102_pos_taggers_tagging_tagger103drug - social - social media - media - health25103_drug_social_social media_media104gender - translation - bias - gender bias - mt24104_gender_translation_bias_gender bias105job - resume - skills - skill - soft21105_job_resume_skills_skillDue to the improved saving procedure, training on large datasets generates small model sizes. In the example below, a BERTopic model was trained on 100,000 documents, resulting in a ~50MB model keeping all of the original’s model functionality. For inference, the model can be further reduced to only ~3MB!The benefits of this integration are particularly notable for production use cases. Users can now effortlessly deploy BERTopic models into their existing applications or systems, ensuring seamless integration within their data pipelines. This streamlined workflow enables faster iteration and efficient model updates and ensures consistency across different environments.safetensors: Ensuring Secure Model ManagementIn addition to the Hugging Face Hub integration, BERTopic now supports serialization using the safetensors library. Safetensors is a new simple format for storing tensors safely (instead of pickle), which is still fast (zero-copy). We’re excited to see more and more libraries leveraging safetensors for safe serialization. You can read more about a recent audit of the library in this blog post.An example of using BERTopic to explore RLHF datasetsTo illustrate some of the power of BERTopic let's look at an example of how it can be used to monitor changes in topics in datasets used to train chat models. The last year has seen several datasets for Reinforcement Learning with Human Feedback released. One of these datasets is the OpenAssistant Conversations dataset. This dataset was produced via a worldwide crowd-sourcing effort involving over 13,500 volunteers. Whilst this dataset already has some scores for toxicity, quality, humour etc., we may want to get a better understanding of what types of conversations are represented in this dataset. BERTopic offers one way of getting a better understanding of the topics in this dataset. In this case, we train a model on the English assistant responses part of the datasets. Resulting in a topic model with 75 topics. BERTopic gives us various ways of visualizing a dataset. We can see the top 8 topics and their associated words below. We can see that the second most frequent topic consists mainly of ‘response words’, which we often see frequently from chat models, i.e. responses which aim to be ‘polite’ and ‘helpful’. We can also see a large number of topics related to programming or computing topics as well as physics, recipes and pets. databricks/databricks-dolly-15k is another dataset that can be used to train an RLHF model. The approach taken to creating this dataset was quite different from the OpenAssistant Conversations dataset since it was created by employees of Databricks instead of being crowd sourced via volunteers. Perhaps we can use our trained BERTopic model to compare the topics across these two datasets?The new BERTopic Hub integrations mean we can load this trained model and apply it to new examples. topic_model = BERTopic.load("davanstrien/chat_topics")We can predict on a single example text: example = "Stalemate is a drawn position. It doesn't matter who has captured more pieces or is in a winning position"topic, prob = topic_model.transform(example)We can get more information about the predicted topic topic_model.get_topic_info(topic)CountNameRepresentation024022_chess_chessboard_practice_strategy['chess', 'chessboard', 'practice', 'strategy', 'learn', 'pawn', 'board', 'pawns', 'play', 'decks']We can see here the topics predicted seem to make sense. We may want to extend this to compare the topics predicted for the whole dataset. from datasets import load_datasetdataset = load_dataset("databricks/databricks-dolly-15k")dolly_docs = dataset['train']['response']dolly_topics, dolly_probs = topic_model.transform(dolly_docs)We can then compare the distribution of topics across both datasets. We can see here that there seems to be a broader distribution across topics in the dolly dataset according to our BERTopic model. This might be a result of the different approaches to creating both datasets (we likely want to retrain a BERTopic across both datasets to ensure we are not missing topics to confirm this). Comparison of the distribution of topics between the two datasetsWe can potentially use topic models in a production setting to monitor whether topics drift to far from an expected distribution. This can serve as a signal that there has been drift between your original training data and the types of conversations you are seeing in production. You may also decide to use a topic modelling as you are collecting training data to ensure you are getting examples for topics you may particularly care about. Get Started with BERTopic and Hugging Face HubYou can visit the official documentation for a quick start guide to get help using BERTopic. You can find a starter Colab notebook here that shows how you can train a BERTopic model and push it to the Hub. Some examples of BERTopic models already on the hub:MaartenGr/BERTopic_ArXiv: a model trained on ~30000 ArXiv Computation and Language articles (cs.CL) after 1991.MaartenGr/BERTopic_Wikipedia: a model trained on 1000000 English Wikipedia pages.davanstrien/imdb_bertopic: a model trained on the unsupervised split of the IMDB datasetYou can find a full overview of BERTopic models on the hub using the libraries filterWe invite you to explore the possibilities of this new integration and share your trained models on the hub!