task_categories:text-generation language:en transformers region:us
---
task_categories:
- text-generation
language:
- en
pretty_name: lex-llm
tags:
- transformers
---

# Intro
This dataset represents a compilation of audio-to-text transcripts from the Lex Fridman Podcast. The Lex Fridman Podcast, hosted by AI researcher at MIT, Lex Fridman, is a deep dive into a broad range of topics that touch on science, technology, history, philosophy, and the nature of intelligence, consciousness, love, and power. The guests on the podcast are drawn from a diverse range of fields, providing unique and insightful perspectives on these subjects.

The dataset has been formatted in ShareGPT format for use with conversational large language models (LLMs) like Vicuna, WizardVicuna, etc.

This dataset can be an invaluable resource for training and refining language models, offering a rich source of nuanced, intellectual, and thought-provoking dialogue. Furthermore, the diversity of topics covered provides a broad spectrum of language usage, idiomatic expressions, and subject matter expertise.

### 3 versions
1. _original: original dataset where each item is an entire episode
2. _chunked: chunked dataset where episodes are formated into chunks of approximately 1200 words(roughly < 2048 tokens)
3. _chunked_gpt: change "lex" & "guest" to "human" & "gpt" in _chunked dataset to fit Vicuna training
    

# What I did
1. Fetch all episode links of Lex Fridman Podcast
2. For each episode, transform the transcript in html to json format (Vicuna ShareGPT format)
3. remove the first few sentences from Lex for each episode to remove the introduction and ads.

# Problems & Concerns

1. These are audio-to-text transcriptions, which contain inaccurate detections
2. Although the speakers are professionals, these are verbal conversations which contain oral languages
3. The dataset may contain ads and personal opinions from Lex Fridman and the speakers
4. more ...

# Next Steps
1. finetune LLaMA, WizardVicuna, Vicuna models using this dataset