language:en license:apache-2.0 region:us
---
license: apache-2.0
language:
- en
viewer: false
---

# Dataset Name: LexiGPT-Podcast-Corpus

This dataset has been created using the transcripts from [lexicap](https://karpathy.ai/lexicap/).
Each transcript has been partitioned into chunks of max 1000 tokens. 
GPT-3.5 has been used to augment the chunks with a description and context field.

The features provided are: title, description, context, transcript.

# Description: 
The LexiGPT-Podcast-Corpus dataset offers a comprehensive collection of transcripts from the Lex Fridman podcast, thoughtfully curated and enhanced using GP-T3.5.

# Use:

First download the dataset to the directory: 'LexiGPT-Podcast-Corpus/dataset.json'

```python
# Load the dataset
dataset = load_dataset('json', data_files='LexiGPT-Podcast-Corpus/dataset.json', field='data')

# Define your custom formatting function
def custom_format(example):
    formatted_text = f"### INSTRUCTIONS:\n\nGenerate the video transcript '{example['Title']}':\n\n{example['Description']}\n\n### CONTEXT: {example['Context']}\n\n### TRANSCRIPT:\n\n{example['Transcript']}"
    return {"text": formatted_text}

# Add the new field using the custom formatting function
dataset = dataset.map(custom_format)

# Access and print a specific row
example = dataset["train"]["text"][0]
print(example)
```