<jupyter_start><jupyter_text>Many captions have a new line in them so replace that with a space<jupyter_code>def bytesToSubtitle(captionBytes):
    captionStr = captionBytes.decode('utf-8')
    for subtitle in srt.parse(captionStr):
        subtitle.content = subtitle.content.replace('\n', ' ')
        yield subtitle<jupyter_output><empty_output><jupyter_text>Lets just join all the text together, ignoring time info<jupyter_code>from functools import reduce
def subToText(subtitles):
    def joinSub(sub1, sub2):
        if sub1 == '':
            return sub2.content
        return f'{sub1} {sub2.content}'
    return reduce(joinSub, subtitles, '')
def bytesToText(captionBytes):
    return subToText(bytesToSubtitle(captionBytes))
df['caption'] = df['caption'].apply(bytesToText)
df
dunk = df['caption'].iloc[7]
print(dunk)<jupyter_output>It's real good. [ banjo music ] Banjo Kazooie is one of those rare titles that was actually made with a glowing spark of inspiration and passion for gaming. It's not one of these soulless, droning cash grabs. It's an illustrious, unforgettable experience. These games are older than dinosaur dirt and yet people still fondly remember Banjo. Even if you never played the games, you know about 'em. It's a bear wearing a backpack that he jammed a little bird into. Who comes up with this? [ shotgun pump ] It's amazing! You're dropped right into a cheery, colorful world bursting with characters and liveliness. There's hundreds of things to collect and discover. You start humming along to that classic music and boom: You've been playing for five hours and it went by just like that [ snaps fingers ]. The game is fun as hell because it offers up a lot of simple pleasures. Like the little jinjos. You walk into them and it goes [ imitating jinjo fanfare ]. It's just so immediate and satisfying. You[...]
{'fraction_non_alphanumeric': 0.07679826933477556, 'fraction_numerical': 0.004326663061114116, 'mean_word_length': 4.4411764705882355, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '122388', 'license': 'FTL', 'n_tokens_mistral': 536, 'n_tokens_neox': 513, 'n_words': 231, 'path': 'notebooks/1.0-JM-CaptionParsing.ipynb', 'repo_name': 'JohnReillyMurray/youtube-transcript-generator', 'size': '1849'}