<commit_before># ytdlrc

> Downloads videos and metadata with `youtube-dl` and moves each file on
> completion to an `rclone` remote

`ytdlrc` is a simple shell script wrapper for `youtube-dl` and `rclone`. It
downloads videos and metadata via `youtube-dl` and moves each file on
completion to an `rclone` remote, e.g. Google Drive.

<details>
  <summary>
    <strong>Table of Contents</strong>
  </summary>
  <ul>
    <li><a href="#about">About</a>
    <li><a href="#example-output">Example Output</a>
        <ul>
          <li><a href="#example-initial-execution">Example: Initial execution
          <li><a href="#example-downloading-all-videos-from-ytuserkurzgesagt">Example: Downloading all videos from ytuser:kurzgesagt
            <ul>
              <li><a href="#first-run">First run</a>
              <li><a href="#second-run">Second run</a>
            </ul>
        </ul>
    <li><a href="#installation">Installation</a>
    <li><a href="#usage">Usage</a>
    <li><a href="#requirements">Requirements</a>
    <li><a href="#license">License</a>
  </ul>
</details>

## About

*   Designed to be executed via cron job.

*   Ideal for use on VPS's with small disk space as it moves files on
    completion rather than copying them. In testing I downloaded / uploaded
    ~500GB of videos on a VPS with a 15GB SSD.

    *   This can easily be changed to keep the files if you have the disk
        space; see the `rclone_command` variable to do so.

*   Includes metadata (xattrs, \*.info.json, \*.description, \*.jpg).

*   Reads a file named `snatch.list` to know which URL's / channels /
    playlists you want to download and monitor for new videos.

    *   Lines beginning with `#` are ignored.

*   Completed video ID's are saved in an file named `archive.list`; this
    prevents `youtube-dl` from re-downloading videos that have already been
    processed and moved to the rclone remote.

*   By default, it creates / uses the following structure:

    *   `~/ytdlrc/stage`           - download directory
    *   `~/ytdlrc/snatch.list`     - list of usernames / URL's to monitor / download
    *   `~/ytdlrc/archive.list`    - list of completed video ID's<commit_msg>Fix typo<commit_after># ytdlrc

> Downloads videos and metadata with `youtube-dl` and moves each file on
> completion to an `rclone` remote

`ytdlrc` is a simple shell script wrapper for `youtube-dl` and `rclone`. It
downloads videos and metadata via `youtube-dl` and moves each file on
completion to an `rclone` remote, e.g. Google Drive.

<details>
  <summary>
    <strong>Table of Contents</strong>
  </summary>
  <ul>
    <li><a href="#about">About</a>
    <li><a href="#example-output">Example Output</a>
        <ul>
          <li><a href="#example-initial-execution">Example: Initial execution
          <li><a href="#example-downloading-all-videos-from-ytuserkurzgesagt">Example: Downloading all videos from ytuser:kurzgesagt
            <ul>
              <li><a href="#first-run">First run</a>
              <li><a href="#second-run">Second run</a>
            </ul>
        </ul>
    <li><a href="#installation">Installation</a>
    <li><a href="#usage">Usage</a>
    <li><a href="#requirements">Requirements</a>
    <li><a href="#license">License</a>
  </ul>
</details>

## About

*   Designed to be executed via cron job.

*   Ideal for use on VPS's with small disk space as it moves files on
    completion rather than copying them. In testing I downloaded / uploaded
    ~500GB of videos on a VPS with a 15GB SSD.

    *   This can easily be changed to keep the files if you have the disk
        space; see the `rclone_command` variable to do so.

*   Includes metadata (xattrs, \*.info.json, \*.description, \*.jpg).

*   Reads a file named `snatch.list` to know which URL's / channels /
    playlists you want to download and monitor for new videos.

    *   Lines beginning with `#` are ignored.

*   Completed video ID's are saved in a file named `archive.list`; this
    prevents `youtube-dl` from re-downloading videos that have already been
    processed and moved to the rclone remote.

*   By default, it creates / uses the following structure:

    *   `~/ytdlrc/stage`           - download directory
    *   `~/ytdlrc/snatch.list`     - list of usernames / URL's to monitor / download
    *   `~/ytdlrc/archive.list`    - list of completed video ID's
{'fraction_non_alphanumeric': 0.12018348623853212, 'fraction_numerical': 0.0022935779816513763, 'mean_word_length': 2.9359205776173285, 'pattern_counts': {'":': 0, '<': 83, '<?xml version=': 0, '>': 87, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '999091', 'n_tokens_mistral': 1383, 'n_tokens_neox': 1318, 'n_words': 478}