<commit_before>---
layout: post
title: "youtube-rss-dl"
description: "a ruby script to automatically download new videos from chosen YouTube channels"
date: 2016-09-14
comments: true
share: false
---

This is the project pipeline of a ruby script to automatically check and download new videos from chosen Youtube channels.

### Environment

* Ubuntu 16.04
* ruby 2.3.1
* [youtube-dl](https://rg3.github.io/youtube-dl/) 2016.08.31
* [pyxattr](http://pyxattr.k1024.org) 0.5.5
* [ffmpeg](https://ffmpeg.org) 2.8.6
* [simple-rss](https://github.com/cardmagic/simple-rss) 1.3.1
* [youtube-dl.rb](https://github.com/layer8x/youtube-dl.rb) 0.3.1.2016.08.31

### Tools

With a fresh Ubuntu 16.04 system, I choose to install the apt version of `ruby` and `pip`.

```
$ sudo apt-get install ruby-full python-pip
```

Through `pip` then, I got `youtube-dl` and `pyxattr`, which will be crucial to extract the metadata from YouTube and add them to the downloaded video. It requires a couple additional libraries and a tool.

```
$ sudo apt-get install libffi-dev libattr1-dev atomicparsley
$ sudo pip install youtube-dl pyxattr
```

From Ubuntu 15, `ffmpeg` is back in the system packages. This tool will be necessary to convert the videos in the proper format.

```
$ sudo apt-get install ffmpeg
```

The two gems then:

`simple-rss` will parse the rss feed generated by YouTube. I choose to use an external gem instead of the standard library `RSS` because it didn't like the YouTube feed tags very much.

`youtube-dl.rb` will give us the ruby bindings for youtube-dl. The gems ships with a version of youtube-dl, but I preferred to install the pip version anyway to have the command line tool too.

```
sudo gem install simple-rss youtube-dl.rb
```

### Process

The whole script is a the end of the page.

Here I will break it in pieces to better understand the process, in case you want to add or change something.

#### Shebang

The script will be called regularly by cron, therefore we need to tell the shell this is a ruby script.

Use the `env` convention to make it portable and not guess where the ruby bin is located.

```ruby
#!/usr/bin/env ruby
```

#### Require statements

```ruby
require 'json'
require 'open-uri'
require 'simple-rss'
require 'youtube-dl.rb'
```

In addition to the two gems (`open-uri` is used with `simple-rss` to open the feed url), we'll need to manipulate a json to keep track of the downloaded videos.

#### YouTube channels
Here's the most customizable part: the YouTube channels you want to follow.

For each one of them, you need to get the `channel-external-id` tag in the source of the channel page. For example the [**Vsauce**](https://www.youtube.com/user/Vsauce) id is `UC6nSFpj9HTCZ5t-N3Rm3-HA`.

We will group them in an hash, where the key will be an arbitrary name (that will come handy to organize the json) and the value the actual channel id.

```ruby
CHANNELS = {
  'vsauce' => 'UC6nSFpj9HTCZ5t-N3Rm3-HA',
  'vihart' => 'UCOGeU-1Fig3rrDjhm9Zs_wg'
}
```

We can also declare a costant that will store the base url for the feed.

```ruby
RSS_BASE_URL = 'https://www.youtube.com/feeds/videos.xml?channel_id='
```
<commit_msg>fixed some typos in youtube-rss-dl post<commit_after>---
layout: post
title: "youtube-rss-dl"
description: "a ruby script to automatically download new videos from chosen YouTube channels"
date: 2016-09-14
comments: true
share: false
---

This is the project pipeline of a ruby script to automatically check and download new videos from chosen Youtube channels.

### Environment

* Ubuntu 16.04
* ruby 2.3.1
* [youtube-dl](https://rg3.github.io/youtube-dl/) 2016.08.31
* [pyxattr](http://pyxattr.k1024.org) 0.5.5
* [ffmpeg](https://ffmpeg.org) 2.8.6
* [simple-rss](https://github.com/cardmagic/simple-rss) 1.3.1
* [youtube-dl.rb](https://github.com/layer8x/youtube-dl.rb) 0.3.1.2016.08.31

### Tools

With a fresh Ubuntu 16.04 system, I choose to install the apt version of `ruby` and `pip`.

```
$ sudo apt-get install ruby-full python-pip
```

Through `pip` then, I got `youtube-dl` and `pyxattr`, which will be crucial to extract the metadata from YouTube and add them to the downloaded video. It requires a couple additional libraries and a tool.

```
$ sudo apt-get install libffi-dev libattr1-dev atomicparsley
$ sudo pip install youtube-dl pyxattr
```

Since Ubuntu 15, `ffmpeg` is back in the system packages. This tool will be necessary to convert the videos in the proper format.

```
$ sudo apt-get install ffmpeg
```

The two gems then:

`simple-rss` will parse the rss feed generated by YouTube. I choose to use an external gem instead of the standard library `RSS` because it didn't like the YouTube feed tags very much.

`youtube-dl.rb` will give us the ruby bindings for youtube-dl. The gems ships with a version of youtube-dl, but I preferred to install the pip version anyway to have the command line tool too.

```
sudo gem install simple-rss youtube-dl.rb
```

### Process

The whole script is a the end of the page.

Here I will break it in pieces to better understand the process, in case you want to add or change something.

#### Shebang

The script will be called regularly by cron, therefore we need to tell the shell this is a ruby script.

Use the `env` convention to make it portable and not guess where the ruby bin is located.

```ruby
#!/usr/bin/env ruby
```

#### Require statements

```ruby
require 'json'
require 'open-uri'
require 'simple-rss'
require 'youtube-dl.rb'
```

In addition to the two gems (`open-uri` is used with `simple-rss` to open the feed url), we'll need to manipulate a json to keep track of the downloaded videos.

#### YouTube channels
Here's the most customizable part: the YouTube channels you want to follow.

For each one of them, you need to get the `channel-external-id` tag in the source of the channel page. For example the [**Vsauce**](https://www.youtube.com/user/Vsauce) id is `UC6nSFpj9HTCZ5t-N3Rm3-HA`.

We will group them in an hash, where the key will be an arbitrary name (that will come handy to organize the json) and the value the actual channel id.

```ruby
CHANNELS = {
  'vsauce' => 'UC6nSFpj9HTCZ5t-N3Rm3-HA',
  'vihart' => 'UCOGeU-1Fig3rrDjhm9Zs_wg'
}
```

We can also declare a constant that will store the base url for the feed.

```ruby
RSS_BASE_URL = 'https://www.youtube.com/feeds/videos.xml?channel_id='
```

{'fraction_non_alphanumeric': 0.1040553285130462, 'fraction_numerical': 0.02169129204652625, 'mean_word_length': 4.0260663507109005, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 12, 'lorem ipsum': 0, 'www.': 4, 'xml': 2}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4992756', 'n_tokens_mistral': 2176, 'n_tokens_neox': 1959, 'n_words': 875}