<commit_before>---
layout: post
title: "Docker meet firewall - finally an answer"
description: "Integrating docker with your firewall has always been a pain. But those times are coming to an end!"
category:
tags: ['docker', linux', 'iptables', 'firewall', 'security']
---

One of the most annoying things with Docker has been how it interacts with iptables. And ufw. And firewalld. Most firewall solutions on Linux assume they are the source of truth. But increasingly thats not a sensible assumption. This inevitably leads to collisions - restarting the firewall or Docker will end up clobbering *something*. Whilst it was possible to get it working, it was a pain. And always a bit dirty. **I don't want to have to restart Docker after tweaking my firewall!** Recently a new solution has [presented itself](https://docs.docker.com/engine/userguide/networking/#links) and it looks like things are going to get a lot better:

> In Docker 17.06 and higher, you can add rules to a new table called DOCKER-USER, and these rules will be loaded before any rules Docker creates automatically. This can be useful if you need to pre-populate iptables rules that need to be in place before Docker runs.

You can read more about it in the [pull request](https://github.com/docker/libnetwork/pull/1675) that added it.

So how do we make use of that? Searching for an answer is still hard - there are 3 years of people scrambling to work around the issue and not many posts like this one yet. But by the end of this post you will have an iptables based firewall that doesn't clobber Docker when you apply it. Docker won't clobber it either. And it will make it easier to write rules that apply to non-container ports and container ports alike.

Starting from an Ubuntu 16.04 VM that has Docker installed but has never had an explicit firewall setup before. If you've had any other sort of Docker firewall before, undo those changes. Docker should be allowed to do its own iptables rules. Don't change the `FORWARD` chain to `ACCEPT` from `DROP`. There is no need any more. On a clean environment before any of our changes this is what `iptables-save` looks like:

```
$ sudo iptables-save
# Generated by iptables-save v1.6.0 on Tue Aug 15 04:02:08 2017
*nat
:PREROUTING ACCEPT [1:64]
:INPUT ACCEPT [1:64]
:OUTPUT ACCEPT [8:488]
:POSTROUTING ACCEPT [10:616]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.19.0.0/16 ! -o br-68428f03a4d1 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p tcp -m tcp --dport 9200 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-68428f03a4d1 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER ! -i br-68428f03a4d1 -p tcp -m tcp --dport 9200 -j DNAT --to-destination 172.19.0.2:9200
COMMIT
# Completed on Tue Aug 15 04:02:08 2017
# Generated by iptables-save v1.6.0 on Tue Aug 15 04:02:08 2017
*filter
:INPUT ACCEPT [174:13281]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [138:16113]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-68428f03a4d1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-68428f03a4d1 -j DOCKER
-A FORWARD -i br-68428f03a4d1 ! -o br-68428f03a4d1 -j ACCEPT
-A FORWARD -i br-68428f03a4d1 -o br-68428f03a4d1 -j ACCEPT
-A FORWARD -o docker_gwbridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker_gwbridge -j DOCKER
-A FORWARD -i docker_gwbridge ! -o docker_gwbridge -j ACCEPT
-A FORWARD -i docker_gwbridge -o docker_gwbridge -j DROP
-A DOCKER -d 172.19.0.2/32 ! -i br-68428f03a4d1 -o br-68428f03a4d1 -p tcp -m tcp --dport 9200 -j ACCEPT
-A DOCKER-ISOLATION -i br-68428f03a4d1 -o docker0 -j DROP
-A DOCKER-ISOLATION -i docker0 -o br-68428f03a4d1 -j DROP
-A DOCKER-ISOLATION -i docker_gwbridge -o docker0 -j DROP
-A DOCKER-ISOLATION -i docker0 -o docker_gwbridge -j DROP
-A DOCKER-ISOLATION -i docker_gwbridge -o br-68428f03a4d1 -j DROP
-A DOCKER-ISOLATION -i br-68428f03a4d1 -o docker_gwbridge -j DROP
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
```

The main points to note are that `INPUT` has been left alone by Docker and that there is (as documented) a `DOCKER-USER` chain that has been set up for us. All traffic headed to a container goes to the `FORWARD` chain and this lets `DOCKER-USER` filter that traffic before the Docker rules are applied.


## A firewall that doesn't smoosh Docker iptables rules

So a super simple firewall. Create a new `/etc/iptables.conf` that looks like this:

```
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FILTERS - [0:0]
:DOCKER-USER - [0:0]

-F INPUT
-F DOCKER-USER
-F FILTERS

-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp --icmp-type any -j ACCEPT
-A INPUT -j FILTERS

-A DOCKER-USER -i ens33 -j FILTERS

-A FILTERS -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FILTERS -m state --state NEW -s 1.2.3.4/32
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 23 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A FILTERS -j REJECT --reject-with icmp-host-prohibited

COMMIT
```

You can load it into the kernel with:

```
iptables-restore -n /etc/iptables.conf
```

That `-n` flag is crucial to avoid breaking Docker.

Whats going on here?

 * This firewall avoids touching areas Docker is likely to interfere with. You can restart Docker over and over again and it will not harm or hinder our rules in `INPUT`, `DOCKER-USER` or `FILTERS`.

 * We explicitly flush `INPUT`, `DOCKER-USER` and `FILTERS`. This means we don't end up smooshing 2 different versions of our iptables.conf together. Normally this is done implicitly by `iptables-restore`. But its that implicit flush that that clobbers the rules that Docker manages. So we will only ever load this config with `iptables-restore -n /etc/iptables.conf`. The `-n` flag turns off the implicit global flush and only does our manual explicit flush. **The Docker rules are preserved - no more restarting Docker when you change your firewall**.
 
 * We have an explicit `FILTERS` chain. This is used by the `INPUT` chain. But Docker traffic actually goes via the `FORWARD` chain. And thats why `ufw` has always been problematic. This is where `DOCKER-USER` comes in. We just add a rule that passes any traffic from the external physical network interface to our `FILTERS` chain. This means that when I want to allow my home IP (in this example `1.2.3.4`) access to every port I update the FILTERS chain once. I don't have to add a rule in `INPUT` and a rule in `DOCKER-USER`. I don't have to think about which part of the firewall a rule will or won't work in. My `FILTERS` chain is **the** place to go.
 
 
## Starting the firewall at boot

You can load this firewall at boot with systemd. Add a new unit - `/etc/system/system/iptables.service`:

```
[Unit]
Description=Restore iptables firewall rules
Before=network-pre.target

[Service]
Type=oneshot
ExecStart=/sbin/iptables-restore -n /etc/iptables.conf

[Install]
WantedBy=multi-user.target
```

And enable it:

```
$ sudo systemctl enable --now iptables
```

The firewall is now active, and it didn't smoosh your docker managed iptables rules. You can reboot and the firewall will come up as it is right now.


## Updating the firewall

Pop open the firwall in your favourite text editor, add or remove a rule from the `FILTERS` section, then reload the firewall with:

```
$ sudo systemctl restart iptables
```<commit_msg>Fixes #1: Fix silly mistakes in examples and clarify --now flag<commit_after>---
layout: post
title: "Docker meet firewall - finally an answer"
description: "Integrating docker with your firewall has always been a pain. But those times are coming to an end!"
category:
tags: ['docker', linux', 'iptables', 'firewall', 'security']
---

One of the most annoying things with Docker has been how it interacts with iptables. And ufw. And firewalld. Most firewall solutions on Linux assume they are the source of truth. But increasingly thats not a sensible assumption. This inevitably leads to collisions - restarting the firewall or Docker will end up clobbering *something*. Whilst it was possible to get it working, it was a pain. And always a bit dirty. **I don't want to have to restart Docker after tweaking my firewall!** Recently a new solution has [presented itself](https://docs.docker.com/engine/userguide/networking/#links) and it looks like things are going to get a lot better:

> In Docker 17.06 and higher, you can add rules to a new table called DOCKER-USER, and these rules will be loaded before any rules Docker creates automatically. This can be useful if you need to pre-populate iptables rules that need to be in place before Docker runs.

You can read more about it in the [pull request](https://github.com/docker/libnetwork/pull/1675) that added it.

So how do we make use of that? Searching for an answer is still hard - there are 3 years of people scrambling to work around the issue and not many posts like this one yet. But by the end of this post you will have an iptables based firewall that doesn't clobber Docker when you apply it. Docker won't clobber it either. And it will make it easier to write rules that apply to non-container ports and container ports alike.

Starting from an Ubuntu 16.04 VM that has Docker installed but has never had an explicit firewall setup before. If you've had any other sort of Docker firewall before, undo those changes. Docker should be allowed to do its own iptables rules. Don't change the `FORWARD` chain to `ACCEPT` from `DROP`. There is no need any more. On a clean environment before any of our changes this is what `iptables-save` looks like:

```
$ sudo iptables-save
# Generated by iptables-save v1.6.0 on Tue Aug 15 04:02:08 2017
*nat
:PREROUTING ACCEPT [1:64]
:INPUT ACCEPT [1:64]
:OUTPUT ACCEPT [8:488]
:POSTROUTING ACCEPT [10:616]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.19.0.0/16 ! -o br-68428f03a4d1 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p tcp -m tcp --dport 9200 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-68428f03a4d1 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER ! -i br-68428f03a4d1 -p tcp -m tcp --dport 9200 -j DNAT --to-destination 172.19.0.2:9200
COMMIT
# Completed on Tue Aug 15 04:02:08 2017
# Generated by iptables-save v1.6.0 on Tue Aug 15 04:02:08 2017
*filter
:INPUT ACCEPT [174:13281]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [138:16113]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-68428f03a4d1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-68428f03a4d1 -j DOCKER
-A FORWARD -i br-68428f03a4d1 ! -o br-68428f03a4d1 -j ACCEPT
-A FORWARD -i br-68428f03a4d1 -o br-68428f03a4d1 -j ACCEPT
-A FORWARD -o docker_gwbridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker_gwbridge -j DOCKER
-A FORWARD -i docker_gwbridge ! -o docker_gwbridge -j ACCEPT
-A FORWARD -i docker_gwbridge -o docker_gwbridge -j DROP
-A DOCKER -d 172.19.0.2/32 ! -i br-68428f03a4d1 -o br-68428f03a4d1 -p tcp -m tcp --dport 9200 -j ACCEPT
-A DOCKER-ISOLATION -i br-68428f03a4d1 -o docker0 -j DROP
-A DOCKER-ISOLATION -i docker0 -o br-68428f03a4d1 -j DROP
-A DOCKER-ISOLATION -i docker_gwbridge -o docker0 -j DROP
-A DOCKER-ISOLATION -i docker0 -o docker_gwbridge -j DROP
-A DOCKER-ISOLATION -i docker_gwbridge -o br-68428f03a4d1 -j DROP
-A DOCKER-ISOLATION -i br-68428f03a4d1 -o docker_gwbridge -j DROP
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
```

The main points to note are that `INPUT` has been left alone by Docker and that there is (as documented) a `DOCKER-USER` chain that has been set up for us. All traffic headed to a container goes to the `FORWARD` chain and this lets `DOCKER-USER` filter that traffic before the Docker rules are applied.


## A firewall that doesn't smoosh Docker iptables rules

So a super simple firewall. Create a new `/etc/iptables.conf` that looks like this:

```
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:FILTERS - [0:0]
:DOCKER-USER - [0:0]

-F INPUT
-F DOCKER-USER
-F FILTERS

-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp --icmp-type any -j ACCEPT
-A INPUT -j FILTERS

-A DOCKER-USER -i ens33 -j FILTERS

-A FILTERS -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FILTERS -m state --state NEW -s 1.2.3.4/32 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 23 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A FILTERS -j REJECT --reject-with icmp-host-prohibited

COMMIT
```

You can load it into the kernel with:

```
iptables-restore -n /etc/iptables.conf
```

That `-n` flag is crucial to avoid breaking Docker.

Whats going on here?

 * This firewall avoids touching areas Docker is likely to interfere with. You can restart Docker over and over again and it will not harm or hinder our rules in `INPUT`, `DOCKER-USER` or `FILTERS`.

 * We explicitly flush `INPUT`, `DOCKER-USER` and `FILTERS`. This means we don't end up smooshing 2 different versions of our iptables.conf together. Normally this is done implicitly by `iptables-restore`. But its that implicit flush that that clobbers the rules that Docker manages. So we will only ever load this config with `iptables-restore -n /etc/iptables.conf`. The `-n` flag turns off the implicit global flush and only does our manual explicit flush. **The Docker rules are preserved - no more restarting Docker when you change your firewall**.

 * We have an explicit `FILTERS` chain. This is used by the `INPUT` chain. But Docker traffic actually goes via the `FORWARD` chain. And thats why `ufw` has always been problematic. This is where `DOCKER-USER` comes in. We just add a rule that passes any traffic from the external physical network interface to our `FILTERS` chain. This means that when I want to allow my home IP (in this example `1.2.3.4`) access to every port I update the FILTERS chain once. I don't have to add a rule in `INPUT` and a rule in `DOCKER-USER`. I don't have to think about which part of the firewall a rule will or won't work in. My `FILTERS` chain is **the** place to go.


## Starting the firewall at boot

You can load this firewall at boot with systemd. Add a new unit - `/etc/system/system/iptables.service`:

```
[Unit]
Description=Restore iptables firewall rules
Before=network-pre.target

[Service]
Type=oneshot
ExecStart=/sbin/iptables-restore -n /etc/iptables.conf

[Install]
WantedBy=multi-user.target
```

And enable it:

```
$ sudo systemctl enable --now iptables
```

If your version of `systemctl` doesn't support this you can do it the old way:

```
$ sudo systemctl enable iptables
$ sudo systemctl start iptables
```

The firewall is now active, and it didn't smoosh your docker managed iptables rules. You can reboot and the firewall will come up as it is right now.


## Updating the firewall

Pop open the firwall in your favourite text editor, add or remove a rule from the `FILTERS` section, then reload the firewall with:

```
$ sudo systemctl restart iptables
```
{'fraction_non_alphanumeric': 0.08902717022980716, 'fraction_numerical': 0.04429794837040232, 'mean_word_length': 3.627137970353478, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 20, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5882714', 'n_tokens_mistral': 6173, 'n_tokens_neox': 5493, 'n_words': 2497}