Docker, UFW and iptables: a security flaw you need to solve now

Diagram to facilitate the understanding of how Docker works with Iptables. Source: own work

I was about to enter to production mode with my dockerized services in risk even though I had carefully crafted a UFW strategy. Couldn’t believe it! Then, the worst came: iptables configuration and a whole bunch of networking-savy post all over the internet.

Next, I’ll share what I’ve learned from this journey to Firewall secure my bare-metal dockerized infrastructure in the hope others can advance faster than me and avoid serious security risks.

We’ll see a case study where UFW is not accurate in describing the actual packets policy, then an introduction to how Docker works with iptables and finally a fast approach to put in place a firewall strategy.

As a bonus at the end are some useful references for your study.

Case study

My stack

Bare-metal servers running Ubuntu 18.04, with docker version >19.03 and dockerized MySQL servers.

My failed strategy

Some MySQL servers users accepting logins from “%”, so, from anywhere. But no problem, I have UFW there to limit that “anywhere” configured using Ansible roles, so I don’t have to mess around with more than a dozen MySQL servers. Smart game! Well… didn’t have time to QA.

The “what!? oops!”

A colleague using her “%” MySQL user from home, not from her “approved” server to access the databases.

Diagnosis

Coded a python script to query the databases from my local pc, and effectively, the MySQL user with “%” client didn’t have the UFW firewall in front. The query was positive.

Used nmap to port scanning the machines with nmap 127.0.0.1 -v and:

Diagnosis ran with nmap

But when ufw status numbered you can tell it is impossible to tell the security risk we are incurring in, because even the default incoming policy is shown in deny:

UFW is not a 100% accurate source of information about your real protection.

Introduction on how Docker works with iptables

Docker adds a rule to the PREROUTING chain to modify the destination, so it maps to the container’s internal IP and port. This PREROUTING chain:

“Alters packets before routing. i.e Packet translation happens immediately after the packet comes to the system (and before routing)”. Source: Ramesh Natarajan (2011)

Once changed, the package continues to the FORWARD chain in the filter table, where the DOCKER-USER chain is waiting right in the top for you to create real and functional Firewall policies, just before the DOCKER chain takes action, which is key not to modify because it is a tool for Docker engine to automatically accomplish its fine networking job. As it is a bad idea to manually modify it, it is not recommended to keep Docker from working with the iptables, because, as the documentation says:

“Setting iptables to false will more than likely break container networking for the Docker engine”.

As you may notice in the image below, DOCKER-USER and DOCKER chains have higher position in the chain than any ufw rules. There I have my security strategy: way below the floating line.

Why does it goes to the FORWARD chain and not to the INPUT one? I am not 100% sure at this point, but my hypothesis is that in PREROUTING / DNAT the packet is tagged so the kernel identifies that it is not going to a local listening socket but that it needs to be forwarded somewhere else.

Put in place a firewall strategy

As a first and simple approach, the next solution is in process of being escalated using Ansible roles. If you have advances in that, please share in the comments.

Let’s begin by understanding how iptables chains do their job: they work in the same logical way as data structures LIFO approach: the topmost rule is the first to be applied. If it does not match with the packet characteristics, then the second is applied, and if it does not match, the third is applied, and so on…

Our strategy

Block all incomings except IPs 123.45.67.89 and 987.65.432.1 aiming at port 3306 from the DOCKER-USER chain.

A drastic closure at the end of the chain

At the bottom of our chain, we need a rule to drop all connections so if any added rule does not match with the packet, this will stop it from moving forward to the MySQL server:

sudo iptables --insert DOCKER-USER --in-interface eth0 --jump DROP

Here, you added a rule to the chain DOCKER-USER that will match all incoming packets from eth0 and drop them. Pretty drastic, huh?

The interface eth0 is found with ifconfig command in Linux. You may want to pipe it with grep your.public.ip -B 2 to get the exact interface name to reference.

Guest list in place

Here comes the tricky part, specially because networking knowledge comes into play. To open up the doors we’ll need to use a fancy iptables extension called conntrack, which:

“…when combined with connection tracking, allows access to the connection tracking state for this packet/connection”. Source: man2html

iptables --insert DOCKER-USER --in-interface eth0 --source 123.45.67.89/32 --protocol tcp --match conntrack --ctorigdstport 3306 --ctdir ORIGINAL --jump ACCEPT
iptables --insert DOCKER-USER --in-interface eth0 --source 987.65.432.1/32 --protocol tcp --match conntrack --ctorigdstport 3306 --ctdir ORIGINAL --jump ACCEPT

Change the bold ips to your allowed Ip and -ctorigdstport 3306 to the port you want to be followed. And for this rule to only apply to packets going in the client → MySQL direction, and not in the MySQL → client direction, the -ctdir ORIGINAL statement should be placed:

“...--ctorigdstport matches not the destination port on the packet being filtered, but on the packet that initiated the connection… In fact, all rules allowing the connection should also have --ctdir added for exactly expressing their meaning…” Source: Przemek Wesołek (2019)

You may add as many guest list as desired, tunning the Ips and the ports if you have different Docker services being exposed.

Bonus: useful references

To better refresh your understanding of DNAT process:

To refresh your understanding of Iptables. It has very handy commands and good diagrams:

Linux Firewall Tutorial: IPTables Tables, Chains, Rules Fundamentals

Good introduction to iptables if you’re totally lost:

How to make simple iptables rules:

How to configure iptables on Ubuntu

Good discussions with the response almost done, but not in-depth explained:

Steps for limiting outside connections to docker container with iptables?

Docker chain in iptables overrides INPUT rules

Github issues to get more discussion:

Docker Network bypasses Firewall, no option to disable · Issue #22054 · moby/moby

Guide for static iptables rules for DOCKER-USER on CentOS 7? · Issue #8087 ·…

Iptables man page:

iptables(8) - Linux man page

References:

1. RAMESH NATARAJAN (2011) Linux Firewall Tutorial: IP Tables Tables, Chains, Rules Fundamentals. Available at: https://www.thegeekstuff.com/2011/01/iptables-fundamentals/#:~:text=PREROUTING%20chain%20%E2%80%93%20Alters%20packets%20before,for%20DNAT%20(destination%20NAT).
2. man2html (2019) iptables-extensions. Available at: https://ipset.netfilter.org/iptables-extensions.man.html
3. Przemek Wesołek (2019) Available at: https://github.com/moby/moby/issues/22054#issuecomment-466663033