IVPN is a fairly popular and generally trustworthy VPN provider. In this post, I will walk you through how to use the official IVPN client in a ProxyVM on Qubes OS. We will deviate from the official guide by using systemd path to handle DNAT. This will provide the same robustness as their approach to modify /opt/ivpn/etc/firewall.sh, while avoiding the risk that the modifications will be overwritten by a future app update. We will also be using a TemplateVM for IVPN ProxyVMs instead of using Standalone VMs.

Preparing your TemplateVM

I recommend that you make a new TemplateVM based on the latest Fedora GNOME template and remove all unnecessary packages that you might not use. This way, you can minimize the attack surface while not having to deal with missing dependencies like on a minimal template. With that being said, if you do manage to get the minimal template to fully work with IVPN, feel free to open a discussion on GitHub or contact me directly and I will update the post accordingly.

I run this script on my template to trim it down.

Next, you need to create the bind directories for IVPN’s configurations:

sudo mkdir -p /etc/qubes-bind-dirs.d
echo 'binds+=( '\'''/etc/opt/ivpn/mutable''\'' )' | sudo tee /etc/qubes-bind-dirs.d/50_user.conf 

Installing the IVPN App

Inside of the TemplateVM you have just created, do the following:

sudo dnf config-manager addrepo --from-repofile=https://repo.ivpn.net/stable/fedora/generic/ivpn.repo
sudo dnf install -y ivpn-ui

IVPN needs to restart systemd-resolved and run /usr/lib/qubes/qubes-setup-dnat-to-ns every time IVPN modifies /etc/resolv.conf. Create the following files:

• /etc/systemd/system/dnat-to-ns.service

[Unit]
Description=Run /usr/lib/qubes/qubes-setup-dnat-to-ns
StartLimitIntervalSec=0

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl restart systemd-resolved
ExecStart=/usr/lib/qubes/qubes-setup-dnat-to-ns

• /etc/systemd/system/dnat-to-ns.path

[Unit]
Description=Run /usr/lib/qubes/qubes-setup-dnat-to-ns when /etc/resolv.conf changes

[Path]
PathChanged=/etc/resolv.conf
Unit=dnat-to-ns.service

[Install]
WantedBy=multi-user.target

• /etc/systemd/system/dnat-to-ns-boot.service

[Unit]
Description=Run /usr/lib/qubes/qubes-setup-dnat-to-ns
After=qubes-network-uplink.service

[Service]
Type=oneshot
ExecStart=sleep 15
ExecStart=/usr/lib/qubes/qubes-setup-dnat-to-ns

[Install]
WantedBy=multi-user.target

Create /etc/systemd/system/systemd-resolved.conf.d/override.conf to disable rate limiting on systemd-resolved restarting:

[Unit]
StartLimitIntervalSec=0

Next, enable the systemd path and service to run at boot:

sudo systemctl enable dnat-to-ns.path
sudo systemctl enable dnat-to-ns-boot.service

Finally, shut down the TemplateVM:

sudo shutdown now

Creating the ProxyVM

Create an AppVM based on the TemplateVM you have just created. Set sys-firewall (or whatever FirewallVM you have connected to your sys-net) as the net qube. If you do not have such FirewallVM, use sys-net as the net qube. Next, go to the advanced tab and tick the provides network access to other qubes box.

Open the IVPN and select Settings → DNS → Force management of DNS using resolv.conf.

Go to the IVPN Firewall section and tick the box Allow LAN traffic when IVPN Firewall is enabled. Due to some strange interaction between qubes services and IVPN, certain apps will get internet connections while others do not if this toggle is not enabled. This option will not actually allow AppVMs connected to the ProxyVM to connect to the local network.

Enable Always-on firewall to ensure that the killswitch stays on even when the tunnel is disconnected.

Additional Assurances

For additional assurances against VPN leaks, you can optionally add these 2 lines to /rw/config/qubes-firewall-user-script:

nft add rule qubes custom-forward oifname eth0 counter drop
nft add rule ip6 qubes custom-forward oifname eth0 counter drop

This is not strictly necessary, as I have not observed any leaks with the VPN killswitch provided by the app.

Notes

With this current setup, the ProxyVM you have just created will be responsible for handling Firewall rules for the qubes behind it. This is not ideal, as this is still a fairly large VM, and there is a risk that IVPN or some other apps may interfere with its firewall handling.

Instead, I highly recommend that you create a minimal Mirage FirewallVM and use it as a firewall behind the IVPN ProxyVM. Other AppVMs then should use the Mirage Firewall as the net qube instead. This way, you can make sure that firewall rules are properly enforced.

Before We Start…

I have been aware of this issue for awhile now (since at least Android 11), though I have not done enough testing to see what actually causes the leak nor do I have any workaround at the moment. My guess is that applications which launch early when you log into a secondary profile can bypass the VPN killswitch.

I have reported it on Google’s issue tracker.

The Leak

You can reproduce the leak by doing the following:

1. Create a new user profile (you need to create a secondary user profile for this, as it is not reproducible on your owner profile or a work profile). Do not log into your Google account at this stage.
2. Sideload a VPN app. The leak happens with every VPN provider I have tried (since it is likely a platform issue), though if you do not have a VPN subscription I would recommend getting a free one with ProtonVPN.
3. Setup the VPN and the Android VPN killswitch.
4. Log into your Google account through Play Services.
5. Restart the phone. Open the secondary user profile again.
6. Go to Google’s My Devices page. Observe that one of the sessions for your phone has your actual location obtained with GeoIP. In some cases, your actual IP address will be shown there as well.

Notes

1. It is unlikely that this is caused by Play Services being privileged applications. This issue is reproducible on GrapheneOS with the Sandboxed Play Services (which runs as a normal, unprivileged application) as well.

2. More testing is needed to find the root cause of the problem. I do not think that this is Play Services specific. Unfortunately, I do not have access to a router to do a packet capture right now. I would appreciate it if someone can help me get to the bottom of this. You can find my contact information here.

I sent Proton an email regarding this issue in late August 2022 and was told they are working on fixing it, though it will take some time as it requires some architectural changes in how the killswitch works.

The Leak

Ideally, when implementing a killswitch, a VPN client should drop all connections on non-VPN interfaces except when the connection is to the VPN provider’s servers. This is necessary to prevent accidental leaks, at least by unprivileged applications. Unfortunately, the ProtonVPN client does not currently do this.

Effectively, any application that binds to the connected physical interface (as opposed to the VPN’s virtual interface) on your Linux system will expose your actual IP address, regardless of the killswitch state. This is problematic, especially for certain applications like Torrent clients, as they tend to use whatever interfaces they can access (rather than just the default one) to connect to the internet. You can check this with curl:

curl --interface <physical interface> https://ipinfo.io

This will return your actual IP address.

The Workaround

Qubes OS

On Qubes OS, you generally should not have a problem if you use the ProtonVPN client in a ProxyVM. While the same issue still exists within the ProxyVM itself, it is unlikely to manifest as you should not be running any other applications in the same Qube anyways, and apps in an AppVM cannot bind to the public interface of the ProxyVM. I have not observed any leaks from an AppVM behind a ProtonVPN ProxyVM.

General Linux Distributions

On a general Linux distribution, the workaround is to configure OpenVPN manually and setup a killswitch yourself.

Since ProtonVPN does not support IPv6, you should disable it in your kernel settings:

echo 'net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1' | sudo tee /etc/sysctl.d/10-disable-ipv6.conf
sudo sysctl -p

Next, download your OpenVPN configuration files from account.protonvpn.com. In those configuration files, you should see a list of IP addresses and ports of ProtonVPN’s servers.

Finally, set up the VPN killswitch. The rules I posted here are based on this discussion.

Firewalld

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter FORWARD 0 -o tun+ -j ACCEPT
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter FORWARD 0 -i tun+ -j ACCEPT

sudo firewall-cmd --direct --permanent --add-rule ipv6 filter INPUT 0 -j DROP
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 -i lo -j ACCEPT
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 1 -i tun+ -p tcp -j ACCEPT
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 1 -i tun+ -p udp -j ACCEPT
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 999 -j DROP

sudo firewall-cmd --direct --permanent --add-rule ipv6 filter OUTPUT 0 -j DROP
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 -o lo -j ACCEPT
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 -o tun+ -j ACCEPT

#You will need to add each of the IP address and port with the following command:
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 1 -p udp -m udp --dport $PORT -d $IP -j ACCEPT

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 999 -j DROP

sudo firewall-cmd --reload

UFW

sudo ufw default deny incoming
sudo ufw default deny outgoing

#You will need to add each of the IP address and port with the following command:
sudo ufw allow out to $IP port $PORT proto udp

sudo ufw allow out on tun0 from any to any

Note that Ethernet connections can still be tracked via switch ports, and WiFi connections can be broadly localized by access point.

Furthermore, MAC address spoofing and randomization depends on firmware support from the interface. Most modern network interface cards support the feature.

There are three different aspects of MAC address randomization in NetworkManager, each with their own configuration flag:

WiFi scanning

[device]
wifi.scan-rand-mac-address=yes

WiFi connections

[connection]
wifi.cloned-mac-address=<mode>

Ethernet connections

[connection]
ethernet.cloned-mac-address=<mode>

Mode options

random: Generate a new random MAC address every time a connection is activated

stable: Assign each connection a random MAC address that will be maintained across activations

preserve: Use the MAC address already assigned to the interface (such as from macchanger), or the permanent address if none is assigned

permanent: Use the MAC address permanently baked into the hardware

Setting a default configuration

It’s best to create a dedicated configuration file, such as /etc/NetworkManager/conf.d/99-random-mac.conf, to ensure package updates do not overwrite the configuration. In general, I recommend the following:

[device]
wifi.scan-rand-mac-address=yes

[connection]
wifi.cloned-mac-address=random
ethernet.cloned-mac-address=random

This configuration randomizes all MAC addresses by default. These settings can of course be overridden on a per-connection basis.

After editing the file, run sudo nmcli general reload conf to apply the new configuration.

Per-connection overrides

Connection-specific settings take precedence over configuration file defaults. They can be set through nm-connection-editor (“Network Connections”), a DE-specific network settings GUI, nmtui, or nmcli.

Look for “Cloned MAC address” under the “Wi-Fi” or “Ethernet” section:

In addition to the four mode keywords, you can input an exact MAC address to be used for that connection.

For a home or another trusted network, it can be helpful to use stable or even permanent, as MAC address stability can help avoid being repeatedly served a new IP address and DHCP lease (though not all DHCP servers work this way).

For public networks with captive portals (webpages that must be accessed to gain network access), the stable setting can help prevent redirection back to the captive portal after a brief disconnection or roaming to a different access point.

Seeing the randomized MAC address

Activate the connection in question, and then look for GENERAL.HWADDR in the output of nmcli device show. This represents the MAC address currently in use by the interface, whether randomized or not. It is also visible as “Hardware Address” (or similar) in NetworkManager GUIs under active connection details.

$ nmcli device show
GENERAL.DEVICE:                         enp5s0
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         XX:XX:XX:XX:XX:XX

GENERAL.DEVICE:                         wlp3s0
GENERAL.TYPE:                           wifi
GENERAL.HWADDR:                         XX:XX:XX:XX:XX:XX

Remove static hostname to prevent hostname broadcast

sudo hostnamectl hostname "localhost"

An empty (blank) hostname is also an option, but a static hostname of “localhost” is less likely to cause breakage. Both will result in no hostname being broadcasted to the DHCP server.

Disable sending hostname to DHCP server

This configuration will leak your hostname on first connection. Setting a generic or random hostname is strongly recommended if possible.

Due to limitations in NetworkManager, it is not possible to reliably disable sending hostnames by default. This setup is very much a hack.

Due to being leaky, this configuration is virtually useless without also randomizing MAC addresses by default. Your MAC address and hostname will not be correlated starting with the second connection, assuming the first connection used a random MAC address.

Create /etc/NetworkManager/dispatcher.d/no-wait.d/01-no-send-hostname.sh as follows:

#!/bin/sh

if [ "$(nmcli -g 802-11-wireless.cloned-mac-address c show "$CONNECTION_UUID")" = 'permanent' ] \
        || [ "$(nmcli -g 802-3-ethernet.cloned-mac-address c show "$CONNECTION_UUID")" = 'permanent' ]
then
    nmcli connection modify "$CONNECTION_UUID" \
            ipv4.dhcp-send-hostname true \
            ipv6.dhcp-send-hostname true
else
    nmcli connection modify "$CONNECTION_UUID" \
            ipv4.dhcp-send-hostname false \
            ipv6.dhcp-send-hostname false
fi

The script must have specific file permissions and a symlink to take effect:

cd /etc/NetworkManager/dispatcher.d/
sudo chown root:root no-wait.d/01-no-send-hostname.sh
sudo chmod 744 no-wait.d/01-no-send-hostname.sh
sudo ln -s no-wait.d/01-no-send-hostname.sh ./

This script will be automatically triggered on connection events to modify the connection’s dhcp-send-hostname settings. If the connection’s cloned MAC address is explicitly overridden to permanent, the hostname will be sent to the DHCP server on future connections. In all other cases, the hostname will be masked on future connections, so the DHCP server will only see the MAC address.

Verifying proper operation

After initiating first connection with a network:

$ nmcli c show <connection> | grep dhcp-send-hostname
ipv4.dhcp-send-hostname:                no
ipv6.dhcp-send-hostname:                no

<connection> can be the connection name (usually the SSID for WiFi networks) or UUID, obtained from nmcli c show [--active].

Recall that these setting values are set based on the previous connection activation and take effect for the next connection activation.

Sources

• ArchWiki — NetworkManager
• hostnamectl man page
• MAC Address Spoofing in NetworkManager 1.4.0
• NetworkManager.conf man page
• NetworkManager-dispatcher man page
• NetworkManager: Disable Sending Hostname to DHCP Server
• nmcli man page

Mullvad is a fairly popular and generally trustworthy VPN provider. In this post, I will walk you through how to use the official Mullvad client in a ProxyVM on Qubes OS. This method is a lot more convenient than the official guide from Mullvad (which recommends that you manually load in OpenVPN or Wireguard profiles) and will let you seamlessly switch between different location and network setups just as you would on a normal Linux installation.

Preparing your TemplateVM

I recommend that you make a new TemplateVM based on the latest Fedora GNOME template and remove all unnecessary packages that you might not use. This way, you can minimize the attack surface while not having to deal with missing dependencies like on a minimal template. With that being said, if you do manage to get the minimal template to fully work with Mullvad, feel free to open a discussion on GitHub or contact me directly and I will update the post accordingly.

I run this script on my template to trim it down.

Next, you need to create the bind directories for Mullvad’s configurations:

sudo mkdir -p /etc/qubes-bind-dirs.d
echo 'binds+=( '\'''/etc/mullvad-vpn''\'' )' | sudo tee /etc/qubes-bind-dirs.d/50_user.conf 

Installing the Mullvad App

Inside of the TemplateVM you have just created, do the following:

sudo dnf config-manager addrepo --from-repofile=https://repository.mullvad.net/rpm/stable/mullvad.repo
sudo dnf install -y mullvad-vpn

To work around issue 3803, we will be using systemd path to run /usr/lib/qubes/qubes-setup-dnat-to-ns every time Mullvad modifies /etc/resolv.conf. Create the following files:

• /etc/systemd/system/dnat-to-ns.service

[Unit]
Description=Run /usr/lib/qubes/qubes-setup-dnat-to-ns
StartLimitIntervalSec=0

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl restart systemd-resolved
ExecStart=/usr/lib/qubes/qubes-setup-dnat-to-ns

• /etc/systemd/system/dnat-to-ns.path

[Unit]
Description=Run /usr/lib/qubes/qubes-setup-dnat-to-ns when /etc/resolv.conf changes

[Path]
PathChanged=/etc/resolv.conf
Unit=dnat-to-ns.service

[Install]
WantedBy=multi-user.target

Create /etc/systemd/system/systemd-resolved.conf.d/override.conf to disable rate limiting on systemd-resolved restarting:

[Unit]
StartLimitIntervalSec=0

Next, enable the systemd path:

sudo systemctl enable dnat-to-ns.path

Finally, shut down the TemplateVM:

sudo shutdown now

Creating the ProxyVM

Create an AppVM based on the TemplateVM you have just created. Set sys-firewall (or whatever FirewallVM you have connected to your sys-net) as the net qube. If you do not have such FirewallVM, use sys-net as the net qube. Next, go to the advanced tab and tick the provides network access to other qubes box.

Open the Mullvad VPN app. Go to Settings → VPN settings and toggle Local network sharing. Due to some strange interaction between qubes services and Mullvad VPN, certain apps will get internet connections while others do not if this toggle is not enabled. This toggle will not actually allow AppVMs connected to the ProxyVM to connect to the local network.

Enable Lockdown mode to ensure that the killswitch stays on even when the tunnel is disconnected.

Additional Assurances

For additional assurances against VPN leaks, you can optionally add these 2 lines to /rw/config/qubes-firewall-user-script:

nft add rule qubes custom-forward oifname eth0 counter drop
nft add rule ip6 qubes custom-forward oifname eth0 counter drop

This is not strictly necessary, as I have not observed any leaks with the VPN killswitch provided by the app.

Notes

With this current setup, the ProxyVM you have just created will be responsible for handling Firewall rules for the qubes behind it. This is not ideal, as this is still a fairly large VM, and there is a risk that Mullvad or some other apps may interfere with its firewall handling.

Instead, I highly recommend that you create a minimal Mirage FirewallVM and use it as a firewall behind the Mullvad ProxyVM. Other AppVMs then should use the Mirage Firewall as the net qube instead. This way, you can make sure that firewall rules are properly enforced.

Before we start…

Some of the sections will include mentions of unofficial builds of packages like linux‑hardened, akmod, hardened_malloc, and so on. These are not endorsements — they are merely to show that you have options to easily obtain and update these packages. Using unofficial builds of packages means adding more parties to trust, and you have to evaluate whether it is worth doing so for the potential privacy/security benefits or not.

During Installation

Drive Encryption

Most Linux distributions have an option within its installer for enabling LUKS full disk encryption. If this option isn’t set at installation time, you will have to backup your data and re-install, as encryption is applied after disk partitioning but before filesystem creation.

By default, cryptsetup does not set up authenticated encryption. If you are configuring partitioning using the command line, you can enable integrity with the --integrity argument.

Encrypted Swap

Consider using encrypted swap or ZRAM instead of unencrypted swap to avoid potential security issues with sensitive data being pushed to swap space. While ZRAM can be set up post-installation, if you want to use encrypted swap, you should set it up while partitioning your drive.

Depending on your distribution, encrypted swap may be automatically set up if you choose to encrypt your drive. Fedora uses ZRAM by default, regardless of whether you enable drive encryption or not.

Privacy Tweaks

NetworkManager Trackability Reduction

Most desktop Linux distributions including Fedora, openSUSE, Ubuntu, and so on come with NetworkManager by default to configure Ethernet and Wi-Fi settings.

wj25czxj47bu6q has detailed guide on trackability reduction with NetworkManager which I highly recommend you check out.

In short, if you use NetworkManager, add the following to your /etc/NetworkManager/conf.d/00-macrandomize.conf:

[device]
wifi.scan-rand-mac-address=yes

[connection]
wifi.cloned-mac-address=random
ethernet.cloned-mac-address=random

Then, restart your NetworkManager service:

sudo systemctl restart NetworkManager

Finally, set your hostname to localhost:

sudo hostnamectl hostname "localhost"

Note that randomizing Wi-Fi MAC addresses depends on support from the Wi-Fi card firmware.

Other Identifiers

There are other system identifiers which you may wish to be careful about. You should give this some thought to see if it applies to your threat model:

Username

Your username is used in a variety of ways across your system. Consider using generic terms like “user” rather than your actual name.

Machine ID

During installation a unique machine ID is generated and stored on your device. Consider setting it to a generic ID.

System Counting

Many Linux distributions send some telemetry by default to count how many systems are using their software. Consider disabling this depending on your threat model.

The Fedora Project offers a “countme” variable to much more accurately count unique systems accessing its mirrors without involving unique IDs. While currently disabled by default, you could add countme=false to /etc/dnf/dnf.conf in case the default changes in the future. On rpm‑ostree systems such as Fedora Silverblue and Kinoite, the countme option can be disabled by masking the rpm-ostree-countme timer.

openSUSE uses a unique ID to count systems, which can be disabled by emptying the /var/lib/zypp/AnonymousUniqueId file.

Zorin OS also uses a unique ID to count systems. You can opt‑out by running sudo apt purge zorin-os-census and optionally holding the package with sudo apt-mark hold zorin-os-census to avoid accidental reinstallation.

snapd (Snap) assigns a unique ID to your installation and uses it for telemetry. While this is generally not a problem, if your threat model calls for anonymity, you should avoid using Snap packages and uninstall snapd. Accidental reinstallation on Ubuntu can be prevented with sudo apt-mark hold snapd.

Of course, this is a non‑exhaustive list of telemetry on different Linux distributions. If you are aware of other tracking mechanisms used by these or other distributions, feel free to make a pull request or discussion post detailing them!

Keystroke Anonymization

You could be fingerprinted based on soft biometric traits when you use the keyboard. The Kloak package could help you mitigate this threat. It is available as a .deb package from Kicksecure’s repository and an AUR package.

With that being said, if your threat model calls for using something like Kloak, you are probably better off just using Whonix.

Application Confinement

Some sandboxing solutions for desktop Linux distributions do exist; however, they are not as strict as those found in macOS or ChromeOS. Software installed with distro package managers (DNF, APT, etc.) typically have no sandboxing or confinement whatsoever. Several projects which aim to tackle this problem are discussed here.

Flatpak

Flatpak aims to be a distribution-agnostic package manager for Linux. One of its main goals is to provide a universal package format which can be used in most Linux distributions. It provides some permission control. With that being said, Flatpak sandboxing is quite weak.

You can restrict applications further by setting Flatpak overrides. This can be done with the command line or by using Flatseal. To deny common dangerous Flatpak permissions globally, run the following commands:

sudo flatpak override --system --nosocket=x11 --nosocket=fallback-x11 --nosocket=pulseaudio --nosocket=session-bus --nosocket=system-bus --unshare=network --unshare=ipc --nofilesystem=host:reset --nodevice=input --nodevice=shm --nodevice=all --no-talk-name=org.freedesktop.Flatpak --no-talk-name=org.freedesktop.systemd1 --no-talk-name=ca.desrt.dconf --no-talk-name=org.gnome.Shell.Extensions
flatpak override --user --nosocket=x11 --nosocket=fallback-x11 --nosocket=pulseaudio --nosocket=session-bus --nosocket=system-bus --unshare=network --unshare=ipc --nofilesystem=host:reset --nodevice=input --nodevice=shm --nodevice=all --no-talk-name=org.freedesktop.Flatpak --no-talk-name=org.freedesktop.systemd1 --no-talk-name=ca.desrt.dconf --no-talk-name=org.gnome.Shell.Extensions

To allow Flatseal to function after applying the overrides above, run the following command:

flatpak --user override com.github.tchx84.Flatseal --filesystem=/var/lib/flatpak/app:ro --filesystem=xdg-data/flatpak/app:ro --filesystem=xdg-data/flatpak/overrides:create

Note that this only helps with lax, high‑level default permissions and cannot solve the low‑level issues like /proc and /sys access or an insufficient seccomp blacklist.

Some sensitive permissions of note:

• --share=network: network and internet access
• --socket=pulseaudio: the PulseAudio socket, grants access to all audio devices (including inputs)
• --socket=session-bus: access to the entire session bus, which can be used to break out of the sandbox by abusing dangerous D‑Buses.
• --socket=system-bus: access to the entire system bus, which can be used to break out of the sandbox by abusing dangerous D‑Buses.
• --device=all: access to all devices (including webcams)
• --talk-name=org.freedesktop.secrets: D‑Bus access to secrets stored on your keychain
• --talk-name=org.freedesktop.Flatpak: D‑Bus access to run flatpak run. This D‑Bus is a sandbox escape.
• talk-name=org.freedesktop.systemd1: D‑Bus access to systemd. This D‑Bus can be used to load in systemd services with arbitary code and run them.
• --talk-name=ca.desrt.dconf: D‑Bus access to dconf. This D‑Bus can be abused to run arbitary commands by changing key bindings.
• --talk-name=org.gnome.Shell.Extensions: D‑Bus access to install and manage GNOME shell extensions. This D‑Bus can be abused to add malicious extensions to GNOME. If an application works natively with Wayland (not running through the XWayland compatibility layer), consider revoking its access to X11 (--nosocket=x11) and the inter‑process communications (IPC) socket (--unshare=ipc) as well.

Many Flatpak apps ship with broad filesystem permissions such as --filesystem=home and --filesystem=host. Some applications implement the Portal API, which allows a file manager to pass files to the Flatpak application (e.g. VLC) without specific filesystem access privileges. Despite this, many of them still declare --filesystem=host.

My strategy to deal with this is to revoke all filesystem access first, then test if an application works without it. If it does, it means the app is already using portals and no further action is needed. If it doesn’t, then I start granting permission to specific directories.

As odd as this may sound, you should not enable (blind) unattended updates of Flatpak packages. If you or a Flatpak frontend (app store) simply executes flatpak update -y, Flatpaks will be automatically granted any new permissions declared upstream without notifying you. Enabling automatic updates with GNOME Software is fine, as it does not automatically update Flatpaks with permission changes and notifies the user instead.

Snap

Snap is another distribution-agnostic package manager with some sandboxing support. It is developed by Canonical and heavily promoted in Ubuntu.

Snap packages come in two variants: classic, with no confinement, and strictly confined, where AppArmor and cgroups v1 are used to facilitate sandboxing. If a snap uses classic confinement (“classic snap”), you are better off installing an equivalent package from your distribution’s repository if possible. If your system does not have AppArmor, then you should avoid Snap entirely. Additionally, most modern systems outside of Ubuntu and its derivatives use cgroups v2 by default, so you have to set systemd.unified_cgroup_hierarchy=0 in your kernel parameters to get cgroups v1 working.

Snap permissions can be managed via the Snap Store or Ubuntu’s custom patched GNOME Control Center.

On Ubuntu, you can replace various .deb packages with strictly confined snaps to minimize the attack surface. Some examples of these packages are the printing stack or ufw:

One caveat with Snap packages is that you only have control over the interfaces declared in their manifests. For example, Snap has separate interfaces for audio-playback and audio-record, but some packages will only declare the legacy pulseaudio interface which grants access to both play and record audio. Likewise, some applications may work perfectly fine with Wayland, but the package maintainer may only declare the X11 interface in their manifest. For these cases, you need to reach out to the maintainer of the snap to update the manifest accordingly.

Firejail

Firejail is another method of sandboxing. As it is a large setuid binary, it has a large attack surface which increase susceptibility to privilege escalation vulnerabilities. Madaidan offers additional details on how Firejail can worsen the security of your device.

If you do use Firejail, Firetools can help to quickly manage application permissions and launch sandboxed applications. Note that Firetools configurations are temporary with no option to save profiles for long‑term use.

Firejail can also confine X11 windows using Xpra or Xephr, something that Flatpak and Snap cannot do. I highly recommend checking out their documentation on X11 sandboxing.

One trick to launch applications with their Firejail profile is to use the sudo firecfg command. This will create a symlink /usr/local/bin/app_name_here pointing to Firejail, which will get used automatically by most .desktop files (which do not specify the absolute paths of their binaries) to use will launch the application through the symlink and have Firejail sandbox them this way. Of course, this is bypassable if you or some other applications launch the application directly from /usr/bin/app_name_here instead.

Mandatory Access Control

Common Linux mandatory access control (MAC) frameworks require policy files in order to force constraints on the system. The two most notable are SELinux (used on Android and Fedora‑based distributions) and AppArmor (used on Debian‑based distributions and most openSUSE variants).

Fedora includes SELinux preconfigured with some policies to confine system daemons (background processes). You should keep it in enforcing mode.

openSUSE gives the choice of SELinux or AppArmor during the installation process. You should stick to the default for each variant (AppArmor for Tumbleweed and SELinux for MicroOS). openSUSE’s SELinux policies are derived from Fedora.

Arch and its derivatives often do not come with a mandatory access control system, and you must manually install and configure AppArmor.

Note that, unlike Android, traditional desktop Linux distributions typically do not have full system Mandatory Access Control policies; only a few system daemons are actually confined.

Making Your Own Policies/Profiles

You can make your own AppArmor profiles, SELinux policies, bubblewrap profiles, and seccomp blacklists to have better confinement of applications. This is an advanced and sometimes tedious task, but there are various projects you could use as reference:

• Krathalan’s AppArmor profiles
• roddhjav’s AppArmor profiles
• noatsecure’s SELinux templates
• Seirdy’s bubblewrap scripts

Securing Linux Containers

If you’re running a server, you may have heard of containers. They are more common in server environments where individual services are built to operate independently. However, you may sometimes see them on desktop systems as well, especially for development purposes.

Docker is one of the most popular container solutions. It does not offer a proper sandbox, meaning there is a large kernel attack surface. You should follow the Docker and OCI Hardening guide to mitigate this problem. In short, there are things you can do like using rootless containers (via configuration changes or Podman), using a runtime which provides a pseudo‑kernel for each container (gVisor), and so on.

Another option is Kata Containers which masquerades virtual machines as containers. Each Kata container has its own kernel and is isolated from the host.

Security Hardening

Ubuntu Pro

If you are using Ubuntu LTS, consider subscribing to Ubuntu Pro. Canonical currently allows up to 5 machines with the free subscription.

You will also gain access to the Canonical Livepatch Service, which provides livepatching for certain kernel variants. Note that the Hardware Enablement (HWE) kernel is not supported.

While livepatching is less than ideal and I still recommend regularly rebooting your computer, it is quite nice to have.

Umask 077

On distributions besides openSUSE, consider changing the default umask for both root and regular users to 077 (symbolically, u=rwx,g=,o=). On openSUSE, a umask of 077 can break snapper and is thus not recommended.

On Ubuntu, the “Software & Update” application will not work properly if the repository lists in /etc/apt/sources.list.d have the 600 permission. You should make sure that they have the 644 permission instead.

The configuration for this varies per distribution, but typically it can be set in /etc/profile, /etc/bashrc, or /etc/login.defs.

Note that, unlike on macOS, this will only change the umask for the shell. Files created by running applications will not have their permissions set to 600.

Microcode Updates

You should make sure your system receives microcode updates to get fixes and mitigations for CPU vulnerabilities like Meltdown and Spectre.

Debian does not ship microcode updates by default, so be sure to enable the non-free repository and install the microcode package.

On Arch Linux, make sure you have the intel-ucode or amd-ucode package installed.

If you are looking to use the GNU Guix distribution, you should absolutely use the Nonguix channel or similar to get microcode updates.

Avoid the Linux-libre kernel at all costs, as they actively block loading binary‑only microcode.

Firmware Updates

Many hardware vendors offer firmware updates to Linux systems through the Linux Vendor Firmware Service. You can download and install updates using the following commands:

# Update metadata
fwupdmgr refresh

# Download and install firmware updates
fwupdmgr update

Some distributions like Debian do not have fwupd installed by default, so you should check for its existence on your system and install it if needed.

Several graphical frontends integrate with fwupd to offer firmware updates (GNOME Software, KDE Discover, Snap Store, GNOME Firmware, Pop!_OS Settings app). However, not all distributions offer this integration by default, so you should check your specific system and set up scheduled update notifications using systemd timers or cron if needed.

Firewall

A firewall may be used to secure connections to your system.

Red Hat distributions (such as Fedora) and openSUSE typically use firewalld. Red Hat maintains extensive documentation about firewalld and its graphical frontend firewall-config.

Distributions based on Debian or Ubuntu typically use the Uncomplicated Firewall (ufw). As the name suggests, it is much less sophisticated than firewalld. One notable missing feature is the ability to apply different firewall rules for different connections (see zones in firewalld).

You could also set your default firewall zone to drop packets. To implement this with firewalld (with the necessary exceptions for IPv6):

firewall-cmd --set-default-zone=drop
firewall-cmd --add-protocol=ipv6-icmp --permanent
firewall-cmd --add-service=dhcpv6-client --permanent
firewall-cmd --reload

On some distributions, it may be possible for applications running as a wheel or sudo user to make firewall changes through polkit. To prevent this, enable firewalld lockdown mode with sudo firewall-cmd --lockdown-on.

These firewalls use the netfilter framework and therefore cannot (without the help of strict mandatory access control) protect against malicious software running privileged on the system, which can insert their own routing rules that sidestep firewalld/ufw.

There are some per‑binary outbound firewalls such as OpenSnitch and Portmaster that you could use as well. But, just like firewalld and ufw, they are bypassable.

If you are using Flatpak packages, you can set an override to block network access. This is not bypassable.

If you are using non‑classic Snap packages on a system that supports proper confinement (both AppArmor and cgroups v1 present), you can use the Snap Store to revoke network permission. This is also not bypassable.

Kernel Hardening

There are several things you can do to harden the Linux kernel, including setting appropriate kernel parameters and blacklisting unnecessary kernel modules. If you are using Kicksecure or Whonix, most of this hardening is included by default. If you are using Debian, you should consider morphing it into Kicksecure.

This section extensively references Madaidan’s Linux Hardening Guide and in the interest of brevity does not repeat all the information contained there. You are strongly encouraged to read through the relevant sections of Madaidan’s guide (linked for convenience).

Runtime Kernel Parameters (sysctl)

See “2.2 Sysctl” in Madaidan’s guide.

Madaidan recommends that you disable unprivileged user namespaces due to the significant attack surface for privilege escalation. However, some software such as Podman and LXC relies on unprivileged user namespaces. If you wish to use such software, do not disable kernel.unprivileged_userns_clone. Note that this setting does not exist in the upstream kernel and is added downstream by some distributions.

On distributions other than Whonix and Kicksecure, you can copy the configuration file from Tommy’s repository.

Boot Parameters

See “2.3 Boot parameters” in Madaidan’s guide. If desired, formal documentation of boot parameters is available upstream.

Copy these parameters into your bootloader’s configuration. On rpm‑ostree distributions, make sure to use rpm-ostree kargs rather than editing GRUB configuration directly.

CPU mitigations

mitigations=auto,nosmt spectre_v2=on spectre_bhi=on spec_store_bypass_disable=on tsx=off kvm.nx_huge_pages=force nosmt=force l1d_flush=on spec_rstack_overflow=safe-ret gather_data_sampling=force reg_file_data_sampling=on

Simultaneous multithreading (SMT) has been the cause of numerous hardware‑level vulnerabilities and is thus disabled here. If the option is available, you should disable SMT/“Hyper‑Threading” in your firmware as well.

Note however that disabling SMT may have a significant performance impact — for this reason the popular linux‑hardened kernel for Arch does not disable SMT by default. Assess your own risk tolerance, and, if you choose to keep SMT enabled, simply remove all occurrences of nosmt and nosmt=force from these parameters.

Kernel

slab_nomerge init_on_alloc=1 init_on_free=1 pti=on vsyscall=none ia32_emulation=0 page_alloc.shuffle=1 randomize_kstack_offset=on debugfs=off oops=panic quiet loglevel=0

Kicksecure does not enforce either module.sig_enforce=1 or lockdown=confidentiality by default as they lead to a lot of hardware compatibility issues; consider enabling these if possible on your system. Additionally, mce=0 is no longer recommended.

Entropy generation

random.trust_cpu=off random.trust_bootloader=off

Some implementations of the RDRAND instruction (by which the CPU offers a random number generator to the OS) have proven to be vulnerable or outright defective. RDRAND is also impossible to audit, being part of the CPU itself.

As a precaution for the integrity of cryptographic operations, the CPU and bootloader should not be used as credited entropy sources. Note that this change will increase boot time.

Further reading:

• systemd: Random Seeds
• Madaidan: RDRAND
• Linux kernel mailing list
• Hacker News discussion
• NixOS discussion (also cites many additional sources)

DMA mitigations

intel_iommu=on amd_iommu=force_isolation efi=disable_early_pci_dma iommu=force iommu.passthrough=0 iommu.strict=1

Direct memory access (DMA) attacks can be mitigated via IOMMU and disabling certain kernel modules. Furthermore, strict enforcement of IOMMU TLB invalidation should be applied so devices will never be able to access stale data contents.

These parameters do not provide comprehensive DMA protection. In early boot (before the kernel has loaded), only the system firmware can enforce IOMMU and thus provide DMA protection. A DMA attack in early boot can patch the kernel in memory to completely undermine these parameters.

Note that disabling the busmaster bit on all PCI bridges during very early boot (efi=disable_early_pci_dma) can cause complete boot failure on certain systems with inadequate resources. Therefore, as always, ensure you have a fallback option to boot into the system whenever modifying any kernel parameters.

Further reading:

• IOMMU Groups, inside and out
• IOMMU introduction
• intel IOMMU driver analysis
• Avoiding gaps in IOMMU protection at boot
• Madaidan: DMA attacks

Kernel Modules

See “2.5.2 Blacklisting kernel modules” in Madaidan’s guide.

On distributions other than Whonix and Kicksecure, you can copy the configuration file from secureblue’s repository into /etc/modprobe.d/.

There are a few things in this config to keep in mind:

• Bluetooth is disabled. Comment out the install bluetooth and install btusb lines to use Bluetooth.
• Thunderbolt is disabled. Comment out the install thunderbolt line to use Thunderbolt devices.
• The cdrom and sr_mod modules are merely blacklisted; they can still be loaded at runtime with modprobe. If you have no intention to ever use CD‑ROM devices, they should be disabled by adding the lines install cdrom /bin/false and install sr_mod /bin/false to the config. (More about how this works on the ArchWiki)
• Apple filesystems are disabled. While generally fine on non‑Apple systems, if you are using an Apple device you must check the filesystem of your EFI partition and comment out the relevant install line, otherwise your Linux install will not boot. For example, comment out the install hfsplus line if your ESP filesystem is HFS+.

Restricting access to /proc and /sys

See “2.4 hidepid” and “2.7 Restricting access to sysfs” in Madaidan’s guide.

Disabling access to /sys without a proper whitelist will lead to various applications breaking. Developing such a whitelist will unfortunately be extremely tedious for most users. Kicksecure, and by extension Whonix, has the experimental proc-hidepid and hide-hardware-info services which do just this. From my testing, these work perfectly fine on minimal Kicksecure installations and both Qubes-Whonix-Workstation and Qubes-Whonix-Gateway.

linux-hardened

Some distributions like Arch Linux offer the linux‑hardened kernel package. It includes hardening patches and more security-conscious defaults.

linux‑hardened disables unprivileged user namespaces (kernel.unprivileged_userns_clone) by default. This may impact some software.

grsecurity

Grsecurity offers a set of kernel patches that attempt to improve security of the Linux kernel. Payment is required, but grsecurity is worth using if you have a subscription.

Hardened Memory Allocator

The hardened memory allocator (hardened_malloc) from GrapheneOS can be used on general Linux distributions, though only for some programs.

On Fedora and Red Hat Enterprise Linux, secureblue provides a Copr repository with both x86_64 and aarch64 architecture support. Divested Computing Group has a similar build for Fedora, but with only x86_64 support. Using secureblue’s repository is recommended, as the Divested repository is known to block certain IP addresses.

On Arch-based systems, hardened_malloc is available through the AUR.

Disabling XWayland

To disable XWayland with GNOME, create /etc/systemd/user/org.gnome.Shell@wayland.service.d/override.conf with the following content:

[Service]
ExecStart=
ExecStart=/usr/bin/gnome-shell --no-x11

Mountpoint Hardening

Consider adding the mount options nodev, noexec, and nosuid to mountpoints which do not need the respective capabilities. Typically, these can be applied to /boot, /boot/efi, and /var. These flags could also be applied to /home and /root, however noexec will prevent applications that require binary execution in those locations from working (including Flatpak and Snap).

It should be noted that noexec is not foolproof and actually quite easy to bypass.

If you use Toolbox, do not set any of these mount options on /var/log/journal. From my testing, the Toolbox container will fail to start if you have nodev, nosuid, or noexec on said directory. If you are on Arch Linux, you probably do not want to set noexec on /var/tmp, as some AUR packages will then fail to build.

Disabling SUID

SUID allows a user to execute an application as the owner of that application, which in many cases is the root user. Vulnerable SUID executables could lead to privilege escalation vulnerabilities.

It is desirable to remove SUID from as many binaries as possible; however, this takes substantial effort and trial and error on the user’s part, as some applications require SUID to function.

Kicksecure, and by extension Whonix, has an experimental permission hardening service and application whitelist to automate SUID removal from most binaries and libraries on the system. From my testing, these work perfectly fine on minimal Kicksecure installations and both Qubes-Whonix-Workstation and Qubes-Whonix-Gateway.

DNSSEC

Most Linux distributions do not enable DNSSEC by default. I recommend that you enable it to make sure that the responses to your DNS queries are authentic. You will need a DNS provider that supports DNSSEC. Ideally, you should use a VPN which provides this feature with its DNS servers so that you can also blend in with other people.

On systems with systemd-resolved, you can edit the /etc/systemd/resolved.conf file and add DNSSEC=yes to enable it. Run systemctl restart systemd-resolved after you are done editing to apply your configuration.

If you are a Whonix or Tails user, you can disregard setting up DNSSEC, as Tor DNS resolution does not support it. Alternatively, you can use a non-Tor resolver, though it is not recommended that you do this for an extended amount of time.

Time Synchronization

Most Linux distributions by default use the unencrypted and unauthenticated Network Time Protocol (NTP) for time synchronization. There are two ways to easily solve this problem:

• Configure Network Time Security (NTS) with chronyd
• Use Kicksecure’s sdwdate on Debian‑based distributions.

If you decide on using NTS with chronyd, consider using multiple, independent time providers and setting minsources to a value greater than 1.

GrapheneOS uses a quite nice chrony configuration for their infrastructure. I recommend that you replicate their chrony.conf on your system.

Next, enable the secommp filter for chronyd. On Fedora and Arch Linux, you will need to edit Chrony’s environment file in /etc/sysconfig/chronyd:

# Command-line options for chronyd
OPTIONS="-F 1"

On Ubuntu and Debian, the environment file is /etc/default/chrony, and the seccomp filter should already be enabled by default.

Pluggable Authentication Modules (PAM)

PAM’s settings can be hardened to improve authentication security (though keep in mind the bypassable nature of PAM as opposed to encryption).

On Red Hat distributions, you can use authselect to configure this, e.g.:

sudo authselect select <profile_id, default: sssd> with-faillock without-nullok with-pamaccess

On systems where pam_faillock is not available, consider using pam_tally2 instead.

If you have a YubiKey or another U2F/FIDO2 authenticator, you can use pam-u2f to implement two‑factor authentication for login. Make sure to use a hardcoded origin and appid as indicated in the ArchWiki. Do not use the default identifier pam://$HOSTNAME which will break if your hostname changes.

Storage Media Handling

Some Linux distributions and desktop environments automatically mount arbitary filesystems upon storage media insertion. This is a security risk, as an adversary can attach a malicious storage device to your computer to exploit vulnerable filesystem drivers.

This behavior is disabled by default on Whonix.

UDisks

GNOME users on systems with UDisks can mitigate this risk by running the following commands:

echo '[org/gnome/desktop/media-handling]
automount=false
automount-open=false' | sudo tee /etc/dconf/db/local.d/automount-disable

echo 'org/gnome/desktop/media-handling/automount
org/gnome/desktop/media-handling/automount-open' | sudo tee /etc/dconf/db/local.d/locks/automount-disable

sudo dconf update

This will disable automounting and prevent users from overriding that setting (without privileges).

Cinnamon uses the same configuration/commands except with cinnamon substituted in place of gnome. Other desktop environments based on GNOME 3 likely follow a similar pattern — use gsettings to investigate.

autofs

On older systems where autofs is used, you should mask the autofs service to disable this behavior.

USB Port Protection

To better protect your USB ports from attacks such as BadUSB and the infamous Hak5 USB Rubber Ducky, I recommend USBGuard. Documentation is available on the USBGuard website and ArchWiki.

If you are using linux‑hardened, you can alternatively use the deny_new_usb kernel parameter — see “Preventing USB Attacks with linux-hardened”.

Secure Boot

Secure Boot can be used to secure the boot process by preventing the loading of unsigned UEFI drivers and bootloaders.

One of the problems with Secure Boot, particularly on Linux, is that only the chainloader (shim), bootloader (GRUB), and kernel are verified in a typical setup. The initramfs is often left unverified and unencrypted, leaving the door open for an evil maid attack.

The firmware on most devices is also preconfigured to trust Microsoft’s keys for both Windows and third‑parties, leading to a large attack surface.

Enrolling your own keys

Please note that this procedure will brick some non‑compliant UEFI implementations. You should research your specific computer/motherboard, looking for reported successes and failures alike, before attempting. Ideally, you should be prepared to reprogram the EEPROM to a known‑good state if something goes catastrophically wrong. Integrated ‘BIOS flashback’ functionality may be an adequate recovery option.

To eliminate the need to trust the OEM’s keys, I recommend using sbctl.

First, you need to boot into your firmware interface and enter Secure Boot setup mode. Then boot back into Linux and follow the instructions to generate and enroll your own keys.

On certain hardware, this will not work. Instead, you will need to export the public key to your EFI partition and manually import it through your firmware interface:

openssl x509 -in /usr/share/secureboot/keys/db/db.pem -outform DER -out /boot/efi/EFI/fedora/DB.der

Unified Kernel Image

On most desktop Linux systems, it is possible to create a unified kernel image (UKI) that contains the kernel, initramfs, and microcode. This unified kernel image can then be signed with the keys created by sbctl.

For Fedora Workstation, you can follow Håvard Moen’s guide which covers sbctl installation, unified kernel image generation with dracut, and automatic signing with systemd‑boot.

On Arch, the process is very similar, though sbctl is already included in the official repositories and you will need to switch from mkinitcpio to dracut. Arch with linux‑hardened works well with sbctl, but some level of tedious pacman hooks are required for appropriately timing the re‑signing of all relevant files every time the kernel or bootloader is updated.

Afterwards, you need to use systemd-cryptenoll and pin your encryption key against certain PCRs to detect tampering against the firmware. At minimum, you should pin PCR 7 for Secure Boot polices. Personally, I pin PCR 0,1,2,3,5,7, and 14.

Whenever you manually generate a UKI, make sure that the kernel is from the distribution vendor, and that initramfs is freshly generated. Reinstall the kernel package if you have to.

In my opinion, this is the most straightforward setup. However, it does not appear to work well with specialized setups such as Fedora Silverblue/Kinoite. More testing is needed to see if they can be made to work.

Notes on Secure Boot

After setting up Secure Boot, you should password-protect your UEFI settings (sometimes called ‘supervisor’ or ‘administrator’ password) as it is good security practice. This does not protect against an attacker with a programmer however - you need to pin PCRs to detect tampering as mentioned above.

These recommendations can make you a little more resistant to evil maid attacks, but they do not constitute a proper verified boot process as found on Android, ChromeOS, or Windows.

The Obvious Problem

The obvious argument against badness enumeration is that there are so many threat actors out there, it is impossible to make a list to block all of them. Even when you make a magical list that somehow includes all of the threat actors that exist today, tomorrow a new threat actor will pop up and attack you anyways. Enumerating badness does not systematically solve the underlying problem. Instead, it is running away from the problem and hoping that a competent adversary will not come after you. Badness enumeration does not work, cannot work, has never worked, and will never work.

Adblocking Extensions

On top of the obvious problem mentioned above, there are various technical reasons why advertisement/tracker blocking extensions cannot provide privacy. One of which is the fact that tracking can be done without any scripts at all. For example, a website only needs to know your session ID using a cookie and save all logs associated with that ID. It can then analyze when you visited the website, how long you visited the website for, which page on the website you spent the most time on, what you looked at, and so on. Another problem is that a website can just host its own tracking code or proxy third party tracking code under its own domain. Just because your adblocker blocks connections to Google Analytics does not mean that you you are actually “safe” from Google Analytics at all. Even if you are successful in doing so, there is nothing stopping the website from sharing the analytics data it collected on its own with Google either.

“Okay, so adblockers are unreliable, but what is the harm?” you may ask.

The problem here is that adblockers (especially with Manifest v2) are highly privileged and have access to all of your data within the browser. All it takes is for the extension developer to turn malicious for your passwords, session ids, TOTP secrets, etc to get compromised. Even if you were to assume that the extension developer is trustworthy, one vulnerability within the extension could still be catastrophic. This is made worse by the fact that adblockers typically use third-party blocklists, extending trust to the blocklist maintainers to not exploit the extension should a vulnerability be found. The “uBlock, I exfiltrate” blog post describes in detail how a CSS injection vulnerability in uBlockOrigin lead to data exfiltration with one single bad filtering rule.

Overall, adblockers increase your attack surface for dubious privacy benefits. If you insist on getting an adblocker however, I highly recommend that you use purely declarative, permission less Manifest V3 ones like uBlock Origin Lite. While they do block fewer ads and trackers than their Manifest V2 counterparts and V3 extensions with “Read and change all your data on all websites”, they pose much less of a threat to your privacy and security while still providing the convenience of blocking annoyances.

DNS Filtering

DNS filtering solutions. while not having any negative impact on security, are trivially bypassable by just hosting the advertisement and trackers under the apex domain instead of a subdomain. For example, instead of hosting advertisement and trackers under ads.example.com, the webmaster can move them to be under example.com/ads and it would be impossible for DNS filters to block. Other bypasses include an application implementing its own DNS resolution instead of relying on the DNS servers set by the operating system, or connecting directly to certain IP addresses without any DNS resolution at all.

It should also be noted that websites can detect which DNS servers a visitor uses. You can look at DNSLeakTest as an example. Using non-network provided DNS servers adds to the fingerprint and make you more identifiable.

The best way to do DNS filtering is to use a VPN provider which has this feature built in like ProtonVPN, Mullvad, and IVPN in order to not standout from other users of the same VPN provider. Even then, DNS filtering is purely a convenience feature and cannot be relied on for privacy and security.

Antiviruses

Antiviruses are highly privileged processes with access to virtually all of your files and data, parsing through them trying to find something that matches a known bad signature. Beyond the fact that you need to trust the Antivirus company to not exfiltrate your sensitive data and that the signature list will never have all of the malware in existence, a vulnerable parser could lead to a system compromise. The Abusing File Processing in Malware Detectors for Fun and Profit research paper by Suman Jana and Vitaly Shmatikov discusses this in detail.

Here are some other examples of Antiviruses being attack surfaces on their own:

• Arbitrary Code Execution with Avast’s Javascript Interpreter
• Memory Corruption with Bitdefender
• Kaspersky in the Middle

The proper way to deal with untrusted applications is not to scan them with an Antivirus, but to confine them in such a way that even if they were malicious, they cannot do much damage at all. This has already been achieved on secure mobile operating systems like Android and iOS with their application sandbox. Typically, attacks against these systems require an exploit chain against the operating system, or for the user to actually mess up and grant an app access to sensitive data. On desktop operating systems, you should utilize virtualization to contain untrusted applications in their own virtual machine. This can be done with a system like Qubes OS, the Windows Sandbox, or just general KVM / HyperV virtual machines.

Default Permit

Surprisingly (or unsurprisingly), the The Six Dumbest Ideas in Computer Security article from almost 20 years ago still holds true today. It explains the problem with Default Permit better than I ever could. In short, when setting up a Firewall or some sort of filter list, it is better to start out by blocking everything, then allowing only the traffic that you need. That way, you don’t have to worry about applications that you didn’t care enough to block turning out to be vulnerable. Sometimes, “goodness enumeration” is the solution to the problem.

Conclusion

By now, I hope I have clearly explained why badness enumeration is never the solution to the problem. Sometimes, it can be a nice-to-have thing, like a VPN provider blocking advertisements and trackers on the DNS level to make the web experience more enjoyable. Other times, it can be harmful to your privacy and security, like with a malicious/vulnerable extension or antivirus. The important thing to keep in mind is that you cannot rely on badness enumeration for true privacy and security, and you should always be aware of the privacy and security implications that certain options may entail.

Lokinet is an Internet overlay network utilizing onion routing to provide anonymity for its users, similar to Tor network. This post will go over how to set it up on Qubes OS.

Before we start…

This post should not be considered an endorsement of Lokinet in any shape or form. Lokinet is currently not in a good state — it has not had a public release since 2022, and most free public exit nodes have gone offline. According to the developers, they are doing major rewrites of the code, and it should not be used in production at the moment.

Creating the TemplateVM

Currently, the Lokinet client seems to work well with only Debian-based distributions. This means that our template will have to be one of the Debian-based ones. Personally, I use this script to trim down the Debian GNOME template and convert it to Kicksecure. Kicksecure reduces the attack surface of Debian with a substantial set of hardening configurations, and a nice feature to go with an anonymity network like Lokinet is Boot Clock Randomization which helps defend against time-based denonymization attacks.

Start by creating the bind directories for Lokinet’s configurations:

sudo mkdir -p /etc/qubes-bind-dirs.d
echo 'binds+=( '\'''/etc/loki''\'' )' | sudo tee /etc/qubes-bind-dirs.d/50_user.conf 

Next, add the Oxen PGP key and the Lokinet template. We will deviate from the official documentation and pin the PGP key to only be used for this repository:

curl --proxy http://127.0.0.1:8082 https://deb.oxen.io/pub.gpg | sudo tee /usr/share/keyrings/oxen.gpg
echo "deb [signed-by=/usr/share/keyrings/oxen.gpg] https://deb.oxen.io $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/oxen.list

Next, lokinet and resolvconf. lokinet-gui was very buggy when I tested it inside my VM, so I recommend installing only the daemon. resolvconf is used by the Lokinet init script but is not declared as a dependency for some reason, so you have to manually install it as well:

sudo apt update
sudo apt install lokinet-gui resolvconf

To work around the problem where Qubes overrides the DNS configuration at boot, create /etc/systemd/system/lokinet-dns-fix.service with the following content:

[Unit]
Description=Fix DNS for Lokinet
After=qubes-network-uplink.service

[Service]
Type=oneshot
ExecStart=/usr/bin/rm /etc/resolv.conf
ExecStart=/usr/bin/ln -s /run/resolvconf/resolv.conf /etc/resolv.conf

[Install]
WantedBy=multi-user.target

Enable the lokinet-dns-fix service:

sudo systemctl enable lokinet-dns-fix

At this stage, you can install any .deb app you want to use with Lokinet in the TemplateVM. I have been unable to get DNS working properly with Lokinet as a network VM, so for now we will have to use a Lokinet in each individual AppVM.

Finally, shut down the TemplateVM:

sudo shutdown now

Creating the AppVM

Create an AppVM based on the TemplateVM you have just created. Set sys-firewall (or whatever FirewallVM you have connected to your sys-net) as the net qube. If you do not have such FirewallVM, use sys-net as the net qube.

Edit /etc/loki/lokinet.ini and add the exit node you want to use. At the moment, the only free exit node that I am aware of is euroexit.loki:

[network]
exit-node=euroexit.loki

Android Devices

Recommended Phones

Google Pixel phones are the only devices I would recommend for purchase. Pixel phones have stronger hardware security than any other Android devices currently on the market, due to proper AVB support for third-party operating systems and Google’s custom Titan security chips acting as the Secure Element.

When purchasing a device, you should buy one as new as possible. The software and firmware of mobile devices are only supported for a limited time, so buying new extends that lifespan as much as possible. Also, beginning with the Pixel 8 and 8 Pro, Pixel devices receive a minimum of 7 years of guaranteed security updates, ensuring a much longer lifespan compared to the 2-4 years competing OEMs typically offer.

Phones to Avoid

Avoid buying the Fairphone 4, which only has just over 2 years of full security updates since its release date despite them advertising 6 years of support. This is because the System on a Chip they use (Snapdragon 750G) only has 3 years of support from Qualcomm, and the SoC was already old when the phone came out. This is not to mention, the Fairphone 4 uses the Android Verified Boot Test Key as their OEM keys, effectively making Verified Boot useless. In general, you should check for how long the SoC a phone uses is supported for and not blindly trust the phone manufacturer’s claims.

You should also avoid buying phones preloaded with /e/ OS (sometimes branded as the Murena phones). /e/ OS in itself is extremely insecure, engaging in security bad practices that include, but are not limited to, not supporting verified boot; shipping userdebug build; shipping months-old version of Chromium; and bundling years-old version of Orbot into their operating system, then marketing it as “Advanced Privacy”. There was also a recent incident where their cloud service mishandled session keys and gave users access to other’s files, then proceeded to mislead users that the server cannot see their files, despite there being no end-to-end encryption.

You should also be very wary of low quality privacy branded phones like the Freedom Phone, BraX2 Phone, Volta Phone, and the like. These are cheap Chinese phones with the Mediatek Helio P60 from 2018, which has already reached or is near end-of-life. Needless to say, you should also avoid any vendor who claims they are Zero-day proof like this:

Android-based Operating Systems

In certain cases, installing a custom Android-based operating system can help increase your privacy and security. This is rather tricky, however, as the vast majority of these operating systems (a.k.a. “custom ROMs”) do exactly the opposite: break the Android security model, thereby ruining your security while providing no or dubious privacy benefits.

I have written a detailed post on selecting your Android-based operating system, which you can find here.

TLDR: If you are using a modern Pixel, install GrapheneOS. Otherwise, stick to your stock operating system. Do not blindly use an OS just because it is advertised as “degoogled”.

Use New Android Versions

It’s important to not use an end-of-life version of Android. Newer versions of Android receive not only security updates for the operating system but also important privacy enhancing updates too. For example, prior to Android 10, any apps with the READ_PHONE_STATE permission could access sensitive and unique serial numbers of your phone such as IMEI, MEID, your SIM card’s IMSI, whereas now they must be system apps to do so. System apps are only provided by the OEM or Android distribution.

Do Not Root Your Phone

Rooting Android phones can decrease security significantly as it weakens the complete Android security model. This can decrease privacy should there be an exploit that is assisted by the decreased security. Common rooting methods involve directly tampering with the boot partition, making it impossible to perform successful Verified Boot. Apps that require root will also modify the system partition meaning that Verified Boot would have to remain disabled. Having root exposed directly in the user interface also increases the attack surface of your device and may assist in privilege escalation vulnerabilities and SELinux policy bypasses.

Use a diceware passphrase, avoid pattern unlock

On Android, the phone unlock (Password, Pin, Pattern) is used to protect the encryption key for your device. Thus, it is vital that your unlock secret is secure and can withstand Bruteforce attacks.

Pattern unlock is extremely insecure and should be avoided at all costs. This is discussed in detail in the Cracking Android Pattern Lock in Five Attempts research paper.

If you trust the hardware enforced rate limiting features (typically done by the Secure Element or Trusted Execution Environment) of your device, a 8+ digit PIN may be sufficient.

Ideally, you should use a randomly generated passphrase of 8 words or longer to secure your phone. These are practically impossible to bruteforce with current technology, regardless of the efficacy of any ratelimiting that may be present.

Setup Auditor

Auditor provides attestation for GrapheneOS phones and the stock operating systems on a number of devices. It uses hardware security features to make sure that the firmware and operating system have not been downgraded or tampered with.

Attestation can be done locally by pairing with another Android 8+ device or remotely using the remote attestation service. To make sure that your hardware and operating system is genuine, perform local attestation immediately after the device has been set up and before connecting to the internet.

Use Global Toggles

Modern Android devices have global toggles for disabling Bluetooth and location services. Android 12 introduced toggles for the camera and microphone. When not in use, you should disable these features. Apps cannot use disabled features (even if granted individual permissions) until re-enabled.

Manage Android Permissions

Permissions on Android grant you control over what apps are allowed to access. Google regularly makes improvements on the permission system in each successive version. All apps you install are strictly sandboxed, therefore, there is no need to install any antivirus apps.

You can manage Android permissions by going to Settings → Privacy → Permission Manager. Be sure to remove from apps any permissions that they do not need.

Enable VPN Killswitch

Android 7 and above supports a VPN killswitch and it is available without the need to install third-party apps. This feature can prevent leaks if the VPN is disconnected. It can be found in Settings → Network & internet → VPN → Block connections without VPN.

Connectivity Check

Connectivity checks on Android do not go through the VPN tunnel (they are not supposed to anyway). This is generally not a cause for concern, however, you should be aware that Google and a network observer on your internet service provider (ISP)’s network can see that there is an Android device with your actual IP address.

On GrapheneOS, connectivity checks by default are done with GrapheneOS’s own servers, instead of with Google ones. A network observer on your ISP’s network can see that you are using a GrapheneOS device. If you are using a VPN and want to appear like a regular Android device to your ISP, go to Settings → Network & internet → Internet connectivity check and select Standard (Google) instead. Note that this will not stop a determined adversarial ISP from finding out you are not using stock OS through your DNS fallback.

If you want to, you can disable connectivity check altogether. Note that this will stop captive portals from working.

• On GrapheneOS, go to Settings → Network & internet → Internet connectivity check and select Disabled.
• On other Android-based operating systems, you can disable captive portal via ADB.

To disable:

adb shell settings put global captive_portal_mode 0

To re-enable:

adb shell settings delete global captive_portal_mode

Restrict USB Peripherals

USB peripherals should be disabled or set to only be allowed when the device is unlocked if possible.

On GrapheneOS, you can adjust this setting in Settings → Security → USB accessories. The OS defaults to “Allow new USB peripherals when unlocked”.

Media Access

Quite a few applications allow you to “share” a file with them for media upload. If you want to, for example, tweet a picture to Twitter, do not grant Twitter access to your “media and photos”, because it will have access to all of your pictures then. Instead, go to your file manager (documentsUI), hold onto the picture, then share it with Twitter.

If you are using GrapheneOS, you should utilize the Storage Scopes feature to force apps that request broad storage access permission to function with scoped storage.

User Profiles

Multiple user profiles can be found in Settings → System → Multiple users and are the simplest way to isolate in Android.

With user profiles, you can impose restrictions on a specific profile, such as: making calls, using SMS, or installing apps on the device. Each profile is encrypted using its own encryption key and cannot access the data of any other profiles. Even the device owner cannot view the data of other profiles without knowing their password. Multiple user profiles are a more secure method of isolation.

Work Profile

Work Profiles are another way to isolate individual apps and may be more convenient than separate user profiles.

A device controller such as Shelter is required, unless you’re using CalyxOS which includes one.

The work profile is dependent on a device controller to function. Features such as File Shuttle and contact search blocking or any kind of isolation features must be implemented by the controller. You must also fully trust the device controller app, as it has full access to your data inside of the work profile.

This method is generally less secure than a secondary user profile; however, it does allow you the convenience of running apps in both the work and personal profiles simultaneously.

Baseband Modem Attack Surface Reduction

By default, your baseband modem will typically be set to support just about every generation of mobile cellular technology, from 2G to 5G. This exposes a large attack surface.

You can reduce this attack surface by limiting the baseband modem to using just the generation that it needs. In most cases, this would be 4G/LTE.

GrapheneOS has the LTE only mode exposed in settings. You can set this by going to Settings → Internet → Your carrier name → Preferred network type → LTE Only.

If your Android-based operating system does not expose this setting in the Settings app, or if you want to set your baseband modem to a less restrictive mode, dial *#*#4636#*#* then hit Phone information. Here, you can set preferred network type to just the generations that you intend to use. For example, if you only want to use 5G and 4G, you can set it to NR/LTE.

Carrier Tracking

Carriers can track your coarse location through various means. At minimum, you need to use airplane mode to turn off the baseband modem, and turn off Wifi-calling which bypasses the system VPN. There may also be additional connections to the carrier’s servers outside of the VPN tunnel, so you need to use Wireshark to verify this for your specific setup.

Exact behavior does differ across SoCs and may vary between carriers as well, so I cannot give exact instructions for every setup. On a Google Pixel 7 Pro running GrapheneOS, you need to do the following:

• Disable Wi‑Fi calling.

• Disable the SIMs/eSIMs in Settings → Network & internet → SIMs. On GrapheneOS, if you are using an eSIM, you will need to enable privileged eSIM management. With certain carriers, there will be an ePDG server defined which the operating system will connect to outside of a VPN tunnel. While unlikely, a malicious carrier can track a user by giving them a unique ePDG server.

• Turn on airplane mode. This will turn off the modem and disable all transmission to cell towers. Note that simply removing SIM cards is not enough — your phone will still connect to cellular networks to permit emergency calling.

• Disable privileged eSIM management after you have disabled all of the eSIMs. With certain carriers, the eSIM management app will connect to the provisioning server to check for eSIM update, even if the eSIMs are disabled.

Where to Get Your Applications

GrapheneOS App Store

GrapheneOS’s app store is available on GitHub. It supports Android 12 and above and is capable of updating itself. The app store has standalone applications built by the GrapheneOS project such as the Auditor, Camera, and PDF Viewer. If you are looking for these applications, I highly recommend that you get them from GrapheneOS’s app store instead of the Play Store, as the apps on their store are signed by the GrapheneOS’s project own signature that Google does not have access to.

Aurora Store

The Aurora Store is a proxy for the Google Play Store. It is great for privacy in the sense that it automatically gives you a disposable account to download apps, and it works on Android-based distributions that do not support Google Play Services. That being said, it lacks security features like certificate pinning and does not support Play Asset Delivery.

My recommendation is to stick with the Google Play Store unless your threat model calls for not logging into Google Services at all.

F-Droid

F-Droid, despite being often recommended in the privacy community, has various security deficiencies. You can read more about them here.

I do not recommend that you use F-Droid at all unless you have no other choice to obtain certain apps. In some rare cases, there may be some apps which require the F-Droid version to work properly without Google Play Services. If you do end up using F-Droid, I highly recommend that you avoid the official F-Droid client (which is extremely outdated and targets API level 25) and use a more modern client with seamless updates such as NeoStore. You should also avoid using the official F-Droid repository as much as possible and stick to the F-Droid repositories hosted by the app developers instead.

GitHub

You can also obtain your apps directly from their GitHub repositories. In most cases, there would be a pre-built APK for you to download. You can verify the signature of the downloaded using apksigner:

• Install Android Studio which includes apksigner. On macOS, apksigner can be found at ~/Library/Android/sdk/build-tools/<version>/apksigner.
• Run apksigner verify --print-certs --verbose myCoolApp.apk to verify the certificate of the apk.

After you have verified the signature of the apk and installed it on your phone, there are several strategies you can use to keep the application up-to-date.

The first strategy is to add the atom feed of the application’s release page to an RSS Reader like ReadYou to get notified of new releases. You will still need to download and install the new releases manually. If you are confused, here is a video that could help with this process:

The second strategy is to use the IzzyOnDroid F-Droid repository with a modern F-Droid client like NeoStore, as mentioned above. The IzzyOnDroid repository pulls new releases from various GitHub repositories to their server, which can then be automatically downloaded and installed by NeoStore. The downside of this strategy is that not every application on GitHub is on IzzyOnDroid, and sometimes IzzyOnDroid fails to pull a new release, resulting in you not getting any updates at all.

It should be noted that since Android has automatic signature checking for existing applications on the system; that is, you only need to manually check the signature of the apk the first time you install an application. If you do use IzzyOnDroid to update applications, you will need to manually confirm the first update of an application to authorize the NeoStore as the installation source. After that, future updates will be seamless.

Google

If you are using a device with Google services, either with the stock operating system or an operating system that safely sandboxes Google Play Services like GrapheneOS, there are a number of additional changes you can make to improve your privacy.

Enroll in the Advanced Protection Program

If you have a Google account we suggest enrolling in the Advanced Protection Program. It is available at no cost to anyone with two or more hardware security keys with FIDO2 support.

The Advanced Protection Program provides enhanced threat monitoring and enables:

• Stricter two factor authentication; e.g. that FIDO2 must be used and disallows the use of SMS OTP, TOTP and OAuth
• Only Google and verified third-party apps can access account data
• Scanning of incoming emails on Gmail accounts for phishing attempts
• Stricter safe browser scanning with Google Chrome
• Stricter recovery process for accounts with lost credentials

If you use non-sandboxed Google Play Services (common on stock operating systems), the Advanced Protection Program also comes with additional benefits such as:

• Not allowing app installation outside of the Google Play Store, the OS vendor’s app store, or via adb
• Mandatory automatic device scanning with Play Protect
• Warning you about unverified applications

Google Play System Updates

In the past, Android security updates had to be shipped by the operating system vendor. Android has become more modular beginning with Android 10, and Google can push security updates for some system components via the privileged Play Services.

If you have an EOL device shipped with Android 10 or above (shipped beginning 2020), you may be better off sticking with the stock OS in the short term as opposed to running an insecure alternative operating system. This will allow you to receive some security fixes from Google, while not violating the Android security model and increasing your attack surface. You should still upgrade to a supported device as soon as possible.

Disable Advertising ID

All devices with Google Play Services installed automatically generate an advertising ID used for targeted advertising. Disable this feature to limit the data collected about you.

On Android distributions with Sandboxed Google Play, go to Settings → Apps → Sandboxed Google Play → Google Settings → Ads, and select Delete advertising ID.

On Android distributions with privileged Google Play Services (such as stock OSes), the setting may be in one of several locations:

• Settings → Google → Ads
• Settings → Privacy → Ads

You will be given the option to either delete your advertising ID or Opt out of interest-based ads — this varies between OEM distributions of Android. If presented with the option to delete the advertising ID, choosing that is preferred. If not, then make sure to opt out and reset your advertising ID.

Google Messages

Google is currently pushing for the adoption of RCS with end to end encryption to compete with iMessage. On certain Android devices, especially Google Pixels with stock OS, Google Messages is set as the default SMS app to provide this feature.

If you are on an OS with Play Services installed, I highly recommend that you use Google Messages as the SMS app to get opportunistic end to end encryption with your contacts. It works fairly well on GrapheneOS with Sandboxed Play Services, too.

You can disable telemetry in Google Messages by tapping the profile in the top right → Messages settings → Help Improve Messages and toggling it off. There are also some other configurations in Messages settings → RCS chats that you might want to go over, such as Show typing indicators or Send read receipts.

If you have trouble connecting to RCS, try disabling your VPN and the VPN killswitch first, then reconnect to RCS. Once you have connected to the server, you can re-enable your VPN and the killswitch, and it should work just fine across reboots. I am not sure what is causing this issue, but it might be related to this bug.

Google Fi

Google Fi provides opportunistic end‑to‑end encryption for phone calls between Fi users on Android and includes a VPN service. Fi also implements a unique privacy‑bolstering virtual carrier network (VCN) architecture on supported devices, but it is temporarily disabled.

This is not without its caveats:

• Google Fi requires Play Services and the Fi app to work properly. Without Play Services, all of the features mentioned above, along with visual voicemail, will not work. SMS messages will have random strings added at the end of each of them.
• The Google Fi app needs to be installed in the owner profile for SIM/eSIM activation.
• Google Fi Wi‑Fi calling does not work behind a VPN with the killswitch enabled in the owner profile.

If you live in the United States and use the stock operating system, I highly recommend using Google Fi as your carrier to take advantage of the end to end encrypted calls and Fi VPN. People using a Pixel 4 and above will benefit the most from the VCN as mentioned.

If you use GrapheneOS and do not mind installing Sandboxed Play Services, Fi is still a better option than other providers thanks to Google’s general good security practices and the fact that you can enroll in the Advanced Protection Program to have much better protection for your account. Some other providers do not even have multi-factor authentication support, and most will not let you enforce FIDO2 as the authentication method.

Should I use a VPN?

Yes, unless you are already using Tor. A VPN does two things: shifting the risks from your Internet Service Provider to itself and hiding your IP from a third-party service.

VPNs cannot encrypt data outside of the connection between your device and the VPN server. VPN providers can see and modify your traffic the same way your ISP could. And there is no way to verify a VPN provider’s “no logging” policies in any way.

However, they do hide your actual IP from a third-party service, provided that there are no IP leaks. They help you blend in with others and mitigate IP based tracking.

What about encryption?

Encryption offered by VPN providers are between your devices and their servers. It guarantees that this specific link is secure. This is a step up from using unencrypted proxies where an adversary on the network can intercept the communications between your devices and said proxies and modify them. However, encryption between your apps or browsers with the service providers are not handled by this encryption.

In order to keep what you actually do on the websites you visit private and secure, you must use TLS. This will keep your passwords, session tokens, and queries safe from the VPN provider. Consider enabling “HTTPS everywhere” in your browser to mitigate downgrade attacks like SSL Strip.

Should I use encrypted DNS with a VPN?

Unless your VPN provider hosts the encrypted DNS servers, no. Using DOH/DOT (or any other form of encrypted DNS) with third-party servers will simply add more entities to trust and does absolutely nothing to improve your privacy/security. Your VPN provider can still see which websites you visit based on the IP addresses and other methods. Instead of just trusting your VPN provider, you are now trusting both the VPN provider and the DNS provider.

A common reason to recommend encrypted DNS is that it helps against DNS spoofing. However, your browser should already be checking for TLS certificates with HTTPS and warn you about it. If you are not using HTTPS, then an adversary can still just modify anything other than your DNS queries and the end result will be little different.

Needless to say, you shouldn’t use encrypted DNS with Tor. This would direct all of your DNS requests to a single entity and make you stand out from the rest of Tor users who would be using the exit node’s DNS configuration.

What if I need anonymity?

VPNs cannot provide anonymity. Your VPN provider will still see your real IP address, and often has a money trail that can be linked directly back to you. You cannot rely on “no logging” policies to protect your data. Use Tor instead.

Should I use Tor over VPN?

By using Tor over VPN, you are creating essentially adding an extra node in the beginning of the circuit. This provides zero additional benefits to you, while increasing the latency of your connection dramatically. If you wish to hide your Tor usage from your ISP or your government, consider using Tor bridges with obfuscation methods.

What about VPN over Tor?

By using VPN over Tor, you are adding an extra node at the end of a circuit, which is always controlled by the same entity. If you pay for the VPN using the traditional banking system, it essentially breaks the anonymity that the three hops in front of it would provide. If you pay for the VPN subscription using cash or a private cryptocurrency like Monero, your privacy is reduced to that of pseudonymity, since the VPN provider still knows the connections being made are from the same individual, they just do not know who you really are. Even if you are using a free VPN, you would still break Stream Isolation, one of Tor’s important anonymity features. There are very few use cases where it would make sense to add a VPN server at the end of the chain.

What about VPN providers that provide Tor nodes?

Do not use that feature. The point of using Tor is that you do not trust your VPN provider. Currently Tor only supports the TCP protocol. Through Tor, UDP (used in WebRTC for voice and video sharing, the new HTTP3/QUIC protocol, etc), ICMP and other packets will be dropped. To compensate for this, VPN providers typically will route all non-TCP packets through their VPN server (your first hop). This is the case with ProtonVPN. Additionally, like VPN over Tor, you lose control over other important Tor features like Stream Isolation. Thus, this feature should be viewed as a convenient way to access the Tor Network, not to stay anonymous. For true anonymity, use the Tor Browser Bundle, TorSocks, or a Tor gateway.

When are VPNs useful?

A VPN is useful in a variety of scenarios, such as:

• Hiding your traffic from only your Internet Service Provider.
• Hiding your downloads (such as torrents) from your ISP and anti-piracy organizations.
• Hiding your IP from third-party websites and services, preventing IP based tracking.

When you buy an Android phone, the device’s default operating system often comes with invasive integration with apps and services that are not part of the Android Open-Source Project. An example of such is Google Play Services, which has irrevocable privileges to access your files, contacts storage, call logs, SMS messages, location, camera, microphone, hardware identifiers, and so on. These apps and services increase the attack surface of your device and are the source of various privacy concerns with Android.

This problem could be solved by using a custom Android-based operating system that does not come with such invasive integration. Unfortunately, many custom Android-based operating systems often violate the Android security model by not supporting critical security features such as AVB, rollback protection, firmware updates, and so on. Some of them also ship userdebug builds which expose root over ADB and require more permissive SELinux policies to accommodate debugging features, resulting in a further increased attack surface and weakened security model.

When choosing a custom Android-based operating system, you should make sure that it upholds the Android security model. Ideally, the custom operating system should have substantial privacy and security improvements to justify adding yet another party to trust.

Baseline Security

Verified Boot

Verified Boot is an important part of the Android security model. It provides protection against evil maid attacks, malware persistence, and ensures security updates cannot be downgraded with rollback protection.

On Android, only your data (inside of the /data partition) is encrypted, and the operating system files are left unencrypted. Verified Boot ensures the integrity of the operating system files, thereby preventing an adversary with physical access from tampering or installing malware on the device. In the unlikely case that malware is able to exploit other parts of the system and gain higher privileged access, Verified Boot will prevent and revert changes to the system partition upon rebooting the device.

Unfortunately, OEMs are only obliged to support Verified Boot on their stock Android distribution. Only a few OEMs such as Google support custom AVB key enrollment on their devices. Additionally, some AOSP derivatives such as LineageOS or /e/ OS do not support Verified Boot even on hardware with Verified Boot support for third-party operating systems. In most cases, trading off verified boot for simply not having Play Services is not worth it.

Firmware Updates

Firmware updates are critical for maintaining security and without them your device cannot be secure. OEMs have support agreements with their partners to provide the closed-source components for a limited support period. These are detailed in the monthly Android Security Bulletins.

On a custom Android distribution, it is the responsibility of the operating system vendor to extract the firmware from the stock operating system, test it against their Android builds, then ship them to the user.

Unfortunately, many custom Android distributions, including extremely popular ones like LineageOS and /e/ OS do not ship firmware updates for most of their supported device. Instead, they expect the user to keep track of stock OS updates, extract and flash the firmware themselves. Beyond the lack of testing, this is extremely burdensome and not feasible for most end users and is yet another reason to not use these distributions.

Patch Levels

As the Android Security Bulletins is updated every month, Android-Based operating systems are expected to apply all security fixes before the next bulletin update comes out. Beside extracting the firmware, testing it and shipping it to the end user as described above, the AOSP based system also need to be updated.

This is a particularly challenging thing to do, especially around the time of a new major Android release since there are a lot of changes. Sometimes, newer firmware versions require newer major versions of AOSP, and if the developer takes too long to update their base operating system to the next major AOSP version, they cannot ship firmware updates either, leaving users vulnerable.

This has happened to CalyxOS during the Android 11 to Android 12 transition. It took them 4 months to update to Android 12; and during those 4 months, they could not ship any firmware updates at all, leaving the user vulnerable during that time period.

It would be much better if you just stick to the stock operating system (which got updated to Android 12 shortly after the AOSP 12 release) instead of using a custom operating system which could not keep up with updates as described.

Chromium Webview Updates

Android comes with a system webview, a component that many apps rely on to use as part of their activity layout. It effectively behaves like a minimal browser, opening random websites with arbitrary code the internet. Thus, it is very important that this component is consistently kept up to date.

Some Android-based operating systems, including ones like CalyxOS, often fall behind on security updates for this component. Particularly, this has gotten so bad that they actually fell behind for 3 months back in January 2022 and 2 months in June 2022. It is a good indication that these operating systems cannot keep up with security updates and should not be used.

User Builds

As mentioned above, userdebug builds expose root over ADB and require more permissive SELinux policies to accommodate debugging features. They violate the Android security model and are really only meant for developers to test out their android builds during development.

End users should be using the production user builds. Distributions which do not ship user builds like LineageOS or /e/ OS should be avoided, especially if your device has not reached end of life.

SELinux in Enforcing Mode

SELinux is a critical part of the Android security model, having the Linux kernel enforcing confinement for all processes, including system processes running as root.

In order for a system to be secure, it must have SELinux in Enforcing mode, accompanied by fine-grained SELinux policies.

Unfortunately, many custom Android-based operating system builds (especially unofficial LineageOS builds) disables SELinux or set it into Permissive mode. You can check whether SELinux is in enforcing mode or not by executing getenforce in the ADB shell (the expected output is Enforcing). You should avoid any Android-based operating system builds that do not have SELinux in enforcing mode at all cost.

Recommended Android-Based Operating Systems

Currently, I am only aware of two Android-based operating systems that should be used over the stock operating systems:

GrapheneOS

GrapheneOS is the only custom Android-based operating system you should buy a new phone for. It provides additional security hardening and privacy improvements over the stock operating system from Google. It has a hardened memory allocator, network and sensor permissions, and various other security feature. GrapheneOS also comes with full firmware updates and signed builds, so verified boot is fully supported. Here is a quick video demonstrating the network and sensors permissions:

For usability purposes, GrapheneOS supports Sandboxed Google Play, which runs Google Play Services fully sandboxed like any other regular app. This means you can take advantage of most Google Play Services, such as push notifications, while giving you full control over their permissions and access, and while containing them to a specific work profile or user profile of your choice. Most interestingly, the In-app Billing API, Google Play Games, Play Asset Delivery, FIDO2 all work exceptionally well. Most Advanced Protection Program features, except for Play Protect and restricted app installation, also work.

Because GrapheneOS does not grant any Google Apps and Services apart from the opt-in eSIM action app privileged access to the system, Play Protect cannot disable or uninstall known malicious applications when it detects them. As for restricted app installation, this feature is not that useful on stock operating system anyways, since it is bypassable with adb push.

Recently, GrapheneOS has also added the Storage Scopes feature, allowing you to force apps that request broad storage access permission to function with scoped storage. With this new feature, you no longer have to grant certain apps access to all of your media or files to use them anymore. You can watch a video of Storage Scope in action here:

Currently, Google Pixel phones are the only devices that meet GrapheneOS’s hardware security requirements.

DivestOS

DivestOS is a great aftermarket operating system for devices that have gone end-of-life or are near end-of-life. Note that this is a harm reduction project, ran by one developer on the best effort basis, and you should not buy a new device just to run DivestOS.

Being a soft-fork of LineageOS, DivestOS inherits many supported devices from LineageOS. It has signed builds, making it possible to have verified boot on some non-Pixel devices. Unlike its upstream, it does ship user builds.

It comes with substantial hardening over AOSP. DivestOS has automated kernel vulnerability (CVE) patching, fewer proprietary blobs, a custom hosts file, and various security features ported from GrapheneOS. A non-exhaustive list of this includes:

• A hardened webview. Mulch comes with some patches from GrapheneOS’s Vanadium browser and the Bromite project. It gets updated fairly quickly and does not fall behind nearly as much as Bromite did.
• Kernel patches from GrapheneOS and enables all available kernel security features via defconfig hardening. All kernels newer than version 3.4 include full page sanitization and all ~22 Clang-compiled kernels have -ftrivial-auto-var-init=zero enabled.
• GrapheneOS’s INTERNET and SENSORS permission toggle.
• Hardened memory allocator
• Secure Exec-Spawning
• Partial bionic hardening patchsets from GrapheneOS
• GrapheneOS’s per-network full MAC randomization option on version 17.1 and higher
• Automatic reboot/Wi-Fi/Bluetooth timeout options

With that being said, DivestOS is not without its faults. The developer does not have all of the devices he is building for, and for a lot of them he simply publishes the builds blind without actually testing them. Firmware update support varies across devices. DivestOS also takes a very long time to update to a new major Android, and actually took longer than CalyxOS did as mentioned above. It does not tend to fall behind on Chromium updates like CalyxOS, however.

Also, please note that I am only recommending DivestOS here, and not any of its related apps. For instance, I would not recommend using Mull, since it is just a Firefox Android fork with better defaults and still inherits many security deficiencies from its upstream, including the lack of support for site isolation and isolatedProcess.

Defining a threat

To make a threat model, we must first define a threat. A common mistake made by people who are just getting into the privacy space is to define the threat as “big-tech companies.” There is a fundamental problem with this definition:

Why are we not trusting “big-tech companies,” but then shifting our trust to “small-tech companies”? What happens if those “small-tech companies” turn out to be malicious? What happens when our favorite “small-tech company” becomes successful and grows exponentially? The proper way to define the threat here is the “service provider,” not “big-tech.”

Generally, there are four primary threats a person would want to protect themselves from:

• A service provider spying their users
• Cross site/service tracking and data sharing, a.k.a. “mass surveillance”
• An app developer spying on users through their malicious software
• A hacker trying to get into users’ computers

A typical person would have several of these threats in their threat model. Some of these threats may weigh more than others. For example, a software developer would have a hacker stealing their source code, signing keys, and secrets as their primary threat, but beyond that they would also want privacy from the websites they visit and so on. Likewise, an average Joe may have their primary threat as mass surveillance and service providers, but beyond that they also need to have decent security to prevent a hacker from stealing their data.

For whistleblowers, the threat model is much more extreme. Beyond what is mentioned above, they also need anonymity. Beyond just hiding what they do, what data they have, not getting hacked by hackers or governments, they also have to hide who they are.

Privacy from service providers

In most setups, our “private” messages, emails, social interactions are typically stored on a server somewhere. The obvious problem with this is that the service provider (or a hacker who has compromised the server) can look into your “private” conversations whenever and however they want, without you ever knowing. This applies to many common services like SMS messaging, Telegram, Discord, and so on.

With end-to-end encryption, you can alleviate this issue by encrypting communications between you and your desired recipients before they are even sent to the server. The confidentiality of your messages is guaranteed, so long as the service provider does not have access to the private keys of either party.

In practice, the effectiveness of different end-to-end encryption implementations varies. Applications such as Signal run natively on your device, and every copy of the application is the same across different installations. If the service provider were to backdoor their application in an attempt to steal your private keys, that could later be detected using reverse engineering.

On the other hand, web-based end-to-end encryption implementations such as Proton Mail’s webmail or Bitwarden’s web vault rely on the server dynamically serving JavaScript code to the browser to handle cryptographic operations. A malicious server could target a specific user and send them malicious JavaScript code to steal their encryption key, and it would be extremely hard for the user to ever notice such a thing. Even if the user does notice the attempt to steal their key, it would be incredibly hard to prove that it is the provider trying to do so, because the server can choose to serve different web clients to different users.

Therefore, when relying on end-to-end encryption, you should choose to use native applications over web clients whenever possible.

Even with end-to-end encryption, service providers can still profile you based on metadata, which is typically not protected. While the service provider could not read your messages to see what you’re saying, they can still observe things like who you’re talking to, how often you message them, and what times you’re typically active. Protection of metadata is fairly uncommon, and you should pay close attention to the technical documentation of the software you are using to see if there is any metadata minimization or protection at all, if that is a concern for you.

Protection from cross site/service tracking

You can be tracked across websites and services using some form of identifiers. These are typically:

• Your IP address
• Browser cookies
• Your browser fingerprint
• Data you submit to websites
• Payment method correlation

Your goals should be to segregate your online identities from each other, to blend in with other people, and simply to avoid giving out identifying information to anyone as much as possible.

Instead of relying on privacy policies (which are promises that could be violated), try to obfuscate your information in such a way that it is very difficult for different providers to correlate data with each other and build a profile on you. This could come in the form of using encryption tools like Cryptomator prior to uploading your data to cloud services, using prepaid cards or cryptocurrency to protect your credit/debit card information, using a VPN to hide your IP address from websites and services on the internet, etc. A privacy policy should only be relied upon as a last resort, when you have exhausted all of your options for true privacy and need to put complete trust in your service provider.

Bear in mind that companies can hide their ownership or share your information with data brokers, even if they are not in the advertising business. Thus, it makes little sense to solely focus on the “ad-tech” industry as a threat in your threat model. Rather, it makes a lot more sense to protect yourself from service providers as a whole, and any kind of corporate surveillance threat that most people are concerned about will be thwarted along with the rest.

Limiting Public Information

The best way to ensure your data is private is to simply not put it out there in the first place. Deleting information you find about yourself online is one of the best first steps you can take to regain your privacy.

On sites where you do share information, checking the privacy settings of your account to limit how widely that data is spread is very important. For example, if your accounts have a “private mode,” enable it to make sure your account isn’t being indexed by search engines and can’t be viewed by people you don’t vet beforehand.

If you have already submitted your real information to a number of sites which shouldn’t have it, consider employing disinformation tactics such as submitting fictitious information related to the same online identity to make your real information indistinguishable from the false information.

Protection from malware and hackers

You need security to obtain any semblance of privacy: Using tools which appear private is futile if they could easily be exploited by attackers to release your data later.

When it comes to application security, we generally do not (and sometimes cannot) know if the software that we use is malicious, or might one day become malicious. Even with the most trustworthy developers, there is generally no guarantee that their software does not have a serious vulnerability that could later be exploited.

To minimize the potential damage that a malicious piece of software can do, you should employ security by compartmentalization. This could come in the form of using different computers for different jobs, using virtual machines to separate different groups of related applications, or using a secure operating system with a strong focus on application sandboxing and mandatory access control.

Mobile operating systems are generally safer than desktop operating systems when it comes to application sandboxing. Apps cannot obtain root access and only have access to system resources which you grant them.

Desktop operating systems generally lag behind on proper sandboxing. ChromeOS has similar sandboxing properties to Android, and macOS has full system permission control and opt-in (for developers) sandboxing for applications, however these operating systems do transmit identifying information to their respective OEMs. Linux tends to not submit information to system vendors, but it has poor protection against exploits and malicious apps. This can be mitigated somewhat with specialized distributions which make heavy use of virtual machines or containers, such as Qubes OS.

Web browsers, email clients, and office applications all typically run untrusted code sent to you from third-parties. Running multiple virtual machines to separate applications like these from your host system as well as each other is one technique you can use to avoid an exploit in these applications from compromising the rest of your system. Technologies like Qubes OS or Microsoft Defender Application Guard on Windows provide convenient methods to do this seamlessly, for example.

If you are concerned about physical attacks you should use an operating system with a secure verified boot implementation, such as Android, iOS, ChromeOS, or macOS. You should also make sure that your drive is encrypted, and that the operating system uses a TPM or Secure Enclave or Secure Element for rate limiting attempts to enter the encryption passphrase. You should avoid sharing your computer with people you don’t trust, because most desktop operating systems do not encrypt data separately per-user.

Bad Practices

As a beginner, you may often fall into some bad practices while making a threat model. These include:

• Solely focusing on advertising networks instead of service providers as a whole
• Heavy reliance on privacy policies
• Blindly shifting trust from one service provider to another
• Heavy reliance on badness enumeration for privacy instead of systematically solving the problem
• Blindly trusting open-source software

As discussed, focusing solely on advertising networks and relying solely on privacy policies does not make up a sensible threat model. When switching away from a service provider, try to determine what the root problem is and see if your new provider has any technical solution to the problem. For example, you may not like Google Drive as it means giving Google access to all of your data. The underlying problem here is the lack of end to end encryption, which you can solve by using an encryption tool like Cryptomator or by switching to a provider who provides it out of the box like Proton Drive. Blindly switching from Google Drive to a provider who does not provide end to end encryption like the Murena Cloud does not make sense.

You should also keep in mind that badness enumeration does not work, cannot work, has never worked, and will never work. While things like ad blockers and antiviruses may help block the low hanging fruits, they can never fully protect you from the threat. On the other hand, they often increase your attack surface and are not worth the security sacrifice. At best, they are merely convenience tools and should not be thought of as part of a defense strategy.

Another thing to keep in mind is that open-source software is not automatically private or secure. Malicious code can be sneaked into a package by the developer of a project, contributors, library developers or the person who compiles the code. Beyond that, sometimes, a piece of open-source software may have worse security properties than its proprietary counterpart. An example of this would be traditional Linux desktops lacking verified boot, system integrity protection, or a full system access control for apps when compared to macOS. When doing threat modeling, it is vital that you evaluate the privacy and security properties of each piece of software being used, rather than just blindly trusting it because it is open-source.

One of the biggest parts of the Free and Open Source Software definitions is the freedom to study a program and modify it; in other words, access to editable source code. I agree that such access is essential; however, far too many people support source availability for the wrong reasons. One such reason is that source code is necessary to have any degree of transparency into how a piece of software operates, and is therefore necessary to determine if it is at all secure or trustworthy. Although security through obscurity is certainly not a robust measure, this claim has two issues:

• Source code describes what a program is designed to do; it is unnecessary and insufficient to determine if what it actually does aligns with its intended design.
• Vulnerability discovery doesn’t require source code.

I’d like to expand on these issues, focusing primarily on compiled binaries. Bear in mind that I do not think that source availability is useless from a security perspective (it certainly makes audits easier), and I do think that source availability is required for user freedom. I’m arguing only that source unavailability doesn’t imply insecurity, and source availability doesn’t imply security. It’s possible (and often preferable) to perform security analysis on binaries, without necessarily having source code. In fact, vulnerability discovery doesn’t typically rely upon source code analysis.

I’ll update this post occasionally as I learn more on the subject. If you like it, check back in a month or two to see if it has something new.

PS: this stance is not absolute; I concede to several good counter-arguments in a dedicated section!

How security fixes work

I don’t think anyone seriously claims that software’s security instantly improves the second its source code is published. The argument I’m responding to is that source code is necessary to understand what a program does and how (in)secure it is, and without it we can’t know for sure.

Assuming a re-write that fundamentally changes a program’s architecture is not an option¹, software security typically improves by fixing vulnerabilities via something resembling this process:

1. Someone discovers a vulnerability
2. Developers are informed of the vulnerability
3. Developers reproduce the issue and understand what caused it
4. Developers patch the software to fix the vulnerability

Source code is typically helpful (sometimes essential) to Step 3. If someone has completed Step 3, they will require source code to proceed to Step 4. Source code isn’t necessary for Steps 1 and 2; these steps rely upon understanding how a program misbehaves. For that, we use reverse engineering and/or fuzzing.

Reverse engineering

Understanding how a program is designed is not the same as understanding what a program does. A reasonable level of one type of understanding does not imply the other.

Source code² is essential to describe a program’s high-level, human-comprehensible design; it represents a contract that outlines how a developer expects a program to behave. A compiler or interpreter³ must then translate it into machine instructions. But source code isn’t always easy to map directly to machine instructions because it is part of a complex system:

• Compilers (sometimes even interpreters) can apply optimizations and hardening measures that are difficult to reason about. This is especially true for Just-In-Time compilers that leverage run-time information.

• The operating system itself may be poorly understood by the developers, and run a program in a way that contradicts a developer’s expectations.

• Toolchains, interpreters, and operating systems can have bugs that impact program execution.

• Different compilers and compiler flags can offer different security guarantees and mitigations.

• Source code can be deceptive by featuring sneaky obfuscation techniques, sometimes unintentionally. Confusing naming patterns, re-definitions, and vulnerabilities masquerading as innocent bugs have all been well-documented: look up “hypocrite commits” or the Underhanded C Contest for examples.

• All of the above points apply to each dependency and the underlying operating system, which can impact a program’s behavior.

Furthermore, all programmers are flawed mortals who don’t always fully understand source code. Everyone who’s done a non-trivial amount of programming is familiar with the feeling of encountering a bug during run-time for which the cause is impossible to find…until they notice it staring them in the face on Line 12. Think of all the bugs that aren’t so easily noticed.

Reading the source code, compiling, and passing tests isn’t sufficient to show us a program’s final behavior. The only way to know what a program does when you run it is to…run it.⁴

Special builds

Almost all programmers are fully aware of their limited ability, which is why most already employ techniques to analyze run-time behavior that don’t depend on source code. For example, developers of several compiled languages⁵ can build binaries with sanitizers to detect undefined behavior, races, uninitialized reads, etc. that human eyes may have missed when reading source code. While source code is necessary to build these binaries, it isn’t necessary to run them and observe failures.

Distributing binaries with sanitizers and debug information to testers is a valid way to collect data about a program’s potential security issues.

Dynamic analysis

It’s hard to figure out which syscalls and files a large program program needs by reading its source, especially when certain libraries (e.g. the libc implementation/version) can vary. A syscall tracer like strace(1)⁶ makes the process trivial.

A personal example: the understanding I gained from strace was necessary for me to write my bubblewrap scripts. These scripts use bubblewrap(1) to sandbox programs with the minimum permissions possible. Analyzing every relevant program and library’s source code would have taken me months, while strace gave me everything I needed to know in an afternoon: analyzing the strace output told me exactly which syscalls to allow and which files to grant access to, without even having to know what language the program was written in. I generated the initial version of the syscall allow-lists with the following command⁷:

strace name-of-program program-args 2>&1 \
	| rg '^([a-z_]*)\(.*' --replace '$1' \
	| sort | uniq

This also extends to determining how programs utilize the network: packet sniffers like Wireshark can determine when a program connects to the network, and where it connects.

These methods are not flawless. Syscall tracers are only designed to shed light on how a program interacts with the kernel. Kernel interactions tell us plenty (it’s sometimes all we need), but they don’t give the whole story. Furthermore, packet inspection can be made a bit painful by transit encryption⁸; tracing a program’s execution alongside packet inspection can offer clarity, but this is not easy.

For more information, we turn to core dumps, also known as memory dumps. Core dumps share the state of a program during execution or upon crashing, giving us greater visibility into exactly what data a program is processing. Builds containing debugging symbols (e.g. DWARF) have more detailed core dumps. Vendors that release daily snapshots of pre-release builds typically include some symbols to give testers more detail concerning the causes of crashes. Web browsers are a common example: Chromium dev snapshots, Chrome Canary, Firefox Nightly, WebKit Canary builds, etc. all include debug symbols. Until 2019, Minecraft: Bedrock Edition included debug symbols which were used heavily by the modding community.⁹

Dynamic analysis example: Zoom

In 2020, Zoom Video Communications came under scrutiny for marketing its “Zoom” software as a secure, end-to-end encrypted solution for video conferencing. Zoom’s documentation claimed that it used “AES-256” encryption. Without source code, did we have to take the docs at their word?

The Citizen Lab didn’t. On 2020-04-03, it published a report revealing critical flaws in Zoom’s encryption. It utilized Wireshark and mitmproxy to analyze networking activity, and inspected core dumps to learn about its encryption implementation. The Citizen Lab’s researchers found that Zoom actually used an incredibly flawed implementation of a weak version of AES-128 (ECB mode), and easily bypassed it.

Syscall tracing, packet sniffing, and core dumps are great, but they rely on manual execution which might not hit all the desired code paths. Fortunately, there are other forms of analysis available.

Binary analysis

Tracing execution and inspecting memory dumps can be considered forms of reverse engineering, but they only offer a surface-level view of what’s going on. Reverse engineering gets much more interesting when we analyze a binary artifact.

Static binary analysis is a powerful way to inspect a program’s underlying design. Decompilation (especially when supplemented with debug symbols) can re-construct a binary’s assembly or source code. Symbol names may look incomprehensible in stripped binaries, and comments will be missing. What’s left is more than enough to decipher control flow to uncover how a program processes data. This process can be tedious, especially if a program uses certain forms of binary obfuscation.

The goal doesn’t have to be a complete understanding of a program’s design (incredibly difficult without source code); it’s typically to answer a specific question, fill in a gap left by tracing/fuzzing, or find a well-known property. When developers publish documentation on the security architecture of their closed-source software, reverse engineering tools like decompilers are exactly what you need to verify their honesty (or lack thereof).

Decompilers are seldom used alone in this context. Instead, they’re typically a component of reverse engineering frameworks that also sport memory analysis, debugging tools, scripting, and sometimes even IDEs. I use the Rizin framework, but Ghidra is also popular. Their documentation should help you get started if you’re interested.

Example: malware analysis

These reverse-engineering techniques—a combination of tracing, packet sniffing, binary analysis, and memory dumps—make up the workings of most modern malware analysis. See this example of a fully-automated analysis of the Zoom Windows installer. It enumerates plenty of information about Zoom without access to its source code: reading unique machine information, anti-VM and anti-reverse-engineering tricks, reading config files, various types of network access, scanning mounted volumes, and more.

To try this out yourself, use a sandbox designed for dynamic analysis. Cuckoo is a common and easy-to-use solution, while DRAKVUF is more advanced.

Extreme example: the truth about Intel ME and AMT

The Intel Management Engine (ME) is a mandatory subsystem of all Intel processors (after 2008) with extremely privileged access to the host system. Active Management Technology (AMT) runs atop it on the subset of Intel processors with “vPro” branding. The latter can be disabled and is intended for organizations to remotely manage their inventory (installing software, monitoring, remote power-on/sleep/wake, etc).

The fact that Intel ME has such deep access to the host system and the fact that it’s proprietary have both made it the subject of a high degree of scrutiny. Many people (most of whom have little experience in the area) connected these two facts together to allege that the ME is a backdoor, often by confusedly citing functionality of Intel AMT instead of ME. Is it really impossible to know for sure?

I picked Intel ME+AMT to serve as an extreme example: it shows both the power and limitations of the analysis approaches covered. ME isn’t made of simple executables you can just run in an OS because it sits far below the OS, in what’s sometimes called “Ring -3”.¹⁰ Analysis is limited to external monitoring (e.g. by monitoring network activity) and reverse-engineering unpacked partially-obfuscated firmware updates, with help from official documentation. This is slower and harder than analyzing a typical executable or library.

Answers are a bit complex and…more boring than what sensationalized headlines would say. Igor Skochinsky (the developer of me-tools) and Nicola Corna (the developer of me_cleaner) presented their analysis of ME in Intel Me: Myths and Reality; Vassilios Ververis thoroughly analyzed AMT in Security Evaluation of Intel’s Active Management Technology. Interestingly, the former pair argues that auditing binary code is preferable to potentially misleading source code: binary analysis allows auditors to “cut the crap” and inspect what software is truly made of. However, this was balanced by a form of binary obfuscation that the pair encountered; I’ll describe it in a moment.

Simply monitoring network activity and systematically testing all claims made by the documentation allowed Ververis to uncover a host of security issues in Intel AMT. However, no undocumented features have (to my knowledge) been uncovered. The problematic findings revolved around flawed/insecure implementations of documented functionality. In other words: there’s been no evidence of AMT being “a backdoor”, but its security flaws could have had a similar impact. Fortunately, AMT can be disabled. What about ME?

This is where some binary analysis comes in. Neither Skochinsky’s ME Secrets presentation nor Intel Me: Myths and Reality seem to enumerate any contradictions with official Intel documentation.

Unfortunately, some components are poorly understood due to being obfuscated using Huffman compression with unknown dictionaries. Understanding the inner workings of the obfuscated components blurs the line between software reverse-engineering and figuring out how the chips are actually made, the latter of which is nigh-impossible if you don’t have access to a chip lab full of cash. However, black-box analysis does tell us about the capabilities of these components: see page 21 of “ME Secrets”. Thanks to zdctg for clarifying this.

Skochinsky’s and Corna’s analysis was sufficient to clarify (but not completely contradict) sensationalism claiming that ME can remotely lock any PC (it was a former opt-in feature), can spy on anything the user does (they clarified that access is limited to unblocked parts of the host memory and the integrated GPU, but doesn’t include e.g. the framebuffer), etc.

While claims such as “ME is a black box that can do anything” are misleading, ME is not without its share of vulnerabilities. My favorite look at its issues is a presentation by Mark Ermolov and Maxim Goryachy at Black Hat Europe 2017: How to Hack a Turned-Off Computer, or Running Unsigned Code in Intel Management Engine.

In short: ME being proprietary doesn’t mean that we can’t find out how (in)secure it is. Binary analysis when paired with runtime inspection can give us a good understanding of what trade-offs we make by using it. While ME has a history of serious vulnerabilities, they’re nowhere near what borderline conspiracy theories claim.¹¹

(Note: Intel is not alone here. Other chips typically have equivalents, e.g. AMD Secure Technology).

Fuzzing

Manual invocation of a program paired with a tracer like strace won’t always exercise all code paths or find edge-cases. Fuzzing helps bridge this gap: it automates the process of causing a program to fail by generating random or malformed data to feed it. Researchers then study failures and failure-conditions to isolate a bug.

Fuzzing doesn’t necessarily depend on access to source code, as it is a black-box technique. Fuzzers like American Fuzzy Loop (AFL) normally use special fuzz-friendly builds, but other fuzzing setups can work with just about any binaries. In fact, some types of fuzz tests (e.g. fuzzing a web API) hardly need any implementation details.

Fuzzing frequently catches bugs that are only apparent by running a program, not by reading source code. Even so, the biggest beneficiaries of fuzzing are open source projects. cURL, OpenSSL, web browsers, text rendering libraries (HarfBuzz, FreeType) and toolchains (GCC, Clang, the official Go toolchain, etc.) are some notable examples.

I’ve said it before but let me say it again: fuzzing is really the top method to find problems in curl once we’ve fixed all flaws that the static analyzers we use have pointed out. The primary fuzzing for curl is done by OSS-Fuzz, that tirelessly keeps hammering on the most recent curl code.

• Daniel Stenberg | A Google grant for libcurl work

If you want to get started with fuzzing, I recommend checking out the quick-start guide for American Fuzzy Loop. Some languages like Go 1.18 also have fuzzing tools available right in the standard library.

Example: CVE-2022-0185

A recent example of how fuzzing helps spot a vulnerability in an open-source project is CVE-2022-0185: a Linux 0-day found by the Crusaders of Rust a few weeks ago. It was discovered using the syzkaller kernel fuzzer. The process was documented on Will’s Root:

CVE-2022-0185 - Winning a $31337 Bounty after Pwning Ubuntu and Escaping Google’s KCTF Containers by willsroot

I highly encourage giving it a read; it’s the perfect example of fuzzing with sanitizers to find a vulnerability, reproducing the vulnerability (by writing a tiny C program), then diving into the source code to find and fix the cause, and finally reporting the issue (with a patch!). When source isn’t available, the vendor would assume responsibility for the “find and fix” steps.

The fact that some of the most-used pieces of FLOSS in existence have been the biggest beneficiaries of source-agnostic approaches to vulnerability analysis should be quite revealing. The source code to these projects has received attention from millions of eyes, yet they still invest in fuzzing infrastructure and vulnerability-hunters prefer analyzing artifacts over inspecting the source.

Good counter-arguments

I readily concede to several points in favor of source availability from a security perspective:

• Source code can make analysis easier by supplementing source-independent approaches. The lines between the steps I mentioned in the four-step vulnerability-fixing process are blurry.

• Patching vulnerabilities is important. Source availability makes it possible for the community, package maintainers, or reporters of a vulnerability to patch software. Package maintainers often blur the line between “packager” and “contributor” by helping projects migrate away from abandoned/insecure dependencies. One example that comes to mind is the Python 2 to Python 3 transition for projects like Calibre.¹² Being able to fix issues independent of upstream support is an important mitigation against user domestication.

• Some developers/vendors don’t distribute binaries that make use of modern toolchain-level exploit mitigations (e.g. PIE, RELRO, stack canaries, automatic variable initialization, CFI, etc.¹³). In these cases, building software yourself with these mitigations (or delegating it to a distro that enforces them) requires source code availability (or at least some sort of intermediate representation).

• Closed-source software may or may not have builds available that include sanitizers and debug symbols.

• Although fuzzing release binaries is possible, fuzzing is much easier to do when source code is available. Vendors of proprietary software seldom release special fuzz-friendly builds, and filtering out false-positives can be quite tedious without understanding high-level design.

• It is certainly possible to notice a vulnerability in source code. Excluding low-hanging fruit typically caught by static code analysis and peer review, it’s not the main way most vulnerabilities are found nowadays (thanks to X_CLI for reminding me about what source analysis does accomplish.

• Software as a Service can be incredibly difficult to analyze, as we typically have little more than the ability to query a server. Servers don’t send core dumps, server-side binaries, or trace logs for analysis. Furthermore, it’s difficult to verify which software a server is running.¹⁴ For services that require trusting a server, access to the server-side software is important from both a security and a user-freedom perspective

Most of this post is written with the assumption that binaries are inspectable and traceable. Binary obfuscation and some forms of content protection/DRM violate this assumption and actually do make analysis more difficult.

Beyond source code, transparency into the development helps assure users of compliance with good security practices. Viewing VCS history, patch reviews, linter configurations, etc. reveal the standards that code is being held up to, some of which can be related to bug-squashing and security.

Patience on Matrix also had a great response, which I agree with and adapt below:

Whether or not the source code is available for software does not change how insecure it is. However, there are good security-related incentives to publish source code.

• Doing so improves vulnerability patchability and future architectural improvement by lowering the barrier to contribution. The fixes that follow can be shared and used by other projects across the field, some of which can in turn be used by the vendor. This isn’t a zero-sum game; a rising tide lifts all boats.
• It’s generally good practice to assume an attacker has full knowledge of a system instead of relying on security through obscurity. Releasing code provides strong assurance that this assumption is being made. It’s a way for vendors to put their money where their mouth is.

Both Patience and Drew Vault argue that given the above points, a project whose goal is maximum security would release code. Strictly speaking, I agree. Good intentions don’t imply good results, but they can supplement good results to provide some trust in a project’s future.

Conclusion

I’ve gone over some examples of how analyzing a software’s security properties need not depend on source code, and vulnerability discovery in both FLOSS and in proprietary software uses source-agnostic techniques. Dynamic and static black-box techniques are powerful tools that work well from user-space (Zoom) to kernel-space (Linux) to low-level components like Intel ME+AMT. Source code enables the vulnerability-fixing process but has limited utility for the evaluation/discovery process.

Don’t assume software is safer than proprietary alternatives just because its source is visible; come to a conclusion after analyzing both. There are lots of great reasons to switch from macOS or Windows to Linux (it’s been my main OS for years), but security is low on that list.

All other things being mostly equal, FLOSS is obviously preferable from a security perspective; I listed some reasons why in the counter-arguments section. Unfortunately, being helpful is not the same as being necessary. All I argue is that source unavailability does not imply insecurity, and source availability does not imply security. Analysis approaches that don’t rely on source are typically the most powerful, and can be applied to both source-available and source-unavailable software. Plenty of proprietary software is more secure than FLOSS alternatives; few would argue that the sandboxing employed by Google Chrome or Microsoft Edge is more vulnerable than Pale Moon or most WebKitGTK-based browsers, for instance.

Releasing source code is just one thing vendors can do to improve audits; other options include releasing test builds with debug symbols/sanitizers, publishing docs describing their architecture, and/or just keeping software small and simple. We should evaluate software security through study rather than source model. Support the right things for the right reasons, and help others make informed choices with accurate information. There are enough good reasons to support software freedom; we don’t need to rely on bad ones.