Hardening a fresh VM: a pre-use baseline and project isolation

What this is. Take a fresh cloud VM from "just provisioned" to "safe to put a real workload on" — and, if the workload runs untrusted or model-authored code, cage it so a compromise can't phone home, exfiltrate, or lift its own limits.

Target: a modern Ubuntu LTS on any commodity provider (DigitalOcean, Vultr, Hetzner, Linode, …). Default to 24.04 LTS (supported to May 2029) — as of mid-2026 the conservative choice. 26.04 LTS ("Resolute Raccoon", released Apr 2026, kernel 7.0) is fine and forward-looking, ideally once the 26.04.1 point release ships; 22.04 LTS still works but its standard support ends May 2027. Minor tweaks noted inline where providers differ. Assumes: first login is as root via an SSH key you added at creation time.

What it pairs. A baseline every VM should have with the service-isolation patterns that cage an untrusted workload — plus the hard-won gotchas that actually bit on a real build.

How to read it. Part A applies to every VM — do all of it. Part B is opt-in: do it only if your workload runs untrusted / model-authored code, ingests untrusted input, or just deserves a bounded blast radius. Part C is the list of things that actually went wrong — read it before you touch nftables or PAM. Not every rule fits every project; where a choice is a judgment call, the rationale is stated so you can make it yourself.

Conventions

Substitute these throughout:

Placeholder	Means
`admin`	your human sudo administrator — the account you SSH into (key-only)
`svc`	the dedicated unprivileged user the workload runs as (Part B). No key, no sudo
`myapp`	your project name → code at `/srv/myapp`, config/secrets at `/etc/myapp`
`<vm>`	the SSH host alias in your laptop's `~/.ssh/config`

UIDs shown (admin 1001, svc 1002) are illustrative — use whatever adduser assigns and read the real one with id -u svc. nftables needs the literal uid, so capture it before you write rules.

Threat model

Be explicit about what you're defending against — it tells you which rules matter for your project:

Drive-by scanners — constant SSH brute-force and exposed-service probes.
Compromised credentials — a stolen key, leaked token, reused password.
Lateral movement from a compromised service — an app bug must not mean root.
Outbound abuse (the one the generic guides miss) — a compromised or prompt-injected workload that phones home, exfiltrates data, or reaches the cloud metadata service for credentials. If your workload runs untrusted code or ingests untrusted input, this is your primary risk — and inbound firewalls do nothing about it. → Part B.
Operator mistakes — you, locking yourself out or leaving a port open.
Provider-layer risk — someone with console access, or snapshot theft.

Out of scope for a commodity cloud VM: nation-state physical access, hypervisor 0-days. If those are your model, you shouldn't be on shared cloud.

Philosophy

The VM is the isolation boundary. Treat it as disposable — rebuildable from a snapshot + offsite backup. Nothing on it should be irreplaceable.
Minimize attack surface — nothing public that doesn't have to be; no service as root that a system user could run.
Defense in depth — no single control is the only thing standing between you and a bad day.
Verify, don't assume — every phase ends by checking the effective state (sshd -T, sysctl read-back, a real login from a fresh terminal), not by trusting that editing a file worked.
The only control that survives a full root compromise is enforced off the box. On-box firewalls, file perms, and caps all fall if root falls. Design accordingly (see Part C).

Sequencing

The single most common way to brick a remote box is doing the right steps in the wrong order.

Create admin and verify you can log in as them from a fresh terminal before disabling root.
Harden SSH in two stages: password-off + root key-only first (keeps a way in during setup), root off entirely only at the very end after admin is proven.
Do all installs and pre-pull any container images before locking egress (Part B) — a network cage blocks the pulls you still need.
Cut over to Tailscale and confirm it works before closing public SSH.
Never start the workload until secrets are set, the sandbox is smoke-tested, and the pre-use audit passes. Keep the scheduler disabled until then.

Part A — Baseline hardening

ssh root@<vm-ip>
apt update && apt upgrade -y && apt autoremove --purge -y
hostnamectl set-hostname myapp          # also add "127.0.1.1 myapp" to /etc/hosts
timedatectl set-timezone UTC            # UTC always — log correlation is painful otherwise
systemctl enable --now systemd-timesyncd
 
# Human sudo admin, key-only.
adduser --disabled-password --gecos "" admin
usermod -aG sudo admin
install -d -m 700 -o admin -g admin /home/admin/.ssh
cp /root/.ssh/authorized_keys /home/admin/.ssh/authorized_keys
chmod 600 /home/admin/.ssh/authorized_keys

Passwordless sudo is a deliberate choice, not laziness — if your only auth factor is the SSH key (no passwords anywhere), then there is no password protecting sudo to begin with, so NOPASSWD keeps admin-over-SSH non-interactive without weakening anything:

printf 'admin ALL=(ALL) NOPASSWD:ALL\n' > /etc/sudoers.d/90-admin && chmod 440 /etc/sudoers.d/90-admin

If you do keep passwords anywhere, drop NOPASSWD and keep faillock (A7).

Verify before going further — do not skip. New terminal on your laptop:

ssh admin@<vm-ip> 'sudo whoami'    # must print: root

Remove any provider default user once admin is proven (DigitalOcean sometimes none; Vultr ships linuxuser): deluser --remove-home <provider-default>.

A2 — SSH hardening

Providers often drop a PasswordAuthentication yes override in /etc/ssh/sshd_config.d/50-cloud-init.conf — neutralize it and write your own managed drop-in /etc/ssh/sshd_config.d/00-hardening.conf:

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowUsers admin
X11Forwarding no
AllowTcpForwarding yes
AllowAgentForwarding no
PermitEmptyPasswords no
MaxAuthTries 3
MaxSessions 4
LoginGraceTime 20
ClientAliveInterval 60
ClientAliveCountMax 30
AddressFamily inet       # IPv6 disabled by default (A4); delete this line if you keep v6

Service box vs. dev box — the one real fork:

Service / agent box (you don't interactively code on it): the values above. AllowAgentForwarding no — do not expose your laptop's keys to a box running untrusted code. AllowTcpForwarding yes is still fine and useful: it's how you tunnel a loopback-only dashboard back to your laptop (ssh -N -L 8787:localhost:8787 <vm>); forwarded ports ride inside the SSH channel and never open on the VM's interface.
Dev box (VSCode/Cursor/JetBrains remote): set AllowAgentForwarding yes (lets git push on the box use your laptop's agent instead of a key living on the box) and keep MaxSessions 4 (editors open several channels per connection). The keepalive pair ClientAliveInterval 60 × CountMax 30 = a 30-minute grace so your editor survives a closed lid or a wifi switch.

AllowAgentForwarding yes is a trust statement — anyone who can read $SSH_AUTH_SOCK on the box (i.e. root) can use your keys while you're connected. Fine on a solo box you control; never on a shared or untrusted one.

Validate, then restart — and test a fresh login before closing your session:

sudo sshd -t && sudo systemctl restart ssh
sudo sshd -T | grep -iE 'permitroot|passwordauth|allowusers|agentforward'   # confirm EFFECTIVE values

Don't hand-pin Ciphers / KexAlgorithms / MACs. OpenSSH on current Ubuntu already defaults to strong, post-quantum key exchange (mlkem768x25519-sha256 on 26.04's 10.2p1; sntrup761x25519 on 24.04's 9.6p1) and has dropped the weak primitives (DSA removed, plain finite-field DH off, AES-GCM preferred). A copy-pasted "hardening" cipher block from an old blog usually disables the PQ default and invites regressions — leave the defaults unless a specific compliance list forces your hand.

Laptop-side (`~/.ssh/config`, keep it in your dotfiles)

Host <vm>
  HostName 100.x.y.z            # Tailscale IP or MagicDNS name (A15)
  User admin
  IdentityFile ~/.ssh/<vm>-key
  IdentitiesOnly yes            # don't offer every key in your agent to this host
  ForwardAgent no              # yes ONLY for dev boxes you fully trust
  ServerAliveInterval 30
  ServerAliveCountMax 6
  ControlMaster auto            # reuse one connection — instant subsequent ssh/scp/editor channels
  ControlPath ~/.ssh/sockets/cm-%r@%h:%p
  ControlPersist 10m
  HostKeyAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256
  PubkeyAcceptedAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256

mkdir -p ~/.ssh/sockets && chmod 700 ~/.ssh/sockets once. Force a fresh connection with ssh -O exit <vm>.

A2b — Editor default

For the human context only — admin, plus root when you sudoedit / visudo. The svc service user (B1) is non-interactive and needs none of this. Do it early if you hand-edit configs on the box, so your editor is ready for the phases below.

Install (apt is simplest and gets security patches via unattended-upgrades, A6):

sudo apt install -y neovim        # prefer Vim? -> sudo apt install -y vim   (full vim, not vim.tiny)

Ubuntu's packaged Neovim lags — notably on 24.04 (ships 0.9.x vs upstream 0.12.x); 26.04 is closer (0.11.x). For a current build use the PPA (sudo add-apt-repository ppa:neovim-ppa/stable — note it's community-maintained, not by the Neovim team) or a checksum-verified release tarball (nvim-linux-x86_64.tar.gz) from github.com/neovim/neovim/releases (then you own its updates; apt/PPA auto-update). Skip the AppImage on a hardened box unless you want FUSE — and on 24.04+ it needs libfuse2t64 (the old libfuse2 name is 22.04-and-earlier).

Make it the default — set all three layers (swap nvim→vim throughout if you prefer Vim):

# 1) Debian 'editor' alternative — what sudoedit, visudo, and crontab use in ROOT's minimal env.
sudo update-alternatives --install /usr/bin/editor editor /usr/bin/nvim 60
sudo update-alternatives --set     editor /usr/bin/nvim          # or interactive: --config editor
 
# 2) Shell env — git, `crontab -e`, and most CLIs read these. Add to admin's ~/.bashrc (or ~/.profile):
export EDITOR=nvim
export VISUAL=nvim
export SUDO_EDITOR=nvim                                          # sudoedit / sudo -e honor this
 
# 3) Git:
git config --global core.editor nvim

Why both the alternative and the env vars: env vars live only in your interactive shell, so sudo -e and visudo — which run in root's minimal environment — fall back to the editor alternative. Set only one and root-context edits will surprise you with nano.

Config: pull yours from your dotfiles (~/.config/nvim/), same as your laptop — don't hand-roll it on the box. Verify: sudo update-alternatives --display editor shows nvim, echo "$EDITOR" is nvim, and sudo -e /tmp/verify opens nvim (quit without saving).

A3 — Firewall (UFW)

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw enable
grep IPV6 /etc/default/ufw          # must be IPV6=yes
sudo ufw status verbose             # must show "(v6)" on the deny defaults

Provider cloud firewall (DO/Vultr/etc.) is inbound-only and sits in front of the VM — it short-circuits UFW for traffic it blocks. Pick one as the source of truth for inbound, or treat them as explicit layers. Neither provider firewall gives you outbound enforcement — that's Part B.

A4 — IPv6 posture

Default: disable it unless a workload needs it. Most small cloud boxes get no usable public v6, and disabling shrinks surface + simplifies the egress rules — the A5 disable block and A2's AddressFamily inet are both on by default. "Locked down on v4, wide open on v6" is the most common cloud-VM misconfiguration, so the one thing not to do is leave it ambiguous.

If a workload needs v6: keep it and let UFW cover it (it already does, v4+v6 from A3) — skip the A5 disable block, delete AddressFamily inet from A2, and make sure nothing binds only to :: by accident (sudo ss -tulpn | grep ':::'). Check what you have with ip -6 addr show.

A5 — Kernel & network sysctl

/etc/sysctl.d/99-hardening.conf:

# Spoofing / source routing
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
# ICMP redirects (MITM vector)
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
# SYN floods / smurf
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.log_martians = 1
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.default.accept_ra = 0
# Kernel hardening
kernel.kptr_restrict = 2
kernel.dmesg_restrict = 1
kernel.yama.ptrace_scope = 2
kernel.unprivileged_bpf_disabled = 1
net.core.bpf_jit_harden = 2
# Filesystem hardening
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_fifos = 2
fs.protected_regular = 2
fs.suid_dumpable = 0
# Core dumps leak secrets
kernel.core_pattern = |/bin/false

Disable IPv6 (the A4 default) — append this block; skip it only if a workload needs v6:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

sudo sysctl --system
sudo sysctl kernel.kptr_restrict kernel.yama.ptrace_scope net.ipv4.tcp_syncookies   # spot-verify

A6 — Automatic updates

sudo apt install -y unattended-upgrades apt-listchanges needrestart
sudo dpkg-reconfigure --priority=low unattended-upgrades   # or enable via /etc/apt/apt.conf.d/20auto-upgrades
grep -A3 "Allowed-Origins" /etc/apt/apt.conf.d/50unattended-upgrades   # confirm security-only

needrestart tells you when a library patch needs a service restart — a silent patch that isn't loaded doesn't help you.

Auto-reboot for kernel/libc patches (recommended for an unattended box). Updates that need a reboot otherwise sit unapplied indefinitely. Set a fixed reboot window in /etc/apt/apt.conf.d/52unattended-reboot:

Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:00";
# Set ::Automatic-Reboot-WithUsers "false" if you'd rather not reboot during an active login.

A oneshot timer's Persistent=true (B5) catches any run missed during the reboot; for a long-running server, pick a low-traffic window.

A7 — Session limits & faillock

/etc/login.defs: UMASK 027, PASS_MAX_DAYS 90, PASS_MIN_DAYS 1, PASS_WARN_AGE 14.

/etc/security/limits.d/99-hardening.conf — cap resources against fork bombs / fd exhaustion:

*   hard   nproc   2048     # 4096 if you run containers
*   hard   nofile  8192     # 65536 if you run containers
*   hard   core    0

faillock — stage it, but think before wiring it into PAM. With key-only SSH + NOPASSWD sudo there is no password surface to protect, and a bad edit to /etc/pam.d/common-auth on a remote box is a real lockout risk. Recommended: write /etc/security/faillock.conf (deny=5, unlock_time=900, fail_interval=900) but leave it out of PAM unless you actually keep passwords somewhere. If you do wire it, keep a root session open and test in a second terminal first.

A8 — AppArmor

sudo aa-status        # profiles should be loaded + enforcing

For any process that handles untrusted input, prefer a confining profile over running unconfined.

A9 — fail2ban

sudo apt install -y fail2ban && sudo systemctl enable --now fail2ban && sudo fail2ban-client status

Its value drops sharply once you're key-only + Tailscale (no public SSH to brute-force). Keep it for anything you do expose publicly later; otherwise it's cheap and harmless.

On 26.04, OpenSSH does some of this itself. PerSourcePenalties (built-in throttling of misbehaving sources) is default-on since OpenSSH 9.8, so it's active on 26.04's 10.2p1 but not on 24.04's 9.6p1. On 26.04 it reduces reliance on fail2ban for SSH; on 24.04, fail2ban still earns its keep.

A10 — auditd

sudo apt install -y auditd

/etc/audit/rules.d/hardening.rules:

-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudo_changes
-w /etc/sudoers.d/ -p wa -k sudo_changes
-w /etc/ssh/sshd_config -p wa -k sshd_config
-a always,exit -F arch=b64 -S execve -k exec
-e 2

sudo augenrules --load && sudo systemctl restart auditd   # query later: ausearch -k identity

A11 — DNS (pinned + DoT)

/etc/systemd/resolved.conf.d/99-hardening.conf:

[Resolve]
DNS=1.1.1.1 9.9.9.9
FallbackDNS=1.0.0.1 149.112.112.112
DNSOverTLS=yes
DNSSEC=allow-downgrade
Cache=yes

sudo systemctl restart systemd-resolved && resolvectl status | grep 'DNS Server'

This pinning holds because we keep Tailscale MagicDNS off (A15, --accept-dns=false). With MagicDNS on, Tailscale seizes /etc/resolv.conf (resolver 100.100.100.100) and these resolvers no longer apply — and you'd then need the port-53 CGNAT exception in the B3 cage (C-3).

Gotcha: the provider's DHCP resolver often keeps winning. If resolvectl still shows the DHCP one, add a netplan drop-in /etc/netplan/60-dns.yaml with dhcp4-overrides.use-dns: false + nameservers.addresses: [1.1.1.1, 9.9.9.9], then netplan apply. Also note (C-4): resolvectl is the systemd-resolved view; glibc and most tools read /etc/resolv.conf, which can say something else — check both, especially before you firewall any IP range.

A12 — Package integrity

sudo apt install -y debsums && sudo debsums -c    # clean = no shipped files modified; run periodically

A13 — File integrity (AIDE)

Worth it for long-lived, high-value, or compliance-bound boxes; skip for a truly disposable throwaway. Detects tampering with binaries and config. Add dev-tool exclusions before building the baseline or it drowns in churn — create /etc/aide/aide.conf.d/99_dev_exclusions excluding ~/.vscode-server, ~/.cursor-server, ~/.cache, ~/.config, ~/.local/{state,share}, language managers (~/.nvm ~/.rustup ~/.cargo/registry …), shell history, and your git-tracked code tree. Keep watching /etc, /usr/{bin,sbin}, /bin, /sbin, /usr/local, /boot, and ~/.ssh ~/.bashrc ~/.profile ~/.zshrc (login-time tampering targets). Then:

sudo aideinit && sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
# daily /etc/cron.daily/aide-check → log file; even better, ping a Healthchecks.io URL so a SILENT
# failure (the check not running) alerts you. Refresh the baseline after every change you MADE.

A14 — Off-box log shipping

Logs that live only on the VM vanish with it — including during an incident (rm -rf /var/log is an early attacker move). For a disposable box, journald with size caps is a legitimate choice (nothing external, journalctl covers "what it did / why it errored"). If the box is long-lived or high-value, ship journald + auditd + ufw logs off-box over Tailscale: rsyslog → remote listener (simple), promtail → Loki (moderate), or a managed sink (Grafana Cloud / Better Stack / Axiom).

A14b — Liveness / dead-man's switch

A dashboard or a log tail can't show you the one failure that matters most: the box or its workload silently stopped (crash, OOM, hung run, dead VM) — and a dashboard on a dead box serves no page. Close that with a dead-man's-switch: something external that expects a regular signal and alerts you when it stops arriving.

Emit the signal from the thing whose liveness you care about, and on success — a scheduled job pings a URL from the last line of its script/unit; a daemon pings on a heartbeat. Pinging on success (not just on start) means a run that half-fails still trips the alarm.
Watch it off-box, so a dead box can't mask its own death: healthchecks.io (hosted, free), self-hosted Uptime Kuma, cronitor / Better Stack, or a provider alarm. Set the expected period = your run interval + grace, and point the alert at your phone/email.
The ping URL is a credential — whoever holds it can fake liveness. Keep it in the secrets file (A17), never in git.
If you cage egress (B3), the monitor's host must be in the allowlist or the ping never leaves.

Cheapest high-value alert you can wire — do it before you walk away.

A15 — Private access (Tailscale)

Puts SSH behind a WireGuard mesh; removes ~all drive-by noise and gives ACL-based access.

Required in this guide: every box joins the tailnet and public SSH is closed (below). SSH auth stays native OS keys (A2's AllowUsers admin), not Tailscale SSH — so you're not locked out if Tailscale is down for maintenance, and not tied to their identity layer. If you'd rather manage access purely in tailnet ACLs, use tailscale up --ssh and drop the OS-key path instead — just accept that coupling.

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --accept-dns=false   # MagicDNS off (keeps A11's pinned DoT); note its 100.x.y.z

For reproducible provisioning, use a tagged auth key: sudo tailscale up --authkey=tskey-… --advertise-tags=tag:vm --accept-routes=false --accept-dns=false.

Cut over in the right order — connect over Tailscale and confirm it works first:

# from a Tailscale-originated session:
sudo ufw allow in on tailscale0
sudo ufw delete allow OpenSSH
sudo ufw reload && sudo ufw status verbose    # confirm public :22 now times out

Also close root SSH entirely now (A2's PermitRootLogin no — final stage). Break-glass if Tailscale ever fails = the provider's web console (unaffected by tailnet ACLs). Write Tailscale ACLs in the console — the default "anyone in the tailnet reaches anything" is usually too permissive.

tailscale up resets any flag you don't restate (see C-5) — so always restate --accept-dns=false (plus your tags/routes). We keep MagicDNS off so A11's pinned DoT resolvers stay in force; enable it only if you need tailnet-hostname resolution, and then add the port-53 CGNAT exception to the B3 cage (C-3).

A16 — Filesystem layout

sudo install -d -m 750 -o admin -g admin /srv/myapp        # code (git-managed, reproducible)
# data + logs as needed; back up data (A18), cap logs via the log driver

Don't run services out of /home — harder to back up, mixes code with user config. (Part B moves the service tree to svc ownership.)

A17 — Secrets

Never bake credentials into images; never commit .env. Default — a root-owned 0600 env file, referenced from a systemd EnvironmentFile:

sudo install -m 0600 /dev/null /etc/myapp/app.env    # edit, reference from systemd EnvironmentFile

Upgrades when you outgrow a flat file: sops+age (encrypted in git, decrypted at deploy) or a managed store (1Password CLI / Infisical / Doppler / Vault / cloud secret manager). Part B tightens the ownership so the workload can read but not rewrite its own config.

A18 — Snapshots & backups

Default: treat the box as stateless and disposable. If its real state lives in git and off-box (code reproducible, data in a managed DB / object store), all you need is a provider snapshot as a known-good rollback — take one now labeled post-hardening-baseline. A snapshot is rollback, not backup: it lives with the provider, so a compromised provider account takes it too.

Only if the box holds state you can't recreate (a local DB, generated artifacts, an on-box ledger), add an offsite backup of just that state: restic to S3/B2, repo password in your secrets manager, on a systemd timer. Test a restore at least quarterly — an untested backup is not a backup.

A19 — Pre-use audit

Run this before any real workload touches the box. Anything surprising, stop and fix.

sudo ss -tulpn                                          # no unexpected listeners
sudo sshd -T | grep -iE 'permitroot|passwordauth|pubkey|allowusers|x11|agentforward'
sudo ufw status verbose                                 # v4 + v6, minimal rules
ip -6 addr show                                         # IPv6 firewalled or disabled, not ambiguous
systemctl is-active ssh unattended-upgrades auditd fail2ban systemd-timesyncd
sudo sysctl kernel.kptr_restrict kernel.yama.ptrace_scope net.ipv4.tcp_syncookies fs.protected_hardlinks
sudo find /etc -type f -perm -o+w                       # no world-writable config
awk -F: '$3==0 {print $1}' /etc/passwd                  # UID 0 == root only
awk -F: '$7 !~ /nologin|false/ {print $1,$7}' /etc/passwd  # only expected users can log in
sudo debsums -c                                         # package integrity
sudo ls -la /etc/cron.*/ /var/spool/cron/crontabs/      # no surprise cron

Part B — Project isolation

B0 — When to do Part B

Do Part B if any of these is true; otherwise Part A is enough:

The workload runs code it wrote itself or fetched (LLM agents, plugin runners, CI that executes untrusted PRs).
It ingests untrusted input (scrapes/fetches the web, processes user uploads, reads a public inbox).
It holds or can reach money, signing keys, or high-value credentials.
You simply want its blast radius bounded so one bug isn't game-over.

The guiding idea: the workload is the risk actor, so cage it, not the whole box. Everything below is scoped to the svc user, so it doesn't fight your admin access, Tailscale, or apt.

Two choices you then make per box (deliberately not baked into this guide): (1) whether svc is a regular or a --system user — B1; (2) admin's SSH posture, dev vs service box — A2. Both follow from what the workload actually is, so decide them once you know it.

B1 — Dedicated service user

First decide svc's type — it follows from whether you'll run the B4 rootless-container sandbox:

Using the sandbox (B4) → a regular user (uid ≥ 1000). Rootless Podman needs per-user subuid/subgid ranges (auto-assigned by adduser), a lingering user session, and XDG_RUNTIME_DIR=/run/user/<uid> — none of which a --system user gets.
No sandbox / no rootless containers → a conventional --system daemon user is cleaner (no home, no login shell, no subuid ranges).

# pick ONE, per the choice above — no SSH key, no sudo, ever, either way:
adduser --disabled-password --gecos "" svc      # regular user  (needed for the B4 rootless sandbox)
# adduser --system --group svc                   # system user   (if you're NOT running rootless containers)
 
install -d -m 750 -o svc   -g svc   /srv/myapp   # code the service runs
install -d -m 750 -o root  -g svc   /etc/myapp   # config/secrets: root owns, svc-group can traverse

admin administers; svc runs the workload and can touch nothing else. Act as it with sudo -u svc -H …. This one split is what makes every control below meaningful — a control scoped to "the workload" needs the workload to be a distinct principal.

B2 — Read-only secrets

The subtle, high-value one. Set the env/config file root:svc 0640 — owned by root, readable by the svc group, writable by neither the service nor anyone but root:

sudo install -m 0640 -o root -g svc /dev/null /etc/myapp/app.env

Why it matters: if the service could write its own config, it could flip its own resource limits, budget/rate caps, or an ISOLATION=off switch on its next run — quietly undoing your controls. Read-only means the limits are set by you, above it. (This bit for real once: an env file mistakenly svc:svc would have let the workload disable its own sandbox.) The very highest-value secrets go further — see B6.

B3 — Egress allowlist

Inbound firewalls do nothing against a compromised workload phoning home or exfiltrating. Close that by letting svc reach only the handful of destinations it genuinely needs, via a loopback proxy, and dropping everything else at the kernel. Proxy-based (not IP-based) so it survives CDN IP churn; uid-scoped so it doesn't disturb the rest of the box.

Recommended posture: start tight, broaden deliberately. Ship with a strict domain allowlist, and open up to broad reads only after the proxy-uid caging (below) is in place. Broadening the allowlist before that caging briefly exposed the cloud metadata service — do it in the opposite order.

Two layers:

(1) A loopback domain-allowlist proxy. tinyproxy on 127.0.0.1:8888, /etc/tinyproxy/tinyproxy.conf:

Listen 127.0.0.1
Port 8888
Allow 127.0.0.1
FilterDefaultDeny Yes
Filter /etc/tinyproxy/filter
FilterExtended On           # deprecated in newer tinyproxy — 1.11.3+ prefers "FilterType ere" (same ERE behavior)
ConnectPort 443

/etc/tinyproxy/filter = the domains the workload actually needs (regex, anchored):

(^|\.)api\.your-provider\.com$
(^|\.)registry\.npmjs\.org$
# add your package registry, API host(s), RPC/webhook host, healthcheck host — nothing else

Ubuntu's AppArmor confines tinyproxy and will block reading the filter file — allow it via /etc/apparmor.d/local/tinyproxy (file r /etc/tinyproxy/filter,) then sudo apparmor_parser -r /etc/apparmor.d/usr.bin.tinyproxy. Restart tinyproxy.

(2) An nftables owner-match confining the svc uid to loopback only, loaded at boot by a tiny service. Get the uid first (id -u svc) and put the literal in /etc/myapp/egress.nft:

table inet app_egress {
    chain output {
        type filter hook output priority 0; policy accept;
        meta skuid != 1002 accept          # <-- svc's REAL uid; police ONLY the workload
        oifname "lo" accept
        ip daddr 127.0.0.0/8 accept        # -> loopback proxy + local DNS stub only
        log prefix "app-egress-drop " counter drop
    }
}

Load it via a oneshot unit ordered after ufw with no global flush (so it coexists with UFW): nft -f /etc/myapp/egress.nft.

Make the workload use the proxy. For Node ≥ 24: NODE_USE_ENV_PROXY=1 + HTTPS_PROXY=http://127.0.0.1:8888 (global fetch honors it — no code change). For other stacks set HTTPS_PROXY / https_proxy (and HTTP_PROXY/http_proxy) in the service env. Prefer https:// URLs — some runtimes only route https through the env proxy, so a bare http:// URL bypasses it and dead-ends at DNS under the cage.

Verify four ways as svc:

sudo -u svc -H bash -lc 'curl -s -o /dev/null -w "%{http_code}\n" https://api.your-provider.com'   # 200
sudo -u svc -H bash -lc 'curl -s -o /dev/null -w "%{http_code}\n" https://example.com'             # blocked
sudo -u svc -H bash -lc 'HTTPS_PROXY= curl -s https://api.your-provider.com'                       # direct egress: fails

If you deliberately relax the domain allowlist (e.g. the workload needs broad read access), you have not removed the cage's most important job — you've moved it. Now the proxy's own uid can reach the tailnet, other loopback services, RFC1918, and — critically — the cloud metadata service (169.254.169.254, which hands out cloud credentials). Add a second nftables rule set that denies the proxy's uid (id -u tinyproxy, often 105) those ranges, placed first in the chain (see C-2, C-3):

        # ONLY if you run MagicDNS (NOT the default — see A15): its resolver 100.100.100.100 sits inside
        # 100.64/10, so let it through on port 53 BEFORE the drop, or the proxy resolves nothing:
        meta skuid 105 ip daddr 100.100.100.100 udp dport 53 accept
        meta skuid 105 ip daddr { 169.254.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10 } drop

And add the same blocks to any app-layer fetch/SSRF guard you write: reject loopback, RFC1918, link-local/metadata (169.254.0.0/16), and the Tailscale CGNAT range 100.64.0.0/10 (second octet 64–127) plus IPv6 ULA fc00::/7. The app guard and the network layer are belt-and-suspenders; the network layer matters more because it also covers any code path that bypasses your guard.

B4 — Run-command sandbox

If the workload executes commands the model (or any untrusted source) wrote, an un-isolated exec can read your secrets and overwrite the very files that enforce safety — voiding every other control. Run each such command in a throwaway container.

Rootless Podman (not Docker — avoids the root-equivalent docker group, and rootless is what makes this safe as svc):

sudo apt install -y podman uidmap slirp4netns fuse-overlayfs passt
sudo loginctl enable-linger svc
# as svc — PRE-PULL before you lock egress (B3); --network=none blocks on-demand pulls:
sudo -u svc -H podman pull docker.io/library/node:22-bookworm-slim

Rootless networking: pasta (from passt) is the default on Podman 5+ (26.04); on 24.04's Podman 4.9 the default is still slirp4netns — installing both covers either. fuse-overlayfs is optional now too (native rootless overlayfs works on kernel ≥ 5.13). With --network=none (below) no backend is used at all, so this only matters for network-enabled runs.

Run template (workspace-only mount, no network, read-only rootfs, all caps dropped, no privilege gain):

podman run --rm --network=none --read-only --cap-drop=ALL --security-opt=no-new-privileges \
  --pids-limit=256 -v "$WORKSPACE:/work:rw" -w /work <image> <cmd>
# Podman applies its default seccomp profile unless you pass seccomp=unconfined — leave it on.

Fail closed: if no container runtime is present, the workload must refuse to run untrusted commands — never silently fall back to host exec. If a command genuinely needs the network, give it the network inside the egress cage (still bounded by B3) and treat that run as tainted/untrusted — don't punch a hole in the cage. Smoke-test: confirm the sandbox has no network and cannot see the host FS.

B5 — Scheduling (systemd timer)

For a periodic workload, run it as a oneshot on a timer rather than a long-lived daemon — the timer catches up runs missed while the box was off and kills a hung run.

/etc/systemd/system/myapp.service:

[Unit]
After=network-online.target tinyproxy.service app-egress.service
Wants=network-online.target
[Service]
Type=oneshot
User=svc
Group=svc
WorkingDirectory=/srv/myapp
EnvironmentFile=/etc/myapp/app.env
ExecStart=/usr/local/bin/<runner> <task>
TimeoutStartSec=900          # kill a hung run (RuntimeMaxSec is ignored for oneshot)
MemoryMax=1400M
# XDG_RUNTIME_DIR=/run/user/1002 in app.env + linger (B4) so rootless Podman works under systemd.
# Do NOT add NoNewPrivileges=/SystemCallFilter= here if you use rootless Podman — they break its
# setuid newuidmap (see C-6). Isolation = egress cage + containerized exec + non-root user.
[Install]
WantedBy=multi-user.target

/etc/systemd/system/myapp.timer:

[Timer]
OnCalendar=*-*-* 0/4:00:00    # e.g. every 4h
Persistent=true               # run once on boot if a slot was missed
[Install]
WantedBy=timers.target

Keep the timer disabled until the pre-use audit passes and a dry/real run succeeds. Go live: sudo systemctl start myapp.service (one run on demand) → watch logs → sudo systemctl enable --now myapp.timer. Keep the host on UTC+NTP if any window/quota is wall-clock. (cron alternative if you drop systemd: flock -n /tmp/myapp.lock timeout 900 <runner> <task>.)

Long-running service instead of a scheduled job? If the workload is a persistent server (web / API / bot), drop the timer and make the service Type=simple (or notify) with a restart policy — same User, EnvironmentFile, After= egress/sandbox ordering, and the same rootless-Podman caveat as above:

[Service]
Type=simple
User=svc
Group=svc
WorkingDirectory=/srv/myapp
EnvironmentFile=/etc/myapp/app.env
ExecStart=/usr/local/bin/<runner> <serve>
Restart=on-failure
RestartSec=5
MemoryMax=1400M
# same as the oneshot: NO NoNewPrivileges/SystemCallFilter with rootless Podman (C-6)

(Reuse the [Unit] and [Install] blocks from the oneshot above — only [Service] changes.) Then sudo systemctl enable --now myapp and journalctl -u myapp -f. Everything else in Part B (service user, secrets perms, egress cage, sandbox) applies unchanged — only the unit shape differs. Public inbound for such a server is a separate decision (reverse proxy / Tailscale Funnel / an opened port); the egress cage still governs what it can reach out regardless.

B6 — Crown-jewel credential

The highest-value secret — a signing key, a prod-DB master, a push-capable deploy key — should not be reachable by the workload at all, and ideally not live on the box:

On-box, root-only: a root:root 0600 file the svc user cannot read; a separate helper process (as root or another uid) uses it and exposes only a narrow, capped operation to svc over loopback. The workload references the capability by handle, not by key.
Off-box (better for anything irreversible, e.g. money movement): a broker/managed service enforces the policy — hard caps, destination allowlist, velocity limits, revocation — outside the workload. A raw key inside the process is the documented disaster mode for autonomous/financial agents; every large loss in that space was a key sitting where injected code could reach it.

Remember the philosophy: on-box controls fall if root falls; only off-box enforcement survives a root compromise. Match the custody to the stakes.

Part C — Hard-won gotchas

C-1 — Test on the target OS, not just your laptop. A blocking syscall against an unwritable path (e.g. a synchronous mkdir at a path that lost write perms) hangs on Linux where it threw on macOS — and a try/catch can't rescue a hung syscall. If a stalled write sits in a hot path, it can wedge the one component that matters. Exercise the real deploy path on the real OS before trusting it.
C-2 — nftables rule ORDER is everything. A chain whose first rule is meta skuid != <svc> accept short-circuits everything that isn't the workload — so any rule you append after it (e.g. to cage the proxy uid) is never reached. Order-sensitive rules must come first.
C-3 — DNS can live inside the range you're about to block. With the default MagicDNS-off posture (A15) this is moot, but if you enable MagicDNS its resolver 100.100.100.100 sits inside the CGNAT range 100.64.0.0/10 — block that range without a port-53 exception and the workload resolves nothing → total, silent egress failure. The general lesson stands regardless: always cat /etc/resolv.conf before firewalling any IP range, and add any DNS exception before the drop.
C-4 — resolvectl lies (politely). It shows the systemd-resolved view; glibc/most tools read /etc/resolv.conf, which may point elsewhere. Check the file, not just the tool.
C-5 — tailscale up resets flags you don't restate. Re-running it to change one thing silently reverts --accept-dns, --accept-routes, tags, etc. to defaults. State every flag you care about, every time.
C-6 — systemd hardening breaks rootless Podman. NoNewPrivileges=yes / SystemCallFilter= block the setuid newuidmap rootless Podman needs. Don't add them to a unit that runs rootless containers; get your isolation from the egress cage + containerized exec + non-root user instead.
C-7 — Pre-pull images before locking egress. --network=none (and a tight allowlist) blocks on-demand pulls. Pull every image you'll need while the network is still open.
C-8 — faillock + remote PAM = lockout risk. Editing /etc/pam.d/common-auth wrong locks you out of a box you can only reach remotely. With key-only + NOPASSWD there's nothing to protect; stage faillock but don't wire it unless you keep passwords (A7).
C-9 — Host-level egress is open by design, and that's a real limit. The cage confines the workload; it does not confine root. A full root compromise can tear down any on-box firewall. The only outbound control that survives that is enforced upstream (a NAT/proxy gateway off the box). Know where your boundary actually is.
C-10 — Egress drops are a high-signal alert. The log prefix "app-egress-drop " line is gold: a caged workload trying to reach somewhere new is often the first visible sign of injection or compromise. Alert on it once you're live.
C-11 — Don't let your own testing poison the workload's signals. If the workload reacts to observed state (traffic counters, an inbox, a queue), your own curl/probe pollutes exactly the signal it reads and it'll "learn" from noise. Give probes an excluded path (/health), and reset any real counter before go-live.
C-12 — Verify effective state, not the file you edited. sshd -T, sysctl <key>, a fresh login from a new terminal. A provider drop-in, an AppArmor deny, or a DHCP override can quietly beat your edit.

Decommission / teardown

Plan the exit before you start — especially for a long-running or autonomous workload.

Clean stop (graceful): build a kill switch the workload checks each run — e.g. a STOP=1 flag in its env/config, or systemctl disable --now myapp.timer. The switch must be enforced outside the workload's control (a root-owned file it can read but not write — B2 — or the systemd timer it can't touch). If the workload's objective could ever read as "keep running," never let it hold its own off switch, and judge a stopped run by its behavior and true state, not by what it reports about itself.

Full teardown checklist:

Stop the scheduler (systemctl disable --now myapp.timer) and the service.
Take a final snapshot / offsite backup if the state matters.
Revoke every credential that touched the box — API keys (rotate, don't just delete), deploy keys, the RPC/webhook URLs, the healthcheck ping URL (the URL is the credential).
Remove the node from the tailnet (admin console) and delete its auth key.
Destroy the VM at the provider, then confirm it's gone (and any attached volumes/snapshots you don't want to keep).
Rotate anything the box could have read even if you think it wasn't exposed.

What NOT to do

Leave root SSH enabled "just for now."
Use password auth anywhere; change the SSH port and call it security.
Open a port publicly "for testing" — it stays open.
Run a service as root when a system user would do; add yourself to docker on a shared box (≈ root).
Put a signing key / crown-jewel secret where the workload can read it.
Relax the egress allowlist without caging the proxy uid + metadata service (B3).
Trust provider snapshots as your only backup, or ignore IPv6 because "we only use v4."
Skip the pre-use audit (A19) because "it worked last time."

Final checklist

Part A (every VM):

Part B (if B0 applies):

Dedicated svc user, no key/no sudo (B1)
Config root:svc 0640 — service can read, not write (B2)
Egress allowlist verified four ways; proxy uid + metadata caged if allowlist relaxed (B3)
Container runtime installed, image pre-pulled, exec sandbox fail-closed + smoke-tested (B4)
systemd oneshot + timer installed, timer disabled until go-live (B5)
Crown-jewel credential off the box or root-only, referenced by handle (B6)
Read Part C before touching nftables / PAM / DNS

Secure by default, hardened in depth, private by network, caged where it counts, verified before use. Part A is the floor every VM starts from. Part B is what turns "a hardened box" into "a box I'd let run code it wrote itself." Everything past this — per-service seccomp, an off-box egress gateway, an IDS — is an upgrade on this foundation.