podman container hardening

[0xlua]

NixOS

There is a greate overview with guides and additional ressources to harden the base OS in the Wiki.

Basic Hardening

• Non-Root User: eg. user = “1000:100”
  • Advanced: user = "nobody:nogroup"
• drop capabilities, restart, read the error if it crashes, add back the minimum
  • run podman inspect (check capabilities: NET_RAW, SYS_CHROOT, MKNOD, ...)
  • add capabilities = {ALL = false} to everything
  • Caddy, Vaultwarden, (maybe more) need NET_BIND_SERVICE for port 80/443
  • PostgreSQL: needs to chown data directories (CHOWN, SETUID, SETGID, ...)
• --security-opt=no-new-privileges
• read-only filesystem: --read-only
  • in additon some (e.g. PostgresQL) need --read-only-tmpfs
• make sure these are false:
  • --privileged
  • --tty

Podman Networks

• don't put everything on the same network: Don't use the default bridge network
• Use Reverse Proxy for HTTPS
• expose only containers which are made for being exposed
• databases have own networks: internal: true (zero internet access, only specific app)
• Reverse proxy gets its own network
• Inter-service communication goes through a separate network, eg. mail

    # after: database isolated, no internet
    networks:
      default:
        name: myapp_db
        internal: true
      web_ingress:
        external: true

Related: use per-service database container. I already do this though. Just in case, two services should share a database: Dedicated database + role per service, connection limits per role, and then revoked CONNECT from PUBLIC on every database.

Resource limits

• disable swap per container (memswap_limit = mem_limit): only container gets killed, not whole box
  • mem_limit: educated guess from stats or do real profiling?
• add PID limits: prevent fork bomb
• tiered CPU with cpu_shares:
  • Reverse proxy and databases get highest priority
  • App services get medium
  • Background workers get lowest

Health checks

• replace "is the process alive" with real HTTP check
• each runtime needs its own approach:
  • Node: no curl, use Node's http module inline
  • Python slim: no curl, use urllib
  • Postgres: pg_isready just works

Pinned versions & Auto Updates

This adds little to no security because version tags are still mutable. However it would make it safer/easier to use podman auto-update.

When using the podman home-manager module, I can activate auto-updating on a weekly basis like this:

per container: services.podman.containers.<name>.autoUpdate

services.podman.autoUpdate = {
  enable = true;
  onCalendar = "Sun *-*-* 00:00";
};

For maximum security I'd have to pin every image to SHA256 digests, Tradeoff: manual updates

HTTP Headers

e.g. HSTS, CSP, ...

Check Security for each service: Mozilla Observatory

Fail2Ban

Enable Logs for services to be used in fail2ban jails. Look for service-specific guides.

Secrets

• I already use sops for secrets
• when possible no plain-text env vars: prefer Files / Podman Secrets
• API Keys: Use scoped tokens where possible

Fix once

• enable firewall and block all access from countries you are not from, see fail2ban: jails don't seem to work #208
• use OIDC everywhere: 2FA/Passkeys
• use volumes only for data, move config to this flake: caddy-etc, stalwart-etc, ...
• use named volumes instead of file mounts. Also relevant for backups, see Backups #124

Reverse Proxy

Use TLS 1.2 minimum, Restricted ciphers (already default in caddy).

Maybe switch to nginx with lego or certbot for HTTPS certs?

Sources

• https://reddit.lua.one/r/selfhosted/comments/1sqi04i/beyond_the_basics_what_are_your_nonnegotiable/
• https://reddit.lua.one/r/selfhosted/comments/1sgvwep/after_my_last_post_blew_up_i_audited_my_docker/

[0xlua]

Progress

I'll use this comment to track which containers have been hardened according to the checklist above. I'll also list container specific guides when available.

galileo

• Atuin: I need to update and want to migrate from Postgres to SQLite, Healthcheck
• Bore: Won't fix, since I want to switch anyways, see caddy: add l4 plugin #211
• Caddy: also block common malicious access patterns
• KitchenOwl: Healthcheck, also see KitchenOwl: disable username/password login #117
• Linkding: Healthcheck
• Littlelink: Healthcheck
• Miniflux: Healthcheck, see PostgreSQL
• Pocket ID
• rustypaste
• Stalwart: Healthcheck
• Upvote-RSS
• Vaultwarden

Do once:

• Seperate Podman Networks
• Set Ressource Limts

ganymede

• Caddy: Also use caddy inside LAN
• Jellyfin: OIDC
• Calibre: OIDC
• Mafl: everything read-only
• Paperless: Security, OIDC + Redis -> Valkey
• HomeAssistant
• MusicAssistant
• Arr
  • Gluetun
  • Deluge: bind to tun0, see: https://github.com/qdm12/gluetun-wiki/blob/main/setup/popular-apps.md
  • FlareSolverr
  • Prowlarr
  • Radarr
  • Sonarr
  • Bazarr

Do once:

• Seperate Podman Networks
• Set Ressource Limts