# Incident: Nightly backup saturated the corral NFS mount ("corral jail")

[wmobley]

• Status: Open — root cause identified; corral access still failing (Permission denied as of 2026-07-01)
• Severity: High — corral mount unresponsive/inaccessible; WebODM media I/O at risk; backups non-functional
• Reported window: ~2026-06-09 through present
• Host: webodm (ubuntu.tacc.utexas.edu), corral NFS 129.114.52.151:/corral/main/tacc/aci/PT2050/projects/PTDATAX-263
• Owner: wmobley

Summary

The nightly host cron backup (/usr/local/bin/webodm-backup.sh, installed by
WebODM/setup.sh:setup_backup) tars the entire WebODM media tree off the corral NFS
mount and writes the tarball back to the same mount every night at 02:00. That I/O load
saturated the corral NFS server. Because corral is mounted hard (timeo=600), every backup
process that hit an unresponsive server went into uninterruptible sleep and blocked — one
stuck job per night — while WebODM's own media I/O stalled on the same mount. On 2026-06-18 the
failure mode changed from an I/O hang to an outright Permission denied on /corral, which
persists.

Separately, the script's database backup has never worked since install (wrong DB name),
and the script reports success regardless because it never checks exit codes.

Timeline (reconstructed from the cron mail spool, /var/spool/mail/wmobley)

Backup filenames embed the job start time (DATE=$(date +%Y%m%d_%H%M%S)); the cron mail is
sent when the job finishes. The gap between the two is the hang.

• 2026-06-09 → 2026-06-17: nightly 02:00 jobs start on schedule but do not complete. They
  block on corral NFS I/O and stack up (uninterruptible sleep).
• 2026-06-17 13:34:00 → 13:34:46: the mount recovers and nine backlogged jobs
  (started 06-09, 06-10, 06-11, 06-12, 06-13, 06-14, 06-15, 06-16, 06-17) all flush their
  completion mail within 46 seconds. This burst is the signature of the mount hang clearing.
• 2026-06-18 02:00:01 onward: failure mode changes to
  mkdir: cannot create directory '/corral': Permission denied. corral is still mounted rw
  but is not accessible to the user.
• 2026-06-22 18:01: all WebODM containers (db, webapp, worker, broker) recreated
  (rebuild/update), which wiped the Docker logs from the incident window.
• 2026-07-01 (now): /corral/webodm/backups, df /corral → Permission denied. Cron
  still runs nightly (305 mails on file); backups still failing.

Root cause

1. Primary: the backup design generates heavy, repeated NFS I/O against corral — a full
   re-tar of the entire media tree every night, sourced from and written to the same mount —
   with no throttling (nice/ionice), no timeout, and no reachability preflight. On a
   hard NFS mount this stacks uninterruptible jobs and can take the mount (and WebODM media)
   down with it.
2. Secondary (current): a corral allocation/permissions/squash change on ~2026-06-18 makes
   /corral inaccessible to the user. This is a TACC-side allocation/permissions issue, not a
   code bug, and is tracked separately (needs a TACC ticket).

Contributing bugs in webodm-backup.sh

Installed script (verified on host):

BACKUP_DIR="/corral/webodm/backups"          # writes to corral
docker exec db pg_dump -U postgres webodm > $BACKUP_DIR/webodm_$DATE.sql
tar -czf $BACKUP_DIR/media_$DATE.tar.gz -C /corral/webodm media/   # reads+writes corral
find $BACKUP_DIR -type f -mtime +7 -delete    # more corral I/O

• Wrong DB name — DB backup never worked. Actual database is webodm_dev; the script dumps
  webodm, producing FATAL: database "webodm" does not exist on every run since install
  (confirmed back to 2025-09). The .sql files are empty/garbage.
• No exit-code checking / no set -euo pipefail. pg_dump fails but the script still prints
  "Database backed up to …", so cron mail looks successful unless you read the pg_dump line.
  The empty .sql is left on disk regardless (> created it before pg_dump ran).
• No I/O throttling. No nice/ionice; full-tree tar competes directly with WebODM.
• No timeout / no preflight. Nothing detects an unresponsive/denied corral before starting,
  so jobs hang (hard mount) or run partially and report success.
• Backup target is on the failing mount. DB dump, media tarball, retention sweep all live on
  corral — the exact resource under stress — so backups are useless precisely when needed.
• No real log file. Output goes only to cron mail; there is no on-disk backup log to audit.

Impact

• Backups non-functional: DB never backed up; media backups blocked/failing since ~06-09.
• corral mount hangs/inaccessibility risked WebODM media reads/writes during the window.
• Incident diagnosis was nearly lost: Docker logs wiped on 06-22 rebuild; the cron mail spool
  was the only surviving record.

Evidence sources

• /var/spool/mail/wmobley — 305 backup cron mails back to 2025-09; June 17 13:34 backlog burst.
• mount — corral hard,timeo=600,retrans=2 NFS export …/PTDATAX-263.
• docker inspect — containers recreated 2026-06-22T18:01.
• /usr/local/bin/webodm-backup.sh, WebODM/.env (WO_MEDIA_DIR=/corral/webodm/media,
  WO_DB_DIR=/db), WebODM/setup.sh:setup_backup.

Immediate actions

• Open TACC ticket for corral PTDATAX-263 Permission denied (blocking; not a code fix).
• Disable the nightly cron until the script is hardened, to stop stacking hung/failing
  jobs against corral (crontab -e, comment the webodm-backup.sh line).
• Verify no orphaned uninterruptible (D-state) tar/pg_dump processes remain
  (ps aux | awk '$8 ~ /D/').
• Confirm WebODM media integrity once corral access is restored.

Follow-up

• Harden the backup per the design doc: docs/design/2026-07-01-backup-hardening-corral-safe.md.
• Add a corral-reachability preflight to WebODM startup so media-mount problems surface early.

[wmobley]

Status update — 2026-07-08

Nightly backup cron disabled (temporary mitigation).

The webodm-backup.sh cron job has been commented out via crontab -e on the host. This stops new hung/failing jobs from stacking against the corral NFS mount each night, and removes the immediate risk of additional I/O pressure while corral access is still degraded.

Immediate actions from the issue checklist completed:

• Disable the nightly cron (crontab -e, webodm-backup.sh line commented out)

Still open:

• Open TACC ticket for corral PTDATAX-263 Permission denied
• Verify no orphaned uninterruptible (D-state) tar/pg_dump processes remain
• Confirm WebODM media integrity once corral access is restored
• Harden the backup script per docs/design/2026-07-01-backup-hardening-corral-safe.md before re-enabling