Backups

Model

• Target = one restic repository for one account (own random 40-char password in the vault, target/<id>; ADR 0013 §7). Kinds: local (/var/lib/rc-backups/<account>/<id>, root 0700), connection (any storage connection through restic's rclone backend, path below the connection's base path) and s3 (a private bucket + two keys created on the built-in store through the public storage.* actions: a backup key and a prune key, both in the vault). Creating a target installs restic, runs init and fails cleanly (everything created is rolled back) when it cannot write/read.
• Immutability (immutable_days on s3): the bucket is created with S3 object lock (COMPLIANCE, default retention). Proven in the lab: a permanent delete of a snapshot object version with the target's own credentials is AccessDenied, the panel refuses to force-delete the bucket, accounts.deleted keeps the bucket. Honest limit: SeaweedFS has no "write without delete" action, so a plain S3 DELETE (what restic forget does) only adds a delete marker - the data stays recoverable until retention ends but restic no longer sees that snapshot. Prune cannot free space inside the window. Backup, check, snapshots, ls, stats use --no-lock; forget/prune must lock (restic refuses otherwise), so on a locked bucket a lock file may outlive the run (see follow-ups).
• Job = scope (paths relative to the home, excludes as restic globs, databases: export every database first) + schedule (interval >= 15 min, daily, weekly, monthly, account time zone, up to 5 min stable jitter) + retention (keep-last/hourly/ daily/weekly/monthly/yearly, default 7d/4w/3m, daily capped by backups.retention_days) + targets (each gets every run, per-target result). backups.schedule.tick (every minute, system actor) advances next_run before running, so a slow run never repeats, runs at most 5 due jobs per tick, a 5 % restic check per target weekly, a test restore monthly, and raises backups.stale after 48 h without success (once a day).
• Run (backups.job.run, or the tick): per-target busy flag (one backup/restore/check per target, stale after 3 h) -> databases.db.export into <home>/rc-backup/db/<db>.sql.gz (a database that fails to export fails the run: no silent partial backups) -> restic backup <home> --one-file-system --exclude-caches under nice 19 / ionice idle -> 3 attempts with backoff -> stats -> retention (forget + prune with the prune credential, only after success, only snapshots tagged rc of that account; pre-restore safety snapshots are tagged pre-restore and never pruned). Limits: backups.manual_runs_per_day (per target-run), backups.storage_gb (local and built-in S3 targets; skipped run + backups.quota.exceeded, nothing deleted).
• Restore: restore.files (default into <home>/restore-<time>/ with the snapshot's relative tree, ownership/mode/symlinks kept, --verify; inplace over the originals; other into another account on the same server, re-owned), restore.database (replace with a safety dump first, new database, download the dump), restore.account (plan via dry run, typed account name, safety snapshot first, files in place then every database import, rollback within 24 h via restore.rollback). All dry-runnable.
• verify.run (restic check, optional subset), verify.test_restore (random 5 files into a scratch dir, hash verify, cleanup), retention.preview (forget --dry-run), recovery.export (admin, critical: repository locations + passwords + S3 keys; audited).

Verified in the lab (2026-10-09)

Connect external S3 (fake cloud) -> init repo through rclone -> job with files + MariaDB -> run twice (second run added 4.7 kB of 3 MB) -> delete files and drop a table -> alternate-folder restore (sha256 equal, mode 640 kept), in-place restore, database replace (row data back) -> whole-account restore after deleting www/report.bin and dropping a table, then rollback -> verify 100 %, test restore, retention keep-last 2 removed 2 of 4 snapshots on local and S3 targets -> immutable target proof -> restore into a second account -> scheduler ran the due job -> account deletion removed targets, jobs, local repo, keys and connection config.

Known gaps / next (see handover)

Agent op timeout is 60 s (module SDK cannot set it) so big backups will fail until Host.Agent takes a timeout; no progress/cancel; mail and containers volumes, PITR, server-level repo + DR restore, tar.zst archive download, mailbox restore, domain remap, cron expressions, restoring deletes nothing that is not in the snapshot (a rollback does not remove files the restore added), PostgreSQL "replace" needs a clean import (databases.db.import does not drop existing tables), restic/rclone from the distro (0.16.4 / 1.60.1), per-snapshot delete.