Backups and restore · Joripo documentation

The storage root is the record, so it is what the server backs up. The database is not in these backups: what it indexes is rebuilt from the files, and what only it holds needs a backup of its own (below).

What runs on its own

Every 24 hours (JORIPO_BACKUP_INTERVAL_HOURS), first about five minutes after a start, the server writes one zip of the whole storage root into its backup folder (JORIPO_BACKUP_DIR, by default _backups inside the storage root).

Files exactly as stored. A file sealed at rest goes in sealed, never opened, so reading a backup needs the same data key.
Verified. A manifest in the zip records every file's size and SHA-256 as it was read; the zip is then opened again from the disk and every file re-read and checked. Only then does it count; one that fails is deleted.
Kept. The newest three verified backups (JORIPO_BACKUP_KEEP).
Room on the disk. A backup is refused, not squeezed in, if writing it would leave less than 20% of the disk (JORIPO_BACKUP_RESERVE_PERCENT) and at least 1 GB free, because a full disk would refuse submissions. To make room it first removes older verified backups, never the last one. A refusal turns Server health › Backups red, saying what it needed.
One at a time. A second backup while one runs is refused; a backup cut off by a restart never blocks the next.
Left out: the backup folders themselves, the published app installers (publish them again after a restore), image previews, temporary and lock files, and the maintainers' file.

JORIPO_BACKUP_INTERVAL_HOURS=0 turns the schedule off; Back up now still works.

Off the machine

A backup on the same disk protects against mistakes (a wrong delete, a bad migration), not against losing the disk.

A second disk: set JORIPO_BACKUP_COPY_DIR to a USB drive, a network share or another volume. Each verified backup is copied there and checked. A folder on the data's own disk is not "off the machine"; the backups page says so.
Download: Server administration › Backups offers every backup, with its SHA-256. Download the newest regularly and keep it somewhere else; every download is recorded.
Cloud storage (below): every file copied to two cloud providers as it is written.

Cloud storage

With JORIPO_CLOUD_MODE set, every file of the storage root is copied within seconds to a Cloudflare R2 bucket and a Backblaze B2 bucket, two independent companies, and the database is dumped to both every day. Everything uploaded is sealed with the server's data key first, so neither provider can read it; the server refuses to start in a cloud mode without a key.

mirror keeps every file on the disk as well.
primary keeps attachments that have not been opened lately in the cloud only, once the files on the disk pass JORIPO_LOCAL_CACHE_MAX_MB: a server with a small disk never fills it. An attachment kept in the cloud is fetched back, checked, when someone opens it; pages and checks read the answers from the database, so they are as fast as before.
While a bucket cannot be reached, files wait on the disk and are sent later. If the disk fills to 85% meanwhile, new submissions are refused for a while; the apps keep them and send them again, so nothing is lost.
Server administration › Backups shows what is in the buckets, what is waiting, and the last database backup; Back up now runs the database backup.
The local zip backups above are not made while cloud storage is on.

Restoring a lost disk, a lost database or a lost provider from a bucket: scripts/restore_from_cloud.py, with the runbook in docs/OPERATIONS.md, "Cloud storage".

The database

Back up PostgreSQL where it lives, for example with pg_dump --format=custom, or the hosting provider's own database backups. Restore with pg_restore. With cloud storage on, the server does this itself every day.

Not yet exercised here: the pg_dump and pg_restore commands are standard PostgreSQL, but have not been run as part of Joripo's own restore drill, which used the test database. Test your database restore before you need it.

Restoring

Restore only into a new or empty folder, never over the live one. The restore runbook below has been walked end to end on a test server (a sealed tree and its database both lost, restored, rebuilt, counts and answers checked); it has not yet been done on a hosted server.

Pick the backup. From Server administration › Backups, or python -m app.backup list. Anything received after it was taken is not in it.

Check it (writes nothing). It re-reads and hashes every file, and says what the backup holds, which data keys sealed it, and whether this machine has them:

python scripts\restore_backup.py <backup.zip> --to <new folder>

Restore it. Each file is checked against its SHA-256 as it is written, and the restore ends by counting what is on the disk against the backup:

python scripts\restore_backup.py <backup.zip> --to <new folder> --apply

Point the server at it. Set STORAGE_ROOT to the new folder and start the engine with the same data key; it refuses to start with keys that cannot open the tree. The old folder is left as it was.

The index. If the database survived, there is nothing to do: the start-up re-index adds what it lacks. If it was lost, restore its own backup, then (or instead) rebuild the index from the files.

Check the counts, then open Server health: Submissions, Storage, Encryption at rest and Schema should be green.

python scripts\restore_backup.py <backup.zip> --check-index

Finally, publish the apps again on Server administration › Apps: installers are not in backups.