Backups and restore — Colloq docs

What is kept on disk

A standard k3s instance uses /var/lib/colloq/data and /var/lib/colloq/workspace. Next to them sit the private configuration, persistent secrets and release records. Local PVs survive a Pod being replaced, but not a lost disk or a deleted VM.

The portable archive contains SQLite, the workspace, configuration, application and runtime keys, release/catalog files and checksums. It holds secrets: keep it private and off the source machine. Container image layers are not in the archive, so the registry must keep the digests you need.

Choose a backup mode

Mode	What it does	Cost
`consistent`	Stops the app, the broker and every kernel before copying.	Downtime; Python memory is lost.
`live`	Takes an SQLite snapshot and copies changing files separately.	The database and workspace are not one atomic snapshot; files that change during the copy can make it fail.

sudo env MODE=consistent scripts/backup.sh
# After checking the backup, resume explicitly:
sudo scripts/cluster.sh start

# Or back up while running:
sudo env MODE=live scripts/backup.sh

The archive is saved to backups/colloq-<time>-<mode>.tar.gz in the tools directory, and a named backup to backups/<NAME>/; set OUT= for another path. By default a consistent backup leaves writing stopped; with RESUME=1 the app starts again right after a successful backup. If the backup fails, find out why first; don't restart automatically. For a remote Vast VM installed from a release, use vast-sync with the same RELEASE=… as for vast-up: without RELEASE the Makefile takes the older vast-legacy.sh path, which makes a backup in the old format.

Restore on Linux

Use a trusted archive, the matching deployment name and a release that is explicitly compatible. On the prepared target server, run from its tools directory:

sudo scripts/cluster.sh prepare --release /path/release.json
sudo bash scripts/restore.sh \
  --archive /path/colloq-consistent.tar.gz \
  --release /path/release.json --replace
sudo scripts/cluster.sh prepare --release /path/release.json
sudo scripts/cluster.sh start

The example above is for a backup made without NAME. For a named backup, pass the same name when restoring, for example sudo env NAME=hse bash scripts/restore.sh …. The name must match the archive. --replace explicitly allows existing trees to be replaced; even the first prepare creates secrets and configuration. The tools check the archive and compatibility first, stop all writers, then install whole trees. The previous trees are kept privately in replaced-…; don't delete them until you have checked the restore.

The second prepare reconciles the restored runtime secrets and old catalog revisions. Don't pass a new --env-file at this step unless you mean to replace the restored configuration.

Check the result

The expected rooms open, and so do old teacher and student links.
Notebooks, publications, data and files match the backup you chose.
Two rooms run independently, and kernels use the right revisions.
In a GPU environment, a real CUDA operation succeeds.
You can restore even after the source machine's disk is lost.

Checksums and SQLite quick_check are required for integrity, but they don't replace checking the materials and a real class workflow.

If a restore was interrupted

An unfinished operation leaves .restore-in-progress behind. Until it finishes, starting and changing the deployment are blocked. Repeat the restore with the same verified archive and --recover --replace, or, from a full checkout, through make with RECOVER=1 REPLACE=1. Don't delete the marker by hand to get around the guard.

An explicit restore can bring back deleted rooms: it reconciles files with the persistent runtime reservations of deleted IDs. Regular start, stop and update don't reset those reservations.

Old formats and k3s

make restore-legacy DB=old.db FILES=old-files.tar.gz is for the old local format and is refused on top of an installed cluster. Migrating an old service needs a separate check of the schema and catalog.

A Colloq backup is not a backup of the k3s datastore. Before changing the k3s version, save its datastore and server token separately, following the k3s documentation.

A class on your own computer

If Colloq is installed with pip install colloq, colloq backup makes the backup. It is saved as two files in ~/.colloq/backups/: colloq-<date>-<time>.db and colloq-<date>-<time>-files.tar.gz. The backup includes the database, class files, the link-signing key, the setup token and the package lists of your own environments. Built environment images are not included, and neither is the .env file with the oracle key, RELAY_* and PUBLIC_URL. You need the sqlite3 tool; you can back up in the middle of a class.

colloq backup
colloq stop
colloq restore --legacy --db ~/.colloq/backups/colloq-DATE.db \
  --files ~/.colloq/backups/colloq-DATE-files.tar.gz

Restore won't run while a class is running. The existing database is moved aside to data/colloq.db.replaced-…, and files from the archive are copied over workspace/ without a separate backup. When the command doesn't run in a terminal, the --replace flag gives consent. From a checkout, make backup-legacy and make restore-legacy DB=… FILES=… do the same. This format doesn't work for a k3s install.