The ready-made image: make vast-vm

The quick way is make vast-vm HOST=class1 from a checkout of the repository, with VAST_TOKEN and an SSH key in .env. It rents a Vast VM over the API, starts the ready-made colloq-vast image on it (the same image as colloq-server) under colloq-host, and prints the class address and the owner link. Each room gets its own kernel container on that VM's Docker. The class address usually comes from your relay; fill in the settings — the relay token, the AI key, COLLOQ_IMAGE — once in backups/vast-onstart.sh. On the VM itself, looking after it — colloq-host status, backup, update — works as on a university server; the rental details are in deploy/vast/README.md. The data lives on the VM's disk and goes with the rental: copy the backups off before you delete it.

For GPU rooms or GPU submissions, rent the cards and the disk the CUDA images need: make vast-vm HOST=class1 GPUS=2 GPU_RAM=24 DISK=100 asks for at least two cards of 24 GB each and 100 GB of disk, since the VM holds base-gpu (about 9.9 GB) and kaggle-gpu (about 2.8 GB more). Both are published with every release by make kernel-publish (RELEASING.md), and the VM pulls them when they are listed in KERNEL_PRELOAD; it builds them itself only when its version's GPU kernels are not published yet, or their lists were edited on it. Whatever the class, make vast-vm rents only offers whose Max CUDA is 13.0 or more: the GPU environments' torch 2.14.1 is built for CUDA 13.0 and needs an NVIDIA driver of 580 or newer. DISK= and MAX_PRICE= override .env for that one rental, and OFFER= rents the offer with that number wherever it stands in the market. In backups/vast-onstart.sh set KERNEL_PRELOAD=base,base-gpu,kaggle-gpu, and once the class is up give submissions their cards in the panel; see a card for submissions.

Submission machines: make vast-runner

A Vast VM can also be rented not for a class but as a submission machine of another Colloq server: it takes that server's competition submissions and runs the notebooks, while the answers and the metric stay on the server. The server's owner mints the join code (Resources → Machines → Connect a machine, or colloq-host runner-code); then, from a checkout of the repository, with the same VAST_TOKEN and SSH key in .env:

make vast-runner COUNT=3 CODE=<code> URL=https://colloq.example.edu
make vast-runner COUNT=2 GPU="RTX 4090" GPUS=2 DISK=150 CODE=<code> URL=https://colloq.example.edu DESTROY_UNJOINED=1

make runners-update URL=https://colloq.example.edu   # after a server update: the machines onto its version's code
make runners-down URL=https://colloq.example.edu     # the machines leave the server, their VMs are destroyed

make vast-runner rents COUNT VMs, 1 to 16, with the same filters and the same Max CUDA 13.0 floor as make vast-vm; the disk is 120 GB by default. Each VM's on-start holds only the server's address and the code, no relay token: the machine calls the server over HTTPS by itself and opens no address. The machines are labelled colloq-runner, make runners-update and make runners-down find them by that label, and URL= keeps only that server's machines. A VM rented but never joined bills like any other: the command names it at the end, and DESTROY_UNJOINED=1 destroys it. runners-update first waits up to a minute for each machine's fresh word from its server, and leaves alone the machines whose code matches the server's (FORCE=1 updates them all the same). runners-down waits for a machine to leave only while its server answers: after 10 minutes of its silence the VM is destroyed all the same, and the command says so. With NOW=1 neither command waits for the submissions in progress, which run again, spending no participant's attempts. Submission machines are no deployments: make vast-up, make vast-down and make vast-status do not take them (vast-status only says how many are rented), and a deployment name that would give a VM their label (NAME=runner, runner-…) is refused. The details are under Machines for submissions and classes.

Before you rent

The k3s path uses a full Vast VM with systemd, not a regular Docker instance. The VM runs a single k3s node. Check that VM offers are available, and check for a working NVIDIA driver, a registered SSH key, the price and disk space.

On a trusted operator machine you need the release manifest and a clone of the public repository github.com/colloq-edu/colloq that contains its exact sourceCommit. vast-up packs the tools from exactly that commit. The app never gets your Vast account key.

In the configuration, prepare VAST_TOKEN, an SSH key and the settings of an existing relay. For private images, set VAST_REGISTRY_CONFIG with separate pull-only credentials and VAST_REGISTRY_READ_ONLY=1. The file must contain only Docker auths, with no credential helpers. A syntax check doesn't prove the credentials work — the preflight checks the pull.

Launch and monitor

For a release whose images really are available anonymously:

VAST_PUBLIC_IMAGES=1 RELEASE=/absolute/path/release.json \
  NAME=hse HOST=hse.example.edu make vast-up
NAME=hse scripts/vast.sh status
NAME=hse SINCE=2h scripts/vast.sh logs

Replace the name, domain and path. VAST_PUBLIC_IMAGES=1 is an explicit statement that the images are available, not a way to make private images public. With mixed registries, it applies to the images you pass no credentials for.

The make vast-… targets choose a path by RELEASE: without it they call the old scripts/vast-legacy.sh, meant for older Docker installs without k3s. For a k3s setup, call scripts/vast.sh directly or pass RELEASE=… to every target. App and kernel logs are saved to logs/hse/<date>/.

The tool checks the manifest before it touches your account, offers a VM and asks you to confirm the price. Only the release tools and the allowed configuration are sent. On an update the remote configuration is kept: not every change to your local .env reaches the VM.

When the GPU counts as ready

An HTTP response from the app and the output of nvidia-smi are not enough. The installer checks the RuntimeClass and allocatable GPUs and runs a real PyTorch CUDA operation in a throwaway non-root Pod. vast-up also runs the cluster smoke test with independent rooms.

A GPU room requests one device; there is no automatic sharing and no fallback to a shared kernel. The GPU check needs a free card. Passing the repository's local tests doesn't prove that a particular rental works.

Tested configuration

On September 9, 2026 we tested a dedicated Vast VM with an RTX 3090 (24 GB), Ubuntu 22.04, NVIDIA driver 580.95.05 and k3s v1.36.4+k3s1. In a GPU class, two clients got the result of a matrix multiplication in PyTorch 2.14.0 / CUDA 13.0 through a shared notebook.

We also tested room separation, a GPU tensor surviving an app and broker restart, and updating and restoring a remote class from a consistent backup. We used NVIDIA Container Toolkit 1.20.0-1 and device plugin 0.19.3. The test VM was deleted afterwards.

These results apply to that configuration. Other GPUs, external registries, a public relay and restricted access to node services need their own checks. Release updates and backup restores stop kernels; Python memory is not guaranteed to survive them.

Copy data off the VM

# While running: the database and files are not one atomic snapshot
NAME=hse MODE=live scripts/vast.sh sync

# Stop all writers and get a consistent backup
NAME=hse MODE=consistent scripts/vast.sh sync

# Explicitly allow resuming after a successful backup
NAME=hse MODE=consistent RESUME=1 scripts/vast.sh sync

Archives are saved to backups/hse/ on the operator machine. Each archive is verified on the VM and again after transfer. It contains secrets, configuration, release/catalog files and the workspace; images stay in the registry.

Colloq data lives on the VM's disk, and deleting the rental can destroy it. Don't count on local PVs being kept automatically or on being able to attach a separate Vast Volume to the VM: check the limits of the product you chose with the provider.

Restore and update

Run these commands from the repository clone on the operator machine. To update, run make vast-up again with the same NAME and a new RELEASE: on a machine with a release installed it runs cluster.sh update — a consistent backup, stopping kernels, the new release and a smoke test. On a new empty VM, vast-up itself verifies and restores the newest backup from backups/hse/; if you want a clean setup, move the backups out of that directory first. For a manual restore, follow the portable restore procedure with the right name and a compatible release, then repeat prepare and the room checks. Change the k3s version as a separate operation.

Old root-service/Compose installs and unmanaged non-empty data are not migrated automatically. Keep the old machine, export its data with the tools made for it and check the move separately.

Before you delete the rental

Get a consistent backup off the VM, check it and make sure the images you need are available. Only then run NAME=hse scripts/vast.sh down and confirm deleting that rental. A stopped cluster is not a backup.

For diagnostics, use durations such as SINCE=30m or SINCE=2h. Logs of room Pods that were already deleted may be unavailable; app and cell output may contain class data.