Machines for submissions and classes

What it is and when it helps

A submission machine is any machine with Docker that runs the notebooks of your Colloq server's competition submissions: a rented Vast VM, a university server, a lab workstation. The machine comes to the server by itself over HTTPS, takes a submission from the shared queue, runs the notebook in the same throwaway sandbox with no network that the server uses, and sends the result back. The server scores it: the materials, the queue, the hidden answers and the metric stay with it. The machine runs the same colloq-server image in its runner mode — no server, no relay tunnel and no database, just one agent container, colloq-runner.

Machines help in three cases:

• A big competition. A hundred participants send notebooks in the last minutes, and the server has a few places. The machines add places of their own; the queue stays one queue, fair per person, as without them (see the queue).
• GPU notebooks the server cannot hold. The server has no cards, or too few: GPU submissions go to machines with cards, such as Vast VMs with two RTX 4090s.
• A small server. A university VM with 8 vCPU holds the class, not the submission queue. Set Notebooks on this server to metrics only, and the server only scores the answers the machines send back, while its cores stay with the class (see below). A submission can also be given more cores than the server has (up to 32): a machine with enough cores takes such a notebook (see how a submission is executed).

For a participant almost nothing changes: the same competition page, the same time and memory limits, the same outcomes. While no place can take their submission yet — the machines are still downloading its environment image or data, or no machine that fits is online — they see “getting ready” instead of a place in the queue, and “being scored” while the server runs the metric.

Only the server's owner connects and sees the machines: in the panel they are in the Machines block of the Resources tab. A machine the owner opened to classes holds class kernels too; see classes on machines.

What stays on the server and what a machine sees

Back from the machine come the answer file submission.csv, the executed notebook and a report of the run. The server takes them as participant data: it checks their sizes and checksums, lays the files out itself, and writes the texts a participant reads — the brief ones under a hidden test too — in its own words. More on the boundaries under Isolation and limits.

The metric of a competition with a GPU runs on the server in the light kaggle-base image (numpy, pandas, scikit-learn, scipy), not in the competition's 10–13 GB GPU image: a metric needs two tables, not a card. The image is pinned to the competition at its first metric and does not change after that — neither for new submissions, nor on a rescore, nor when the competition's card is turned off later. The metric moves to the current kaggle-base only when asked to: in the editor of a competition with a card, with the Metric image row of the Metric section, and in the console, with Move the metric to the current kaggle-base next to a submission whose metric failed because the pinned image is no longer on the server. Every submission of the competition is then rescored, and no notebook runs again. So keep kaggle-base ready on the server before the first metric (KERNEL_PRELOAD=base,kaggle-base): a competition whose first metric ran without it stays on the submissions' own pinned images, as before this version; a baseline checked on the CPU before the card was turned on pins nothing. A metric that imports torch fails here as metric code — METRIC FAILED — and can be rescored once fixed. A competition whose metric never ran with a card still scores in the submission's pinned image.

Connect a machine

1. In the panel, open the Resources tab and, in the Machines block, press Connect a machine.
2. Set how many machines join on one code, how many hours the code is valid (up to 24), the places per machine (empty: by cores, see places) and, if needed, Trust the machines with the hidden test, which is off by default. The Take classes line opens class kernels to the machines too (see classes on machines); it never goes together with trust.
3. Press Show the commands. The code is shown once, inside three ready commands. Run one of them: the first two on the machine itself, as root; the third on your own computer, from a checkout of the repository.

# Docker is installed: Linux amd64, Docker Engine 28+; a GPU machine also needs nvidia-container-toolkit
docker run -d --name colloq-runner --restart unless-stopped --network host --stop-timeout 90 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/colloq-runner:/var/lib/colloq-runner \
  -e COLLOQ_RUNNER_JOIN='https://colloq.example.edu#<code>' \
  ghcr.io/colloq-edu/colloq-server:X.Y.Z runner

# A bare machine (Ubuntu): installs Docker and colloq-host, which installs NVIDIA Container Toolkit
command -v docker || curl -fsSL https://get.docker.com | sh
docker run --rm -v /usr/local/sbin:/host ghcr.io/colloq-edu/colloq-server:X.Y.Z install-host /host
colloq-host runner up https://colloq.example.edu <code>

# Vast VMs, from a checkout of the repository (see below)
make vast-runner COUNT=3 CODE=<code> URL=https://colloq.example.edu

The code is 32 hexadecimal characters. In the first command it stands in the address after #: that part of the address never goes into a request line. --network host puts the agent on the host's network: that is how it reaches class kernels, should the machine be opened to classes (see classes on machines). --stop-timeout 90 gives the agent a minute and a half to stop: in that time it hands the submissions it holds back to the server, and they do not count as lost. The code is valid for the number of machines you set and for a day at most. The server keeps only its fingerprint, so the code cannot be shown a second time — mint a new one. Send it to whoever sets the machine up, not to a group chat: while the code is valid, any machine joins with it, and that machine sees participants' notebooks.

The codes still valid are listed in the block under Join codes: for how many machines, how many have joined, until when the code is valid and who minted it. Cancel ends a code at once: no machine can join with it any more, not even one that already has the command pasted, and the machines that joined keep working. Disconnect all cancels every code still valid too.

Under the commands the window says Waiting for the first machine… and then names the machines as they join. You can close the window; the machines appear in the list by themselves. When the address in the commands is this server's own loopback (localhost), the window warns that a machine anywhere else cannot reach it: set PUBLIC_URL. A machine that has joined starts downloading the environment images and the data of open competitions at once, and takes a competition's submissions once everything for it is ready (see readiness). The server accepts at most 10 joins from one address in 10 minutes: join machines behind one NAT in batches.

Without the panel. When the panel is out of reach — for example, the server is reached over SSH through Teleport only — colloq-host runner-code on the server's machine mints the same code. It prints the code once, inside the same three commands:

colloq-host runner-code --machines 3 --hours 12
colloq-host runner-code --machines 2 --trusted --slots 4 --prefix lab

--machines is how many machines join on the code, 1 to 100 (1 by default); --hours, how many hours it is valid, 1 to 24 (24); --trusted gives the machines the hidden test from the start; --classes makes them take classes too (never together with --trusted); --slots caps the places on each machine, 1 to 64; --prefix names the machines on the server (lab-1, lab-2…; without it a machine is named by its host name, and the panel does not change that name); --url is the address the machines reach the server at (PUBLIC_URL by default).

Which image the commands name. The one the server runs: a published release as ghcr.io/colloq-edu/colloq-server:<version>, an image from your own registry or mirror in full, and then colloq-host runner up gets it as its third argument. In the second case there is no make vast-runner command: rented machines take the published image, not yours.

A private registry. The first two commands pull the image without logging in to the registry, so log the machine's Docker in first, as root, with a read-only token: docker login <registry>. colloq-host runner up and runner update log in by themselves too when given COLLOQ_REGISTRY_USER and COLLOQ_REGISTRY_TOKEN, but only to the agent image's registry, and the variables themselves are not kept in runner.env. The agent pulls kernels by digest from the registry the server took them from (KERNEL_IMAGE_REPO), with root's login, which colloq-host passes to the agent's container: when the kernels are in another private registry, log in to that one too. An agent started by the first command does not get that login: it pulls private kernels only once colloq-host has recreated its container with a first runner update (see updating).

What a machine needs

• Linux on x86-64 and Docker Engine 28 or newer. The image is built for amd64 only.
• A dedicated machine. The agent holds the Docker socket, which is root on the machine. It gives every card it finds to submissions and counts free memory over the whole machine, knowing nothing of a Colloq server's rooms on the same Docker. So do not put it next to a server that runs classes: colloq-host runner up warns when it finds a server container there, and with --classes refuses. A university lends the competition a machine it can give over whole.
• Disk: at least 40 GB without cards and 120 GB with them, as rented or sold. The file system on a disk shows a few percent less, and the check allows 5 % for that. The GPU kernels take about 10–13 GB, and the cache of data and packages and the runs themselves come on top. A smaller disk shows in the panel as low on disk; when space runs out, the machine removes what has not been needed for a long time by itself.
• For GPUs, an NVIDIA driver of 580 or newer and NVIDIA Container Toolkit. The GPU kernels are built for CUDA 13.0, and on an older driver torch does not see the card. colloq-host runner up installs the toolkit itself, through apt, but not the driver, and says right away whether the driver will do. The machine checks a GPU image with a short torch run on a card with a share free, and takes no GPU submissions until that check passes: the panel then says driver 535, 580+ needed: the GPU image will not start. While submissions hold every share, the check waits and counts as no failure. The machine still takes CPU submissions.
• Outbound connections only. HTTPS to the server's address (PUBLIC_URL; http:// only for a server on the same network), to the image registry (ghcr.io or yours) and to the package index the server took participants' own packages from; when the index cannot be reached, the machine takes the packages from the server. No inbound port is needed. Pass an outbound proxy and an internal CA (HTTPS_PROXY, NO_PROXY, NODE_EXTRA_CA_CERTS) to the colloq-host runner up call: it writes them to /etc/colloq/runner.env. Images are pulled by the machine's Docker itself, which gets its proxy separately, as on the server (see the outbound proxy).
• The agent's directory is /var/lib/colloq-runner, at the same path on the host and in the container, with mode 0700; it holds the machine's identity (runner.json, mode 0600) and the cache of data and packages. COLLOQ_RUNNER_STATE names another one — a directory of its own only, such as /srv/colloq-runner: the agent refuses system directories and the state directory of a Colloq server.

For the server machines add requirements of their own about images. A competition's environment image must be on the server when the server pins the environment to the competition — at the first run of its notebook, usually the baseline check — and when it prepares participants' own packages; the machines pull the same image from the registry. After that, a GPU competition whose notebooks only machines run (notebooks are off on the server, or the server has no card for submissions) needs no image of its own on the server to take uploads when three things hold: its metric runs in kaggle-base (which the server then needs instead), the environment's revision is in a registry, and a machine with cards that fits one of its submissions was heard within the last hour — the machine's memory less 2 GB holds the submission's memory plus 256 MB, and its cores less one the submission's cores. A competition whose metric stays on the submissions' own images (its first metric ran without kaggle-base; see the metric) needs its own image on the server as before, and a competition on the CPU always does: its metric runs in it.

Places, cards and trust

Places are how many submissions a machine runs at once. By default, auto: as many as the machine offers by its own cores and memory. It keeps one core and 2 GB of memory for itself and Docker, and takes a submission only when the competition's limits — the cores and the memory, plus 256 MB on top — fit into what is left. A number of your own, 1 to 64, is set in the connect window (Places → per machine), with Places… in the row's menu, or on the machine's page; empty is auto again. The server fills a machine's free places in the same fair queue order as its own.

The panel counts places in submissions of the size there are: a machine with 61 vCPU offers 60 places, a core per submission, while submissions of 2 cores fit on it 30 times by its cores, and fewer when their memory, with 256 MB each on top, does not fit 30 times into what is left of its memory; the panel says the smaller number. The competition console and the editor's Runs on line count places in the competition's own submissions, the list of machines in the submissions waiting or running now, and without them in the largest submission of an open competition.

Cards. A machine gives every card it has to submissions — unlike the server, where the owner ticks the cards for submissions. How many submissions share one card is set with Runs per GPU… in the machine's menu; empty means as on the server, under GPUs for submissions (Runs per card). The share of video memory is the card's memory divided by that number, and it is held as on the server: by the PyTorch cap inside and the watchdog outside (see a card for submissions). A card speeds up only what the notebook moves to CUDA itself, its model and tensors: a notebook written for the CPU runs on the CPU even on a machine with a card. A machine without cards takes no GPU submissions.

A trusted machine is the only kind that gets the hidden test. Trust is given when connecting (Trust the machines with the hidden test) or later, with Trusted in the row's menu or on the machine's page. Taking trust away sends its runs with the hidden test back to the queue, cuts a download of the test short, and purges the test from the machine. While no machine is trusted, submissions with the hidden test are run by the server when its notebooks are on, and wait otherwise: waiting for a trusted machine: hidden test. Better not trust a rented machine with the hidden test: whoever runs the host sees everything on it.

The list of machines and the console

The Machines block shows a summary on top — how many machines are online, how many places and cards they have — and below it a row per machine: its state, resources, places (busy of how many), readiness and a menu.

Readiness. For every competition that is open, opens within a day or has waiting submissions, the machine downloads the environment image, the data and, when trusted, the hidden test in advance, checking the checksums. It takes a competition's submissions only once everything is in place; participants' packages are downloaded in advance too, and a missing one is fetched while the submission is prepared — download time is not counted in the notebook's time. In the row this reads as image ↓ 45%, data ✓, hidden test: trusted machines only. A competition whose environment was built on this server is marked only this server runs it: its image is not in the registry, and machines never get it (see environments from the registry). Preparation that failed names its reason: driver 535, 580+ needed: the GPU image will not start, torch sees no CUDA: the GPU image will not start, the image did not download, the data did not download, or its checksums did not match, low on disk.

The machine's page (Machine page in the menu): what the machine said about itself — cores, memory, disk, cards, driver, Docker; its places, runs per GPU and trust; its readiness for each competition; what it runs now, with Stop the submission — a stop reaches the machine with its next call, within 15 s, and a submission already exporting or sending its result back can no longer be stopped on the machine; the last 50 events, and counters of the submissions it ran, its losses and its declines.

The competition console groups the executors: This server and each machine with its places and cards; a running submission shows which machine and card it runs on, and the console's header counts this server's metrics apart, as METRICS · 2 OF 2. A waiting submission carries a reason when it waits for more than its turn: machines are downloading the image, machines are downloading the data, waiting for a trusted machine: hidden test, waiting for a GPU: none here or on the machines, fits no machine online, nobody to run it: this server only scores, and no machine takes work, nobody to run it: too big for this server, and no machine takes work, back: vast-1 went silent. A submission whose notebook has run and waits for its metric is being scored.

Running now on the Resources tab shows a competition whose notebooks run on machines as a row of its own — with the machines under it and its memory as “0 MB here”: submissions on machines hold none of the server's memory; only their rows in the queue and the metric of the answers that came back stay here. Stop the submissions on machines cancels them: each stops on its machine with that machine's next call, within 15 s, and its participant spends no attempt.

The competition editor, in its Runs on line, says who will take the competition's submissions: this server, how many machines and places, which machines are still preparing and which do not fit. It also warns when the environment was built on this server (this server only) or when nobody can run the submissions with the hidden test.

Notebooks on this server, and scores

Above the list of machines are two fields of the Resources form; the form's Save changes saves them.

• Notebooks on this server. By default, run them here too: both the server and the machines run notebooks. With metrics only, only the machines run notebooks, the server scores the answers that come back, and its cores stay with the class; the memory scale on the same tab then counts only the scoring places for competitions. If no machine is online under metrics only, submissions wait and the block says so; the form saves that setting only after asking. Submissions of an environment built on this server, and submissions with the hidden test while no machine is trusted, have nobody to run them under metrics only — the competition editor warns about that.
• Scores at once: how many metrics the server runs at once, 1 to 8, 2 by default. A metric takes up to 2 minutes and 2 GB per answer. The metrics of answers from the machines and rescores use these places, not the notebook places. Notebooks and metrics share the server's memory: when it fits only one of them, the one that has waited longer goes first (work sent on time before late work); an answer from a machine waits from when its notebook was sent, a rescore from when it was asked for. A paused queue holds new notebooks and rescores, but the metrics of answers already back from machines run under the pause too. In a burst of submissions that machines run, these places are the bottleneck: on a server with cores and memory to spare, raise their number (see the check on Vast).

Updating

A machine takes new submissions only while its code matches the server's — the agent's build from the same release. A release that leaves the machines' code alone keeps them working as they are. A release that changes it makes them outdated: they finish the submissions they hold and take no new ones, and the row says, for example, code 0.23.0, server 0.24.0: takes no new submissions. So the order is: the server first, then the machines.

# 1. The server, as usual
colloq-host update ghcr.io/colloq-edu/colloq-server:X.Y.Z

# 2. Vast machines: every machine of this server, from a checkout of the repository
make runners-update URL=https://colloq.example.edu

#    or one machine with colloq-host, over SSH, as root
colloq-host runner update

colloq-host runner update learns the server's version from its /api/health, pulls that version's image, drains the machine — the submissions it holds finish, and it takes no new ones — and starts the agent on the new image; colloq-host itself is updated along the way. When the machine itself cannot reach /api/health, the agent names the version, but only from a sync made after the update began: the command waits up to a minute for one (COLLOQ_RUNNER_FRESH_SEC), and without it asks for the version. The drain waits no longer than the longest run the protocol allows (COLLOQ_RUNNER_DRAIN_SEC, 15,720 s by default). With --now it does not wait: the submissions it holds go back to the queue and run again, with no loss and no participant's attempt spent. A version can also be named: colloq-host runner update 0.24.0. An image without the agent — a release from before submission machines, after a rollback of the server, say — is refused by runner update and runner up before anything is drained or started: a running agent stays on its old image.

A machine joined with the first command (docker run) has no colloq-host. Install it from the same image — docker run --rm -v /usr/local/sbin:/host <image> install-host /host — and the first colloq-host runner update, leave, status or logs takes its container over: it reads the image, the server's address, the code and the directory out of it, and from then on manages the machine as one set up with runner up.

make runners-update does the same on every Vast machine at once. On each machine the update runs in the background and survives a dropped SSH connection, and running the command again follows an update already under way. First the command waits up to a minute for each machine's agent to sync after the command began: a machine whose server said in that sync that its code is the server's own is left alone (FORCE=1 updates it all the same), and a machine with no such fresh word is updated, with colloq-host on it deciding the rest. NOW=1 is --now; VERSION= is one version for all.

A restart of the server does not cut off the submissions on machines: the machines finish them and send the results once the server is back, and for the first 3 minutes after a start the server counts no machine as gone. Still, update between competitions, and when that is not possible, update the machines right after the server. Machines do not update themselves in this version. An update does not interrupt the classes on a machine. A machine set up with colloq-host before 0.25 needs two updates to take classes (make runners-update makes both), and one joined with docker run needs colloq-host first; see a machine with classes.

Disconnecting and leaving

• colloq-host runner leave on the machine (on a machine joined with the first command, install colloq-host first; see updating): it finishes what it holds, takes no new submissions, revokes itself on the server and removes the agent's container. The cache of data and packages stays in /var/lib/colloq-runner; remove the directory to free the space. The kernel images stay in the machine's Docker (colloq-revision:<id>); docker image rm or docker image prune -a removes them. A leave ends with the server's answer: when the server has been silent for 10 minutes, colloq-host stops waiting and says so, while the agent goes on leaving — it takes no new submissions, and revokes itself as soon as the server answers. Only a new code brings the machine back: it does not join again on the old one, even when a Vast on-start repeats it at boot. A machine with classes on it asks first; see a machine with classes.
• make runners-down URL=https://colloq.example.edu does the same for this server's Vast machines, and then their VMs are destroyed with their disks; the command asks first. A machine that did not leave (its server was silent for 10 minutes, say) is destroyed all the same, and the command says the server still lists it: disconnect it in the panel. With NOW=1 the VMs are destroyed at once, without waiting: the server loses the machine after 3 minutes of silence and runs its submissions again, and the machine stays in the list until it is disconnected.
• Disconnect in the row's menu (or Disconnect all under the list): the server stops accepting the machine's secret at once. Its submissions go back to the queue without spending any quota, late answers are refused, and at its next call the agent purges the hidden test and stops, so that it does not start again. That does not destroy a rented VM, which bills until it is destroyed. The machine does not come back by itself, but the code it joined with stays valid until it expires, is used up, or is cancelled under Join codes; Disconnect all cancels every code still valid too.

A machine leaving costs participants nothing: the submissions it did not finish run again elsewhere, and the leaving spends neither the daily quota nor attempts. A disconnected machine, or one that left, stays in the list for another 30 days, and then the server removes it with its events.

When a machine fails

• Silent. A machine the server has not heard for 3 minutes loses its runs: they go back to their place in the queue — ahead of those who came after them — and run again, on another machine or on the server. The participant reads: “The machine running this submission stopped running it. It is back in its place in the queue and will run again; no attempt was spent.” The server counts a machine as gone only when it knows the way to the machines works: another machine answered within the last minute, or its own check of its address passed. When the way itself is gone — the server's tunnel or proxy is down — nothing is lost for up to 15 minutes, and once the way is back every machine gets its full 3 minutes again.
• Declines. A machine that cannot start a submission — no image, data or packages, not enough disk, memory or card share — gives it back before the container starts. The submission goes to another place at once, and that machine is not offered it for 10 minutes. A decline costs the participant nothing.
• Loses it again and again. A submission lost by three different machines (or six times on any) is not queued again: it ends as a failure of the server — “This is a failure of the server, not of your notebook; no attempt was spent” — and spends neither the quota nor the ceiling on failed submissions; it can be run again from the console. A submission three different machines declined ends the same way.
• Hangs. A run that goes past its ceiling — 10 minutes to prepare, the notebook's time, 2 minutes more and 10 minutes to send the result — is taken back by the server and counted as lost, even while the machine keeps answering.
• Quarantine. Three losses or five declines on one machine within 30 minutes, and the server drains it by itself: it finishes what it holds and takes nothing else, and a banner appears above the list. Runs lost at one moment — every run of one agent restart, say — are one loss, and only losses after the container started count. Look at the events on the machine's page and, once the cause is fixed, press Resume: the quarantine count starts over.
• An agent restart. When the agent crashed or the machine rebooted, Docker starts it again. At start the agent first makes sure that no run of its earlier life is still going, and removes their containers; the results it had already written down are still sent, and the rest of its runs the server counts as lost at once and queues again. A stopped agent (docker stop, colloq-host runner update --now) hands them back to the server first, with no loss. To Docker, docker kill colloq-runner is a manual stop: such an agent does not restart by itself, the server requeues its runs after 3 minutes of silence, and the submissions' containers stay on the machine until the agent is started again — with docker start colloq-runner, or the same colloq-host runner up that joined the machine; at start it removes them. Class containers it does not remove at start but takes up, with their variables (see classes on machines).

Classes on machines

A machine the owner opened to classes (Takes classes) holds, beside submissions, the kernels of classes too: those whose Where it runs says On machines. It is the same agent, the same image and the same join command, and such a machine takes submissions as before. For each class it runs:

• the class notebooks' kernel: the shared notebooks and the terminal, one container per class; a class whose environment uses a GPU gets a whole card of the machine;
• the personal notebooks' kernels: every personal notebook of the class in one container, up to 60 kernels, with no GPU;
• the council pool: the workers of the attempts, when the class's council runs on machines; one container per class, with no GPU (see the council on machines);
• a copy of the class folder: /var/lib/colloq-runner/rooms/<class>/, one for the class's containers; the council pool also has /var/lib/colloq-runner/council/<class>/, the notebooks' snapshots and the attempts' own folders. The attempts' folder (…/scratch) belongs to root, mode 1733: the agent itself makes it so before every start of the pool's container, so that attempts cannot find each other's folders (see security).

The council pool loads a snapshot of the notebook and runs students' attempts at the same time, each in a worker of its own. Its container takes, for each worker, Memory per attempt, about 256 MB for the kernel itself and the unpacked part of the snapshot, the strings, lists and small objects each worker holds on its own; once, the snapshot's numeric files, which all the workers share; 512 MB of headroom; and a CPU core per worker; it grows and shrinks with the pool. It sees the class folder read-only, and the server brings the files attempts write to their cards. Without machines, with the council In the shared kernel, and while the pool is starting, attempts run as before, one by one in the notebook's kernel, wherever that runs.

Everything else stays on the server: the database, the notebooks with their output, the class files themselves (the machine has a copy), the queue of cells. The kernels' Jupyter listens on the machine's 127.0.0.1 only, and the server reaches it through a tunnel: one WebSocket that the agent opens by itself to /api/runners/v1/tunnel at the server's address, so the machine still needs no inbound port. Submissions and classes share the machine's memory and cores: classes take them first, but a running submission is never stopped for a class. Variables live where the kernel is: a broken link does not wipe them, and a move of the kernel elsewhere does (see moves).

Opening a machine to classes. Only the owner can, and only on a machine without trust: a machine that takes classes is never given the hidden test. In the connect window it is the line Take classes — the machine will see students' code, data, files and personal notebooks (Trust the machines with the hidden test is then unavailable); on a machine that has joined, Takes classes in the row's menu or on the machine's page, with a confirmation. Without the panel, colloq-host runner-code --classes mints such a code; together with --trusted it refuses. Stop taking classes moves the machine's classes off at once — variables are lost, files stay — and the machine takes submissions as before. Your own or a university machine will do; give a rented one teaching data only: whoever runs its host sees everything on it (see isolation).

What a machine needs beyond what submissions need:

• Docker run by root that manages iptables itself (the DOCKER-USER chain): rules hooked into it fence each class's network. Rootless Docker, "iptables": false in daemon.json and the nftables backend will not do.
• cgroup v2: it shows how much memory a kernel took and why it died.
• No Colloq server on this Docker: none of its container, its rooms or their network.
• 20 GB more disk for the class folders, on top of a submission machine's 40 or 120 GB.
• The agent on the host's network (--network host): that is how it reaches the kernels' Jupyter on 127.0.0.1. Every join command of this version starts it so; for machines joined earlier, see below.

For a code that takes classes, the connect window's second command is colloq-host runner up URL CODE --classes: it checks the first four points before it pulls or starts anything, and names everything that is wrong. A machine that has joined tells the server what it can do, and the Classes: line in the list of machines names what is missing: needs --network host — restart the machine with the command from the panel, perimeter: rootless Docker or nftables only — no classes, this machine runs the Colloq server — no classes, image ↓ 45% — takes classes once base is downloaded. Before it takes its first class, the machine checks its sandbox from the inside, with a throwaway container under a class's hardening: the process is not root, has no capabilities and cannot gain privileges, sees neither the Docker socket nor the agent's files, cannot reach the cloud metadata, the machine itself, its local network, IPv6 or another class's port, and reads and writes the class folder. Until that check passes, no class goes to the machine: the admission check failed: …, and while the machine holds no class image yet, the admission check runs once a class image is on the machine.

The owner's settings are the Classes on machines block of the Resources tab; they take effect at once:

• Teachers send classes to machines (on): a teacher picks Where it runs for their own class; when off, only the owner decides.
• Default for new classes: the preset a new class starts with, Everything here by default. A class whose environment uses a GPU gets Everything on a machine with a card under any preset but Everything here. Existing classes do not change.
• Move after: 60 s, 30 to 300; the delay after which a class moves until it is given one of its own (see moves).
• File from a machine — up to: 200 MB; a larger file a kernel wrote on the machine stays there (see files).
• Class files: 4 MB/s, and Submission downloads during classes: 2 MB/s. While classes run on machines, class files go both ways no faster than the first, and competition data goes to machines no faster than the second (without classes, up to 8 MiB/s by default), for all machines together. Files a cell's run waits for go at up to 8 MB/s. Cells, output and the terminal are not held to these: the machine only smooths its output to 6 MiB/s.

Where a class runs

Where to compute is chosen for each class on its own: in its settings in the panel, the Where it runs block of the Resources part, and in the New class window. Presets come first:

• Everything here: as before, nothing leaves. Classes from before 0.25 run this way, and new ones too while the owner's Default for new classes is Everything here.
• Council and personal notebooks on machines: the class notebooks stay here, and the lecture has no link delay, while the thirty kernels of personal notebooks and the council's attempts leave the server: a pool on a machine runs the attempts, on a snapshot of the class notebook (see the council on machines).
• Everything on machines: only notebooks and files stay here. A class whose environment uses a GPU is offered Everything on a machine with a card instead: a whole card of the machine for the class notebooks; personal notebooks and the council pool never get a card, and a snapshot of the notebook holding data on the GPU keeps the attempts in the class notebook's kernel, on the card.

Classes that were given Personal notebooks on machines (the second preset's name then), Everything on machines or Everything on a machine with a card in 0.25 — at creation, through the owner's Default for new classes too, or later in Where it runs — and were not set back to Everything here are stored with the council On machines: once the server runs 0.26, a pool runs their attempts.

Below them are the same parts one by one: Class notebooks and Personal notebooks, Here or On machines, and Council, In the shared kernel (attempts one by one in the class notebook's kernel, wherever that is) or On machines. A council on machines has two numbers more: Attempts at once, auto (2 to 16 workers, as many as fit) or a number of your own up to 32, and Memory per attempt, 1 GB by default: each attempt's limit. Two more rows:

• If no machine fits — here (on): when no machine fits now, the kernel starts on this server if memory allows, and the kernel log says: “No machine can take the kernel now: it starts on this server.” When off, the start refuses: “No machine can take this class now, and it may not compute on this server.”
• If the machine disappears: move after 60 s (30 to 300), or wait for the machine (see moves).

Under the form are the machines that fit the class now: Fit: vast-3 (41 GB free), lab-2 (18 GB), marked far: +317 ms per cell when the link to the machine (its p95) is over 150 ms, or why none fits: not enough memory, no image yet, not online, outdated, quarantined, went silent recently… The class's own refusals: The environment was built on this server — machines do not have it, since only the colloq environments from the registry go to machines (base, base-gpu, kaggle-base, kaggle-gpu; see environments from the registry); A graphics card is needed: on a machine with a card; Machines do not take classes — Resources → Machines.

Which machine the server takes. A machine fits when it is online, of the server's own build, open to classes, not quarantined for classes, with its tunnel up, its sandbox checked and the class's image (ready or downloading), with memory free for the kernel plus 256 MB more, enough cores, and, for a class with a card, a whole free card. Of those the server takes the one that already holds another kernel of the class, then one with the image ready, then the shorter link and the more free memory. The council pool goes first of all where the class notebooks run: its snapshot then never crosses the network. Machines download the class's environment image in advance, as soon as the class is sent to machines.

Who chooses, and when it takes effect. The class's teachers while Teachers send classes to machines is on, the owner alone otherwise; a teacher then reads The owner decides where classes run for now…. A new choice takes effect at the next kernel start: a running kernel stays where it is until it is stopped or moved. Kernels on a machine hold the machine's memory, not the server's: on the Resources tab's memory scale and in Running now they take 0 MB here. A server in Kubernetes (the Helm chart) has no such block: there every room has a Pod of its own.

How much memory and how many cores. A class's memory and cores are bounded by the place of every part they reach: this server for a part here, the biggest machine open to classes for a part on machines (its memory less what the machine keeps for itself, its cores less one), and the tighter of the two when the number reaches both. Under the numbers the form gives the machines' line: On machines: memory up to 61.8 GB, cores up to 31; the biggest of them, vast-3, gives no more. The server saves no number past the bound and names the bound; it refuses to move a class to a place that cannot give its numbers until they are lowered. The personal notebooks' memory and cores are bounded the same way. These numbers do not reach the council pool: its container follows from Memory per attempt and the number of workers.

What the room and the panel show

Students see only that the kernel is on a machine: Kernel on a machine in the notebook's bar, without the machine's name. Teachers see the name and the link's delay: Kernel · vast-3 · 52 ms. Cells, output, interrupt, restart, input(), running a file, help and the terminal work as on the server; hover help and completion wait a little longer for their answer, up to a second more, for the way to the machine. When a kernel takes longer than 8 seconds to start on a machine, the kernel log says “Starting the kernel on a machine…”, and teachers see the same above the notebook with the machine's name and the image's progress: “Starting the kernel on machine vast-3… image ↓ 45%”.

No link. When the link to the machine is gone — its tunnel broke, or nothing came from the machine for 12 seconds — the notebook's bar turns amber: No link to the machine · 0:12. After 10 seconds without the link the kernel log says: “No connection to the machine · 0:12. The kernel and its variables there are intact — waiting for the connection. New runs will wait.” Queued cells wait, marked waiting for the machine; restart and stopping a cell are refused meanwhile, to whoever pressed: “No link to the machine — a restart cannot reach it now. Press again when the link is back.”, “No link to the machine — the cell cannot be stopped now. The time limit will take effect once the link is back.” Once the link is back, runs go on in the same kernel, and the log says “The connection to the machine came back after 37 s.” When a break ate a cell's result, the cell says so: “The connection to the machine broke — the cell’s result is lost. The kernel and its variables are intact; run the cell again.” A restart whose answer was lost in a break is settled once the link is back: “The restart went through: the variables were reset.” or “The restart did not get through — the variables are still there.”

The teacher's banner. After 10 seconds without the link, teachers see above the notebook what happens next — for example “Machine vast-3 has not answered since 14:32:05. Waiting until 14:33:05, then the kernel starts again (the variables will be lost).” — and buttons. Move now moves the kernel at once, without waiting for the deadline: the variables are lost, and the kernel starts at the next run. Wait for the machine sets the class to wait for the machine: the kernel does not move by itself until someone presses Move now (“Waiting for the machine: the kernel will not move by itself until you press “Move now”.”), and the deadline can be brought back in Where it runs; that button is shown to whoever may change Where it runs. While the machine's agent updates, the log says “The agent on the machine is updating. The kernel and its variables there are intact — new runs will wait.”

In the panel. In the Machines list a machine has two chips, submissions and classes, and a line such as Classes: 3 classes · 22 of 60 GB · card 0 held by a class · link 48 ms (p95 80). The machine's page lists its classes — each one's kernels, memory, CPU, link and files — and its events: moves with their reason, tunnel takeovers with the addresses, the quarantine. Move classes in the machine's menu moves all its classes off at once — variables are lost, files stay — and the machine keeps taking classes. Running now shows a class with kernels on machines with class, personal and council sub-rows (the pool's with its number of workers, 3 workers): where they run, how much memory and CPU they take, and how their files go — files: in sync, files: ↑ 3, 12 MB, files: 2 only on the machine; such kernels take 0 MB here. A stop of kernels on a machine without a link happens once the machine is back.

A class's files on a machine

The server keeps the class's files, the machine has a copy, and the copy keeps up with the server both ways. What changed here — an upload, a save in the editor, a write by a kernel on the server — goes to the machine a second later, and before every run of a cell, a file or an attempt on the machine the server checks that the copy has caught up. When nothing changed, a run waits for nothing. When a big file is on its way, the cell shows “Files are on their way to the machine: 12 of 150 MB…”, and when nothing moves for 30 seconds, or 2 minutes have passed, it does not run: “The files did not reach the machine: 30 s without progress — the cell was not run.”

What a kernel wrote on the machine comes here: after every run, and every 5 seconds while the kernel computes (every 30 when idle). A file is taken once it has not changed for 2 seconds, and a new folder once everything in it has settled, so an unpacking or a pip install never comes half done. A new file shows in the files panel once it is here; a file whose newer version is still on its way is marked A newer version is on its way from the machine, and a download waits for it.

• Both changed one file. The server's version stays, and the machine's lands beside it as a copy: “data.csv was changed here and on the machine at the same time. The server’s version was kept; the machine’s is “data (from the machine 14-32).csv”.”
• Deleted here, rewritten there. The delete wins: the file does not come back, and the kernel log says “old.csv was deleted on the server; the changes made to it on the machine were dropped.”
• Renamed on the machine. It is renamed here too, with nothing sent again.
• Notebooks. The class's notebook files belong to the server: a kernel on the machine cannot rewrite or delete them, and the server puts its own version back.

What stays on the machine. A file over File from a machine — up to (200 MB), or one that does not fit the class's limit (All of a class's files up to, MAX_SESSION_MB, 1 GB), does not come here: the files panel marks it on the machine, and a teacher can Fetch it, which brings a file over 200 MB too as long as it fits the class's limit and the server's proxy lets such a body through (see the reverse proxy). A file whose body the proxy turned away (its 413) stays on the machine too, until it changes: the kernel log says “model.pt (150 MB) stays on the machine: the proxy in front of this server refused a file this large. …”, and Fetch refuses until the proxy's limit is raised. Files past 5000 in the folder stay on the machine as well. Packages installed into the class folder (pip install -t .) never come at all, and show as one row, packages (pip -t) — on the machine, 2.1 GB. Links, hidden files, __pycache__ and temporary files are never synced. Whatever is on the machine is lost with it, as the kernel log says once: “model.pt (2.1 GB) is over the 200 MB limit and stays on the machine. If the machine is lost, the file is lost too.” Such a file is deleted like any other: Delete in the files panel (a teacher's) deletes it on the machine, and the row goes as soon as the machine has deleted it.

Before the end of a class (up to 2 minutes), the build of its page (up to 15 seconds), a move of a kernel off a machine that answers, and turning off its Takes classes (up to 10 seconds), the server waits for the files from the machine; a machine that leaves (runner leave) sends its classes' files first by itself (up to 10 seconds). When a class's kernels on the machine are stopped — the end of the class, the owner's stop, idling — the changes that had not arrived are named in the kernel log: “The kernels on the machine were stopped; the file changes that had not reached the server were dropped: out.csv, model.pt.” After a move, the files are as they last arrived (“the files are as of 14:31:58” in the move's note), and a machine that comes back after a move overwrites nothing here: its late changes are dropped — “The machine came back after the move; its file changes after 14:32 were not accepted: out.csv, model.pt.” — and its copy of the folder is removed.

When a class's machine is lost

A broken link is no loss yet. The agent brings its tunnel back by itself, and what was on the way is sent again when the break lasted under two minutes; the kernel and its variables on the machine stay intact all the while. A kernel moves only when the server is sure the machine is gone:

• The machine is silent — no tunnel and no calls — for longer than the class's delay (move after 60 s), while the way to the machines works: another machine is online, or the check the server makes of its own address every 5 seconds while classes run on machines passed (see the server). When the server's own way is gone, nothing moves: nobody could reach the class anyway.
• The tunnel does not come up while the machine answers its calls: after 2 minutes, since it cannot run the class anyway.
• At once: the machine was disconnected, Takes classes was turned off, it left (runner leave), its network got out of control, or a teacher pressed Move now.

Nothing moves in the first 3 minutes after the server starts, nor in the first 2 minutes after an agent went away to update (colloq-host runner update). A class set to wait for the machine never moves by time, only on Move now.

When it moves, the variables are lost, the queue is cleared (“The queue was cleared: its cells counted on variables that no longer exist.”), and the running cell says “This cell was running on the machine when the machine was lost. Whether it finished is unknown; the new kernel has none of it.” The room's kernel log says “The machine has not answered since 14:32:05. The kernel will start again: the variables are gone, the files are as of 14:31:58. Run the cells again.”, and teachers read the same with the machine's name. The kernel starts at the next run: on another machine that fits — it does not go back to the one it left for half an hour — otherwise on this server, if If no machine fits — here is on and memory allows; the log says where (“The kernel was started on another machine.”). When the personal notebooks' machine is lost, the class notebooks are not touched: only personal notebooks lose their variables, and their kernels start at their next run. If the council pool's machine is lost while the class notebooks run elsewhere — here, as with Council and personal notebooks on machines, or on another machine — the notebook's kernel is not touched: an attempt that was running there runs once more in the notebook's kernel, the waiting ones go there too, and the pool starts again wherever it is placed and loads the snapshot anew. But the pool goes to the class notebooks first, and with Everything on machines and Everything on a machine with a card it usually shares their machine: then losing it is a move for the notebooks too. Their variables are lost as above, the attempts run again in the notebook's new kernel, without the teacher's variables, and the pool starts from a snapshot of the new kernel, so the setup cells have to be run again (see the council on machines). A kernel that died by itself on the machine (out of memory, say) starts again there, with the same words as on the server.

What survives.

• An agent restart, of an agent that crashed or was killed with kill -9 inside its container: Docker starts the agent again, and it takes up the running class containers. Variables and the terminal stay.
• colloq-host runner update: the old agent tells the server as it goes that it will be back, the server holds its classes for 2 minutes, and the new agent takes up their containers. Variables stay.
• A server restart: kernels on machines keep running, the server connects to them again, and variables stay; the queue of cells is lost, as for kernels on the server. Files a kernel wrote while the server was away arrive after it starts.
• docker kill colloq-runner is a manual stop to Docker: the agent does not come back by itself, and its classes move after the class's delay.

Class quarantine. A machine lost twice within 30 minutes (silent, or its tunnel stuck; moves by hand do not count) gets no new classes: above the list of machines, “Machine “vast-3” went silent twice in 30 minutes — classes are no longer sent to it.” with Resume, and Resume classes in the machine's menu. It takes submissions as usual: this is not the submission quarantine.

Link takeovers. A machine has one tunnel: a new one closes the old, and replacements go into the machine's events with both addresses. When the link was taken over from different addresses more than three times within 10 minutes, the list shows “The link of machine “vast-3” was taken over 4 times in 10 minutes from different addresses: 5.6.7.8, 9.10.11.12. Check whether its key leaked.” — which is what a stolen machine secret looks like. Disconnect the machine and join it with a new code.

The machine's network is checked every minute while it holds classes: a missing rule is installed again, and after three failures in a row the machine stops its class containers by itself, the classes move, and a banner appears above the list of machines.

A machine with classes: updates, leaving, status

• colloq-host runner status prints the agent's JSON line as before, and beside it, on stderr, the tunnel and the classes: tunnel: up · RTT 48 ms and classes: room 1 · own 1 · council 0, how many class containers the machine holds. When the agent does not run on the host's network, the command warns that it takes no classes.
• colloq-host runner update does not interrupt classes: the agent is drained, goes away marked as a restart, and the new agent takes up the class containers. An update that changes the tunnel's protocol while classes run on the machine is refused with nothing touched, unless the new image is the release the server already runs. So the order stays the same: the server first, then the machines. make runners-update updates the same way.
• A drained machine — Drain in the menu, or the start of runner update or runner leave — finishes its submissions and takes neither new submissions nor new classes, while the classes already on it go on. The agent's own drain, which runner update starts (docker exec colloq-runner colloq-vast runner drain), and its leave wait for submissions only: they do not count classes.
• colloq-host runner leave on a machine with classes refuses first and changes nothing (exit code 3): leaving moves the classes, and their variables are lost. With --force the machine leaves as usual: it finishes its submissions, sends the server what its classes' kernels wrote (up to 10 seconds), and when it revokes itself its classes move, and the agent removes their containers and folder copies. make runners-down answers --force for you: it has already asked whether to destroy the machines.

Moving from 0.24. A machine joined before 0.25 runs its agent on Docker's network, not the host's. Until its first update it is outdated to a 0.25 server, as after any release: it takes no new submissions and no classes. That update is run by the old colloq-host, which installs the new one only at its end, so after it the agent is still off the host's network: it takes submissions and no classes, the machine's line says needs --network host — restart the machine with the command from the panel, and runner status warns about it. A second colloq-host runner update, by the new colloq-host, moves it. make runners-update makes the second update by itself, and a Vast VM is also fixed by a reboot: its on-start installs a fresh colloq-host and says runner up again. On a machine joined with the docker run command, install colloq-host from the new version's image (see updating): its first runner update moves the agent onto the host's network at once.

Machines on Vast

From a checkout of the repository, with VAST_TOKEN and an SSH key in .env, as for make vast-vm:

make vast-runner COUNT=3 CODE=<code> URL=https://colloq.example.edu
make vast-runner COUNT=2 GPU="RTX 4090" GPUS=2 DISK=150 CODE=<code> URL=https://colloq.example.edu

The command rents COUNT VMs, 1 to 16, the way make vast-vm does: with the same offer filters (GPU=, GPUS=, GPU_RAM=, CPU=, RAM=, DISK=, MAX_PRICE=) and the same Max CUDA 13.0 floor. It first says what it will rent and the most it pays per hour for each machine (MAX_PRICE), and asks to confirm (FORCE=1 does not ask); each machine names the offer it found, and its price, only after that. The machines are rented one after another — each takes the cheapest offer that fits — and boot together; each waits until its agent has joined. The code must allow at least COUNT machines; URL= can be written once into .env as VAST_RUNNER_URL.

The disk is 120 GB by default, the floor of a machine with cards: Vast has no VMs without cards, and a machine with cards is asked for the GPU kernels. A machine for CPU submissions is a cheaper VM with the cores you need (CPU=), and its card takes GPU submissions as well.

A rented VM's on-start holds only the server's address and the code: no relay token and no other key. The host's operator sees the on-start and Vast keeps it in its console, while the code is worth nothing after a day at most, or once every machine it was minted for has joined (the command in the connect window sets COUNT to that number). The machine learns the image version from the server, so a server update needs no new on-start — it needs make runners-update.

The machines are labelled colloq-runner on Vast (colloq-runner-<NAME> with NAME=), and make runners-update and make runners-down find them by that label. They first ask each machine over SSH which server it serves; with URL= only that server's machines are taken, without it every machine of the account with the label.

You pay. The machines bill to your Vast account until they are destroyed. That includes those rented but never joined (the server did not answer, the code ran out of joins): they are in no list of machines. At the end the command names each such machine and prints the command that destroys it, and with DESTROY_UNJOINED=1 it destroys them itself. A machine disconnected in the panel bills too until it is destroyed — with make runners-down or in the Vast console.

Environments from the registry, and GPU kernels

A machine pulls the environment image not from the server but from the registry — by the digest the server recorded when it got that image itself — and before running checks its layers against the server's image. So machines run only the environments whose image the server pulled from a registry. base and kaggle-base are published with every release; the GPU kernels base-gpu and kaggle-gpu by a separate command, make kernel-publish on a machine with a card, at release time (RELEASING.md). The server pulls them from the registry when they are listed in KERNEL_PRELOAD, for example KERNEL_PRELOAD=base,kaggle-base,base-gpu,kaggle-gpu.

An environment built or edited on this server runs on this server only in this version: its image is not in the registry. The list of machines marks such a competition only this server runs it, the editor this server only. With KERNEL_IMAGE_REPO=none the server builds every environment itself, and machines have nothing to run. A GPU kernel the server built itself before its version's kernel was published stays on the server the same way; how to move to the published one is in the same section of RELEASING.md.

Checked on Vast

On 14 October 2026, in two rounds on two builds of this version before its release. The server was a Vast VM (61 vCPU, 86 GB, an RTX A2000 of 6 GB) behind the relay, with Notebooks on this server set to metrics only: every call of the machines went through the relay. In the first round there was one machine, on the server's own VM (for the test only: in use a machine must be a separate one); in the second a separate VM joined it, with 1 × RTX 4090 of 24 GB, 35 vCPU and driver 580.95.05.

The bottleneck of a burst is the server's metrics. In the first round, with one machine and Scores at once at 2, the tail of a burst of 30 submissions waited for metrics (4.6 s an answer), and the last verdict came after 96 s. The burst of 52 in the second round was over within a minute with 8, but it also had two machines and a newer build, so the two bursts do not show what the 8 alone gave. When the server has cores and memory to spare, raise Scores at once before a competition with machines. And mind CUDA: that reference notebook moves nothing to the card, so on a machine with a card it runs on the CPU — 80 s a submission in the second round.

What the rehearsal found is fixed before the release. One stop of an agent in the middle of 10 runs quarantined the machine at once; the second round's build counts it as one loss, as the agent killed in that round shows above. Three fixes came after both builds and were not checked on Vast: the panel gave places by the machine's cores, not by the competition's submissions; a GPU competition needed its 13 GB image on the server even when only machines ran its notebooks; a server updated by the previous release's colloq-host named the published image in the commands even when it ran its own. The rehearsal also showed that docker kill is a manual stop, not a crash (see failures).

What machines do not do yet

• They do not prepare participants' own packages: the server prepares the sets (on the Docker path it needs Docker Engine 28 or newer), and machines only download the ready ones.
• They give the council pool no GPU, and the server itself has no pool: without machines, attempts run one by one in the notebook's kernel.
• They do not move variables: a kernel that moved off a machine starts afresh.
• They do not spread a class's personal notebooks over machines: all of them are in one container on one machine, while the class notebooks may run on another or here.
• They do not update themselves: after a server update, run make runners-update or colloq-host runner update.
• They do not sign their requests. A machine presents its secret in a header, and TLS ends at the reverse proxy or relay in front of the server: whoever controls those can read the machines' secrets (see isolation).
• They do not pick a run up again after an agent restart: such a run runs again.
• One machine serves one server. Machines are not rented from the panel — only with make vast-runner or by hand.
• A server in Kubernetes (the Helm chart) connects no machines in this version: its panel has no Machines block, and every room there has a Pod of its own.