Environments in production

An environment is a catalog entry with a name and an immutable digest of the kernel image. A class keeps the revision chosen for it. Changing the default environment or releasing a new image does not change the Python of a class that already exists.

The Environments panel shows the available revisions and lets you choose the default environment for new classes. If a pinned revision is unavailable, the kernel fails to start; the server does not quietly substitute whatever image is current.

Add libraries

  1. Edit the environment's requirements in kernel/environments/. The # colloq: from NAME directive sets inheritance, and # colloq: python 3.12 sets the Python version, from 3.10 to 3.13; by default the version comes from ARG PARENT in kernel/Dockerfile, currently 3.11. The version is set by the root of the inheritance chain: a layer on top of a ready image installs wheels for the interpreter it inherits from the base and cannot replace it, so an environment with from that asks for a different version is rejected at build time. The # colloq: gpu directive makes an environment a GPU environment, and environments that inherit from it through from get this flag automatically. If a release contains such an environment, the --gpu-toolkit-version and --gpu-device-plugin-image build options are required. To pip, all these lines are comments.
  2. Pin the source commit and build a new release on a trusted build machine or in release CI.
  3. Publish the images and a catalog with the real digests.
  4. Install the release and check an import and a typical task from the class.

Production never builds an image from a web request and never gets the Docker socket. The package list records the requested dependencies, not a full pip lock: rebuilding with version ranges can produce a different image. The installed digest stays the same either way.

Building a release (maintainers)

The Publish immutable release workflow takes a version tag and the exact tested k3s patch release, builds the chosen environments and attaches the manifest, the tools archive and checksums. The alternative is running scripts/release-build.py locally. An example with values you need to replace:

python3 scripts/release-build.py --version YOUR_VERSION_TAG \
  --registry YOUR_REGISTRY_PREFIX --source-commit FULL_COMMIT_SHA \
  --k3s-version EXACT_TESTED_K3S_VERSION --environments base,cv

For GPU, also pass --gpu-toolkit-version EXACT_NVIDIA_APT_VERSION and --gpu-device-plugin-image NVIDIA_PLUGIN_IMAGE_AT_SHA256. Log in to the registry on the build machine. The k3s version must be v1.33 or newer: a room's memory is changed during class through the pods/resize subresource, and manifest validation rejects an older one. Publishing the artifacts is not, by itself, a test on the target VM.

Memory and CPU for a class

The Resources section is in the new class form and in the settings of an existing class (class menu → Settings). The Memory for the room's kernel, GB field moves in 0.5 GB steps, from 0.5 GB up to the machine's memory minus 1 GB. The Processor, cores field takes a whole number from 1 up to the machine's core count. To go back to the shared value, clear the field or click back to the environment default or back to the instance default.

The hint under the field shows the total memory, the free memory and the selected environment's default. If Docker runs in its own virtual machine, as Docker Desktop and Colima do on macOS, the hint starts with Rooms live in Docker. In that case the memory size, the core count and the fields' upper limits come from Docker, not from the computer. Free memory is Docker's memory minus 1 GB and minus the limits of rooms that are already running. If free memory cannot be determined, the hint leaves it out. The form does not stop you from setting a limit above free memory, but it warns that the kernel may not start. For a running room, only the increase over what it already has is compared with free memory. If the machine has an NVIDIA graphics card, the section shows it and whether the selected environment uses the card. GPU memory is not limited: all rooms on the card share it.

In Docker, a new memory limit applies to the running kernel immediately, without a restart; going back to the environment default takes effect the next time the room's container starts. A new core count also applies to the running kernel immediately, without a restart, and so does going back to the instance default. The numpy and torch thread count (OMP_NUM_THREADS and related variables) is set when the room's container is created and does not change for a running container, not even with a kernel restart: the restarted kernel inherits the same container's environment. The next container of the room gets the new thread count, for example when the room has stood empty for two hours and is opened again. Until then you can raise the thread count right in the notebook: threadpoolctl.threadpool_limits(n) for numpy and scikit-learn, torch.set_num_threads(n) for torch. Without KERNEL_MEM, the default is 4 GB, or 16 GB for a GPU environment. KERNEL_MEM sets the default for all environments, KERNEL_MEM_<ENVIRONMENT> sets it for one environment (for example, KERNEL_MEM_BASE_GPU=16g), and KERNEL_CPUS sets the number of cores (2 by default). The .env file that colloq start creates already contains KERNEL_MEM=4g. How this works in a k3s installation is described below.

Room resources

In a k3s installation, a room Pod gets 2Gi of memory, 2 CPUs and 2Gi of ephemeral storage by default; requests equal limits. The memory and the CPU count from the class's “Resources” section reach the Pod, and you can raise or lower them during class: the broker changes them on the running Pod in place, so the kernel does not restart and variables stay. The back to the environment default and back to the instance default buttons also apply at once. The Pod takes the numpy and torch thread count from its CPU limit when it starts and, as in Docker, does not change it for the running kernel, neither after an in-place change nor after a kernel restart; the room's next Pod gets the new count. The broker has one default for all environments, GPU ones included, and that is the default the form's hint shows. The field's upper limit is the node's memory minus 1 GB. The broker variables RUNTIME_KERNEL_MEMORY and RUNTIME_KERNEL_MEMORY_MAX set the default and the ceiling: put them in the same instance.env as the app settings (see installation configuration), for example RUNTIME_KERNEL_MEMORY=4Gi. Values are whole Mi or Gi from 64Mi to 256Gi, and the ceiling can't be below the default; the installer checks them before stopping rooms and keeps them across updates. Changing memory in place needs Kubernetes 1.33 or newer, and a release with an older k3s fails validation. These are resource limits, not a promise of a certain number of students or a certain model training speed.

A room's memory and cores are reserved in full, even if Python never touches them. That is why the hint counts as free the node's memory minus 1 GB and minus the memory already given to running rooms. If the node can't cover an increase in memory or cores right now, the new value is saved and waits: Kubernetes applies it on its own once the resource frees up, for example when a neighbouring room stops. The broker does not set more than the node can ever provide: the running Pod keeps its previous value. A new room has nowhere to wait: if the node can't cover its memory, CPU or GPU, startup stops after 20 seconds. The teacher sees in the room how much it asked for and what to lower or close; students see a short note that Python can't start right now and that the teacher can see why. If code runs out of memory, k3s ends the room's whole Pod, not just the kernel as in Docker; the next run starts a new Pod, already with the new memory value.

Memory in /dev/shm counts toward the Pod limit: 64 MiB by default for a CPU environment and 1 GiB for a GPU one. The installer sets a limit of 256 PIDs per Pod. A namespace ResourceQuota and any extra admission rules are up to the operator; no ResourceQuota ships with the installation. There is no enforced per-room file quota for the workspace; keep an eye on the separate file partition and on free space.

GPU environments

Each GPU room requests one nvidia.com/gpu and RuntimeClass nvidia. Device sharing is not turned on automatically. Several GPU rooms running at once need a matching number of available devices, or a sharing policy that has been separately designed and tested.

The installer keeps the host driver, installs pinned versions of the NVIDIA Container Toolkit and device plugin, and runs a real PyTorch CUDA operation as a check. The check needs a free GPU; a card that is in use is not taken away from a running class.

sudo scripts/cluster.sh gpu-preflight
sudo k3s kubectl get nodes \
  -o "custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

An application that responds normally does not prove the GPU is ready. If the card is not allocated or CUDA does not work, find the cause before the class starts.

Keep old images

Historical catalog revisions are kept as long as rooms refer to them. Do not delete the corresponding images from the registry. An application backup contains the catalog and the digests, but not the image layers.