Competitions

What this is

A competition is a task with a closed evaluation. You hand the class the data and the brief, an entrant submits an .ipynb notebook as a file, the server executes it from scratch and scores the result with your metric. The answers the score is computed against are never shown to an entrant; their place on the leaderboard is.

A competition does not live in a room: it has its own address, /k/<name>, its own queue for the whole instance and its own entrants. No class is needed — a link is enough. You create and run competitions from the admin panel, under Competitions.

Set a competition up

1. Open Competitions in the panel and press NEW COMPETITION. Give it a title and an address — the address becomes the /k/<name> link the class opens.
2. Write the one-line blurb (it shows in the list) and the description in Markdown: the task, what is in the data, what to submit.
3. Add the open data files. Entrants download them from the task page and see them inside the container as a read-only data/ directory. The ceiling is 200 MB per competition.
4. Add the hidden answers through their own button. On disk the file is always called solution.csv and it lives outside the class workspaces, so no room can read it. The row key must be named id: it is what the rows are split by and what a submitted file is checked against. No other column name can be set yet, and a competition using one will not open — the baseline check catches it.
5. Paste the metric code, or take a preset (MAPE, RMSE, MAE, ROC AUC, F1, accuracy, QWK) — a preset assembles itself from the columns of your answers file. Say which direction is better: lower or higher.
6. Upload the sample notebook — the baseline every entrant starts from — and press Check the whole thing.
7. Set a deadline and press OPEN THE COMPETITION.

Until every step is done, the OPEN button stays disabled and names what is missing. The sample notebook is a gate of its own: until it has gone all the way to a number, the competition cannot be opened at all. A task that does not even solve for its author means a hundred people hunting for a mistake of their own that isn't there.

The metric code

A metric is one function. The server calls it with two pandas frames — your answers file and the submitted submission.csv, already joined on the id column. It must return a number.

import numpy as np
import pandas as pd


class ParticipantVisibleError(Exception):
    pass


def score(solution: pd.DataFrame, submission: pd.DataFrame) -> float:
    if 'orders' not in submission.columns:
        raise ParticipantVisibleError('The submission has no orders column')
    merged = solution.merge(submission, on='id', how='left', suffixes=('_true', '_pred'))
    if merged['orders_pred'].isna().any():
        raise ParticipantVisibleError('Some rows of test.csv have no prediction')
    true = merged['orders_true'].to_numpy(dtype=float)
    pred = merged['orders_pred'].to_numpy(dtype=float)
    return float(np.mean(np.abs((true - pred) / np.maximum(true, 1e-9))))

The text of a ParticipantVisibleError reaches the entrant verbatim — that is how you tell them what is wrong with their file. Any other error is yours alone: the submission is marked metric failed, the entrant reads "the checking code failed", and you get the traceback. After fixing the code press Fix the metric and rescore everyone: notebooks are not re-executed, only the metric runs again.

Check against the baseline runs the metric over the sample notebook's saved answer without executing it again — that turns a code fix into a few seconds of waiting.

The public and the private share

The metric is computed twice: on the public share of the answer rows and on the private one. The public leaderboard is always visible — it is how entrants tell whether they are getting anywhere. The final one stays closed until the deadline: a hidden share you cannot fit to by refreshing the page is the whole point of having one.

The public share is a number in the editor (30 % by default). Rows are split once and for good: the split seed is recorded when the competition is created, so every entrant and every submission faces the same public share. If your answers file has a Usage column holding Public and Private, the split comes from it and the share is not asked for.

What counts is the submission the entrant picked with "Count this one". If they picked none, their best public one counts. The rule can be changed to "best public" or "latest" in the competition's settings.

How a submission is executed

The entrant presses CHOOSE A FILE and sends a notebook. There is no upload straight from a class notebook: a file is the only way in.

1. The submission joins the instance queue.
2. In a throwaway container with no network the notebook runs in full, cell by cell, from scratch: no saved variables, and data/ mounted read-only. The notebook must write submission.csv.
3. That file is taken out of the container and the container is destroyed.
4. In a second throwaway container — this one with neither the entrant's data nor their code — your metric runs. The hidden answers exist only here.

The competition's limits (notebook time, memory, cores) are set in the editor. Time is enforced by killing the container from outside; memory is enforced by Docker. The entrant watches the stages move live and, if the notebook failed, reads their own traceback in full.

The queue and the console

There is one queue for the whole instance; it lives in the database and survives a server restart. By default one submission runs at a time — a class is usually running on the same machine. The queue is fair per person: someone's second submission queues behind everyone else's first, so no single entrant can hold the runner for the whole session.

The competition console (the Submissions tab) shows what is running now, who is waiting, the five summary numbers and the feed of submissions. A row's menu opens the executed notebook, shows the full output, re-runs it, rescores the metric, or drops it from the standings. The runner strip in the competitions list lets you pause the queue, and a run in progress has a Kill button.

To stop anyone from occupying the queue, set a daily submission quota. A submission that failed before its first cell does not spend one.

Entrants and the entry key

An entrant is not a room member: their identity belongs to the whole instance. On joining, a person gives a name and receives an entry key shaped K7Q-M2X-9FD and a link carrying it. The same key brings them back to their submissions from another device, another browser, or after clearing site data.

The key sits in the "YOUR ENTRY KEY" card on the competitions page — shown every time, not once. The key itself is not in the database: what is stored is its fingerprint and a copy encrypted with the instance secret, so a stolen database without that secret opens no one's account.

If an entrant loses their key, issue a new one from the Entrants tab. The old one stops working that same second, together with every tab still open on it.

The deadline and the debrief

When the deadline passes, submissions close by themselves and the final leaderboard opens — if the settings say "open at the deadline". The other option is I will open it by hand, at the debrief: then a button reveals the results and you choose the minute of the class when they appear.

Finish now closes submissions early. For the debrief there is Leaderboard on the projector — a separate window with just the table, without file names or internal marks.