Nodes

Tenant admin

A node is a machine that runs containers. The Nodes page lists them with their identity, capacity and health, alongside the node invites that let new machines connect themselves.

Tenant and global nodes

Tenant nodes belong to your tenant. Global nodes are shared across tenants by a superadmin; they appear in your list, and you can enable them for your tenant, but the object itself is not yours to edit.

Adding a node

The normal route is an invitation, not direct registration: the machine connects outward and enrols itself, so it does not need to be reachable from the internet.

1. Connect new node creates an invite with a label, a validity period and a maximum number of uses.
2. Run the command it gives you on the target machine.
3. The node appears in the list and reports its GPUs.
4. Link it to the queues that should schedule onto it.

A max uses above one lets the same invite enrol several machines - useful for a batch of identical workstations, and worth avoiding otherwise.

Invites expire and are not revivable

An invite past its validity date cannot be extended. Issue a new one. Delete invites you are finished with; an unused invite with uses remaining is a way into your tenant.

Mounts

A Docker node shares host directories with jobs through mounts. The first ones come from the invite; more can be added to a node that is already connected, which is the normal case when each project or student has its own share.

1. Make the directory available on the node, for an NFS share through /etc/fstab, and check that it is mounted.
2. Open the node, enter the directory under Mount Paths and choose Add mount. AiQu creates a storage for it, linked to the node’s queues, with a Storage Bucket Collection and a bucket named after the directory.
3. Under Storage, grant the new bucket to a group or project.

Nothing has to be done on the node itself. A job mounts the bucket from the host directory directly, and the node’s storage service is restarted for you so the file browser sees it.

The dialog also shows a volume argument for the directory, such as -v /mnt/project_share:/mnt/11. It is not needed to use the mount. It keeps the aiquagent container’s own docker run command in step with the /mnt/<n> numbering AiQu has registered, for the next time that container is recreated for some other reason.

The node's storage service restarts

Adding or removing a mount restarts the node’s storage service on the latest version, because it only sees the mounts it was started with. Browsing the node’s buckets is unavailable for up to a minute while it comes back; running jobs are not affected.

Mounts can only be edited by the tenant that enrolled the node, and only on Docker nodes: K8s nodes use PVC volumes instead. System directories such as /etc, /home or /var/lib/docker, and their parents, are refused.

Removing a mount removes its bucket and Storage Bucket Collection too. It is refused while the bucket is in use: granted to a user, group or project, public to the tenant, or mounted in a job that has not ended. Take that away first. Only local mounts can be removed here: a distributed storage is shared with other nodes.

Node health

Online means the node currently holds its tunnel to the central server. Offline nodes take no jobs, and their queues have nothing to schedule onto.

A node going offline is not necessarily a fault - many nodes are workstations that are switched off overnight. It does mean users will see “This queue has no node that can run a job right now” until it returns.

Enable and disable

The enable toggle takes a node out of scheduling without removing it. Prefer it to deletion for maintenance: deletion means the machine has to re-enrol with a new invite.

Deleting

Deleting a node removes it from the tenant. Jobs’ history survives; the machine must enrol again from a fresh invite to come back.