Troubleshooting¶

Most commonly this happens due to issues and outages of computing infrastructure - please alert facility admins.

Another possible cause is an overfilled home directory: sessions cannot start while you are over the home directory quota.

1. Start a session with the Minimal JupyterLab interface option — it should work even when a normal session does not.
2. Check your home directory usage.
3. Move large files (data, Pixi/Conda environments) to /work/ or /depot/ storage — see Storage volumes.

If you selected a full 40 GB A100 GPU, the session may also fail to start simply because all full GPU instances are taken. Try a 5 GB slice instead, or start a CPU-only session.

• You might be trying to use custom Pixi or Conda environments on a slow filesystem. Try moving them to /work/ storage.
• Check whether you are running out of RAM: only the memory you selected at session creation is guaranteed — see Session resources. Restart the session with more RAM if needed.
• Reading many small files from /depot/ or /eos/ can be slow — see Data access for faster access patterns (XCache).

Idle sessions are shut down automatically — see Idle sessions. Sessions may also occasionally get shut down due to unplanned outages, so save your work regularly and keep important code in sync with a Git repository.

Shut down your session (File → Hub Control Panel → Stop My Server), then start a new one. The session image is recreated from scratch every time; only the contents of your storage volumes persist.

Depot is writable only with a Purdue account — see Storage volumes for where to write instead.

Purdue users can write to their own private directories, as well as into group directories to which they have access. If you don't have access to your group's directory, please contact facility admins.

Purdue EOS is mounted read-only — see Writing to EOS.

See Saving outputs of CRAB jobs.

Restart the session (File → Hub Control Panel → Stop My Server), then run the eos-connect command again — see CERNBox access.

This is intentional — see Pixi storage locations.

The kernel found no Pixi project around the notebook, or cannot use the one it found; the error names the cause. See what the kernel needs in Pixi kernels. For a notebook outside any Pixi project, choose Python (pixi global).

See Conda kernels.

Contact support — we regularly update the global Pixi environment. Alternatively, create your own Pixi environment with the packages you need.

Your VOMS proxy is probably missing or expired. Check with voms-proxy-info, and create a fresh proxy if needed.

If no CMS site has the files on disk, a tape recall is necessary: create a Rucio replication rule to subscribe the dataset to Purdue — see Rucio tutorial.

new_cluster() has no time limit. If it has waited for more than a few minutes, interrupt it, stop the pending cluster, and contact us if it happens again.

See the one-cluster limit: shut down your existing cluster (see Shutting down clusters), or wait for it to finish stopping, then try again.

Add prometheus_client to the environment the cluster runs — see Pixi or Conda environments.

• Check that the Pixi/Conda environment passed to new_cluster() is visible to the workers.
• CERN/FNAL users: see the NB_UID / NB_GID note in Environment variables.

Idle clusters are shut down automatically — see Cluster lifetime and timeouts.

See Reading data via XRootD.

Fewer GPUs are free than you ask for: try again later, or ask for up to as many with af-max-workers — see Settings.

A job of yours is running with other settings: wait for it to end, or stop it — see Settings.

A job's status and log are removed some minutes after your last job ends — see Good to know.

Set storage_path to a directory on /work — see Data-parallel training.

Submit from a session started without a GPU — see Before you start.

Your session has no RAY_AUTH_TOKEN: restart it.

See the troubleshooting section of the IDE connection guide. The usual suspects: home directory permissions (chmod 755 ~/), a missing or not-on-PATH websocat binary, or an expired JupyterHub token.