Skip to content

Storage volumes

Overview

The following table summarizes size, access modes, and accessibility of each storage volume (scroll sideways for details).

Storage volume Path Size per user Access mode Mounted in Slurm jobs and Dask/Slurm workers Mounted in Dask/k8s workers Writable by users w/o Purdue account
AF home storage /home/<username>/ 25 GB Read/write ❌ ❌ ✅
Purdue Depot storage /depot/cms/ up to 1 TB Read/write for Purdue users, read-only for others ✅ ✅ ❌
AF work storage /work/users/<username>/ 100 GB Read/write ❌ ✅ ✅
AF shared project storage /work/projects/ up to 1 TB Read/write ❌ ✅ ✅
Purdue EOS /eos/purdue/ up to 100 TB Read-only (writable via gfal/xrdcp) ❌ ✅ ❌
CVMFS /cvmfs/ N/A Read-only ✅ ✅ ❌
CERNBox (CERN EOS) /eos/cern/ N/A Read/write ❌ ❌ ✅

In the JupyterLab file browser, your home directory contains shortcuts to the other volumes: work (/work/), depot (group directories under /depot/cms/), and eos-purdue (/eos/purdue/).

Quotas

  • Home directory — your /home/<username>/ directory (the root directory of the JupyterLab file browser) has a strict quota of 25 GB. If you go over this limit, you will not be able to start a session on Purdue AF. Rather than storing your data, Pixi or Conda environments, etc. in your home directory, use the other storage volumes. You can check your current home directory usage with:

    du -sh $HOME

  • Work directory — the 100 GB quota of /work/users/<username>/ is not enforced by the filesystem, but please stay within it: the facility admins contact users whose usage exceeds it.

Which storage volume should I use?

Below are common storage use cases with recommendations on which volume to use.

Storing analysis code and environments

  • Keep your code in a Git repository (GitHub or GitLab), cloned into one of the writable volumes.

  • Purdue users: use either Depot or /work/ storage:

    • private directories: /depot/cms/users/<username>/ or /work/users/<username>/
    • group directories for shared work: /depot/cms/<group-name>/ or /work/projects/<project-name>/.

  • CERN / FNAL users: use private /work/users/<username>/ directory or a shared project directory /work/projects/<project-name>/.

Code and environments used by Slurm jobs or Dask Gateway workers must be on a volume those workers mount (see the table above) — for the Slurm backend, that means Depot.

Storing custom Pixi or Conda environments

In order for Pixi or Conda environments to appear as JupyterLab kernels, they must be stored in publicly readable directories, so /depot/cms/private/ directories will NOT work. Possible locations:

  • group directories on Depot (for example, /depot/cms/top/);
  • personal directories on work storage: /work/users/<username>/;
  • shared project directories on work storage: /work/projects/<project-name>/.

Transferring official CMS datasets to Purdue

  1. Locate the dataset using DAS (CMS Data Aggregation System).
  2. Use Rucio to "subscribe" the dataset to Purdue.

See Data access for the full picture.

Saving outputs of CRAB jobs

  • The outputs of CRAB jobs are written to your Grid directory: /eos/purdue/store/user/<cern-username>. Note: your CERN username is different from your Purdue username!
  • The Grid directory at Purdue EOS is created by request and only for Purdue-affiliated users. This must be indicated when creating a Purdue Tier-2 account.
  • If you can't see your Grid directory under /eos/purdue/store/user/, please contact support.

Processing ("skimming") CMS datasets

The best storage volume depends on the size of the output:

  • Large outputs (over 100 GB): save to Purdue EOS. Since Purdue EOS is not directly writable, save outputs into /tmp/<username>/ first and then copy them to EOS — see Writing to EOS.

  • Small outputs (under 100 GB):

    • Purdue users should use work storage or Depot.
    • Non-Purdue users should use work storage.

Don't stress Depot from many jobs at once

Avoid writing many files to Depot at the same time, as it may slow Depot down for everyone. If your jobs produce large outputs, first save them into /tmp/<username>/ on the individual Slurm jobs / Dask workers, and then copy them over to EOS — see Writing to EOS.

Shared project directories

Directories under /work/projects/ are intended for groups of users collaborating on the same analysis, regardless of which institution their accounts belong to. If you would like a project directory to be created, contact support.

Other options

  • Git: users can use GitHub or GitLab to store and share their work. The Git extensions in JupyterLab and VSCode interfaces allow you to work with repositories interactively (commit, push, pull, etc.).
  • CERNBox: anyone with a CERN account can mount their CERNBox directory — see CERNBox access.
  • XRootD client is installed and can be used to access data stored at other CMS sites — see Data access.