Training on GPUs with Ray¶
Ray is a Python library that runs your PyTorch script on GPUs outside your session, on one of them or on several at once. You submit the script from a notebook and read its output there.
| To | Each GPU runs | See |
|---|---|---|
| Train a model | The training | One GPU |
| Train a model faster | A copy of the model, on a share of the data | Data-parallel training |
| Try many hyperparameters | A training with one set of them | Hyperparameter search |
| Train a model too large for one GPU | A part of the model | A model larger than a GPU |
| Apply a trained model to a dataset | A copy of the model, on a share of the rows | Batch inference |
When Purdue's GPUs are busy, or you need more of them, a job can run on NRP instead.
Before you start¶
Available to all users.
- Start your session without a GPU.
- Use the Python (pixi global) kernel, which has every package this guide uses, or an environment of your own.
- Keep your data and results on
/workor/depot: a job does not see your home directory.
One GPU¶
A PyTorch script that trains on cuda runs as it is:
# train.py
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
X = torch.randn(100_000, 20)
y = (X.sum(dim=1, keepdim=True) > 0).float()
loader = DataLoader(TensorDataset(X, y), batch_size=512, shuffle=True)
model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 1)).to("cuda")
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.BCEWithLogitsLoss()
for epoch in range(5):
for X_batch, y_batch in loader:
loss = loss_fn(model(X_batch.to("cuda")), y_batch.to("cuda"))
optimizer.zero_grad()
loss.backward()
optimizer.step()
print(f"epoch {epoch}: loss {loss.item():.3f}")
torch.save(model.state_dict(), "/work/users/<username>/model.pt")
Submit it from a notebook in the same directory:
from ray.job_submission import JobSubmissionClient
client = JobSubmissionClient("http://ray-train-gateway:8265")
job = client.submit_job(
entrypoint="python train.py",
# Sends the notebook's directory, the script with it.
runtime_env={"working_dir": "."},
entrypoint_num_gpus=1,
)
The job starts within a minute or two on GPUs of its own, and goes on if you close the notebook. What the script prints is the job's log:
Data-parallel training¶
Every GPU holds a copy of the model and trains it on its own share of each
epoch, and the copies stay in step, as with PyTorch's
DistributedDataParallel.
Ray Train
sets this up. The numbered comments mark what changes in the script of
One GPU:
# train.py
import os
import tempfile
import ray.train
import ray.train.torch
import torch
from ray.train import Checkpoint, FailureConfig, RunConfig, ScalingConfig
from ray.train.torch import TorchTrainer
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
# 1. The training goes into a function.
def train_func():
X = torch.randn(100_000, 20)
y = (X.sum(dim=1, keepdim=True) > 0).float()
loader = DataLoader(TensorDataset(X, y), batch_size=512, shuffle=True)
net = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 1))
# 2. Ray Train puts the model and the batches on the GPU, in place of
# .to("cuda"), and gives each GPU its share of the data.
model = ray.train.torch.prepare_model(net)
loader = ray.train.torch.prepare_data_loader(loader)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.BCEWithLogitsLoss()
# 3. The training starts from its last checkpoint, if it has one.
first_epoch = 0
checkpoint = ray.train.get_checkpoint()
if checkpoint:
with checkpoint.as_directory() as path:
state = torch.load(f"{path}/state.pt")
net.load_state_dict(state["model"])
optimizer.load_state_dict(state["optimizer"])
first_epoch = state["epoch"] + 1
for epoch in range(first_epoch, 5):
for X_batch, y_batch in loader:
loss = loss_fn(model(X_batch), y_batch)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# 4. Every GPU reports the epoch, in place of print, and the first
# one saves a checkpoint with it.
with tempfile.TemporaryDirectory() as tmp:
checkpoint = None
if ray.train.get_context().get_world_rank() == 0:
state = {
"model": net.state_dict(),
"optimizer": optimizer.state_dict(),
"epoch": epoch,
}
torch.save(state, f"{tmp}/state.pt")
checkpoint = Checkpoint.from_directory(tmp)
ray.train.report({"loss": loss.item()}, checkpoint=checkpoint)
# 5. Ray Train runs the function on the GPUs.
trainer = TorchTrainer(
train_func,
scaling_config=ScalingConfig(
# From one GPU to as many as you ask for when you submit.
num_workers=(1, int(os.environ["AF_MAX_WORKERS"])),
use_gpu=True,
),
run_config=RunConfig(
# Where the checkpoints go: a directory of yours on /work.
storage_path="/work/users/<username>/ray_results",
# Lets the training go on when it loses a GPU.
failure_config=FailureConfig(max_failures=10),
),
)
result = trainer.fit()
print(result.checkpoint.path)
Submit it with the number of GPUs to train on, and without
entrypoint_num_gpus:
client = JobSubmissionClient(
"http://ray-train-gateway:8265", headers={"af-max-workers": "4"}
)
job = client.submit_job(entrypoint="python train.py", runtime_env={"working_dir": "."})
The training takes as many of the four GPUs as are free. Its last checkpoint
is the directory the script prints, with the model in state.pt.
For data in files that do not fit in memory,
Ray Data
takes the place of the DataLoader and of prepare_data_loader:
def train_func():
...
data = ray.train.get_dataset_shard("train")
for epoch in range(first_epoch, 5):
# A batch is a dict of tensors by column, on the GPU.
for batch in data.iter_torch_batches(batch_size=512):
...
trainer = TorchTrainer(
train_func,
datasets={"train": ray.data.read_parquet("/work/users/<username>/events")},
...
)
Ray Train has guides of its own for Lightning, Transformers and XGBoost.
Hyperparameter search¶
Every GPU trains the model with a different set of hyperparameters. Ray Tune chooses the sets, stops the trainings that fall behind and returns the best one. The numbered comments mark what changes in the script of One GPU:
# search.py
import torch
from ray import tune
from ray.tune.schedulers import ASHAScheduler
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
# 1. The training goes into a function of its hyperparameters.
def train(config):
X = torch.randn(100_000, 20)
y = (X.sum(dim=1, keepdim=True) > 0).float()
loader = DataLoader(TensorDataset(X, y), batch_size=512, shuffle=True)
model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 1)).to("cuda")
optimizer = torch.optim.Adam(model.parameters(), lr=config["lr"])
loss_fn = nn.BCEWithLogitsLoss()
for epoch in range(5):
for X_batch, y_batch in loader:
loss = loss_fn(model(X_batch.to("cuda")), y_batch.to("cuda"))
optimizer.zero_grad()
loss.backward()
optimizer.step()
# 2. It reports each epoch, in place of print.
tune.report({"loss": loss.item()})
# 3. Ray Tune runs the function once per set of hyperparameters, each on a GPU.
tuner = tune.Tuner(
tune.with_resources(train, {"gpu": 1}),
param_space={"lr": tune.loguniform(1e-4, 1e-1)},
tune_config=tune.TuneConfig(
metric="loss",
mode="min",
num_samples=20,
# Stops the trainings that fall behind.
scheduler=ASHAScheduler(),
),
run_config=tune.RunConfig(
storage_path="/work/users/<username>/ray_results",
# Lets a training start again when it loses its GPU.
failure_config=tune.FailureConfig(max_failures=10),
),
)
best = tuner.fit().get_best_result()
print(best.config, best.metrics["loss"])
Submit it as for data-parallel training:
af-max-workers is the number of trainings to run at once.
Ray Tune can also run a list of values, choose them with Optuna and search over data-parallel trainings.
A model larger than a GPU¶
On two GPUs¶
The script puts each part of the model on a GPU. The numbered comments mark what changes in the script of One GPU:
# train.py
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
X = torch.randn(100_000, 20)
y = (X.sum(dim=1, keepdim=True) > 0).float()
loader = DataLoader(TensorDataset(X, y), batch_size=512, shuffle=True)
# 1. Each part of the model goes on a GPU of its own.
first = nn.Sequential(nn.Linear(20, 4096), nn.ReLU()).to("cuda:0")
second = nn.Linear(4096, 1).to("cuda:1")
optimizer = torch.optim.Adam([*first.parameters(), *second.parameters()], lr=1e-3)
loss_fn = nn.BCEWithLogitsLoss()
for epoch in range(5):
for X_batch, y_batch in loader:
# 2. The data follows the model from one GPU to the other.
hidden = first(X_batch.to("cuda:0"))
loss = loss_fn(second(hidden.to("cuda:1")), y_batch.to("cuda:1"))
optimizer.zero_grad()
loss.backward()
optimizer.step()
print(f"epoch {epoch}: loss {loss.item():.3f}")
Submit it with two GPUs:
client = JobSubmissionClient(
"http://ray-train-gateway:8265", headers={"af-gpus-per-worker": "2"}
)
job = client.submit_job(
entrypoint="python train.py",
runtime_env={"working_dir": "."},
entrypoint_num_gpus=2,
)
The two GPUs are T4s, with 16 GB each.
On more GPUs¶
FSDP is data-parallel training in which every GPU keeps only its share of the model's parameters, and gets the rest of a layer from the others when it computes that layer. It is slower than training a model that fits. The numbered comments mark what changes in the data-parallel script:
train.py with FSDP
# train.py
import os
import tempfile
import ray.train
import ray.train.torch
import torch
from ray.train import Checkpoint, FailureConfig, RunConfig, ScalingConfig
from ray.train.torch import TorchTrainer
from torch import nn
from torch.distributed.fsdp import FullStateDictConfig, StateDictType
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.wrap import ModuleWrapPolicy
from torch.utils.data import DataLoader, TensorDataset
def train_func():
X = torch.randn(100_000, 20)
y = (X.sum(dim=1, keepdim=True) > 0).float()
loader = DataLoader(TensorDataset(X, y), batch_size=512, shuffle=True)
net = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 1))
# 1. Each GPU keeps a share of the parameters, and holds one layer whole
# at a time.
model = ray.train.torch.prepare_model(
net,
parallel_strategy="fsdp",
parallel_strategy_kwargs={"auto_wrap_policy": ModuleWrapPolicy({nn.Linear})},
)
loader = ray.train.torch.prepare_data_loader(loader)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.BCEWithLogitsLoss()
# 2. A checkpoint is loaded through FSDP.
first_epoch = 0
checkpoint = ray.train.get_checkpoint()
if checkpoint:
with checkpoint.as_directory() as path:
state = torch.load(f"{path}/state.pt")
model.load_state_dict(state["model"])
optimizer.load_state_dict(
FSDP.optim_state_dict_to_load(model, optimizer, state["optimizer"])
)
first_epoch = state["epoch"] + 1
for epoch in range(first_epoch, 5):
for X_batch, y_batch in loader:
loss = loss_fn(model(X_batch), y_batch)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# 3. It is saved through FSDP too: the GPUs put the model together,
# and hand it to the first one.
whole = FullStateDictConfig(offload_to_cpu=True, rank0_only=True)
with FSDP.state_dict_type(model, StateDictType.FULL_STATE_DICT, whole):
state = {
"model": model.state_dict(),
"optimizer": FSDP.optim_state_dict(model, optimizer),
"epoch": epoch,
}
with tempfile.TemporaryDirectory() as tmp:
checkpoint = None
if ray.train.get_context().get_world_rank() == 0:
torch.save(state, f"{tmp}/state.pt")
checkpoint = Checkpoint.from_directory(tmp)
ray.train.report({"loss": loss.item()}, checkpoint=checkpoint)
trainer = TorchTrainer(
train_func,
scaling_config=ScalingConfig(
# 4. The number of GPUs is fixed: each holds its share of the model.
num_workers=int(os.environ["AF_MAX_WORKERS"]),
use_gpu=True,
),
run_config=RunConfig(
# Where the checkpoints go: a directory of yours on /work.
storage_path="/work/users/<username>/ray_results",
# Lets the training go on when it loses a GPU.
failure_config=FailureConfig(max_failures=10),
),
)
result = trainer.fit()
print(result.checkpoint.path)
Submit it with af-n-workers in place of af-max-workers, for
exactly that many GPUs.
Batch inference¶
Every GPU holds a copy of a trained model, here the one that One GPU saved, and scores its share of a dataset. Ray Data reads the files, hands the rows to the GPUs in batches and writes the results:
# score.py
import os
import numpy as np
import ray
import torch
from torch import nn
FEATURES = [f"f{i}" for i in range(20)]
class Score:
# 1. Each GPU loads the model once.
def __init__(self):
self.model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 1))
self.model.load_state_dict(torch.load("/work/users/<username>/model.pt"))
self.model.to("cuda")
# 2. It gets the rows in batches, an array per column, and adds the scores.
def __call__(self, batch):
x = torch.as_tensor(np.stack([batch[name] for name in FEATURES], axis=1))
with torch.inference_mode():
batch["score"] = self.model(x.to("cuda")).squeeze(1).cpu().numpy()
return batch
# 3. Ray Data reads the files, runs the model on the GPUs and writes the results.
events = ray.data.read_parquet("/work/users/<username>/events")
events.map_batches(
Score,
num_gpus=1,
batch_size=4096,
compute=ray.data.ActorPoolStrategy(
# From one GPU to as many as you ask for when you submit.
min_size=1,
max_size=int(os.environ["AF_MAX_WORKERS"]),
),
).write_parquet("/work/users/<username>/scores")
Submit it as for data-parallel training:
af-max-workers is the number of GPUs to score on.
More GPUs on NRP¶
NRP Nautilus is a national
GPU cluster with many more GPUs than Purdue. A job runs there when you submit
it with af-site. Available to all users.
NRP does not see /work or /depot, so you name the directory your job reads
and the directory its results go to. The first is copied to NRP before the job
starts, and what the job writes comes back when it ends.
client = JobSubmissionClient(
"http://ray-train-gateway:8265",
headers={
"af-site": "nrp",
"af-max-workers": "4",
"af-inputs": "/work/users/<username>/events",
"af-output": "/work/users/<username>/results",
},
)
job = client.submit_job(entrypoint="python train.py", runtime_env={"working_dir": "."})
The job finds the two places in AF_INPUTS and AF_OUTPUT. The numbered
comments mark what changes in the script of
Data-parallel training, with its data in Parquet
files:
trainer = TorchTrainer(
train_func,
# 1. The data comes from AF_INPUTS.
datasets={"train": ray.data.read_parquet(os.environ["AF_INPUTS"])},
scaling_config=ScalingConfig(
num_workers=(1, int(os.environ["AF_MAX_WORKERS"])),
use_gpu=True,
),
run_config=RunConfig(
# 2. The checkpoints go to AF_OUTPUT.
storage_path=os.environ["AF_OUTPUT"],
failure_config=FailureConfig(max_failures=10),
),
)
A script that works on plain files copies them next to itself first, and its results back:
# train.py
import os
from pathlib import Path
import pyarrow.fs
import torch
Path("/tmp/inputs").mkdir()
pyarrow.fs.copy_files(os.environ["AF_INPUTS"], "/tmp/inputs")
model = torch.nn.Linear(20, 1).to("cuda")
...
Path("/tmp/results").mkdir()
torch.save(model.state_dict(), "/tmp/results/model.pt")
pyarrow.fs.copy_files("/tmp/results", os.environ["AF_OUTPUT"])
Good to know on NRP:
- Keep your GPUs busy. NRP asks for it: a job with a GPU under 40% busy over 30 minutes is stopped.
- Starting. A job can take up to half an hour to start, and is stopped if it has not by then.
- Workers. Each has one GPU, 4 CPU cores and 16 GB of memory. A job's GPUs
can be of several models: set
af-min-gpu-memory-gbfor the least memory your job needs of each. - Environment. Jobs run the Python (pixi global) environment. Add packages
with
runtime_env={"pip": ["<package>"]}. - Results. They arrive in
af-outputwhen the job ends, also if it fails or you stop it. It reads RUNNING until they have.
Settings¶
The headers of the JobSubmissionClient say what your jobs run on. A
worker is a process with one GPU, and CPU cores and memory beside it. Every
value is a string:
| Header | Sets | Default |
|---|---|---|
af-max-workers |
The most GPUs to use: your jobs get as many of them as are free | "1" |
af-n-workers |
An exact number of GPUs: your jobs start only if all are free | |
af-min-gpu-memory-gb |
The least memory of a GPU: "16" for T4s |
Any GPU |
af-gpus-per-worker |
"2" for a model on two GPUs |
"1" |
af-cpus-per-worker |
CPU cores of each worker, up to "16" |
"8" |
af-ram-per-worker-gb |
Memory of each worker in GB, up to "64" |
"32" |
af-env |
An environment of your own | Python (pixi global) |
af-site |
"nrp" to run on NRP |
Purdue |
af-inputs |
A directory your job reads, in AF_INPUTS |
|
af-output |
A directory for its results, in AF_OUTPUT |
Each job runs with the settings it was submitted with.
Your own environment¶
In a Pixi project of your own on /work or /depot:
| Package | For |
|---|---|
ray[default] |
Submitting jobs and running them |
ray[train] |
Data-parallel training |
ray[tune] |
Hyperparameter search |
ray[data] |
Data in files, batch inference |
torch |
PyTorch, with its GPU libraries |
ipykernel |
The project's notebook kernel |
Then, from a notebook on the Python (pixi project-aware) kernel, submit with the environment's path:
import os
import sys
client = JobSubmissionClient(
"http://ray-train-gateway:8265",
headers={"af-env": os.path.realpath(sys.prefix)},
)
Good to know¶
- GPUs. A GPU is a 5 GB slice of an A100 or a 16 GB T4, whichever is free.
- Shared GPUs. All your GPUs but one can be taken back while a job runs, and return when they are free again. The scripts of this guide carry on when that happens.
- Storage. A job sees
/work,/depot/cms,/eos/purdueand/cvmfsas your session does, and not your home directory. - Logs. A job's log is removed some minutes after it ends, and its status
some days later: have the script write what you keep to
/work.