GPUs on Carina: A Practical Guide

Carina Hardware

Carina is a compact cluster of 5 Dell PowerEdge XE7745 compute nodes. Each node has:

8 NVIDIA L40S GPUs
256 CPUs (512 threads)
1.5 TB RAM
~28 TB local NVMe storage

That’s 40 L40S GPUs and 1,280 CPU cores across the cluster.

The L40S is an Ada-generation GPU built for AI/ML and scientific computing — training, inference, and data processing, plus rendering and video workloads. If your code runs on CUDA-enabled GPUs, it runs on the L40S.

What is a GPU, Actually?

Your CPU is a generalist. It can run your email client, compile code, render video, or solve differential equations — trading raw speed for flexibility.

A GPU is a specialist. It does the same operation on millions of pieces of data at once, trading flexibility for throughput.

The pizza shop analogy: a CPU core is a chef who can make any dish perfectly. A GPU is 10,000 line workers, each putting the same topping on a pizza, all at once. One pizza? The chef wins. A million pizzas? The GPU wins by orders of magnitude.

What GPUs Are Good At

Deep learning — training neural networks on massive datasets
Matrix operations — linear algebra and scientific computing
Image and video processing — applying the same filter to every pixel
Molecular simulations — the same physics calculation across billions of atoms
Data processing at scale — ETL, feature engineering, batch inference

The common thread: the same operation repeated millions of times on different data.

What GPUs Are Terrible At

Sequential work — anything where step 2 depends on the result of step 1
Complex conditionals — “if this, do that; else do something else”
General-purpose logic — code that does different things each step
Small jobs — the cost of copying data to the GPU can outweigh the benefit

If your code is 95% sequential logic and 5% computation, a GPU only speeds up the 5% — you still bottleneck on the rest.

When to Request a GPU

Request a GPU if:

Your software is GPU-accelerated — TensorFlow or PyTorch (with CUDA), GROMACS/AMBER/LAMMPS with GPU support, CUDA libraries like cuDNN or cuBLAS, OpenACC, or custom CUDA code.
You’ve profiled it, not guessed. Time a small test on CPU cores, then the same job on a GPU. A 5x speedup is worth the queue wait; a 2x speedup often isn’t.
The speedup is worth the cost. GPUs are expensive in queue priority and allocation. A job that drops from 1 hour to 12 minutes is worth it; 5 minutes to 4 is not.

Don’t request a GPU if:

Your code doesn’t support GPUs (check its docs)
You haven’t tested it on a GPU
Your CPU job is already fast enough
You’re “just trying it” — explore on CPU cores, save the GPU for production runs
Your code uses only a fraction of the GPU — you’re tying up an expensive resource

The Profiling Workflow

Before you request a GPU:

# 1. Run a small test on CPU cores
sbatch -p normal --cpus-per-task=8 --time=02:00:00 my_job.sh

# 2. Run the same test on a GPU
sbatch -p normal --cpus-per-task=8 --gres=gpu:1 --time=02:00:00 my_job.sh

# Compare the runtimes: is the GPU version at least 3-5x faster?

If the GPU version isn’t faster, you likely have data-transfer overhead eating the benefit, code that isn’t actually using the GPU, or a GPU-unfriendly algorithm. In those cases, stick with CPU cores.

What You Actually Request

A GPU job specifies:

-p normal             # Partition: dev (2h), normal (2 days), or long (5 days)
--gres=gpu:1          # Number of GPUs (usually 1, max 8 per node)
--cpus-per-task=8     # CPU cores to support the GPU
--mem=32G             # Memory for data staging
--time=04:00:00       # Wall-clock limit

GPUs need CPU cores to work — the GPU can’t handle I/O, disk operations, or most setup and teardown. Allocate enough cores to keep it fed; 8-16 per GPU is a reasonable starting point.

Local NVMe Storage

Each node has ~28 TB of fast, local NVMe scratch storage. Use it for:

High-throughput I/O — reading and writing large datasets during a job
Checkpointing — saving model weights and job state
Scratch files — intermediate results

Scratch is ephemeral: it’s local to the node and is wiped when your job ends, so copy anything you need to keep back to $HOME or /projects before the job finishes.

Scratch lives at /local/scratch, and Slurm automatically creates a per-job directory and sets $LOCAL_SCRATCH_JOB to it (/local/scratch/<sunetid>/<jobid>). Use that variable rather than hardcoding a path. See Local Scratch Storage for details.

#!/bin/bash
SCRATCH=$LOCAL_SCRATCH_JOB   # auto-set by Slurm: /local/scratch/<sunetid>/<jobid>

# Copy input data to fast local storage
cp $HOME/my_large_dataset.h5 $SCRATCH/

# Run your job, reading from fast NVMe
python train.py --input $SCRATCH/my_large_dataset.h5 --output $SCRATCH/model.pt

# Copy results back to persistent storage before the job ends
cp $SCRATCH/model.pt $HOME/results/

Common Mistakes

Requesting a GPU but not using it. Check nvidia-smi during your job — utilization should be well above 80%. If the GPU sits idle, you’re holding a resource you don’t need.
Requesting too much. Asking for 8 GPUs and 48 hours when the job needs 1 GPU for 3 hours doesn’t make it faster — it just blocks other researchers.
Ignoring local NVMe. Repeatedly reading a large dataset from shared storage makes your job bandwidth-limited instead of compute-limited. Stage it to $LOCAL_SCRATCH_JOB first.
Mixing CPU and GPU work inefficiently. If a job does two hours of CPU data prep and then 30 minutes on the GPU, you’re still keeping the GPU for 2.5 hours. Prep on CPU first, then submit the GPU job — or move the prep onto the GPU too.

L40S Specifics

The NVIDIA L40S is purpose-built for AI/ML and data processing:

48 GB GDDR6 memory — enough for most training jobs, but watch your batch sizes
NVENC/NVDEC — hardware video encoding/decoding if you need it
Full FP32, TF32, BF16, and FP16 support — flexible precision for training and inference
864 GB/s memory bandwidth — fast enough for most workloads

If your code supports CUDA, it supports the L40S. No special setup needed.

When in Doubt

Start with CPU cores
Time a representative job
Profile to see where the time actually goes
Request a GPU only if the measurements justify it

Carina’s CPU cores are plenty fast for most work. GPUs are the rocket booster — use them when you actually need them.