Local Scratch Storage
Every Carina compute node has a large, fast local storage area called scratch. It is built from four NVMe drives in a RAID0 configuration, which means reads and writes are significantly faster than your $HOME or /projects directories. It is a good place to land large intermediate files during a computation-heavy job.
Where Is It?
Scratch storage is available at /local/scratch on every compute node.
The Critical Rule: Copy Your Results Out
/local/scratch is not automatically cleaned up — but it is only accessible from the node your job is running on. Once your job is done and Slurm releases that node, you cannot get back to your files. Copy anything you need to keep into $HOME or /projects before your job ends.
This is not a place to store data permanently. It is a high-speed workspace for the duration of a job.
When to Use Scratch
Scratch is most valuable when your job:
- Reads or writes many large files (model checkpoints, large datasets, video frames)
- Produces substantial intermediate output that is not itself a result
- Would otherwise generate heavy I/O on your
$HOMEor/projectsdirectories, which are network-mounted
If your job reads a file once and writes a result once, scratch is probably not worth the extra copy steps.
Using Scratch in a Slurm Job
When you request local scratch, some useful environment variables are automatically set, and your local scratch directory is created.
LOCAL_SCRATCH_ROOT=”/local/scratch” /local/scratch
LOCAL_SCRATCH_USER=$LOCAL_SCRATCH_ROOT/$USER /local/scratch/sunetid
LOCAL_SCRATCH_JOB=$LOCAL_SCRATCH_USER/$SLURM_JOB_ID /local/scratch/sunetid/5309
Note that there is no trailing slash on these paths, so you will need to supply it in your script if needed (as in the copying step in our sample below)
Here is a pattern for using scratch in an sbatch script.
#!/bin/bash
#SBATCH --job-name=my-analysis
#SBATCH --output=my-analysis.out
#SBATCH --error=my-analysis.err
#SBATCH --time=4:00:00
#SBATCH --mem=32G
# -------Copy input data to scratch-------
SCRATCH = $LOCAL_SCRATCH_JOB
cp $HOME/my-data/input.tar.gz $SCRATCH/
cd $SCRATCH
tar -xzf input.tar.gz
# -------Run your computation-------
ml python
python3 my_analysis.py --input $SCRATCH/input/ --output $SCRATCH/output/
# -------Copy results back before the job ends-------
cp -r $SCRATCH/output/ $HOME/my-results/
The $SCRATCH directory is removed when your session ends, so your script does not need to do ta clean-up step.
Try It: Scratch Demo
Not sure if scratch is working, or want to see what kind of I/O speeds you can expect? Download these two files, place them in a scratch-demo folder in your $HOME, and submit the batch script.
- scratch-demo.sbatch — the batch script
- generate_data.py — writes and reads a series of 100 MB files on scratch, then reports throughput
mkdir $HOME/scratch-demo
# copy both files into scratch-demo/, then:
sbatch $HOME/scratch-demo/scratch-demo.sbatch
When the job completes, results will be in $HOME/scratch-demo/results-<jobid>/summary.txt.
Checking Available Space
To see how much space is free on scratch while you are on a compute node:
df -h /local/scratch
Each node has approximately 28 TB of raw NVMe storage shared across all users currently running on that node. If you are running a job that needs a very large scratch area, check availability before writing large amounts of data.