System administrator · University of Pittsburgh
Cluster life cycle management Work-load management
I run the research compute cluster for Computational and Systems Biology at Pitt: more than 80 nodes that I install, configure, watch and schedule, for the scientists who depend on them.
Tenth year at the University of Pittsburgh. Thirteen years of molecular simulation before that.
Life cycle management of the compute cluster
Every node is installed, configured and watched the same way.
A compute node passes through three stages, and none of them depend on someone standing in the server room. That's what keeps more than 80 machines identical, and what lets me rebuild any one of them in an afternoon.
-
1Foreman API
Unattended provisioning
A new or failed node is reinstalled remotely. One call to the Foreman API sets its build flag; the node PXE-boots, installs Ubuntu from a preseed file and comes back on its own.
$ curl -X PUT …/api/hosts/node042 \ -d '{"host":{"build":true}}' → PXE boot · preseed · reboot ✓ node042 installed, nobody touched it -
2Ansible · pdsh
Configuration management
Ansible playbooks describe what every node should look like: packages, mounts, users, Slurm daemons. Running them again changes only what drifted. For a quick command across the whole cluster, pdsh.
PLAY RECAP ********************* node041 ok=48 changed=2 failed=0 node042 ok=48 changed=31 failed=0 node043 ok=48 changed=0 failed=0 -
3Icinga · Ganglia
Monitoring
Icinga checks every node and service and tells me when something breaks, often before a user notices. Ganglia keeps the history: load, memory and network over hours and weeks, so slow problems show up too.
- OKssh · nfs · slurmd
- WARNING/scratch 91%
- CRITICALnode017 down
- UNKNOWNgpu-temp
Bash and Perl scripts tie the pieces together.
- Foreman
- Ansible
- Icinga
- Ganglia
- Slurm
- bash
- perl
- python
- Foreman
- Ansible
- Icinga
- Ganglia
- Slurm
- bash
- perl
- python
Work-load management of the compute cluster
One queue. Four kinds of hardware.
The cluster is tens of servers, and they aren't interchangeable. Some have a lot of memory, some have GPUs, most are standard CPU nodes, and file servers hold everyone's data. Slurm decides whose job runs where, and when.
My part is making sure it decides well: partitions for each kind of hardware, limits and priorities so one group can't fill the cluster, and accounting so everyone can see what they used. I install, compile and upgrade Slurm itself, and before Slurm I ran PBS.
- Standard CPUMost jobs: many cores, ordinary memory.
- High memoryGenome assembly, large matrices, anything that needs hundreds of gigabytes in one place.
- GPUDeep learning, molecular dynamics, structure prediction.
- File serversHome to every user's files, and the backup for their data. Most rely on hardware RAID.
Start of a job script for
Memory, CPU and time limits here are examples. Run sinfo on the cluster to see each partition's limits.
- PDpending
- Rrunning
- CDcompleted
- OOMout of memory
Users' problem solving
Some days start with somebody's job that didn't run.
Researchers want results, not a lesson in schedulers. These are the questions I hear most, what I check, and what usually fixes it.
SlurmMy job has been pending since yesterday.
I check
squeue -j 812345 -o "%.10i %.9P %.8T %r"The Reason column says why: waiting for Resources, lower Priority, or a per-user limit.
Usually
The request asks for more than any free node has, or a week of walltime when the job needs a day. Right-sizing it lets the backfill scheduler start it much sooner.
SlurmMy job died after ten minutes with no error.
I check
sacct -j 812345 --format=JobID,State,ExitCode,ReqMem,MaxRSS,ElapsedState OUT_OF_MEMORY or TIMEOUT explains most silent deaths.
Usually
Request memory that matches what the job really used (MaxRSS plus some headroom), or move it to the high-memory nodes.
GPUMy code can't find the GPU.
I check
scontrol show job 812345 | grep -i tresWas a GPU actually requested? Then nvidia-smi inside the job, and the CUDA version against the driver.
Usually
A missing --gres=gpu:1, the wrong partition, or a framework built for a different CUDA version.
SoftwareIt works on my laptop but not on the cluster.
I check
module list; ldd $(which program)Which modules and libraries the job sees on the compute node, not on the login node.
Usually
A module not loaded inside the job script, or a library that isn't installed. I compile and install what's missing so everyone can use it.
PerformanceIt runs slower here than on my desktop.
I check
ssh n010 top -u $USERWhether the program starts sixteen threads on the one core Slurm gave it. Ganglia shows the node's load over the job's lifetime.
Usually
Match --cpus-per-task to the threads the program uses and set OMP_NUM_THREADS from it.
StorageNobody in our lab can save files anymore.
I check
ssh capricorn df -hWhen a whole lab is stuck at once, it's usually their PI's file server. Here, capricorn is at 100%. Then du -sh * | sort -h on its shares shows which directories filled it.
Usually
A few huge job outputs or duplicate copies of a dataset. I show the PI where the space went, the lab cleans up or archives, and if they really need more room we plan bigger disks for the server's RAID array.
Also part of the job: user accounts and SSH access, the department's file servers, Linux desktops, and finding the failed disk or memory module when hardware gives out.
Contact
Find me in Oakland.
afe9@pitt.edu Department of Computational and Systems BiologySchool of Medicine, University of Pittsburgh
3396 Fifth Avenue, 10th floor
Pittsburgh, PA 15213