Job Purpose:
We are seeking a motivated and technically capable High-Performance Computing (HPC) Engineer to join our HPC Infrastructure and Operations team. The role is suitable for early-career engineers with strong Linux fundamentals, practical troubleshooting skills, and a keen interest in GPU computing, distributed systems, containerized workloads, data center networking, and AI infrastructure.
The successful candidate will support the deployment, administration, maintenance, and optimization of large-scale GPU/CPU computing clusters that power cutting-edge Artificial Intelligence (AI), Machine Learning (ML), and scientific research.
Key Responsibilities:
HPC Hardware Operations & Maintenance
- Operate, monitor, and maintain large-scale GPU/CPU computing clusters, including NVIDIA/AMD GPU servers, CPU compute nodes, and enterprise storage systems.
- Perform hardware diagnostics, preventive maintenance, firmware upgrades, and component replacement for GPUs, CPUs, memory, storage devices, network adapters, and other server components.
- Identify and troubleshoot hardware failures, including GPU errors, memory faults, disk failures, thermal issues, and system instability.
- Conduct hardware health checks, performance validation, and stress testing to ensure cluster reliability and availability.
- Coordinate with hardware vendors and internal infrastructure teams for incident resolution, repairs, and hardware lifecycle management.
HPC System Administration & Cluster Management
- Support the deployment, configuration, administration, and maintenance of Linux-based HPC clusters.
- Administer and troubleshoot Slurm workload management, including job scheduling, partitions, QoS, resource allocation, job failures, and user access management.
- Support distributed storage systems, including Lustre or IBM Storage Scale (GPFS), covering filesystem availability, performance monitoring, client connectivity, and storage-related troubleshooting.
- Perform operating system provisioning, patching, driver updates, software installation, and cluster configuration management.
- Monitor cluster health, system performance, resource utilization, and service availability.
GPU Computing & AI Workload Support
- Provide technical support to researchers running AI/ML training, inference, and scientific computing workloads across GPU/CPU clusters.
- Diagnose and resolve job execution failures related to Slurm, CUDA, NVIDIA drivers, NCCL, MPI, PyTorch, distributed training frameworks, and system dependencies.
- Troubleshoot GPU memory issues, resource allocation problems, multi-GPU/multi-node communication failures, and performance bottlenecks.
- Assist researchers with job submission, resource optimization, environment configuration, and best practices for efficient cluster utilization.
- Support the validation and benchmarking of GPU/CPU compute nodes and distributed AI workloads.
Automation, Monitoring & Operational Support
- Develop and maintain scripts and automation tools using Python, Bash, or other programming languages to streamline HPC administration and operational tasks.
- Support automated system provisioning, configuration management, monitoring, and health-check processes.
- Maintain technical documentation, troubleshooting guides, operational procedures, and incident reports.
- Participate in incident response, root-cause analysis, preventive maintenance, and continuous infrastructure improvement.
- Collaborate with researchers, HPC engineers, IT teams, and technology vendors to ensure stable and efficient HPC services.
HPC Containers & Data Center Networking
- Build, maintain, and support HPC container environments using Apptainer (Singularity), including definition files, image builds, GPU and MPI integration, and secure execution of containerized jobs under Slurm.
- Support the deployment and operation of Kubernetes-based platforms for AI/ML services, inference, and research workflows, including GPU scheduling (NVIDIA GPU Operator / device plugin) and integration with HPC storage.
- Maintain container registries and curated base images (e.g., CUDA, PyTorch, MPI, NGC) to provide reproducible, secure, and up-to-date software environments for researchers.
- Operate, monitor, and troubleshoot data center network switches (Ethernet and InfiniBand), including port configuration, VLANs, link aggregation, firmware upgrades, cabling/optics faults, and fabric health checks.
- Support high-performance cluster fabrics (InfiniBand, RoCE/RDMA) and leaf-spine topologies, working with network teams and vendors to resolve latency, congestion, and link-error issues affecting distributed workloads.
Other Duties:
Perform all other duties as reasonably directed by the line manager that are commensurate with these functional objectives.
Qualifications and Criteria:
- Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, Software Engineering, Information Technology, or a related technical discipline.
- Master’s degree in a relevant field, or equivalent practical experience in Linux systems, HPC infrastructure, or distributed computing will be preferred.
Essential:
- Strong understanding of Linux operating systems, command-line tools, system services, processes, permissions, and troubleshooting methodologies.
- 3-5 years of practical experience with Linux server administration, hardware diagnostics, or infrastructure operations.
- Understanding of computer hardware architecture, including CPUs, GPUs, memory, storage, and networking components.
- Familiarity with scripting languages such as Python and Bash for system administration and automation.
- Fundamental knowledge of TCP/IP networking, SSH, DNS, network interfaces, and system connectivity.
- Working knowledge of container technologies such as Apptainer (Singularity), Docker, or Podman, including building, running, and troubleshooting container images.
- Hands-on exposure to data center network switches, including Ethernet switching concepts (VLANs, LACP, MTU/jumbo frames, LLDP) and switch CLI configuration and troubleshooting.
- Ability to investigate technical problems using system logs, monitoring tools, and structured troubleshooting approaches.
- Strong analytical, problem-solving, documentation, and communication skills.
- Ability to work collaboratively with researchers, engineers, and technical support teams.
Preferred:
GPU Computing & AI Infrastructure
- Familiarity with NVIDIA GPU architecture, drivers, CUDA, and GPU monitoring tools such as nvidia-smi and NVIDIA DCGM.
- Exposure to PyTorch, NCCL, or distributed AI/ML training workloads.
- Understanding multi-GPU and multi-node computing environments.
Containers & Orchestration
- Hands-on experience with Kubernetes administration (deployments, Helm, namespaces, RBAC, persistent storage) in GPU-enabled environments.
- Familiarity with the NVIDIA Container Toolkit, NVIDIA GPU Operator, and running containerized MPI/NCCL workloads via Apptainer or Pyxis/Enroot under Slurm.
- Exposure to container registries (e.g., Harbor), CI/CD pipelines, and configuration/infrastructure-as-code tools such as Ansible or Terraform.
Data Center Networking
- Experience with InfiniBand fabrics (e.g., NVIDIA Quantum switches, UFM, Subnet Manager, ibdiagnet) and RDMA/RoCE (PFC, ECN) for GPU clusters.
- Familiarity with data center switch platforms such as NVIDIA Spectrum (Cumulus Linux), Arista EOS, Cisco NX-OS, or Juniper, and leaf-spine/BGP-EVPN architectures.
- Ability to monitor and benchmark network performance using tools such as SNMP, Prometheus/Grafana, perftest (ib_write_bw), and iperf.