HPC Systems Engineer

Job Purpose:

We are seeking a motivated and technically capable High-Performance Computing (HPC) Engineer to join our HPC Infrastructure and Operations team. The role is suitable for early-career engineers with strong Linux fundamentals, practical troubleshooting skills, and a keen interest in GPU computing, distributed systems, containerized workloads, data center networking, and AI infrastructure.

The successful candidate will support the deployment, administration, maintenance, and optimization of large-scale GPU/CPU computing clusters that power cutting-edge Artificial Intelligence (AI), Machine Learning (ML), and scientific research.

Key Responsibilities:

HPC Hardware Operations & Maintenance

• Operate, monitor, and maintain large-scale GPU/CPU computing clusters, including NVIDIA/AMD GPU servers, CPU compute nodes, and enterprise storage systems.
• Perform hardware diagnostics, preventive maintenance, firmware upgrades, and component replacement for GPUs, CPUs, memory, storage devices, network adapters, and other server components.
• Identify and troubleshoot hardware failures, including GPU errors, memory faults, disk failures, thermal issues, and system instability.
• Conduct hardware health checks, performance validation, and stress testing to ensure cluster reliability and availability.
• Coordinate with hardware vendors and internal infrastructure teams for incident resolution, repairs, and hardware lifecycle management.

HPC System Administration & Cluster Management

• Support the deployment, configuration, administration, and maintenance of Linux-based HPC clusters.
• Administer and troubleshoot Slurm workload management, including job scheduling, partitions, QoS, resource allocation, job failures, and user access management.
• Support distributed storage systems, including Lustre or IBM Storage Scale (GPFS), covering filesystem availability, performance monitoring, client connectivity, and storage-related troubleshooting.
• Perform operating system provisioning, patching, driver updates, software installation, and cluster configuration management.
• Monitor cluster health, system performance, resource utilization, and service availability.

GPU Computing & AI Workload Support

• Provide technical support to researchers running AI/ML training, inference, and scientific computing workloads across GPU/CPU clusters.
• Diagnose and resolve job execution failures related to Slurm, CUDA, NVIDIA drivers, NCCL, MPI, PyTorch, distributed training frameworks, and system dependencies.
• Troubleshoot GPU memory issues, resource allocation problems, multi-GPU/multi-node communication failures, and performance bottlenecks.
• Assist researchers with job submission, resource optimization, environment configuration, and best practices for efficient cluster utilization.
• Support the validation and benchmarking of GPU/CPU compute nodes and distributed AI workloads.

Automation, Monitoring & Operational Support

• Develop and maintain scripts and automation tools using Python, Bash, or other programming languages to streamline HPC administration and operational tasks.
• Support automated system provisioning, configuration management, monitoring, and health-check processes.
• Maintain technical documentation, troubleshooting guides, operational procedures, and incident reports.
• Participate in incident response, root-cause analysis, preventive maintenance, and continuous infrastructure improvement.
• Collaborate with researchers, HPC engineers, IT teams, and technology vendors to ensure stable and efficient HPC services.

HPC Containers & Data Center Networking

• Build, maintain, and support HPC container environments using Apptainer (Singularity), including definition files, image builds, GPU and MPI integration, and secure execution of containerized jobs under Slurm.
• Support the deployment and operation of Kubernetes-based platforms for AI/ML services, inference, and research workflows, including GPU scheduling (NVIDIA GPU Operator / device plugin) and integration with HPC storage.
• Maintain container registries and curated base images (e.g., CUDA, PyTorch, MPI, NGC) to provide reproducible, secure, and up-to-date software environments for researchers.
• Operate, monitor, and troubleshoot data center network switches (Ethernet and InfiniBand), including port configuration, VLANs, link aggregation, firmware upgrades, cabling/optics faults, and fabric health checks.
• Support high-performance cluster fabrics (InfiniBand, RoCE/RDMA) and leaf-spine topologies, working with network teams and vendors to resolve latency, congestion, and link-error issues affecting distributed workloads.

 Other Duties:

Perform all other duties as reasonably directed by the line manager that are commensurate with these functional objectives.

Qualifications and Criteria:

• Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, Software Engineering, Information Technology, or a related technical discipline.
• Master’s degree in a relevant field, or equivalent practical experience in Linux systems, HPC infrastructure, or distributed computing will be preferred.

Essential:

• Strong understanding of Linux operating systems, command-line tools, system services, processes, permissions, and troubleshooting methodologies.
• 3-5 years of practical experience with Linux server administration, hardware diagnostics, or infrastructure operations.
• Understanding of computer hardware architecture, including CPUs, GPUs, memory, storage, and networking components.
• Familiarity with scripting languages such as Python and Bash for system administration and automation.
• Fundamental knowledge of TCP/IP networking, SSH, DNS, network interfaces, and system connectivity.
• Working knowledge of container technologies such as Apptainer (Singularity), Docker, or Podman, including building, running, and troubleshooting container images.
• Hands-on exposure to data center network switches, including Ethernet switching concepts (VLANs, LACP, MTU/jumbo frames, LLDP) and switch CLI configuration and troubleshooting.
• Ability to investigate technical problems using system logs, monitoring tools, and structured troubleshooting approaches.
• Strong analytical, problem-solving, documentation, and communication skills.
• Ability to work collaboratively with researchers, engineers, and technical support teams.

Preferred:

GPU Computing & AI Infrastructure

• Familiarity with NVIDIA GPU architecture, drivers, CUDA, and GPU monitoring tools such as nvidia-smi and NVIDIA DCGM.
• Exposure to PyTorch, NCCL, or distributed AI/ML training workloads.
• Understanding multi-GPU and multi-node computing environments.

Containers & Orchestration

• Hands-on experience with Kubernetes administration (deployments, Helm, namespaces, RBAC, persistent storage) in GPU-enabled environments.
• Familiarity with the NVIDIA Container Toolkit, NVIDIA GPU Operator, and running containerized MPI/NCCL workloads via Apptainer or Pyxis/Enroot under Slurm.
• Exposure to container registries (e.g., Harbor), CI/CD pipelines, and configuration/infrastructure-as-code tools such as Ansible or Terraform.

Data Center Networking

• Experience with InfiniBand fabrics (e.g., NVIDIA Quantum switches, UFM, Subnet Manager, ibdiagnet) and RDMA/RoCE (PFC, ECN) for GPU clusters.
• Familiarity with data center switch platforms such as NVIDIA Spectrum (Cumulus Linux), Arista EOS, Cisco NX-OS, or Juniper, and leaf-spine/BGP-EVPN architectures.
• Ability to monitor and benchmark network performance using tools such as SNMP, Prometheus/Grafana, perftest (ib_write_bw), and iperf.