NVIDIA GPU Infrastructure Engineer / HPC Engineer

Stratsol Software Systems · Chennai, Tamil Nadu, India · Full Time · Hybrid

Posted

Hiring: NVIDIA GPU Infrastructure Engineer / HPC Engineer Location: Chennai, India Experience: 7–12 Years Work Mode: Onsite / Hybrid About the Role We are looking for an experienced NVIDIA GPU Infrastructure Engineer / HPC Engineer to build, manage, and optimize large-scale AI/HPC GPU clusters supporting high-performance LLM training and inference workloads.The ideal candidate will have strong hands-on expertise in NVIDIA H100/B-series GPUs, CUDA, Slurm, Kubernetes, NCCL, and GPU performance optimization, with experience supporting enterprise-scale GPU infrastructure. Must-Have Technical Skills GPU Infrastructure: NVIDIA H100/B-series GPUs, large-scale GPU clusters (up to 10 K GPUs), multi-pod architecture, GPU/CPU/Memory/NIC capacity planning, and heterogeneous GPU SKU bin-packing.GPU Virtualization: NVIDIA MIG, v GPU profiles, multi-tenant GPU partitioning, quota management, and resource allocation.GPU Computing: CUDA, cu DNN, Tensor RT, NCCL, UCX, GPU clocks/persistence, and CUDA runtime tuning.Cluster Orchestration: Slurm, Kubernetes (K8s), GPU Device Plugins, Daemon Sets, Node Feature Discovery (NFD), scheduling, accounting, and quotas.GPU Monitoring: NVIDIA DCGM, NVML, nvidia-smi, Prometheus, Grafana, ELK, and Splunk.Automation & Deployment: Helm, Argo CD, golden images, node remediation (cordon/drain/reimage), and auto-healing workflows.High-Performance Networking: Infini Band (IB), Ethernet, RDMA, TOR/leaf connectivity, NUMA/locality optimization, and GPU Direct Storage.Performance Benchmarking: HPL, HPL-AI, ML Perf-like benchmarking, performance baselines, regression analysis, and SLO monitoring.Infrastructure Operations: Rack/power/cooling layouts, firmware and driver lifecycle management, change management, and zero/low-downtime maintenance.Security: MIG isolation, cgroups/device cgroups, mpsd, secure GPU drivers/containers, SBOM, and image scanning. Key Responsibilities Deploy, configure, and operate enterprise-scale NVIDIA GPU clusters supporting AI/ML training and inference.Integrate Slurm/Kubernetes scheduling with GPU resource management and multi-tenant workloads.Optimize CUDA, NCCL, UCX, RDMA, GPU performance, and distributed training efficiency.Manage GPU firmware, drivers, monitoring, cluster health, and capacity planning.Lead P0/P1 production incidents involving GPU, CUDA, networking, NCCL, RDMA, and scheduling failures.Coordinate with Networking, GPU Operations, Platform Engineering, Storage, and Application teams to restore services within SLA targets.Perform root cause analysis (RCA), implement corrective actions, and improve infrastructure reliability.Maintain runbooks, performance baselines, and application tuning documentation.Track GPU utilization, queue wait-time SLOs, failed-job rates, performance baseline adherence, and incident MTTD/MTTR. Qualifications BE/B-Tech or equivalent in Computer Science or Electronics & Communication.7–12 years of hands-on experience building and operating HPC/AI GPU clusters at scale.Deep expertise in CUDA, MIG, and v GPU.Proven experience with NVIDIA H100/B-series GPUs, Slurm and/or Kubernetes device scheduling, and NCCL/UCX performance tuning. Preferred Certifications NVIDIA Certified Professional – AI Infrastructure RHCE / LPIC (Linux).

Apply on Stratsol Software Systems's site