Job Purpose
Build and operate large?scale GPU compute pods to deliver predictable, high?throughput, low?latency training and inference services across 10K GPU cluster.
Roles & Responsibilities
Stand up multi?pod GPU clusters (rack/power/cooling layouts; TOR/leaf connectivity; IB/Ethernet host configs).
Implement GPU partitioning (MIG/vGPU profiles) and quota policies for multi?tenant environments.
Integrate cluster schedulers (Slurm/Kubernetes) with GPU device plugins, node feature discovery, and accounting/quotas.
Own day?2 operations across firmware/driver/DCGM/NVML lifecycles; execute change windows with zero/low downtime.
Capacity planning (GPU/CPU/Memory/NIC) and bin?packing strategies for heterogeneous GPU SKUs.
Tune NCCL/UCX, GPU clocks/persistence, GPU Direct Storage, NUMA/locality, and CUDA runtime parameters.
Drive benchmarking and acceptance (HPL, HPL?AI, MLPerf?like internal suites); track perf regressions with SLOs.
Lead P0/P1 incident response for GPU, CUDA, or scheduler issues; perform root?cause and preventative actions.
Act as the primary technical lead during production outages, coordinating cross-functional teams across Networking, GPU Operations, Platform Engineering, Storage, and Application teams to restore services within SLA targets.
Perform detailed Root Cause Analysis (RCA) for network, GPU, CUDA, NCCL, RDMA, and scheduler-related failures, identifying underlying causes and implementing preventive and corrective actions.
Collaborate with platform and GPU engineering teams to resolve issues impacting CUDA jobs, Kubernetes scheduling, Slurm workload management, GPU resource allocation, and large-scale AI training environments.
Define golden images; implement node remediation (cordon/drain/reimage) and auto?healing workflows.
Enforce GPU tenancy isolation (MIG, cgroup/device cgroup, mpsd), secure drivers/containers, SBOM and image scanning.
Publish runbooks, performance baselines, and application tuning guides for LLM training and inference.
Experience & Educational Requirement
BE/B-Tech or equivalent with Computer Science or Electronics & Communication
NVIDIA Certified Professional (AI Infrastructure); Linux (RHCE/LPIC) preferred.
RELEVANT EXPERIENCE
Tools / Tech
1. CUDA, NCCL, cuDNN, TensorRT; 2. DCGM/NVML, nvidia-smi; 3. Slurm, K8s (device plugin, DaemonSets, NFD); 4. Helm/ArgoCD; 6. GPUDirect Storage; 7. Prometheus/Grafana; 8. ELK/Splunk.
Source: Larsen & Toubro careers — Read the original posting and apply
This role is listed by the employer on its own careers site. Kaam Ki Khoj does not process applications for it.
Larsen & Toubro is an employer in the Construction/Engineering sector with operations in India. This is a directory listing maintained by Kaam Ki Khoj so that candidates can find the organisation; it is not an official company page and Kaam Ki Kh... Read More
Upload your CV once — we read it, build your profile and put you in front of every employer hiring on Kaam Ki Khoj. No forms, no fees.
