ML Systems Engineer - Nebius B.V. - Palo Alto, United States

Nebius Token Factory is building an AI training and model post-training capability for frontier model improvement. This role owns the infrastructure that makes large-scale training and RL experiments possible, reliable, reproducible, and efficient. The work sits at the intersection of distributed systems, GPU performance, model training frameworks, RL pipelines, and production engineering. A Senior ML Systems Engineer owns substantial training or RL infrastructure components end to end. They are deeply hands-on, can debug difficult distributed training failures independently, and can deliver measurable improvements in experiment throughput, stability, and GPU utilization. Your responsibilities: - Build and maintain distributed training infrastructure for SFT, continued pretraining, preference optimization, and RL workloads. - Integrate and extend frameworks such as Megatron-LM, DeepSpeed, PyTorch FSDP/DTensor, Ray, verl, slime, AReaL, OpenRLHF, or equivalent internal systems. - Implement and debug parallelism strategies including tensor, pipeline, sequence/context, expert, and data parallelism. - Build reliable rollout, reward model serving, replay/data buffer, checkpointing, evaluation, and experiment orchestration components for RL training. - Profile and improve GPU utilization, communication efficiency, memory usage, and training throughput. - Diagnose failures across NCCL, CUDA, PyTorch, Ray, schedulers, storage, networking, and checkpointing layers. - Create reproducible training runs, launch scripts, dashboards, runbooks, and operational tooling for research users. - Partner with research scientists to turn algorithmic training recipes into scalable, debuggable systems. - Write clear design docs, incident reports, benchmark reports, and operating guides. Must-haves: - Strong Python and PyTorch engineering skills. - Hands-on experience with distributed model training, large-scale ML systems, or GPU cluster workloads. - Practical understanding of transformer training bottlenecks, memory pressure, gradient/optimizer state, communication overhead, and checkpointing. - Experience debugging production or research training jobs across multiple GPUs or nodes. - Ability to reason quantitatively about throughput, utilization, memory, reliability, cost, and research velocity. - Strong communication skills and ability to collaborate with researchers, ML engineers, platform engineers, and leadership. Nice-to-haves: - Experience with Megatron-LM, DeepSpeed, PyTorch FSDP/DTensor, Ray, Slurm, Kubernetes, or large internal training platforms. - Experience with RL infrastructure frameworks such as verl, slime, AReaL, OpenRLHF, TRL, or custom PPO/GRPO/RLHF systems. - Familiarity with NCCL, CUDA, Triton, Nsight, InfiniBand, RDMA, RoCE, H100/H200/B200 clusters, or storage/network bottlenecks. - Experience supporting SFT, DPO, PPO, GRPO, RLAIF, reward model serving, rollout generation, or agent training workloads. - Open-source contributions to distributed training, RL infrastructure, PyTorch, Ray, Megatron, DeepSpeed, or related systems.