
Senior Machine Learning Engineer
Nebius Token Factory is building an AI training and model post-training capability for frontier model improvement. This role owns the infrastructure that makes large-scale training and RL experiments possible, reliable, reproducible, and efficient. The work sits at the intersection of distributed systems, GPU performance, model training frameworks, RL pipelines, and production engineering. Your responsibilities include designing and running model-training and post-training experiments (including SFT, continued pretraining, DPO/IPO/KTO, RL methods like RLHF/RLAIF, PPO, GRPO), building reward functions and evaluation sets, creating synthetic data and data pipelines, analyzing model-behavior failures, building distributed training and RL infrastructure using Megatron-LM, DeepSpeed, PyTorch FSDP/DTensor, Ray, and more, implementing parallelism strategies, optimizing GPU utilization, and writing experiment plans and runbooks. Must-haves: strong Python and PyTorch engineering, hands-on experience in at least two areas among model training, post-training/RL, data pipelines, or large-scale ML systems, ability to design rigorous experiments, understanding of modern LLM behavior and training bottlenecks, strong communication skills. Nice-to-haves: experience with RL frameworks/pipelines and Megatron-LM/DeepSpeed, CUDA and NCCL familiarity, publications or production impact in LLM domains. In the US, key benefits are Health insurance, 401(k), Parental leave, Remote work reimbursement, Disability & life insurance. Pay: Base Compensation Range $195,200 - $262,200 USD annually. We offer competitive compensation and benefits. This is an onsite role in Palo Alto with US-based benefits.
Job Details
Responsibilities
- Design and run model-training and post-training experiments including SFT, continued pretraining, DPO/IPO/KTO, RL methods such as RLHF/RLAIF, PPO, and GRPO
- Build reward functions, verifiers, task environments, and evaluation sets for reasoning, coding, tool use, and agentic workflows
- Create synthetic data and data pipelines including teacher-student generation, self-play, rejection sampling, filtering, and quality scoring
- Analyze model-behavior failures and turn them into targeted data, reward, or algorithm improvements
- Build and maintain distributed training and RL infrastructure using Megatron-LM, DeepSpeed, PyTorch FSDP/DTensor, Ray, Verl, slime, AReaL, or OpenRLHF
- Implement and debug parallelism strategies (tensor, pipeline, sequence/context, expert, data) and build rollout, reward-serving, checkpointing, and experiment-orchestration components
- Profile and improve GPU utilization, memory usage, communication efficiency, training throughput, and inference/serving performance
- Design rigorous evaluations and ablations for capability, instruction following, reasoning, tool use, safety, and regression risk
- Write experiment plans, design docs, benchmark reports, and runbooks, and partner across research and platform teams
- Must-haves: strong Python and PyTorch engineering, experience with at least two areas among model training, post-training/RL, data pipelines, or large-scale ML systems
- Nice-to-haves: experience with RL frameworks/pipelines and Megatron-LM/DeepSpeed, CUDA, NCCL familiarity, publications or production impact in LLM domains
Requirements
- Strong Python and PyTorch engineering skills
- Hands-on experience across at least two of: model training, post-training/RL, data pipelines, large-scale ML systems
Skills & Technologies

| Location | Active listings |
|---|---|
| Remote - Global | 398 |
| Remote - Europe | 126 |
| Amsterdam, Netherlands | 57 |
| London, United Kingdom | 22 |
| Remote - United States | 20 |
| Remote - Finland | 18 |
| Berlin, Germany | 16 |
| Mäntsälä, Finland | 12 |
| Helsinki, Finland | 10 |
| Lappeenranta, Finland | 10 |
| Prague, Czech Republic | 8 |
| Israel | 6 |
| Amsterdam | 5 |
| United Kingdom | 5 |
| Canada | 4 |
| Tel Aviv, Israel | 3 |
| Singapore | 3 |
| Remote | 3 |
| Abu Dhabi | 2 |
| New York City, United States | 2 |
| London | 2 |
| Paris, France | 2 |
| Austin, United States | 2 |
| Dubai | 2 |
| France, Paris | 2 |
| Canada, Remote - United States | 1 |
| East London, United Kingdom | 1 |
| Singapore, Singapore | 1 |
| Minnesota, United States | 1 |
| Remote - Israel | 1 |
| Munich, Germany | 1 |
| UK | 1 |
| Berlin | 1 |
| New Jersey, United States | 1 |
| Remote - Germany | 1 |
| Abu Dhabi, Dubai | 1 |
| Prague | 1 |
| New Jersey, US | 1 |
| Remote - United Kingdom | 1 |
| Oklahoma, United States | 1 |
| Finland | 1 |
| California, United States | 1 |
| Abu Dhabi, United Arab Emirates | 1 |
| Czechia | 1 |
| Alabama, US | 1 |
| Béthune, France | 1 |
| Netherlands | 1 |
| Austin, Texas | 1 |
| San Francisco Bay Area, United States | 1 |
| Kansas City, United States | 1 |
| Béthune, Pas-de-Calais, France | 1 |
| Philadelphia, United States | 1 |
| Dallas, United States | 1 |
| London, UK | 1 |
| Israel, Israel | 1 |
| Remote - EU | 1 |
| Paris | 1 |
| Role type | Active listings |
|---|---|
| Backend Engineer | 308 |
| Software Engineer | 82 |
| Account Executive | 54 |
| Site Reliability Engineer | 6 |
| Technical Product Manager | 5 |
| Technical Project Manager | 5 |
| Technical Program Manager | 5 |
| Product Manager | 4 |
| Sales Representative | 4 |
| System Engineer | 4 |
| Data Center Technician | 3 |
| ML Engineer | 3 |
| Data Center Operations Technician | 3 |
| IT Technician | 2 |
| Product Designer | 2 |
| Hypervisor Engineer | 2 |
| Delivery Manager | 2 |
| Backend engineers, Frontend engineers, Site reliability engineers | 2 |
| Open Positions at Nebius | 2 |
| Applied AI Researcher | 2 |
| Head of Channel Marketing | 1 |
| Forward Deployment Engineer | 1 |
| Site Selection & Colocation Manager | 1 |
| Network Engineer | 1 |
| DC IT Support Manager | 1 |
| Application Security Engineer | 1 |
| MEP Engineer | 1 |
| Datacenter IT Technician | 1 |
| Partner Solutions Architect | 1 |
| Customer Engineer | 1 |
| Internal Control Business Partner | 1 |
| Solutions Partner | 1 |
| Senior System Engineer | 1 |
| Senior Technical Program Manager | 1 |
| Solutions Architecture Leader | 1 |
| Electrical Engineer | 1 |
| Data Scientist | 1 |
| Compensation Analyst | 1 |
| Network Planning Project Manager | 1 |
| Data Center IT Manager | 1 |
| Technical Account Manager | 1 |
| Security Solutions Engineer | 1 |
| Cloud Solution Architect | 1 |
| AI and ISV Partner Business Development Manager | 1 |
| Structured Cabling Design Engineer | 1 |
| ML Infrastructure Engineer | 1 |
| Senior Applied AI Solutions Engineer | 1 |
| VP of Developer Relations & Community | 1 |
| Senior Support Engineer | 1 |
| Operations Specialist | 1 |
| Project Development Manager | 1 |
| Backend Developer | 1 |
| Data Center Operations Manager | 1 |
| Data Center IT Technician | 1 |
| Mechanical Design Engineer | 1 |
| Generalist | 1 |
| HPC Engineer | 1 |
| Solutions Architect | 1 |
| Data Center Electrical Lead | 1 |
| Pricing Director | 1 |
| Manager, ML Solutions Architecture | 1 |
| Senior Technical Project Manager | 1 |
| Accountant | 1 |
| Product Growth Analytics Lead | 1 |
| Group Product Manager | 1 |
| Infrastructure Security Engineer | 1 |
| Senior Research Scientist | 1 |
| GTM Recruiting Manager | 1 |
| Technical Due Diligence Manager | 1 |
| Technical Support Engineer | 1 |
| Data Engineer | 1 |
| Machine Learning Engineer | 1 |
| Mechanical Data Center Technician | 1 |
| Sales Engineer | 1 |
| Senior Software Developer | 1 |
| AI/ML Specialist Solutions Architect | 1 |
| Instructional Designer | 1 |
| Vendor Security & Standards Manager | 1 |
| ML Solutions Architect | 1 |
| Physical Security Systems Technician | 1 |
| Mechanical Engineer | 1 |
| Vulnerability Operations Center Lead | 1 |
| IT Risk and Control Manager | 1 |
| VP of Strategic Sales | 1 |
| Human Resources Specialist | 1 |
| Educational Content Author | 1 |
| Offensive Security Lead | 1 |
| Data Center Logistics Specialist | 1 |
| Financial Controller | 1 |
| Applied ML Engineer | 1 |
| Field Technical Lead | 1 |
| Data Center Facilities Manager | 1 |
| Principal | 1 |
| Systems HPC Engineer | 1 |
| Backend Engineers | 1 |
| Role level | Active listings |
|---|---|
| Mid-Level | 387 |
| Senior | 67 |
| Manager | 14 |
| Executive | 3 |
Related Opportunities
Discover more opportunities that match your interests and skills