Machine Learning Infrastructure Engineer
True Talent Solutions Partners
- Location
- Onsite (San Francisco, California)
- Compensation
- $200k - $400k/yr
- Employment
- Full-time
- Level
- Senior Level
About the role
Join an AI research team building large-scale models for physical systems, with weather as an initial application. You will develop the infrastructure for distributed model training and inference, helping turn large volumes of observational data into useful predictions and insights.
Skills
Perks
- Relocation support
Full job details
About the Role
Join an AI research team building large-scale models for physical systems, with weather as an initial application. You will develop the infrastructure for distributed model training and inference, helping turn large volumes of observational data into useful predictions and insights.
What You'll Do
Design, deploy, and maintain distributed machine learning training and inference clusters.
Build scalable pipelines for petabyte-scale datasets and manage model training across the machine learning lifecycle.
Test training approaches, including parallelization methods and numerical precision trade-offs across model sizes.
Profile and debug low-level GPU operations to improve performance.
Evaluate research and apply relevant techniques to infrastructure and training workflows.
Analyze training experiments, including results from ablation studies, to inform model development.
What We're Looking For
Two to ten years of experience building large-scale machine learning infrastructure for pretrained or foundation models.
Hands-on experience with compute clusters, data infrastructure, inference systems, and distributed training.
Strong skills in optimizing large-scale training, data, or inference workloads, including GPU performance debugging.
Proficiency with distributed training frameworks such as FSDP or DeepSpeed, and experience with CUDA and JAX.
Experience with Python, C++, Linux, NVIDIA GPUs, and cloud platforms such as GCP, AWS, or Azure.
Familiarity with Docker, Kubernetes, distributed task management, model serving, monitoring, logging, and observability.
Background in physical AI or AI for science, such as robotics, self-driving, or biology, and interest in mission-driven work.
A generalist approach to the machine learning lifecycle and experience delivering in fast-paced, time-sensitive environments.
Compensation & Benefits
Base salary is $200,000 to $400,000 USD annually. Visa sponsorship is not available. Relocation support is provided.
Location
On-site in San Francisco, California, United States, five days per week.