Machine Learning Infrastructure Engineer
True Talent Solutions Partners
- Location
- Onsite (San Francisco, California)
- Compensation
- $200k - $400k/yr
- Employment
- Full-time
- Level
- Senior Level
About the role
Join an AI team building foundation models for physical systems, starting with weather. You will develop the distributed infrastructure that enables large-scale model training and inference, helping advance AI for scientific and real-world applications.
Skills
Perks
- Relocation support
Full job details
About the Role
Join an AI team building foundation models for physical systems, starting with weather. You will develop the distributed infrastructure that enables large-scale model training and inference, helping advance AI for scientific and real-world applications.
What You'll Do
Design, deploy, and maintain distributed machine learning training and inference clusters.
Build scalable end-to-end pipelines for petabyte-scale datasets and model training across the machine learning lifecycle.
Research and evaluate training approaches, including parallelization strategies and numerical precision trade-offs across model scales.
Profile and debug low-level GPU operations to improve performance.
Track relevant research and apply promising ideas to infrastructure and model development.
What We're Looking For
Two to ten years of experience building large-scale machine learning infrastructure for pretrained foundation models.
Experience with distributed training frameworks such as FSDP or DeepSpeed, and with optimizing large-scale training, data, or inference workloads.
Strong background in distributed compute, data infrastructure, inference systems, and model serving.
Familiarity with GPU performance optimization and debugging, including CUDA and JAX, and with Python, C++, Linux, and NVIDIA hardware.
Knowledge of cloud platforms such as GCP, AWS, or Azure, and containerization and orchestration tools such as Docker and Kubernetes.
Experience in physical AI or AI for science, such as robotics, autonomous systems, or biology, and motivation to work on scientific or physical-world applications.
Comfort working across the machine learning lifecycle in fast-paced, time-sensitive environments; experience analyzing training experiments and ablation studies is valuable.
Compensation & Benefits
Base salary: $200,000 to $400,000 annually. Visa sponsorship is not available.
Location
On-site, five days per week in San Francisco, California. Relocation support is provided.