Senior Infrastructure Engineer

The Data Platform Engineering team (Part of Upstart's Decisioning & Data) provides developer tools, frameworks, and scalable data infrastructure as shared services across Upstart. The team's primary objective is to provide Data Analysts, ML scientists, and Software Engineers access to high-quality data and developer tools to create business metrics and training data for respective product verticals. 

You will join the Data Production Experience (PX) team, which owns Upstart’s Lakehouse (Databricks) infrastructure. This is the platform that Machine Learning, Analytics, and Software Engineering teams use to build the training data and metrics behind our lending, fraud, and pricing models—enabling growth and reducing risk across our business verticals.

As the Senior Infrastructure Engineer joining this team, you will focus on:

1. Lakehouse Infrastructure and Security: Own the Databricks platform across multiple environments, including a newly launched regulated environment (Upstart Bank). Responsibilities include expanding infrastructure as code for new Lakehouse features, enforcing a no-click-ops policy, establishing disaster recovery, implementing least-privilege and attribute-based access controls (e.g., based on data classification), ensuring environment isolation, and resolving other known security risks.
2. Compliance: Close control gaps and automate evidence gathering for OCC, SOX and SOC audits across access management, change management, and data quality.
3. Cost and usage efficiency: Reduce Databricks and AWS spend through accurate cost attribution, waste detection, and platform-level optimizations (e.g., compute rightsizing, improving pruning, migrating to Graviton and spot instances, Delta Lake Archival).

How you’ll make an impact 

• Evaluate and adopt new technologies, including open-source lakehouse alternatives, to expand platform capabilities.
• Improve developer efficiency and infrastructure reliability by working with cross-functional stakeholders like Data Engineering, Machine Learning, Analytics, Compliance, and Security. 
• Improve Lakehouse security, usage efficiency, and observability by collaborating closely with the Data Engineering and Security teams.
• Evangelize good coding and engineering best practices while uplifting fellow engineers and mentoring junior team members.
• Help break down complex projects and requirements into quarterly commitments and sprints.
• Own Lakehouse infrastructure components end to end, from technical requirements and design through rollout and production support.

Minimum Qualifications 

• Bachelor’s degree in Computer Science or a related field, or equivalent practical experience.
• 5+ years of experience in infrastructure engineering, DevOps, or data platform engineering.
• Hands-on experience operating a lakehouse or big data platform (e.g., Databricks, Snowflake, Apache Spark, Delta Lake, Apache Iceberg).
• Hands-on experience with AWS (e.g., IAM, S3, EC2, EKS), Python, infrastructure as code (Terraform or CDK), CI/CD, and Kubernetes.

Preferred Qualifications

• Experience implementing security and compliance controls in audited environments (e.g., SOX, SOC 2, GDPR, GLBA, OCC), including access management, encryption, and change management.
• Experience reducing cloud or data platform costs through usage analysis, cost attribution, and infrastructure optimization (e.g., Graviton or spot instances, storage tiering).
• Ability to reason from first principles and work effectively through ambiguity on complex, cross-functional problems.
• Proficient in DevOps engineering, leveraging Continuous Integration/Continuous Delivery (CI/CD) tools such as Kubernetes, Jenkins, CDK, APM, Terraform, and more. Experienced in automation, alerting, monitoring, security, and declarative infrastructure.
• Solid understanding of cloud networking fundamentals.
• Skilled at explaining complex concepts in simple, accessible terms and thriving in fast-paced, ambiguous environments.
• Experience using AI coding assistants or agents to accelerate infrastructure development and operations.
• Familiarity with distributed systems, data architecture design, and big data technologies (e.g., Spark, Kafka, Lakehouse, Databricks)

Travel requirements As a digital first company, the majority of your work can be accomplished remotely. The majority of our employees can live and work anywhere in the U.S but are encouraged to to still spend high quality time in-person collaborating via regular onsites. The in-person sessions’ cadence varies depending on the team and role; most teams meet once or twice per quarter for 2-4 consecutive days at a time.

#LI-REMOTE

#LI-MidSenior