Engineering Director, ML Inference Services

Engineering Director, ML Inference Services at CoreWeave

CoreWeave is seeking an Engineering Director for ML Inference Services to lead the architecture, delivery, and scale of our machine learning inference platform. This role will guide cross‑functional teams, drive system performance, reliability, and operational efficiency for production ML workloads, and collaborate with ML researchers, platform, and product teams to shape the serving roadmap.

Responsibilities

• Lead and mentor a team of ML engineers and software developers focused on ML inference services and deployment infrastructure.
• Own end-to-end architecture of inference pipelines, model serving, monitoring, reliability, and performance optimization.
• Collaborate with ML researchers, platform teams, and product management to set roadmap and deliverables.
• Define coding standards, code reviews, and software development processes to ensure scalable, maintainable code.
• Drive performance, cost efficiency, and operational excellence in production ML workloads.
• Establish metrics, SLAs, incident response, and security/compliance alignment.

Qualifications

• Extensive leadership experience in ML/AI software engineering with a track record of delivering robust ML inference platforms.
• Strong background in distributed systems, high-throughput services, and low-latency inference at scale.
• Proficiency in languages such as Python and C++, and experience with ML frameworks and accelerators (e.g., CUDA, TensorRT).
• Experience with cloud infrastructure, containerization (Docker/Kubernetes), and CI/CD in production environments.
• Excellent communication, strategic thinking, and collaboration skills; ability to mentor and grow teams.
• Bachelor’s or Master’s degree in CS, ML, or related field; advanced degrees preferred.

About CoreWeave

CoreWeave is a leading provider of GPU-accelerated cloud computing, delivering high-performance infrastructure and ML workloads to researchers and enterprises. This position is based in Sunnyvale, CA, and requires on-site presence.

Benefits

• Competitive salary and equity package
• Health, dental, and vision insurance
• Flexible work arrangements and strong work-life balance
• Opportunities to work on cutting-edge ML inference technologies