Technical Program Manager, Frontier Evals

📋 Description

• Manage frontier evaluation projects from research questions to benchmarks.
• Translate model capability questions into eval designs, metrics, and plans.
• Design and manage human data campaigns with QC workflows.
• Hands-on work: prompts, eval pipelines, data analysis, scripting, dashboards.
• Build roadmaps and operating rhythms for fast-moving research.
• Coordinate across research, engineering, data, product, safety, legal, vendors.

🎯 Requirements

• Experience in technical program management, research operations, or data ops.
• Proficient in Python and SQL to analyze data, automate workflows, and inspect outputs.
• Strong understanding of large language models: prompting, evaluation, and failure modes.
• Able to work as IC and program manager: write scripts, align stakeholders, adapt campaigns.
• Turn vague goals into clear plans with milestones and owners.
• Excellent communication with technical and non-technical stakeholders.

🎁 Benefits

• Hybrid work arrangement with 3 days in the office per week.
• Relocation assistance for new hires.
• Reasonable accommodations for applicants with disabilities.
• Opportunity to work on frontier AI evaluation research.