Related skills
ai ml llms agent studio evaluation framework📋 Description
- Define and own the internal AI agent evaluation framework (starting with Agent Studio).
- Build the customer-facing evaluation experience for builders testing and improving agents.
- Make hard calls on evaluation complexity vs. abstraction for balance.
- Partner with the Build Experience PM to integrate evaluation into the builder journey.
- Work with ML engineers and platform teams to ground the framework in reality.
- Establish metrics for internal agent quality and customer evaluation adoption.
🎯 Requirements
- 7+ years in Product Management
- Hands-on experience writing evaluations for AI/ML systems (agents/LLMs)
- Track record of shipping technical products to internal and external users
- Experience driving adoption of frameworks across engineering teams
- Strong written and verbal communication skills
- Bachelor's degree or equivalent experience
🎁 Benefits
- Flexible, trust-oriented culture
- Dynamic, supportive work environment
- Access to a multitude of benefits
- Opportunity to work on AI/ML product features
- Collaborative, cross-functional teams
Meet JobCopilot: Your Personal AI Job Hunter
Automatically Apply to Product Jobs. Just set your
preferences and Job Copilot will do the rest — finding, filtering, and applying while you focus on what matters.
Help us maintain the quality of jobs posted on Empllo!
Is this position not a remote job?
Let us know!