Related skills
bigquery python pandas pytest vertex ai📋 Description
- Own QA for an LLM-based, multi-agent claims adjudication pipeline
- Build a Three-Tier Evaluation Framework for agent reasoning and tools
- Develop semantic validation and grounding data testing libraries
- Audit tool calls with Vertex AI traces and correct arguments
- Validate orchestration across data, knowledge, reasoning, and tooling
🎯 Requirements
- Python SDET Expertise: Python and pytest; custom mocking for Vertex AI/ADK
- AI/LLM Observability: Vertex AI Experiments, Auto-SxS, Cloud Logging
- Data Literacy: SQL (BigQuery) and Pandas; diff datasets for adjudication
- Prompt Engineering for QA: refine prompts based on failed tests
- Architectural Testing: test multi-layer systems with RAG, Vertex AI Search, LangGraph
🎁 Benefits
- Hybrid in-office schedule at Lehi and Plano, requiring two days in the office
- Stock options: 115,000; health insurance, 401k, and PTO
- Professional development and internal mobility opportunities
- Flexible work arrangements and supportive culture
Meet JobCopilot: Your Personal AI Job Hunter
Automatically Apply to Engineering Jobs. Just set your
preferences and Job Copilot will do the rest — finding, filtering, and applying while you focus on what matters.
Help us maintain the quality of jobs posted on Empllo!
Is this position not a remote job?
Let us know!