This job is no longer available

The job listing you are looking has expired.
Please browse our latest remote jobs.

See open jobs →
← Back to all jobs
Added
1 day ago
Location
Type
Full time
Salary
Not Specified

Use AI to Automatically Apply!

Let your AI Job Copilot auto-fill application questions
Auto-apply to relevant jobs from 300,000 companies

Auto-apply with JobCopilot Apply manually instead
Save job

Related skills

nlp aws python mlops openai

Monitor, evaluate, and optimize AI/LLM workflows in production environments. Ensure reliable, efficient, and high-quality AI system performance by building out an LLM Ops platform that is self-serve for the engineering and data science departments.

Key Responsibilities:-

  • Collaborate with data scientists and software engineers to integrate an LLM Ops platform (Opik by CometML) for existing AI workflows
  • Identify valuable performance metrics (accuracy, quality, etc) for AI workflows and create on-going sampling evaluation processes using the LLM Ops platform that alert when metrics drop below thresholds
  • Cross-team collaboration to create datasets and benchmarks for new AI workflows
  • Run experiments on datasets and optimize performance via model changes and prompt adjustments
  • Debug and troubleshoot AI workflow issues
  • Optimize inference costs and latency while maintaining accuracy and quality Develop automations for LLM Ops platform integration to empower data scientists and software engineers to self-serve integration with the AI workflows they build

    • Requirements:-

  • Strong Python programming skills
  • Experience with generative AI models and tools (OpenAI, Anthropic, Bedrock, etc)
  • Knowledge of fundamental statistical concepts and tools in data science such as: heuristic and non-heuristic measurements in NLP (BLEU, WER, sentiment analysis, LLM-as-judge, etc), standard deviation, sampling rate, and a high level understanding of how modern AI models work (knowledge cutoffs, context windows, temperature, etc)
  • Familiarity with AWS
  • Understanding of prompt engineering concepts
  • People skills: you will be expected to frequently collaborate with other teams to help to perfect their AI workflows
  • Experience Level 4-7 years of experience in LLM/AI Ops, MLOps, Data Science, or MLE

    • Use AI to Automatically Apply!

    Let your AI Job Copilot auto-fill application questions
    Auto-apply to relevant jobs from 300,000 companies

    Auto-apply with JobCopilot Apply manually instead
    Share job

    Meet JobCopilot: Your Personal AI Job Hunter

    Automatically Apply to Hybrid Engineering Jobs. Just set your preferences and Job Copilot will do the rest—finding, filtering, and applying while you focus on what matters.
    Activate JobCopilot

    Related Engineering Jobs

    See more Engineering jobs →