Role OverviewThe AI Evaluation Specialist will evaluate enterprise AI tools and capabilities to determine their effectiveness, safety, usability, and readiness for business adoption. They'll partner with product managers, data scientists, AI engineers, software engineers, security and privacy teams, and business stakeholders to define evaluation criteria and conduct structured assessments of generative AI products and platforms.
What You Will Do
The AI Evaluation Specialist will design repeatable evaluation frameworks, test real-world use cases, analyze model and product behavior, and translate findings into clear recommendations. They will be responsible for comparing AI solutions across quality, accuracy, reliability, safety, privacy and security.
Why It Might Be a Fit
The ideal candidate will have 5+ years of experience in model evaluation, model quality assurance with hands-on experience evaluating AI-enabled products or services, and experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
Requirements
- 5+ years of experience in model evaluation, model quality assurance with hands-on experience evaluating AI-enabled products or services
- Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria
- Experience with hands-on generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes
- Experience assessing AI systems across dimensions such as accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost
- Ability to demonstrate familiarity with common generative AI architectures and capabilities, including large language models, retrieval-augmented generation (RAG), AI agents, APIs, enterprise search, and tool or connector integrations
Benefits
- Paid Time Off
- PTO / Vacation
- Paid Holidays
- California residents receive an additional 24 hours of sick leave a year
- Health & Wellness
- Medical
- Dental
- Vision
- Prescription
- Employee Assistance Program
- Short- & Long-Term Disability
- Life and AD&D Insurance
- Spending Account
- Flexible Spending Account
- Health Savings Account
- Health Reimbursement Account
- Dependent Care Spending Account
- Commuter Benefits
- Retirement
- 401k / 401a
- Voluntary Benefits
- Hospital Indemnity
- Critical Illness
- Accident Insurance
- Pet Insurance
- Legal Insurance
- ID Theft Protection
]]>