Role OverviewKR3 Information Systems is seeking an experienced Senior AI Evaluation Specialist to support an AI/ML initiative. The role will involve evaluating leading AI platforms and products, including OpenAI solutions and life sciences models, and communicating findings effectively to both technical and non-technical stakeholders.
What You Will Do
Plan and execute hands-on evaluations of enterprise AI products, develop structured evaluation frameworks, test real-world business scenarios, and translate technical findings into actionable recommendations for leadership.
Why It Might Be a Fit
We are looking for someone who can independently lead hands-on evaluations, think critically, identify AI limitations and risks, and communicate findings effectively to both technical and non-technical stakeholders.
Requirements
- Bachelor's degree in Computer Science, Data Science, Information Technology, Engineering, or a related discipline.
- 5+ years of experience in AI product evaluation, model evaluation, model quality assurance, or evaluating AI-enabled products and services.
- Hands-on experience with Generative AI tools and Large Language Models (LLMs).
- Experience designing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
- Strong understanding of AI evaluation dimensions, including accuracy, relevance, consistency, hallucinations, safety, bias, privacy, security, usability, performance, and cost.
- Practical experience with prompt engineering, systematic testing, output analysis, and identifying AI failure modes.
- Familiarity with LLM architectures, RAG, AI agents, APIs, enterprise search, and tool/connector integrations.
- Ability to independently conduct technical evaluations, analyze results, and communicate findings and recommendations clearly.
]]>