Role OverviewJoin Vacatia and help build the future of AI-powered vacation ownership. As a Senior AI Agent & Evaluations Engineer, you'll design and improve AI agents that directly impact customer experiences, operational efficiency, and business outcomes across the organization.
What You Will Do
Design and improve AI agents, build evaluation frameworks, create guardrails, and continuously improve agent performance. Work on problems that matter, such as customer communications, mortgage outcomes, rental operations, and owner experiences.
Why It Might Be a Fit
If you're passionate about prompt engineering, agent reliability, and creating measurable AI systems that solve meaningful business problems, this role might be a fit.
Requirements
- Proven experience shipping and owning production AI agents or LLM-powered systems beyond proof-of-concept environments
- Deep expertise in prompt engineering, including system prompts, tool usage, context management, output constraints, and agent behavior design
- Hands-on experience building evaluation frameworks using golden datasets, scoring rubrics, LLM-as-judge methodologies, and regression testing
- Strong familiarity with modern AI development tools such as Claude Code, Codex, or similar coding agents
- Experience with agent observability and evaluation platforms such as LangSmith, Langfuse, Arize, Galileo, or comparable solutions
- Ability to distinguish prompt issues from data, tooling, model, or evaluation failures and systematically improve agent performance
- Strong written and verbal communication skills with the ability to work effectively across engineering and business teams
- Demonstrated ownership mindset with a passion for building reliable, measurable, and continuously improving AI systems
Benefits
- Build the future of applied AI
- Work on problems that matter
- Own the intelligence layer
- Measure what matters
- Partner across the business
- Join a small team with outsized impact
]]>