Role OverviewWe're building a dataset to evaluate AI coding agents by creating challenging tasks and evaluation criteria within realistic simulated environments. You'll design tasks, write tests, and iterate on tasks and tests based on QA feedback.
What You Will Do
Create realistic developer environments, design tasks, write tests, and iterate on tasks and tests to evaluate AI agent solutions.
Why It Might Be a Fit
You'll need 5+ years of software development experience, proficiency in Python, JavaScript/TypeScript, and experience writing tests.
Requirements
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
Benefits
- Up to $50/hr equivalent
- Flexible schedule
- 20 hours per task, estimated
]]>