Role OverviewWe're building a dataset to evaluate AI coding agents by creating challenging tasks and evaluation criteria within realistic simulated environments. You'll design tasks, write tests, and iterate on tasks based on QA feedback.
What You Will Do
Create challenging tasks and evaluation criteria, write tests, and iterate on tasks based on QA feedback to evaluate AI coding agents.
Why It Might Be a Fit
You'll need 5+ years in software development, experience with Python, JavaScript, Docker, and writing tests, as well as English proficiency (B2+).
Requirements
- 5+ years in software development
- Experience with Python, JavaScript, Docker, and writing tests
- English proficiency (B2+)
Benefits
- Up to $50/hr equivalent
- Flexible schedule
- Estimated 20 hours per task
]]>