Software Engineering Evaluation Specialist

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment. About the Role: You’ll design coding tasks that challenge frontier AI coding agents. Each task is a self-contained Docker environment with a broken piece of software; an AI agent attempts the fix; automated tests verify the outcome. Responsibilities: - Invent a realistic developer scenario — a real bug, a broken ETL, a missing feature — not a toy problem. - Build a reproducible Docker environment with pinned dependencies. - Write a pytest that verifies outcomes, not specific commands — deterministic, non-flaky, and does not leak the fix. - Write an instruction.md that reads like a Jira ticket a developer would receive. - Write a reference solve.sh proving the task is solvable. - Calibrate difficulty so current state-of-the-art agents solve the task 20–60% of the time. - Iterate based on feedback from expert QA reviewers. Later: review other authors’ tasks as a QA reviewer. Not in scope: Data labeling, prompt engineering, production code to ship, Leetcode puzzles. Requirements: 3+ years production software development; Python + pytest fluency; Docker authoring; Linux & Bash; AI coding agent experience; English — B2+ written. Time commitment: onboarding ~10 hours; steady state ~5 hours per task, 2–4 parallel tasks; Realistic weekly load: 8–20 hours. Compensation: up to $35/hour; task-based compensation; some projects include incentive payments. Apply: Submit CV via Mindrift platform, indicate English level, role, GitHub profile.