Evaluation Scenario Writer - AI Agent Testing Specialist

the Homebase; Remote; GBP 40,000 - GBP 60,000; Part time

Job summary

A leading tech opportunities provider in the UK seeks software engineers for project-based roles focused on testing AI systems. This part-time opportunity requires a strong background in Python and familiarity with Git. Contributors will create test cases, define evaluation standards, and analyze agent performance logs. Payment is up to $50/hour, allowing flexibility in work hours based on project complexity.

Qualifications

• 3+ years of software development experience with a strong Python focus.
• Experience in working with Git and code repositories.
• Comfortable with structured formats like JSON/YAML for scenario description.

Responsibilities

• Create structured test cases that simulate complex human workflows.
• Define gold-standard behavior and scoring logic to evaluate agent actions.
• Analyze agent logs and decision paths.

Skills

Software development experience

Strong Python focus

Experience with Git

Structured formats like JSON/YAML

Understanding core LLM limitations

Familiarity with Docker

English proficiency (B2)

Job description

Please submit your CV in English and indicate your level of English proficiency.

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves

While each project involves unique tasks, contributors may:

• Create structured test cases that simulate complex human workflows
• Define gold-standard behavior and scoring logic to evaluate agent actions
• Analyze agent logs, failure modes, and decision paths
• Work with code repositories and test frameworks to validate your scenarios
• Iterate on prompts, instructions, and test cases to improve clarity and difficulty
• Ensure that scenarios are production-ready, easy to run, and reusable

What we look for

This opportunity is a good fit for software engineers, open to part-time, non-permanent projects. Ideally, contributors will have:

• 3+ of software development experience with strong Python focus
• Experience with Git and code repositories
• Comfortable with structured formats like JSON/YAML for scenario description
• Understanding core LLM limitations (hallucinations, bias, context limits) and how these affect evaluation design
• Familiarity with Docker
• English proficiency - B2

How it works

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Project time expectations

Tasks for this project are estimated to take 6-10 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Payment

• Paid contributions, with rates up to $50/hour*
• Fixed project rate or individual rates, depending on the project
• Some projects include incentive payments

Note: Rates vary based on expertise, skills assessment, location, project needs, and other factors. Higher rates may be offered to highly specialized experts. Lower rates may apply during onboarding or non-core project phases. Payment details are shared per project.