AI Evaluation Scenario Architect

the Homebase; Remote; GBP 40,000 - GBP 60,000; Part time

Job summary

A leading tech opportunities provider in the UK seeks software engineers for project-based roles focused on testing AI systems. This part-time opportunity requires a strong background in Python and familiarity with Git. Contributors will create test cases, define evaluation standards, and analyze agent performance logs. Payment is up to $50/hour, allowing flexibility in work hours based on project complexity.

Qualifications

• 3+ years of software development experience with a strong Python focus.
• Experience in working with Git and code repositories.
• Comfortable with structured formats like JSON/YAML for scenario description.

Responsibilities

• Create structured test cases that simulate complex human workflows.
• Define gold-standard behavior and scoring logic to evaluate agent actions.
• Analyze agent logs and decision paths.

Skills

Software development experience

Strong Python focus

Experience with Git

Structured formats like JSON/YAML

Understanding core LLM limitations

Familiarity with Docker

English proficiency (B2)

Job description

A leading tech opportunities provider in the UK seeks software engineers for project-based roles focused on testing AI systems. This part-time opportunity requires a strong background in Python and familiarity with Git. Contributors will create test cases, define evaluation standards, and analyze agent performance logs. Payment is up to $50/hour, allowing flexibility in work hours based on project complexity.