Component Level Evaluations
This EPIC is a follow up for #15300 which introduced the concept of "behavioral evals" tests to the repository.
Since that issue's creation we have generate 76 behavioral eval tests, we run them for 6 supported Gemini models, and have created some basic infrastructure for naming, running, and tracking the runs over time.
This EPIC tracks:
- Generalizing the behavioral evals infrastructure to support a broader set of tests.
- Introducing component level eval tests for hill climbing specific components.
- Introducing "hero tests" for assessing performance of specific high value scenarios.
- Fleshing out more of the test and analysis tooling to accelerate 1st and 3rd party contributions to Gemini CLI.
Behavioral Evals vs. Component Level Evals
As part of this work the existing behavioral evals tests suite will be expanded and refactored and tagged into the following suites. Each suite will be tagged with a specific feature area for targeted validation on applicable PRs.
-
Behavioral -- a small number of medium sized evaluations of user facing bugs, misbehaviors, and feature-level prompted behaviors. Typically judged deterministically. e.g.: "was tool X called with the expected parameters".
-
Component evals -- A large number of relatively small evaluations that test a specific components performance at a specific tasks. e.g.: "compression prompt recall".
-
Hero tests -- A small number of relatively large evaluations that test the end to end GCLI's performance at specific high value task. e.g.: "does Gemini CLI successfully complete a long horizon task, like building a compiler from scratch over 100+ turns".
Component Level Evaluations
This EPIC is a follow up for #15300 which introduced the concept of "behavioral evals" tests to the repository.
Since that issue's creation we have generate 76 behavioral eval tests, we run them for 6 supported Gemini models, and have created some basic infrastructure for naming, running, and tracking the runs over time.
This EPIC tracks:
Behavioral Evals vs. Component Level Evals
As part of this work the existing behavioral evals tests suite will be expanded and refactored and tagged into the following suites. Each suite will be tagged with a specific feature area for targeted validation on applicable PRs.
Behavioral -- a small number of medium sized evaluations of user facing bugs, misbehaviors, and feature-level prompted behaviors. Typically judged deterministically. e.g.: "was tool X called with the expected parameters".
Component evals -- A large number of relatively small evaluations that test a specific components performance at a specific tasks. e.g.: "compression prompt recall".
Hero tests -- A small number of relatively large evaluations that test the end to end GCLI's performance at specific high value task. e.g.: "does Gemini CLI successfully complete a long horizon task, like building a compiler from scratch over 100+ turns".