What

Gemini CLI tests are currently split into unit and integration tests, with unit tests covering individual modules, and integration tests validating the E2E CLI functionality.

There is also a de facto 3rd form of tests present, such as save_memory.test.ts, which are effectively behavioral evals.

What are behavioral evals

Behavioral evals are tests that accept a prompt and run the agent and then check for some desirable outcome. Typically the validations are simple, like ensuring that a tool or function call happened. These tests are valuable for use as:

• A feedback loop for validating changes to prompts, tool descriptions, and other quality impacting factors.
• A stake in the ground that prevents regressions to behaviors implemented through 'steering' the model as the prompt, model, and toolset change.
• A tool for use in refining model steering behaviors, such as the automatic invocation of subagents in Support subagents in extensions #14846, whether or not to commit user changes, and which memories the agent should choose to store with the save_memory tool.

How are behavioral evals different than industry benchmarks

• Behavioral evals tend to encompass simpler scenarios and validations, acting more akin to unit tests of the behaviors implied by the prompt rather than 'challenges' for the agent to solve.

• Behavioral evals perform simpler validations and so should pass more consistently than typical benchmarks.

• Behavioral evals should generally only be checked in a passing state, though being non-deterministic agent tests, they may not pass 100% of the time. Notably, a behavioral eval might pass for one specific model + prompt combination, but not another.

• Benchmarks measure an agent's performance against a standardized set of challenges. Behavioral evals, however, validate specific behaviors that we wish to preserve and/or refine from build to build and across models, prompts, and configurations.

How are behavioral evals different than integration tests

• Integration tests are expected to succeed consistently. A single integration test failure often blocks the completion of a PR or build.
• Behavioral evals should be run multiple times each run to ensure that the agent exhibits the same behavior with reasonable consistency.
• Behavioral evals will probably not run during the CI build but rather on a schedule or perhaps in response to changes in certain key files.

What are some items that may benefit from behavioral evals

• [Agents] Post V1.0 Work #3132 -- behavioral evals can be used to ensure that system prompt steering as to when to use subagents works in several key scenarios.
• Gemini CLI does not respect linter preferences and/or best-practices #15168 -- behavioral evals can be used to refine the prompts to encourage the model to follow linter rules first and fallback to existing convention only if that fails.
• Agent should not commit its own changes #15122 -- behavioral evals can be used to validate steering to avoid committing changes unless the user asks.
• Output length, density and formatting can be validated by such a test, acting as gate against undesired changes with different models.
• [Behavior Mgmt] Gemini CLI should prefer using automated linting and formatting tools vs. manually making changes #15032 -- behavioral evals can be used to validate that Gemini CLI uses tools like eslint --fix when available rather than attempting large fixes itself.
• Epic: System Prompting & Context Trimming vNext #15328 -- acts as guard-rails to ensure that the prompt changes have the desired effect.
• Epic: Plan Mode #15326 -- ensure that plan mode utilizes the tool to exit under the desired conditions.
• Epic: Agent Skills - Standardized Expertise & Progressive Disclosure #15327 -- validate that skills are consistently used when available.
• Epic: Clarifying Questions #15329 -- refine the prompting around when to ask clarifying questions.
• Epic: Improved Tool Usage and Definitions #15331 -- validate improvements to tool usage and prevent regressions.
• Introspection agent demo #15232 -- validate key behaviors of the agent, e.g.: like that it uses get_internal_docs tool.
• Refine system prompt instructions for save_memory tool #15678 -- save_memory tool is almost never called unless you very explicitly instruct the CLI to "call save_memory".
• The write_todos tool has been disabled. #15246 -- ensure TODO is called consistently with each new model.

Details

• Should use vitest
• Should repeat at least a few times.
• Should not be a required part of the build. Perhaps runs asynchronously, as needed (based on directory changed), or manually and user invoked.
• Should emit logs in a machine readable format that is used to monitor the trend of how many tests pass all iterations over time.