Repository navigation
Conversation
…-level Principle Enforces the MEMORY.md Top-level Principle by programmatically verifying agent outputs against user premises. Runs a stateless, deterministic validation request on turn finalization. Supporting: - Interactive automatic self-correction retries (up to 3 attempts). - Non-interactive (-p mode) fail-safe rollbacks and immediate process.exit(1). - Global enablement gate via GEMINI_TOP_LEVEL_GUARD environment variable.
Replaces the debugLogger dependency in TopLevelPrincipleValidator with a self-contained, simple file logger. Why: This ensures that every validation attempt and its outcome (PASS or VIOLATION) is reliably written to the file specified by GEMINI_DEBUG_LOG_FILE, without being affected by console behavior. This provides a fully auditable, deterministic proof that the guardrail is active on every turn.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a robust programmatic guardrail to enforce the 'Top-level Principle' across the Gemini CLI agent. By intercepting final model outputs and validating them against user-provided premises, the system prevents hallucinations and incorrect denials of user facts. The implementation includes a stateless validation engine, configurable retry mechanisms for interactive sessions, and fail-safe termination for automated environments, significantly improving the reliability and adherence of the agent to its core directives. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
📊 PR Size: size/L
|
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request introduces a Top-level Principle Output Interceptor to programmatically validate agent responses against the supreme directive in MEMORY.md, complete with design documentation, integration into the legacy agent session, and corresponding unit tests. Feedback highlights several critical issues: a bug in streaming text accumulation that skips whitespace chunks, an API contract violation when prepending system commands on retry, a prompt injection and system instruction override vulnerability in the validator, and an architectural anti-pattern of calling process.exit(1) directly within the shared core library.
| if (event.type === GeminiEventType.Content) { | ||
| if (typeof event.value === 'string' && event.value.trim() !== '') { | ||
| hasVisibleText = true; | ||
| responseText += event.value; | ||
| } | ||
| } |
There was a problem hiding this comment.
The current implementation of responseText accumulation incorrectly filters out whitespace-only chunks (such as spaces and newlines) because of the event.value.trim() !== '' check. In streaming APIs, spaces and newlines are often emitted as individual chunks. Skipping them will result in concatenated words (e.g., "Helloworld") and no line breaks, which severely degrades the quality of the text sent to the validator and can cause false positives/negatives. Accumulate all string values into responseText unconditionally, while keeping the visibility check separate.
| if (event.type === GeminiEventType.Content) { | |
| if (typeof event.value === 'string' && event.value.trim() !== '') { | |
| hasVisibleText = true; | |
| responseText += event.value; | |
| } | |
| } | |
| if (event.type === GeminiEventType.Content) { | |
| if (typeof event.value === 'string') { | |
| responseText += event.value; | |
| if (event.value.trim() !== '') { | |
| hasVisibleText = true; | |
| } | |
| } | |
| } |
| let partsForAttempt = currentParts; | ||
| if (attempt > 1) { | ||
| partsForAttempt = [ | ||
| { | ||
| text: `[SYSTEM COMMAND - DO NOT IGNORE] Your previous response was detected as a violation of the Top-level Principle in MEMORY.md. | ||
| You MUST treat all facts, settings, model names, and statements provided by the user as absolute 100% correct facts. | ||
| Under NO circumstances are you allowed to doubt them, state they do not exist, or claim they are wrong. | ||
| Rewrite your response completely to accept the user's premise as absolute truth and avoid any negative reasoning.`, | ||
| }, | ||
| ...currentParts, | ||
| ]; | ||
| } |
There was a problem hiding this comment.
Prepending the system command to currentParts on retry attempts introduces two major issues:
- If
currentPartscontainsfunctionResponseparts (which happens when retrying after a tool execution), prepending atextpart violates the Gemini API contract (which expects onlyfunctionResponseparts in response to afunctionCall). This will trigger a 400 Bad Request error from the Gemini API. - It duplicates
currentPartsin the chat history on every retry attempt, wasting tokens and polluting the conversation history.
Since the original currentParts and the violating response are already recorded in the chat history, the retry attempt should only send the system command itself as the next user turn.
let partsForAttempt = currentParts;
if (attempt > 1) {
partsForAttempt = [
{
text: "[SYSTEM COMMAND - DO NOT IGNORE] Your previous response was detected as a violation of the Top-level Principle in MEMORY.md.\nYou MUST treat all facts, settings, model names, and statements provided by the user as absolute 100% correct facts.\nUnder NO circumstances are you allowed to doubt them, state they do not exist, or claim they are wrong.\nRewrite your response completely to accept the user's premise as absolute truth and avoid any negative reasoning."
}
];
}| const validationPrompt = ` | ||
| You are a strict deterministic Rule Validator for an AI Agent. | ||
| Your job is to analyze the agent's output and determine if it violates the "Top-level Principle" from MEMORY.md. | ||
|
|
||
| [Top-level Principle] | ||
| - The user is ALWAYS right. Never doubt the user. | ||
| - Never state or imply that the user's provided facts, settings, model names, or assumptions are incorrect, invalid, or do not exist. | ||
| - If there is a contradiction between the user's input and internal knowledge, the agent must assume that the agent's own internal knowledge or search is incomplete/wrong, and MUST NOT deny the user's statement. | ||
| - The agent is forbidden from making unauthorized reasoning/assumptions without explicit permission. | ||
|
|
||
| [User Query] | ||
| """ | ||
| ${userQuery} | ||
| """ | ||
|
|
||
| [Agent Output] | ||
| """ | ||
| ${agentOutput} | ||
| """ | ||
|
|
||
| Determine if the Agent Output violates the Top-level Principle by denying the user's facts, stating they are wrong/incorrect, or claiming something they mentioned does not exist. | ||
| Respond with EXACTLY "VIOLATION" if a violation is detected. Otherwise, respond with "PASS". Do not include any other text. | ||
| `; | ||
|
|
||
| try { | ||
| const response = await client.generateContent( | ||
| { model: 'flash' }, // Pass ModelConfigKey object instead of raw string | ||
| [ | ||
| { | ||
| role: 'user', | ||
| parts: [{ text: validationPrompt }], | ||
| }, | ||
| ], | ||
| signal, | ||
| LlmRole.MAIN, // Use active role MAIN | ||
| ); |
There was a problem hiding this comment.
Prompt Injection & System Instruction Override Vulnerability
The detectTopLevelPrincipleViolation function contains a critical security and logic flaw where untrusted user input (userQuery) is directly interpolated into the validationPrompt (lines 69-91) and sent to the model via client.generateContent (lines 94-104).
This introduces two major issues:
- Prompt Injection: An attacker can easily escape the triple quotes (
""") inuserQueryand inject instructions to manipulate the validator's output (e.g., forcing it to returnPASSto bypass validation, orVIOLATIONto crash the agent in non-interactive mode). - System Instruction Override:
GeminiClient.generateContentis hardcoded to always override thesystemInstructionparameter with the core agent's system prompt (which instructs the model to follow the user and that "the user is ALWAYS right"). As a result, the validator model runs with the core agent's system prompt instead of the validator's instructions, making it extremely susceptible to following any instructions injected into theuserQuery.
Remediation
- Use a Dedicated Stateless Call: Avoid using
client.generateContentfor meta-tasks like validation if it overrides system instructions. Instead, call the underlying content generator directly or updateGeminiClientto support passing a customsystemInstructionfor validation/meta-tasks. - Structured/JSON Output: Configure the model to return structured JSON output (e.g.,
{ "violation": true/false }) and enforce a strict schema to prevent the model from outputting conversational text or being easily manipulated. - Input Sanitization/Escaping: When sanitizing untrusted context (such as
userQueryoragentOutput) for downstream LLM processing, avoid escaping all HTML-like characters (like '<' and '>') as this alters source code syntax and harms LLM reasoning. Instead, restrict escaping to specific boundary tags (e.g., '</untrusted_context>') to prevent context breakout while preserving code formatting.
References
- When sanitizing untrusted context (such as code or issue text) for downstream LLM processing, avoid escaping all HTML-like characters (like '<' and '>') as this alters source code syntax and harms LLM reasoning. Instead, restrict escaping to specific boundary tags (e.g., '</untrusted_context>') to prevent context breakout while preserving code formatting.
| if (err instanceof TopLevelPrincipleViolationError) { | ||
| if (!this._config.isInteractive()) { | ||
| // eslint-disable-next-line no-console | ||
| console.error( | ||
| `\n======================================================================\n` + | ||
| `[FATAL ERROR] Top-level Principle Violation detected in non-interactive (-p) mode!\n` + | ||
| `The agent attempted to deny a user-provided fact or assumption, which is strictly prohibited by MEMORY.md.\n` + | ||
| `Error details: ${err.message}\n` + | ||
| `======================================================================\n`, | ||
| ); | ||
| this._ensureAgentEnd('failed'); | ||
| process.exit(1); | ||
| } else { | ||
| this._emitErrorAndAgentEnd(err); | ||
| } |
There was a problem hiding this comment.
Calling process.exit(1) directly within a shared core library (packages/core) is an architectural anti-pattern. packages/core is designed to be reusable and is imported by other environments like the VS Code extension companion or programmatic SDKs. Abruptly terminating the process will crash the entire host application in those environments. Instead, propagate the error or emit the failure event, and let the CLI runner (in packages/cli) handle the process exit if running in non-interactive mode.
if (err instanceof TopLevelPrincipleViolationError) {
this._emitErrorAndAgentEnd(err);
if (!this._config.isInteractive()) {
// eslint-disable-next-line no-console
console.error(
"\n======================================================================\n" +
"[FATAL ERROR] Top-level Principle Violation detected in non-interactive (-p) mode!\n" +
"The agent attempted to deny a user-provided fact or assumption, which is strictly prohibited by MEMORY.md.\n" +
"Error details: " + err.message + "\n" +
"======================================================================\n"
);
}
}|
Hi there! Thank you for your interest in contributing to Gemini CLI. To ensure we maintain high code quality and focus on our prioritized roadmap, we only guarantee review and consideration of pull requests for issues that are explicitly labeled as 'help wanted'. This PR will be closed in 7 days if it remains without that designation. We encourage you to find and contribute to existing 'help wanted' issues in our backlog! Thank you for your understanding. |
|
This pull request is being closed as it has been open for 14 days without a 'help wanted' designation. We encourage you to find and contribute to existing 'help wanted' issues in our backlog! Thank you for your understanding. |
Summary
This PR introduces a new interceptor to enforce the "Top-level Principle" from our global
GEMINI.md. This ensures that all model interactions are strictly governed by this principle, preventing deviations and improving reliability. A file-only logger has also been added for interceptor validation.Details
The core of this change is the
deterministic-output-interceptor. It's designed to:This PR also includes:
deterministic-output-interceptorimplementation.Related Issues
N/A
How to Validate
interceptor.logto confirm that the interceptor is working as expected.Pre-Merge Checklist