Skip to content

feat(caretaker): add triage Cloud Run job workflow - #28468

Merged
gundermanc merged 6 commits into
google-gemini:mainfrom
chadd28:feat/caretaker-triage-cloud-workflow
Aug 7, 2026
Merged

feat(caretaker): add triage Cloud Run job workflow#28468
gundermanc merged 6 commits into
google-gemini:mainfrom
chadd28:feat/caretaker-triage-cloud-workflow

Conversation

@chadd28

@chadd28 chadd28 commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds the Google Cloud Workflow definition (triage-worker-workflow.yaml) for orchestrating the caretaker issue triage pipeline.

This workflow is invoked by the Ingestion Layer upon receiving Pub/Sub events, executes the triage-worker Cloud Run Job with issue payload details, and applies retry logic. Upon job failure, it updates the issue's Firestore document to NEEDS_HUMAN with error metadata, clears the processing lock, and forwards the failed message to a Pub/Sub Dead Letter Queue (DLQ) topic (incoming-issues-dlq).

Details

  • Initialization (init): Extracts GCP project metadata, workflow execution ID, base64 payload, and constructs the target DLQ topic path.
  • Job Execution (run_processing_job): Uses googleapis.run.v1.namespaces.jobs.run to trigger the triage-worker Cloud Run Job in us-west1, passing environment overrides (ISSUE_DETAILS and WORKFLOW_EXECUTION_ID).
  • Retry Mechanism (retry): Implements exponential backoff (initial_delay: 5, max_retries: 1) using a custom predicate for transient job failure recovery.
  • Error Handling (except): Intercepts unhandled terminal errors, updates the target Firestore issue status to NEEDS_HUMAN (clearing processing lock fields lock.holder and lock.expires_at), publishes to Pub/Sub DLQ (incoming-issues-dlq), and raises a terminal exception.
  • Output (success_log): Returns structured execution details and a direct Cloud Console log URL upon successful completion.

IAM & Permissions

  • The Cloud Workflow service account (triage-orchestrator-sa) now requires the Datastore User (roles/datastore.user) role to grant googleapis.firestore.v1.projects.databases.documents.patch permissions.

How to Validate

This is deployed via the Cloud Console UI or using this command:

gcloud workflows deploy triage-worker-workflow \
--source=tools/caretaker-agent/workflows/triage-worker-workflow.yaml \
--location=us-west1

@chadd28
chadd28 requested a review from a team as a code owner July 20, 2026 20:45
@gemini-code-assist

gemini-code-assist Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds a new Google Cloud Workflow to orchestrate the caretaker issue triage process. It automates the execution of a Cloud Run Job, incorporates robust retry logic for transient failures, and ensures system reliability by routing unrecoverable errors to a dedicated Dead Letter Queue for later inspection.

Highlights

  • Workflow Orchestration: Introduced a new Google Cloud Workflow definition to manage the caretaker issue triage pipeline.
  • Job Execution: Implemented automated triggering of the 'triage-worker' Cloud Run Job with environment overrides for issue details.
  • Resilience and Error Handling: Added an exponential backoff retry mechanism and a fallback to a Pub/Sub Dead Letter Queue (DLQ) for failed message processing.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added the size/m A medium sized PR label Jul 20, 2026
@github-actions

github-actions Bot commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown

📊 PR Size: size/M

  • Lines changed: 102
  • Additions: +102
  • Deletions: -0
  • Files changed: 1

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jul 20, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Google Cloud Workflow (triage-worker-workflow.yaml) to trigger a Cloud Run Job and publish failures to a Pub/Sub DLQ. The review feedback correctly identifies a critical issue: calling the Cloud Run Job run API is asynchronous, meaning the workflow will immediately report success without waiting for the job to complete or catching failures. The feedback provides a detailed solution to poll the job execution status until completion.

Comment thread tools/caretaker-agent/workflows/triage-worker-workflow.yaml Outdated
@gemini-cli gemini-cli Bot added the status/need-issue Pull requests that need to have an associated issue. label Jul 20, 2026
@github-actions github-actions Bot mentioned this pull request Jul 20, 2026
Open
@chadd28

chadd28 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Wait to review until DLQ vs Dashboard is discussed.

@chadd28
chadd28 marked this pull request as draft July 20, 2026 21:39
@gemini-cli gemini-cli Bot removed the status/need-issue Pull requests that need to have an associated issue. label Jul 20, 2026
@chadd28
chadd28 force-pushed the feat/caretaker-triage-cloud-workflow branch from e8606bb to cc2e65d Compare July 21, 2026 22:25
@chadd28
chadd28 force-pushed the feat/caretaker-triage-cloud-workflow branch from cc2e65d to ae8a71a Compare July 21, 2026 22:27
@chadd28
chadd28 marked this pull request as ready for review July 21, 2026 22:28
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jul 21, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a Google Cloud Workflow to execute the triage-worker Cloud Run Job, updating Firestore and publishing to a Pub/Sub DLQ on failure, alongside Python database updates to clear lock fields when an issue needs human intervention. Key feedback points out that the Cloud Run Job API call is asynchronous, requiring a polling loop to correctly detect job completion and failures, and suggests updating the updated_at timestamp during the workflow's Firestore patch step to maintain consistency with the database store.

Comment thread tools/caretaker-agent/workflows/triage-worker-workflow.yaml
Comment thread tools/caretaker-agent/workflows/triage-worker-workflow.yaml
@gemini-cli gemini-cli Bot added the status/need-issue Pull requests that need to have an associated issue. label Jul 21, 2026
@chadd28

chadd28 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

ready for review! @adamfweidman @gundermanc

@chadd28 chadd28 mentioned this pull request Aug 7, 2026
Closed
@gundermanc
gundermanc added this pull request to the merge queue Aug 7, 2026
Merged via the queue into google-gemini:main with commit 8cb94fe Aug 7, 2026
34 checks passed
@github-actions github-actions Bot mentioned this pull request Aug 12, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gundermanc gundermanc gundermanc approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

size/m A medium sized PR status/need-issue Pull requests that need to have an associated issue.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@chadd28 @gundermanc