I’m requesting help with a stuck OAuth branding verification.

Project ID: (PII Removed by Staff)
App name: (PII Removed by Staff)
User type: External

Situation:
The Verification Center flagged 3 issues from a previous review:

1. Privacy policy page does not have sufficient content
2. Home page is behind a login page
3. Home page does not explain the purpose of the app

All three issues were already resolved before I first submitted for re-verification. I have re-verified again just now, logged out in an incognito browser, that:

• Our home page loads directly with no login wall, and clearly states the app’s purpose (“Dairy farm record-keeping, made simple… HerdTrack helps farmers track animals, milk records, health, breeding and expenses”)
• Our privacy policy page has detailed, comprehensive content covering data collection, usage, and compliance with India’s DPDP Act 2023

I selected “I have fixed the issues” and clicked Proceed on 14 Sep 2026. I have checked my registered contact email including Spam, and have not received any email from the Trust and Safety team since then. The Verification Center still shows the same stale issues today (15 Sep 2026).

Could someone please help trigger a status refresh, confirm whether my re-verification request was received, or escalate this to the Trust and Safety team?

Thank you.

1 post - 1 participant

Read full topic

First attempt failed with error: “Billing setup can’t be completed - Suspicious activity detected [OR_BACR2_32]”. I resolved this by finding and closing a duplicate payments profile (I had one India profile and one Honduras profile under the same name, which seemed to trigger the fraud check).

After that, subsequent attempts now show error “OR_BAOOC_15” asking to retry after 24 hours. UPI verification (small ₹2 authorization charges) succeeds each time, but the billing account creation never completes successfully.

Has anyone faced this issue or know how to resolve it? Would appreciate any guidance.

1 post - 2 participants

Read full topic

I’m trying to replicate a report that previously used the BigQuery connector (with native date range pushdown) but now needs to pull data from AlloyDB via the native PostgreSQL connector.

**Issue:** When using Custom Query mode and referencing the date range parameters `$DS_START_DATE` / `$DS_END_DATE` (with PostgreSQL casting, e.g. `‘$DS_START_DATE’::DATE`), Looker Studio throws a System Error before the query even reaches the database (Error IDs: 04578c6e, 04f8fd74).

**What I tested:** - Custom Query with `$DS_START_DATE` / `$DS_END_DATE` + casting → System Error (crashes immediately) - Direct table connection (no custom query) → report appears to pull the entire table regardless of the Date Range Control selection - Workaround: static `WHERE part_date >= CURRENT_DATE - INTERVAL ‘X days’` in Custom Query → works, but is NOT dynamic (ignores the report’s date range control)

**Question:** 1. Does the native PostgreSQL connector actually support `$DS_START_DATE` / `$DS_END_DATE` in Custom Query mode? I couldn’t find this confirmed anywhere in the documentation. 2. If supported, what is the correct syntax/casting to avoid the System Error? 3. If not supported, is this a known limitation, and is there a workaround to get dynamic date-range pushdown to a PostgreSQL/AlloyDB source (similar to how the BigQuery connector handles it)?

This is currently blocking a migration from BigQuery to AlloyDB as the report’s underlying data source. Any guidance from the community or Product Experts would be greatly appreciated. Thanks!

1 post - 1 participant

Read full topic

1. Why VPC Service Controls? The Threat Model

Standard cloud security controls operate on identity boundaries (IAM) and network infrastructure boundaries (VPC Firewalls). Because Google Cloud managed services (BigQuery, Cloud Storage, Vertex AI, Cloud SQL) are multi-tenant services hosted on public API endpoints (*.googleapis.com), traditional perimeter defenses fail to prevent data movement across project boundaries.

+-----------------------------------------------------------------------------------+
|                              THE DATA EXFILTRATION RISK                           |
|                                                                                   |
|  [Compromised Identity or Service Account]                                        |
|                       |                                                           |
|                       |  1. IAM Permission Check PASSES (User has Read Access)     |
|                       v                                                           |
|        [Production BigQuery / GCS Bucket]                                         |
|                       |                                                           |
|                       |  2. Attacker runs: gsutil cp gs://prod/data gs://attacker |
|                       v                                                           |
|        [External / Personal GCS Bucket] <--- IAM passes, Firewall bypassed!       |
+-----------------------------------------------------------------------------------+

The Exfiltration Vectors VPC-SC Prevents

• Authorized Insider Copying Data Out: A database administrator with valid IAM roles/bigquery.admin credentials executes a BigQuery extract job from a corporate laptop, copying sensitive PII into a personal Google Cloud account.
• Compromised Service Account Credentials: A stolen service account key with GCS read access is used outside the corporate network to dump database backups into an arbitrary external bucket.
• Stolen Service Agent Tokens: A compromised VM inside a VPC uses its compute engine default service account to write internal training datasets out to an untrusted external endpoint.

VPC-SC mitigates these threats by shifting the security model from Identity-Only to Context-Aware Security at the API Layer. An API call is permitted only if the identity and the network context (IP origin, VPC network, device health, source project) satisfy the perimeter rules.

2. Architecture & Core Primitives

VPC Service Controls operates at the Google API Front End (GFE) layer. When a client issues a request to a protected API, the GFE evaluates the request against the perimeter configuration before routing it to the backend service.

Press enter or click to view image in full size

Core Architectural Components

• Service Perimeter: A logical security boundary enclosing one or more GCP projects. Managed service APIs selected inside the perimeter (e.g., storage.googleapis.com, bigquery.googleapis.com) refuse requests originating from outside the perimeter or crossing into unauthorized projects.
• Access Context Manager (ACM) & Access Levels: ACM defines conditions based on public IP ranges, corporate subnets, user identities, or Chrome Enterprise device posture. Access Levels grant entry through the perimeter boundary.
• Ingress & Egress Rules: Granular API-level access rules defined within the perimeter. They allow identity, project, and method-specific API communication into (Ingress) or out of (Egress) the perimeter.
• Perimeter Bridges: Bi-directional connections between two discrete perimeters that allow projects in separate perimeters to share specific Google APIs without merging the perimeters entirely.
• Restricted VIP (restricted.googleapis.com): A dedicated IP range (199.36.90.4/30) used to route Google API traffic entirely over internal network paths, ensuring protected traffic never traverses public internet routes.

3. Key Enterprise Use Cases

Real-World Use Case 1: Financial & Healthcare Data Isolation (PII/PHI)

Scenario: An enterprise healthcare provider processes electronic health records (EHR) in BigQuery and Cloud Storage.

• Objective: Ensure healthcare data cannot be read by or copied to external GCP accounts, while allowing data scientists on corporate laptops to run queries.

VPC-SC Implementation:

• Place production data projects into a dedicated perimeter.
• Restrict bigquery.googleapis.com and storage.googleapis.com.
• Define an ACM Access Level permitting access only from corporate VPN public IP ranges and corporate-managed devices.
• Block outbound storage.objects.create operations directed at non-approved project buckets via strict Egress policies.

Real-World Use Case 2: Multi-Tenant AI/ML Pipeline Security (Vertex AI & Gemini)

Scenario: An engineering organization trains custom LLMs using proprietary source code and customer datasets inside Vertex AI.

• Objective: Prevent training datasets or fine-tuned model artifacts from being exported to external storage buckets or unauthorized third-party services.

VPC-SC Implementation:

• Enclose Vertex AI, Artifact Registry, Cloud Storage, and Compute Engine APIs in a single AI perimeter.
• Configure Egress rules to block connections to external GCS buckets while allowing internal service agents (service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com) to read from internal staging storage.

4. Overcoming Complex Implementation Challenges

Challenge 1: Third-Party SaaS & Multi-Cloud Integrations

External SaaS tools (Fivetran, Databricks, Snowflake) need to read/write data in BigQuery or GCS inside a protected perimeter.

• Solution (Identity-Bound Ingress Rules): Avoid dynamic public IP whitelisting. Authenticate the SaaS vendor via service accounts or Workload Identity Federation, and restrict access to exact API operations:

# Ingress Rule restricting external SaaS access to specific GCS write operations
ingress_policies {
  ingress_from {
    identity_type = "ANY_IDENTITY"
    identities    = ["serviceAccount:saas-data-ingest@vendor-project.iam.gserviceaccount.com"]
  }
  ingress_to {
    resources = ["projects/987654321012"]
    operations {
      service_name = "storage.googleapis.com"
      method_selectors {
        method = "google.storage.objects.create"
      }
    }
  }
}

Challenge 2: Hidden Managed Service Dependencies

GCP managed services execute API calls across projects using implicit internal service agents (e.g., Dataflow worker nodes communicating with GCS, Compute, and Logging).

• Solution: Map service agent accounts during discovery. Grant explicit Egress rules or include dependent APIs (Compute Engine, Cloud Storage, Artifact Registry) within the service perimeter boundary.

Challenge 3: CI/CD External Runners (GitHub Actions / GitLab CI)

Deployments from cloud-hosted runners outside the network perimeter are blocked when pushing build artifacts or executing Terraform apply.

• Solution A: Use self-hosted runners deployed inside a dedicated management VPC inside the perimeter.
• Solution B: Use Workload Identity Federation paired with an Ingress rule granting the CI/CD identity access solely to deployment endpoints (e.g., artifactregistry.googleapis.com or container.googleapis.com).

6. Phased Migration Methodology

Transitioning to VPC Service Controls without interrupting business operations requires a strict four-phase rollout:

1.Phase 1: Discovery & Log Sink Setup:Prerequisite Phase.

Catalog all projects, active APIs, cross-project service accounts, and external third-party integrations. Create an organization-level Cloud Audit Log sink routing policy logs directly to BigQuery.

2.Phase 2: Network Infrastructure & Dry-Run Deployment:Weeks 1–2.

Configure private DNS zones mapping *.googleapis.com to restricted.googleapis.com (199.36.90.4/30). Deploy all perimeters and Access Levels in DRY_RUN mode using Infrastructure as Code (Terraform).

3.Phase 3: Log Remediation & Policy Iteration:Weeks 2–4.

Query BigQuery to isolate dry-run violations. Differentiate legitimate service interactions from unauthorized calls, and iteratively refine Ingress/Egress policies and Access Context Manager levels.

4.Phase 4: Enforcement & Operational Readiness:Maintenance Window.

Confirm zero unintended violations over 14 consecutive days. Transition perimeters to ENFORCED mode, trigger break-glass procedure drills, and activate SIEM alerting for unexpected blocks.

7. Production Automation & Log Analysis

Terraform Pattern: Perimeter with Scoped Policy & Ingress Rules

To prevent configuration drift, manage all VPC-SC resources declaratively:

#Terraform
resource "google_access_context_manager_service_perimeter" "production_perimeter" {
  parent         = "accessPolicies/${var.access_policy_id}"
  name           = "accessPolicies/${var.access_policy_id}/servicePerimeters/prod_perimeter"
  title          = "Production Data Service Perimeter"
  perimeter_type = "PERIMETER_TYPE_REGULAR"

  status {
    restricted_services = [
      "bigquery.googleapis.com",
      "storage.googleapis.com",
      "secretmanager.googleapis.com"
    ]

    resources = [
      "projects/123456789012" # Production Data Project
    ]

    ingress_policies {
      ingress_from {
        identity_type = "ANY_IDENTITY"
        sources {
          access_level = "accessPolicies/${var.access_policy_id}/accessLevels/corp_vpn_access"
        }
      }
      ingress_to {
        resources = ["*"]
        operations {
          service_name = "bigquery.googleapis.com"
          method_selectors {
            method = "google.bigquery.v2.JobService.InsertJob"
          }
        }
      }
    }
  }
}

Dry-Run Log Diagnostics Query

Execute this query in BigQuery against your centralized Cloud Audit Log dataset to inspect blocked operations during the Dry-Run phase:

SELECT
  timestamp,
  protoPayload.authenticationInfo.principalEmail AS caller_identity,
  protoPayload.serviceName AS target_service,
  protoPayload.methodName AS api_method,
  protoPayload.resourceName AS target_resource,
  protoPayload.metadata.violationReason AS violation_reason,
  protoPayload.metadata.ingressViolations,
  protoPayload.metadata.egressViolations,
  protoPayload.metadata.vpcServiceControlsUniqueId AS violation_id
FROM
  `your_gcp_project.audit_logs.cloudaudit_googleapis_com_policy`
WHERE
  protoPayload.metadata.vpcServiceControlsUniqueId IS NOT NULL
  AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
ORDER BY
  timestamp DESC;

Vpc Service Control

Vpcsc

Gcp Vpc Service Controls

Gcp Vpcsc

1 post - 1 participant

Read full topic

To kick off the very first swag drop of 2026, we’re outfitting you with the ultimate barrier against the elements: The Arcade Weather-Shield Jacket. Throw it on for your morning commute or wear it to work—either way, it’s a flex you actually had to earn.

Placed in the Champion and Legend Tier, here’s why it belongs in your rotation:

• Storm-ready weave: Repels light rain and shuts down gusting winds before they hit your layer.

• Official branding: Clean Google Cloud logo on the left chest paired with the super cloud logo on the sleeve.

• Total draft block: High stand-up collar, custom Velcro cuffs, and a ribbed elastic hem lock warmth in and the weather out.

• Secure cargo: Deep zippered hand-warmer pockets keep your phone, keys, and hands safe from cold snaps.

• Weightless mobility: Technical performance without the heavy winter bulk—made to keep you moving.

Don’t leave this sitting on the table because it is built for whatever the forecast throws at your deploy!

If you’re new to the tier system, here’s how the Prize Tiers work: think of it like a “Snowball”. You never have to pick and choose between tiers when you rank up; your rewards just keep rolling forward and getting bigger:

• Arcade Trooper Tier: This is your foundational milestone. Once you hit this points threshold, you unlock the core swag pack filled with essential Google Skills Arcade gear.

• Arcade Ranger Tier: The natural next step for participants pushing past the basics. It’s designed for players who stay consistent, bank more Arcade Points, and put in that extra effort across challenges—unlocking everything from the Trooper Tier plus an additional bonus reward on top.

• Arcade Champion Tier: A major leap into the upper ranks. Hitting the Arcade Champion Tier takes serious dedication, unlocking a high-tier collection of premium Google Skills Arcade gear that really stands out.

• Arcade Legend Tier: The absolute summit of the Google Skills Arcade. Reserved exclusively for the users who max out their points to finish at the very top, scoring everything from the Champion Tier plus an exclusive Legend-only reward.

In simple terms, you never miss out when you rank up. Reaching Arcade Ranger Tier gives you all the Arcade Trooper Tier swags plus extra gear, and reaching the Arcade Legend Tier gives you the complete Arcade Champion Tier pack plus exclusive rewards waiting at the very top.

Please note that Arcade Trooper and Arcade Ranger prizes won’t snowball into the Arcade Champion and Arcade Legend Tiers.

That means every single badge and point you bank right now isn’t just getting you closer to the jacket—it’s building up the whole pile of gear waiting for you on the other side.

The clock’s ticking, the next challenge is waiting, and that Arcade Legend spot won’t claim itself. Go get it!

See You In The Cloud

11 posts - 9 participants

Read full topic

Our OAuth verification is blocked and we cannot find a way to move it forward.

Project ID: (PII Removed by Staff)
Homepage: (URL Removed by Staff)
Requested sensitive scope: …/auth/contacts
Last review: 6 August 2026

Current status in the Verification Center:

• App functionality: approved
• Branding guidelines: approved
• Homepage requirements: FAILED — “Your homepage is protected by a sign-in page”
• Privacy policy requirements: in review
• Minimum scopes: in review

The homepage issue has been fixed. (URL Removed by Staff) now serves our public
product page directly: it does not redirect to any other URL, it does not
require sign-in, it describes what the app does and why we request access to
the user’s contacts, and it links to our privacy policy. The sign-in page now
lives on a separate URL.

The Verification Center tells us to reply to the email thread with the Trust &
Safety team once the issues are resolved. We never received that email. We have
searched all project owner and editor mailboxes, including spam, for messages
from api-oauth-dev-verification-reply and google-cloud-compliance, and there is
nothing. The console offers no option to resubmit or to request the thread
again.

Could you please either re-trigger the review now that the homepage complies,
or resend the Trust & Safety thread so we can reply to it?

Happy to provide any additional information.

Thanks

2 posts - 2 participants

Read full topic

For example, in my Google Workspace organisation, I have two domains registered in the “Domains” section. I have the primary domain: “company1.co.uk” and an additional domain: “company2.co.uk” (this is all still within the same Google Workspace organisation).

Let’s say I (me@company1.co.uk) create a Gemini Notebook, with a source that is on a shared Google Drive (a Google Doc). I have a requirement to share this Gemini Notebook with another user in our Google Workspace environment (you@company2.co.uk) (you@company2.co.uk has view access to the Google Drive area where the Google Doc is by default because of where it is located).

If I share this with you@company2.co.uk, the sources are not read. It states “Source:0” in the bottom right.

I understand that when Google states that Gemini Notebooks cannot be shared out of the organisation, should that really read “cannot be shared with another domain name”?

The new Gemini sharing options do not help either. If I add “company2.co.uk” as a Trusted Domain in the “Allowlist”, I am advised that you “Can’t add your own domain or domain alias as an allowlisted domain.”

I do not want to enable public sharing.

How can I best enable sharing with additional internal domains for our users?

I am aware of the workaround with Google Groups, but I have not tested this yet. It is not solution I want to adopt.

1 post - 1 participant

Read full topic

• Pinch-to-zoom does nothing. Zoom is only possible through the + and − buttons.
• Dragging with a finger does not pan the map.

Every other map on the device, including the Google Maps app itself and AppSheet’s own Map views, responds to these gestures. The picker is the only place where they are unavailable, so it feels broken rather than restricted.

This matters for field data entry. I use the picker to record the location of agricultural parcels, and the pin has to land on a specific point in open terrain with few landmarks. Reaching that point with only the zoom buttons takes many taps, and without panning I cannot move the view to where the parcel actually is.

The underlying Google Maps SDKs expose these gestures as settings that are enabled by default, so this looks like they are being explicitly turned off in the picker rather than being unavailable.

Could pinch-to-zoom and drag-to-pan be enabled in the LatLong picker?

1 post - 1 participant

Read full topic

1. Automation

The export has no Automation section. The strings “Event name”, “Task name” and “Run this process” do not appear even once in the document.

My bots are visible only indirectly, as the system tables they generate, listed among my data tables with names like:

• Process for Delete Preferences Row by deleting UserAccount
• Delete The preferenced row Output

Their triggering event, conditions, steps and Bypass security setting are nowhere in the document.

2. The Require toggle on columns

“Required_If” appears 314 times but is null for every column where I enabled the simple Require toggle instead of writing an expression. Only expression-based requirements are written out.

Reading the export back, a mandatory field is indistinguishable from an optional one.

3. App-level settings

Theme, brand, localization, offline and sync settings do not appear anywhere in the export.

The document contains only four sections: Tables, Views, Format Rules and Actions.

Security filters and Delete-triggered bots, which I reported as missing earlier, do now appear correctly.

Is the export intended to cover these areas, or is it limited to those four sections by design?

2 posts - 2 participants

Read full topic

3 posts - 2 participants

Read full topic

I’m running into a quirky issue where custom fonts like Inter and Libre Barcode 39 Text in my Google Docs templates for creating PDFs aren’t rendering reliably. Instead of working consistently, the output randomly defaults to Arial font.

While this is just a minor visual glitch for my standard text using the Inter font, it completely breaks my scannable barcodes that use the Libre Barcode 39 Text font, because they just render as regular text!

My current workaround: If I manually re-trigger the automation a couple of times, it eventually catches the fonts and renders the PDF correctly.

Has anyone else run into this, or does anyone know a trick to make sure the fonts cache and render correctly every single time?

2 posts - 2 participants

Read full topic

Architecting for immediate intervention requires real-time telemetry that treats BigQuery not just as a passive data warehouse, but as a high-velocity engine for agentic triggers.

Most operational systems that rely on scheduled batch queries suffer from massive operational delays in scenarios like credit card fraud detection or supply chain bottlenecks. Human investigators piece together log entries and run ad-hoc SQL. Conversely, pushing raw data streams straight into LLMs is cost-prohibitive due to token burn. We need a hybrid model where high-speed, stateful SQL filters out the noise so AI agents only process targeted, enriched anomaly context.

Press enter or click to view image in full size

Mastering Agent Observability with BigQuery

The 3-Pillar Hybrid Architecture: Detect, Route, Resolve

To solve this, Google Cloud provides a three-pillar event-driven architecture-

1. Detection (BigQuery Continuous Queries): Persistent SQL runs continuously over streaming ingestion tables (via the Storage Write API) to evaluate rules and windowed heuristics. Rather than polling, BigQuery uses stateful operations like tumbling windows ( TUMBLE ) and stream-to-stream JOINs to compute rolling metrics and detect complex conditions, such as a customer making transactions in two distant countries within minutes (‘Impossible Travel’). When an anomaly condition is met, EXPORT DATA immediately streams the structured alert payload to a Pub/Sub topic without leaving the data warehouse
2. Routing (Pub/Sub & Single Message Transforms): Pub/Sub intercepts exported events, using lightweight JavaScript UDFs to reshape payloads into the exact schema expected by Vertex AI Agent Engine, and pushes it directly to the agent’s webhook endpoint.
3. Resolution (ADK & Agent Engine): Autonomous agents hosted on Vertex AI Agent Engine receive the event, execute reasoning loops using custom tools, and either resolve the issue or escalate to human-in-the-loop workflows.

Press enter or click to view image in full size

Real-World Scenario: Cymbal Bank Fraud Detection

Architecting for high-stakes environments such as fraud detection requires an event-driven loop where detection triggers autonomous action. In this pattern, BigQuery Continuous Queries monitor a stream of incoming data (e.g., retail transactions) for anomalies like “Impossible Travel.”

When an anomaly is detected, the query uses the EXPORT DATA statement to push the event to Pub/Sub in real-time. This message triggers an ADK Agent via Vertex AI Agent Engine, which can then use its tools to evaluate historical context and resolve the event or escalate it to a human.

Triggering an Agent via Continuous Stream

The following SQL demonstrates the Reverse ETL pattern used to trigger an agent from a continuous stream:

EXPORT DATA OPTIONS (
  format = 'CLOUD_PUBSUB',
  uri = 'https://pubsub.googleapis.com/projects/[PROJECT_ID]/topics/fraud_investigator_trigger'
) AS (
  SELECT
    customer_id,
    transaction_id,
    'Anomaly Detected: Impossible Travel' AS alert_reason
  FROM
    APPENDS(TABLE `project.dataset.transactions`)
  WHERE
    is_anomaly = TRUE
);

The Challenge: Black-Box of Production Agents

When you deploy such autonomous AI agents, traditional logging falls short because of the black box execution, unexpected token burn, silent hallucinations, and unmapped multi-agent decisions, etc.

Agents can make nondeterministic choices: they invoke nested tools, hand off control to sub-agents, pause for human approvals, and retry failed prompts. Without structured tracing, debugging a failed customer interaction feels like looking into a black box. Additionally, without token-level budgeting and latency tracking per tool, API costs and latency spikes can easily spiral out of control. We need an analytics engine that treats agent execution traces as queryable, structured data.

Press enter or click to view image in full size

The BigQuery Agent Analytics plugin for the ADK bridges the observability gap. It provides high-throughput, low-latency telemetry essential for debugging complex multi-agent fleets. By leveraging a Reverse ETL pattern, it allows BigQuery to push insights directly into the agentic workflow, transforming raw logs into actionable intelligence. It logs your AI agents’ operational telemetry, lifecycle events, tool executions, and LLM interactions directly into BigQuery tables.

Before we jump into how BigQuery Agent Analytics gives data practitioners total visbility into agent behavior, lets briefly look at the set up.

The Set Up

>> To set up the BQ Agent Analytics plugin, you have to follow these config options and understand the schema and production setup.

>> The attributes.adk envelope — In the multi-agent workflows (agents that transfer control, checkpoint their state, and compact long histories) and long-running tools that pause and resume across turns, a small metadata envelope, attributes.adkis used to build observability by tying the rows back to the ADK event that produced them.

>> Event Lifecycle Coverage: The plugin automatically provisions a partitioned, clustered agent_events table that records 24 canonical event types. There are default event types and payload generated to trace LLM interactions. These events track the execution of tools by the agent and changes to agent’s state typically triggered by tools.

Unboxing the Black-Box of Agent Analytics

BigQuery Agent Analytics Unboxes Agent Behavior into 5 Observable Pillars:

1. Muti-Agent Tracing & Execution Graphs: Every single action carries a persistent session_id and OpenTelemetry (trace_id, span_id). The plugin stamps an attributes.adk envelope on log rows to reconstruct multi-agent execution graphs (DAG), handle agent transfers, and track long-running pauses.
2. Multimodal Analysis: Automatically offloads large files (such as images and audio) to Google Cloud Storage and exposes them to BigQuery ML via Object Tables.
3. Tool Provenance & Human-in-the-Loop (HITL) Tracing: Tracks the exact origin of tools with explicit provenance tags (LOCAL, MCP, SUB_AGENT, A2A, TRANSFER_AGENT) and records Human-in-the-Loop (HITL) confirmation prompts, credential requests, and user inputs.
4. Fleet Observability to Manage Costs and Performance- Monitoring an enterprise agent fleet requires granular tracking of token consumption and responsiveness. The plugin captures detailed metrics, including usage_prompt_tokens and usage_completion_tokens, allowing for precise cost-attribution strategies. Performance is measured via total_ms and ttft_ms (Time-to-First-Token)
5. Business Decision Intelligence: The Context Graph feature (enabled via the BigQuery Agent Analytics SDK and the bqaa context-graph CLI tool) bridges the gap between raw, unstructured agent event logs and high-level business decision intelligence. The logged events can be materialized into a property graph, allowing operators to query decision trees using Graph Query Language (GQL)

Let’s look at each of these in further detail.

1. Multi-Agent Tracing & Execution Graphs

BigQuery Agent Analytics reconstructs agent sessions using distributed tracing and execution graphs. For the failed sessions, it runs Root Cause Analysis (RCA) using BigQuery AI functions.

1. Granular Telemetry Streaming & Immutable Event Capture: As an agent executes, the plugin captures every operational event (such as AGENT_STARTING, LLM_REQUEST, TOOL_STARTING, and TOOL_ERROR) and streams them asynchronously into BigQuery tables via the high-throughput Storage Write API. This guarantees a complete, chronological record of every step the agent took.
2. Distributed Tracing via trace_id, span_id, and the attributes.adk envelope: Every logged row is stamped with OpenTelemetry-compatible identifiers (trace_id, span_id) and an attributes.adk metadata envelope. parent_span_idis a 16-char HEX span id to reconstruct the hierarchical Directed Acyclic Graph (DAG). For multi-agent workflows, the plugin links agent transfers, state checkpoints (AGENT_STATE_CHECKPOINT), and long-running tool suspensions (TOOL_PAUSED) back to the exact conversation turn that triggered them.
3. Automated Diagnostics via BigQuery AI Functions (AI.GENERATE): The plugin identifies the failed session and reconstructs the full conversation context. Then asks Gemini to diagnose the issue using AI.GENERATE. Instead of manual hunting, Supervisor agents automatically run AI.GENERATE over the aggregated history of a failed session to self-diagnose why an agent hit an API error or hallucinated.

DECLARE failed_session_id STRING;
-- Find a recent failed session
SET failed_session_id = (
    SELECT session_id
    FROM `your-gcp-project-id.your-dataset-id.agent_events`
    WHERE error_message IS NOT NULL
    ORDER BY timestamp DESC
    LIMIT 1
);

-- Reconstruct the full conversation context
WITH SessionContext AS (
    SELECT
        session_id,
        STRING_AGG(CONCAT(event_type, ': ', COALESCE(TO_JSON_STRING(content), '')), '\n' ORDER BY timestamp) as full_history
    FROM `your-gcp-project-id.your-dataset-id.agent_events`
    WHERE session_id = failed_session_id
    GROUP BY session_id
)
-- Ask Gemini to diagnose the issue
SELECT
    session_id,
    AI.GENERATE(
        ('Analyze this conversation log and explain the root cause of the failure. Log: ', full_history),
        endpoint => 'gemini-flash-latest'
    ).result AS root_cause_explanation
FROM SessionContext;

2. Multimodal Analysis

Multimodal GCS Offloading in the BigQuery Agent Analytics plugin solves a core infrastructure challenge: how to capture massive text prompts or rich binary media (such as customer-submitted identity documents, transaction receipts, or voice verification audio) without bloating your analytical database or hitting row-size limits.

1. Generating Secure Signed URLs — Large text prompts (>500KB) or binary media (images, audio) are automatically offloaded to Google Cloud Storage. BigQuery stores an ObjectRef reference in content_parts, allowing operators to inspect images via BigQuery Object Tables or generate signed URLs with OBJ.GET_ACCESS_URL.If an engineer or fraud analyst needs to view a specific image or listen to an audio clip associated with a flagged session, BigQuery can dynamically generate time-bound, secure access URLs using the object reference.
2. Linking Object Tables for Direct SQL Inspection- By connecting your GCS bucket to BigQuery as an Object Table, you can query multimodal assets like traditional relational data — inspecting file sizes, update times, and mime types alongside your agent execution logs.
3. Multimodal Auditing with BigQuery Remote Models (Gemini) — Because the assets are securely referenced via GCS URIs, you can pass offloaded images or documents straight into BigQuery ML remote models (such as Gemini) to perform automated visual audits, OCR on receipts, or anomaly classification at scale

SELECT
  ml_generate_text_result.candidates[0].content.parts[0].text AS ai_image_analysis
FROM
  ML.GENERATE_TEXT(
    MODEL `your-project.your_dataset.gemini_multimodal_model`,
    (
      SELECT
        CONCAT('Analyze this transaction document for signs of tampering: ', 
               ML.SET_DOCK_IMAGE(JSON_VALUE(content_part, '$.object_ref.uri'))) AS prompt
      FROM `your-project.your_dataset.agent_events`,
           UNNEST(JSON_EXTRACT_ARRAY(payload, '$.content_parts')) AS content_part
      WHERE trace_id = 'target_suspicious_trace_id'
    )
  );

3. Tool Provenance & HITL Tracing

BigQuery Agent Analytics plugin provides deep visibility into who executed a tool, where it came from, and when a human had to step in.

When an agent calls a tool, the plugin logs its execution payload and stamps its exact origin classification (origin). This prevents “black-box” confusion when multiple agent tiers or external servers interact. To query tool provenance in BigQuery, query v_tool_completed to audit which tool origins are consuming resources or failing most frequently.

SELECT
  JSON_VALUE(payload, '$.tool_name') AS tool_name,
  JSON_VALUE(payload, '$.origin') AS tool_origin,
  COUNT(1) AS total_invocations,
  AVG(CAST(JSON_VALUE(payload, '$.latency_ms') AS FLOAT64)) AS avg_latency_ms
FROM
  `cymbal-fraud.analytics.v_tool_completed`
GROUP BY
  tool_name,
  tool_origin
ORDER BY
  total_invocations DESC;

3.1 HITL Tracing

In high-stakes financial pipelines like a $2,500 cross-border fraud check, automation must occasionally yield to a human expert. When an agent encounters an ambiguous risk threshold, it pauses execution and triggers a Human-in-the-Loop event . The plugin records dedicated event types for these interactions:— HITL_CREDENTIAL_REQUEST, HITL_CONFIRMATION_REQUEST, HITL_INPUT_REQUEST, *_COMPLETED. Under ADK 2.0 multi-agent workflows, when an HITL request is emitted, the plugin logs a TOOL_PAUSED event and stamps an attributes.adk envelope on the row.

4. Monitoring Fleet Health in Real-Time

Fleet observability shifts the focus from debugging a single isolated agent run to monitoring an entire production ecosystem of enterprise AI agents. In high-volume environments like Cymbal Bank’s fraud prevention pipeline — which processes thousands of concurrent transaction alerts — platform operators need microscopic visibility into two critical pillars: cost attribution (token burn) and responsiveness (latency SLAs).

The BigQuery Agent Analytics plugin streams these vital telemetry markers directly into BigQuery tables, giving engineering and finance teams the granular data needed to optimize performance and control operational spend.

4.1 Granular Token Consumption & Cost-Attribution

LLM API costs scale directly with token volume. Rather than receiving a generic, aggregated monthly bill from your AI provider, the plugin captures detailed usage attributes on every model call:

• usage_prompt_tokens: Measures the exact size of the incoming context window (system instructions, user prompts, retrieved customer profile data, and event payloads).
• usage_completion_tokens: Measures the length of the model’s generated response (reasoning steps, risk scoring rationale, and final decisions).

By storing these metrics natively in BigQuery alongside metadata like trace_id and agent names, users can run precise cost-attribution models such as breakdown token burn by specific departments, identifying which agents or instruction sets are bloated & allowing engineers to optimize system prompts, or set up automated BigQuery alerting thresholds when token consumption spikes.

4.2 Responsiveness & Latency Tracking (total_ms & TTFT)

In real-time fraud detection, seconds matter. If an agent takes too long to analyze a transaction, downstream systems time out or user experience degrades. The plugin tracks performance through two core latency metrics:

• total_ms (Total Execution Latency): Measures the complete end-to-end duration of an LLM request or a tool execution from start to finish.
• ttft_ms (Time-to-First-Token): Measures how quickly the model begins streaming its response back. This is crucial for real-time applications where perceived responsiveness depends on getting that first token instantly.

Users can query the v_llm_response view to track rolling averages, spot performance degradation across regions, and ensure SLAs are met.

SELECT
  JSON_VALUE(payload, '$.model') AS model_name,
  DATE(timestamp) AS execution_date,
  COUNT(1) AS total_requests,
  ROUND(AVG(CAST(JSON_VALUE(payload, '$.latency_ms') AS FLOAT64)), 2) AS avg_total_latency_ms,
  ROUND(AVG(CAST(JSON_VALUE(payload, '$.ttft_ms') AS FLOAT64)), 2) AS avg_ttft_ms,
  MAX(CAST(JSON_VALUE(payload, '$.latency_ms') AS FLOAT64)) AS max_total_latency_ms
FROM
  `cymbal-fraud.analytics.v_llm_response`
GROUP BY
  model_name,
  execution_date
ORDER BY
  execution_date DESC;

4.3 Combining Token Burn and Latency in Fleet Dashboards

Because all fleet metrics reside natively in BigQuery, teams can easily connect BI tools (like Looker or Looker Studio) to build comprehensive executive dashboards showing:

1. Cost vs. Latency Trade-offs: Comparing cheaper, faster models (e.g., flash tiers) against heavier reasoning models to find the optimal balance for fraud classification.
2. Spike Detection: Correlating sudden surges in usage_prompt_tokens with unexpected data bloat or recursive tool loops.

5. Business Decision Intelligence

Instead of treating agent traces as isolated lists of API requests, responses, and tool errors, the Context Graph transforms them into a queryable BigQuery property graph that maps out how decisions are made.

Using a scheduled refresh (bqaa context-graph) or event-driven pipeline, the SDK reads raw agent telemetry and distills session logs into structured decision components: DecisionRequest, DecisionOption, DecisionOutcome

The extracted components are mapped into a native BigQuery property graph schema using declarative table DDL and a CREATE PROPERTY GRAPH statement. Crucially, no separate external graph database or complex ETL pipeline is required—everything lives and runs natively within BigQuery.

Once materialized, you can use standard Graph Query Language (GQL) inside BigQuery via GRAPH_TABLE to traverse the decision network. For example, auditors or engineers can trace the exact chain of reasoning and confidence scores for a transaction review.

SELECT * FROM GRAPH_TABLE (
  cymbal_fraud_graph
  MATCH (req:DecisionRequest) -[eo:evaluatesOption]-> (opt:DecisionOption),
        (req) -[ri:resultedIn]-> (out:DecisionOutcome)
  COLUMNS (
    req.request_id AS transaction_id,
    req.request_text AS alert_summary,
    opt.option_label AS risk_strategy_considered,
    opt.confidence AS confidence_score,
    out.status AS final_decision,
    out.rationale AS audit_rationale
  )
);

Press enter or click to view image in full size

Decision Graph using BigQuery Agent Analytics Plugin

The Future of Autonomous Operations

The shift toward autonomous operations demands that we view telemetry not as an afterthought, but as a core agentic skill. The combination of the BigQuery Agent Analytics SDK and Looker dashboards turns raw operational traces into competitive business intelligence. By materializing execution graphs and utilizing AI-powered RCA, organizations can finally peer inside the “black box” of agentic AI, ensuring that every autonomous action is observable, accountable, and optimized for performance.

Hope this was a helpful read.

1 post - 1 participant

Read full topic

However, when attempting to download that view, the file is empty. Adjusting the filter text so that it EXACTLY matches the full text in the filtered column, results in all those EXACT matching rows (but reduced from the initial filter) being downloaded into the file.

This indicates that there is mis-match when filtering is applied for viewing versus when it is applied for downloading from the same view.

Please update the download filtering so it applies a CONTAINS() type operation to be in sync with the filtering applied for views.

Thanks!

3 posts - 3 participants

Read full topic

In this blog post, we will explore how you can establish centralized governance across your Google Kubernetes Engine (GKE) clusters using GKE Policy Controller and compare it with the open-source OPA Gatekeeper project.

Centralized governance: Fleet-level OPA setup

Manually installing and managing policy engines on a cluster-by-cluster basis is highly inefficient and error-prone. To solve this, Google Cloud allows you to define policy configurations at the fleet level.

Once fleet-level defaults are enabled, any new GKE cluster registered to the fleet automatically inherits:

• Automatic Installation: Policy Controller is deployed automatically on registration.

• Version Management: The fleet centrally manages the controller version.

• Global Settings: Global settings such as audit intervals, exempted namespaces, and referential constraints are consistently applied.

Managed policy bundles

Google Cloud provides ready-to-use Policy Bundles mapped to industry standards. You can apply these to all member clusters simultaneously to achieve instant compliance:

• CIS Kubernetes Benchmark: Align cluster settings with the Center for Internet Security standards.

• Regulated Industries (PCI-DSS / NIST / HIPAA): Compliance-specific bundles designed to meet rigorous regulatory controls.

• Cost & Reliability: A bundle that proactively flags wasteful resource configurations or missing high-availability setups (like missing PodDisruptionBudgets).

Fleet dashboard & observability

Within the GKE Enterprise console, you get a unified dashboard offering complete visibility:

• Violation Summaries: Identify which clusters have the highest compliance risk.

• Security Command Center (SCC) Integration: Violations are pushed to SCC as “findings,” enabling security operations teams to track Kubernetes risks alongside general cloud infrastructure.

[image]

GKE Policy Controller vs. OSS OPA Gatekeeper

While GKE Policy Controller is built on the open-source OPA Gatekeeper engine, Google Cloud adds significant enterprise management capabilities:
Feature
OSS OPA Gatekeeper
GKE Policy Controller (Fleet)
Management
Manual (per cluster)
Centralized (Fleet-wide)
Bundles
Community-maintained
Google-maintained & updated
Installation
Helm / Kubectl
One-click or CLI Fleet enablement
Observability
Prometheus / Grafana
GCP Console + Security Command Center
Support
Community
Google Enterprise Support

Getting started: Enabling Fleet-level policies

Enabling Policy Controller across your GKE fleet is straightforward. Using the gcloud CLI, you can declare your configuration in a YAML file and roll it out instantly.

Step 1: Create a Fleet default configuration file

Create a file named fleet-default.yaml to specify the default configuration and any compliance bundles you wish to pre-install:

policyControllerHubConfig:

installSpec: INSTALL_SPEC_ENABLED

policyContent:

bundles:

  cis-k8s-v1.5.1:

    exemptedNamespaces: \["kube-system"\]

Note: The CLI commands above are provided for explanation, testing, and experiment purposes only. In production environments, policy configurations should be managed using GitOps tools like Config Sync.

Step 2: Apply the configuration to the Fleet

Run the following CLI command to enable Policy Controller using your default configuration:

gcloud container fleet policycontroller enable \

--fleet-default-member-config=fleet-default.yaml

Step 3: Verify the deployment status

You can describe the fleet state to verify that the policies have been successfully rolled out and are active across all member clusters:

gcloud container fleet policycontroller describe

10 Real-world guardrail scenarios

Here are ten common scenarios where Policy Controller helps you maintain strict control over GKE workloads:

1. Enforce organizational labeling

• What it does: Requires every Namespace or Deployment to have specific labels (e.g., team-id, cost-center, or env).

• Why it matters: Essential for accurate cost attribution, department chargebacks, and owner tracing in multi-tenant environments.

2. Restrict ingress domains

• What it does: Restricts Ingress resources to only expose services on pre-approved corporate domains (e.g., hostnames ending in *.yourcompany.com).

• Why it matters: Prevents “shadow IT” and accidental public exposure of internal services on arbitrary or unmapped domains.

3. Image registry whitelisting

• What it does: Blocks any Pod from starting if its container image comes from an untrusted public registry.

• Why it matters: Forces teams to pull verified, scanned container images strictly from your private Artifact Registry.

4. Mandatory resource limits

• What it does: Denies Pods that do not explicitly define CPU/memory limits and requests.

• Why it matters: Prevents “noisy neighbor” issues where a single unconstrained container consumes all node resources, risking cluster stability.

5. Prevent privilege escalation

• What it does: Blocks containers configured with privileged: true, direct hostPath mounts, or running as root.

• Why it matters: Eliminates critical container escape risks that could allow a compromised pod to take control of the host GKE node.

6. Ban specific image tags

• What it does: Enforces version pinning by rejecting volatile image tags like :test or :latest.

• Why it matters: Ensures deployments are predictable, repeatable, and easily rolled back by enforcing immutable digest or semantic tags.

7. Limit service types (No public LoadBalancers)

• What it does: Prevents the creation of Services with type: LoadBalancer, steering teams toward secure internal Ingress pathways.

• Why it matters: Avoids unnecessary external IP costs and secures the cluster’s network perimeter from direct, unauthorized public ingress.

8. Enforce network policies

• What it does: Requires every new Namespace to contain a default “deny-all” NetworkPolicy object on creation.

• Why it matters: Establishes a zero-trust network posture inside the cluster right from the start.

9. Prohibit NodePort Services

• What it does: Disallows Kubernetes Services from exposing node-level ports (type: NodePort).

• Why it matters: Eliminates security gaps by preventing arbitrary, high-numbered ports from being opened on GKE worker nodes.

10. Service port naming for observability

• What it does: Mandates that Service port names follow standard conventions (e.g., starting with http- or grpc-).

• Why it matters: Ensures service meshes (such as Istio/Cloud Service Mesh) and monitoring tools can automatically discover and trace protocol-specific traffic.

Step-by-step: Testing a new constraint

Let’s walk through how to deploy and test a policy that prevents the use of unapproved public container registries.

Step 1: Verify the constraint template exists

Policy Controller comes pre-installed with a large template library. Verify that the K8sAllowedRepos template is available in your cluster:

kubectl get constrainttemplates k8sallowedrepos

Step 2: Define your constraint configuration

Create a file named deny-public-images.yaml. This file configures the K8sAllowedRepos template to watch Pod resources and only allow images originating from your private Google Artifact Registry path:

apiVersion: constraints.gatekeeper.sh/v1beta1

kind: K8sAllowedRepos

metadata:

name: allow-only-internal-registries

spec:

enforcementAction: deny

match:

kinds:

  - apiGroups: \[""\]

    kinds: \["Pod"\]

excludedNamespaces:

  - kube-system

  - gatekeeper-system

parameters:

repos:

  - "gcr.io/your-project-id/"

  - "us-central1-docker.pkg.dev/your-project-id/"

Step 3: Apply the constraint

For a single cluster test, apply the file directly via kubectl:

kubectl apply -f deny-public-images.yaml

(Note: In a production environment, it is best practice to commit this file to a Git repository managed centrally by Config Sync to synchronize policies across all GKE clusters automatically).

Step 4: Test the constraint

Attempt to deploy a Pod using a public image from Docker Hub (which is outside your allowlist):

kubectl run test-public-image --image=nginx:test-tag

Expected result

The cluster’s admission controller should reject your request with a descriptive error message:

Error from server (Forbidden): admission webhook “validation.gatekeeper.sh” denied the request: [allow-only-internal-registries] container has an invalid image repo nginx:test-tag, allowed repos are [“gcr.io/your-project-id/”, …]

[image]

Note: The direct kubectl commands and manual constraint setups shown above are intended for explanation, testing, and experiment purposes only.

Managing the impact of constraints on running workloads

Because Policy Controller acts as an Admission Controller, it acts as a “bouncer” at the door of your cluster. It evaluates requests during the creation or update phase, meaning it will not retroactively terminate pods that are already running.

Here is how a newly applied constraint affects existing workloads:

1. Immediate impact: “Audit” mode

Since the Pod is already running, it continues to run. However, the background Audit engine flags the workload as a violation:

• You will see the violation reported under GKE > Posture Management > Policy in the Google Cloud Console.

• This allows you to identify non-compliant workloads without causing production outages.

Pro Tip: When launching a new constraint, set enforcementAction: dryrun first. This lets you observe violations in the dashboard before switching to deny to block deployments.

2. Future impact: Admission blocking

The next time the workload is modified or restarted, the policy will intercept and block the deployment in the following cases:

• Node Restarts / Pod Crashes: If a node goes down or a pod crashes, Kubernetes will try to recreate the Pod. The Policy Controller will block recreation if the pod does not meet the policy.

• Updates: Modifying labels, environment variables, or other deployment specifications will fail if the underlying Pod template violates the policy.

• Autoscaling: GKE’s Cluster Autoscaling or Horizontal Pod Autoscaling (HPA) will fail to spin up new pods if the templates are non-compliant.

3. Checking active violations from CLI

You can view a list of active violations directly using kubectl:

kubectl get k8sallowedrepos allow-only-internal-registries -o yaml

Under the status.violations section of the output, you will find the namespace and name of every non-compliant Pod currently running.

Scaling policy management with Config Sync

While manual setup via direct CLI commands works for testing, scaling policy management across an entire enterprise requires a declarative, version-controlled approach. Organizations leverage Config Sync to implement GitOps-driven, fleet-wide policy management.

By storing constraints and constraint templates in a central Git repository, Config Sync continuously monitors and automatically reconciles configurations across all member clusters in the fleet. This ensures that policies remain consistent, auditable through standard git workflows, and resistant to configuration drift across dev, staging, and production environments.

Conclusion

Standardizing policy enforcement across your GKE environments is key to achieving robust security and cost-efficiency at scale. By leveraging GKE Policy Controller, platform teams can easily automate enforcement of industry-standard security frameworks across their entire GKE Fleet.

Further reading

• Google Cloud GKE Policy Controller Documentation

• Open-Source OPA Gatekeeper GitHub Repository

1 post - 2 participants

Read full topic

Waiting for a JAX training loop to finish on a Cloud TPU slice while step times spike or accelerator utilization drops is frustrating. Debugging these bottlenecks often pulls ML engineers away from modeling and into operations: setting up self-hosted TensorBoard and XProf servers, configuring SSH port-forwarding, and manually syncing profile traces across storage buckets.

The Google Cloud ML Diagnostics SDK (part of the ML Diagnostics platform) eliminates this overhead. It delivers a fully managed XProf profiling and observability experience directly in the Google Cloud console. Best of all, there is no compute charge for the managed XProf backend deployed by ML Diagnostics—you only pay for Cloud Logging metric ingestion and Cloud Storage trace storage.

In this 5-minute walkthrough, you will provision a Cloud TPU VM, instrument a JAX training script to record metrics and capture XProf profiles, and inspect your hardware telemetry in the Google Cloud console.

Prerequisites

Configure the required APIs, IAM permissions, and Cloud Storage bucket in your Google Cloud project so your workload can stream telemetry without permission errors.

Required APIs and IAM roles

1. Enable required APIs

Enable the required services in your Google Cloud project:

gcloud services enable hypercomputecluster.googleapis.com tpu.googleapis.com compute.googleapis.com

Note: The hypercomputecluster.googleapis.com API powers MLRun registration, telemetry ingestion, and the console UI. You do not need the Cluster Director product to manage your clusters—ML Diagnostics works independently with standalone Cloud TPU VMs and Google Kubernetes Engine (GKE). See the official overview for details.

API enablement in the console

2. Enable Log Analytics on Cloud Logging

The Run Diagnostics dashboard uses Log Analytics to render interactive time-series charts for model and system metrics. Upgrade your _Default log bucket before running your workload (Log Analytics does not backfill historical logs):

gcloud logging buckets update _Default --location=global --enable-analytics --project=YOUR_PROJECT_ID

3. Set up a service account and Cloud Storage bucket

Create a dedicated service account for your Cloud TPU VM and a Cloud Storage bucket to store XProf trace files:

1. Create a service account:

   gcloud iam service-accounts create diagon-walkthrough-sa --display-name "ML Diagnostics Walkthrough Service Account"
   

2. Grant required IAM roles:

   export PROJECT_ID=$(gcloud config get-value project)
   export SA_EMAIL="diagon-walkthrough-sa@${PROJECT_ID}.iam.gserviceaccount.com"
   
   for role in roles/hypercomputecluster.editor roles/logging.logWriter roles/storage.objectUser; do
     gcloud projects add-iam-policy-binding ${PROJECT_ID} --member="serviceAccount:${SA_EMAIL}" --role="$role"
   done
   

3. Create a Cloud Storage bucket:

   export BUCKET_NAME="${PROJECT_ID}-diagon-profiles"
   gcloud storage buckets create gs://${BUCKET_NAME} --location=us-central1
   

Tip: If your organization restricts roles/hypercomputecluster.editor, create a custom IAM role with only the required hypercomputecluster.machineLearningRuns.* permissions. See the ML Diagnostics IAM guide.

Step 1: Provision and connect to your Cloud TPU VM

Provision a single-host Cloud TPU v5e VM (v5litepod-4) in us-central1-a, attaching your service account and enabling the cloud-platform OAuth scope:

gcloud compute tpus tpu-vm create my-diagon-tpu \
  --zone=us-central1-a \
  --accelerator-type=v5litepod-4 \
  --version=v2-alpha-tpuv5-lite \
  --service-account=${SA_EMAIL} \
  --scopes=https://www.googleapis.com/auth/cloud-platform

Connect to your Cloud TPU VM over SSH:

gcloud compute tpus tpu-vm ssh my-diagon-tpu --zone=us-central1-a

Tip: If you encounter SSH timeouts or firewall blocks, see Troubleshooting SSH on Compute Engine.

Step 2: Isolate dependencies with venv

Create and activate an isolated Python virtual environment (Python 3.10+ recommended):

python3 -m venv diagon-sdk-env
source diagon-sdk-env/bin/activate

Tip: If python3 -m venv fails with a missing ensurepip error on Debian or Ubuntu images, run sudo apt-get update && sudo apt-get install -y python3-venv, or install packages in user space with pip install --user.

Step 3: Install the ML Diagnostics SDK and JAX

Install google-cloud-mldiagnostics (version 1.0.6 or higher), google-cloud-logging, and jax[tpu]:

pip install --upgrade pip
pip install "google-cloud-mldiagnostics>=1.0.6" "jax[tpu]" google-cloud-logging \
  -f https://storage.googleapis.com/jax-releases/libtpu_releases.html

Tip: In production containers, pin exact versions of JAX (for example, jax==0.4.35) and the SDK (google-cloud-mldiagnostics==1.0.6) in your requirements.txt or Dockerfile for reproducible builds.

Step 4: Instrument your workload

Initializing machinelearning_run registers your experiment with the ML Diagnostics control plane and automatically records software metadata (JAX version, XLA compiler flags) and hardware topology (TPU device type, slice count).

• Setting log_system_metrics=True streams fine-grained TPU hardware counters (TPU TensorCore utilization, TPU duty cycle, HBM utilization) every 10 seconds to Cloud Logging.
• Wrapping critical sections in a programmatic with xprof(): context manager captures cycle-accurate XProf hardware traces directly to your Cloud Storage bucket.

Create a file named test_workload.py on your Cloud TPU VM:

import time
import logging
import random
import jax
import jax.numpy as jnp
import google.cloud.logging
from google_cloud_mldiagnostics import machinelearning_run, metrics, xprof, metric_types

# 0. Route Python logs to Cloud Logging
logging_client = google.cloud.logging.Client()
logging_client.setup_logging()

# Initialize JAX distributed runtime (auto-detects multi-host TPU slices; safe no-op if single-host)
try:
    jax.distributed.initialize()
except Exception:
    pass
time.sleep(jax.process_index() * 5)  # Stagger multi-host startup to avoid contention

# 1. Initialize the machine learning run
run_name = f"tpu-v5e-walkthrough-run-{int(time.time())}"
my_run = machinelearning_run(
    name=run_name,
    run_group="tpu-getting-started",
    configs={"epochs": 30, "batch_size": 32, "precision": "bfloat16"},
    gcs_path="gs://YOUR_BUCKET_NAME/profiles",  # Replace with your Cloud Storage bucket name
    log_system_metrics=True,                    # Streams libTPU and host telemetry every 10 seconds
    project="YOUR_PROJECT_ID",                  # Replace with your Google Cloud project ID
    region="us-central1",                       # Region where MLRun metadata is stored
)

logging.info(f"MLRun created: {my_run.name} on Worker {jax.process_index()} (Devices: {jax.devices()})")

# Simulate a memory-bound attention block by allocating bfloat16 matrices on TPU HBM
key = jax.random.PRNGKey(42)
x = jax.random.normal(key, (4096, 4096), dtype=jnp.bfloat16)

# 2. Record training loop metrics and capture profiles
learning_rate = 0.001
total_steps = 30
for step in range(1, total_steps + 1):
    step_start = time.perf_counter()
    x = jnp.dot(x, x)
    x.block_until_ready()

    step_time = time.perf_counter() - step_start + random.uniform(0.05, 0.15)
    loss = 2.5 / step + random.uniform(-0.02, 0.02)
    accuracy = min(0.99, 0.50 + (step * 0.015))
    mfu_percent = min(95.0, 52.0 + (step * 0.5))  # MetricType.MFU expects percentage (0-100)

    # Batch record predefined MetricType keys and custom metrics
    metrics.record_metrics([
        {"metric_name": metric_types.MetricType.LOSS, "value": loss},
        {"metric_name": metric_types.MetricType.LEARNING_RATE, "value": learning_rate},
        {"metric_name": metric_types.MetricType.STEP_TIME, "value": step_time},
        {"metric_name": metric_types.MetricType.THROUGHPUT, "value": 32.0 / step_time},
        {"metric_name": metric_types.MetricType.MFU, "value": mfu_percent},
        {"metric_name": "ACCURACY", "value": accuracy},
    ], step=step)

    print(f"[Worker {jax.process_index()}] Step {step}/{total_steps} | Loss: {loss:.4f} | MFU: {mfu_percent:.1f}% | Step Time: {step_time:.3f}s")

    # 3. Capture a hardware performance profile programmatically at step 5
    if step == 5:
        print(f"[Worker {jax.process_index()}] Capturing programmatic XProf trace at step {step}...")
        with xprof():
            for _ in range(5):
                y = jnp.dot(x, x)
                y.block_until_ready()
            time.sleep(2)
        print(f"[Worker {jax.process_index()}] Programmatic profile uploaded to Cloud Storage! Training loop continuing...")

    learning_rate *= 0.95
    time.sleep(2)

Run the instrumented script:

python3 test_workload.py

Practical pro tips

• Multi-host TPU slices: When running across multi-host TPU slices, stagger worker startup (time.sleep(jax.process_index() * 5)) and pass run_workload_id="my-shared-run-id" to machinelearning_run() so all workers group cleanly under a single unified run.
• Unique run names: Standalone Cloud TPU VMs require a unique name for each new run (such as appending int(time.time())), whereas GKE workloads automatically append timestamps via the injection webhook.

Step 5: Verify results in the console

Once your script runs, inspect your telemetry across Cloud Logging, Cloud Storage, and the Run Diagnostics dashboard. For full details on console features and CLI queries, see View machine learning runs with ML Diagnostics.

Note: While Steps 1–4 provision a single-host v5litepod-4 VM for quick testing, the console screenshots below showcase a completed 4-host Cloud TPU v5e slice (tpu-v5e-walkthrough-run-completed, workers t1v-n-fef2b8f2-w-0 through w-3) to illustrate how multi-host system metrics, per-worker Cloud Storage artifacts, and synchronized 4-host XProf timelines render across a full TPU slice.

1. Check Cloud Logging and Cloud Storage

• Cloud Logging: Open Logs Explorer and confirm the MLRun created entry alongside structured metric payloads under the ml_diagnostics_metric log stream.
• Cloud Storage: Browse to your profile bucket path (gs://YOUR_BUCKET_NAME/.../plugins/profile/) and verify that compressed multi-host XPlane trace files (.xplane.pb, .trace.json.gz, and .op_stats_v2.pb) were uploaded automatically for each TPU worker host alongside pre-indexed .SSTABLE cache files generated by Managed XProf:

Cloud Storage profile artifacts

2. Explore the Run Diagnostics dashboard

Open the Run Diagnostics dashboard in the Google Cloud console via either navigation path:

• Cluster Director console: Cluster Director > Run Diagnostics (console.cloud.google.com/cluster-director/diagnostics)
• GKE console: Kubernetes Engine > AI/ML > Run Diagnostics (console.cloud.google.com/kubernetes/aiml/diagnostics)

Runs list

Compare all your machine learning runs side-by-side in the Runs table. Track run status (active in-progress spinner vs. completed checkmark), creation and update timestamps, hardware labels (accelerator_type : tpu), and direct View profiles shortcuts:

Run details

Click your run (tpu-v5e-walkthrough-run-completed) to inspect the Details tab. Review run metadata (Run group: tpu-getting-started, Orchestrator: GCE), auto-attached SDK labels (accelerator_type : tpu, diagon_sdk_version : 1-0-7, framework : jax), and auto-collected software configurations (libtpu_version: 0.0.17, framework: JAX, framework_version: 0.6.2):

Model metrics

Switch to the Model metrics tab to analyze training convergence curves side-by-side. Every metric recorded via metrics.record_metrics() (such as Learning Rate and Loss) is plotted both over Time (left column) and over Steps (right column). In this completed 42-step run, the charts capture schedule transitions including the initial linear warmup ramp (0.0002 to 0.001 over steps 1–4), smooth exponential decay (0.95x per step over steps 4–42), and monotonic loss reduction (2.5 down to 0.38):

Performance metrics

Open the Performance metrics tab to track training speed and stability. Interactive charts for Step Time (in seconds) and Throughput (per second) are plotted over Time and over Steps, with automatic Mean and StdDev dashed horizontal reference lines that highlight step-time jitter at a glance:

System metrics

Select the System metrics tab to inspect fine-grained hardware counters streamed every 10 seconds across all TPU hosts in your slice (t1v-n-fef2b8f2-w-0 through w-3). Correlate per-host Top 5 TPU Duty Cycle (%) (peaking at 100% during active matrix multiplications), Top 5 TPU TensorCore Utilization (%) (peaking around ~2.8% for this lightweight synthetic benchmark), Top 5 HBM Utilization (%) (~41% peak), and Top 5 Host CPU Utilization (%) against Mean and StdDev baselines:

Captured profiles (programmatic XProf capture)

The Profiles tab lists every profile session captured programmatically during the run via with xprof(): (such as 2026_09_12_10_36_04), displaying capture timestamps and tracer levels (Host tracer level: Info, Device tracer level: Enabled). Click any session link to launch the interactive Managed XProf Trace Viewer directly in your browser:

Note: On standalone Cloud TPU VMs, profile sessions are captured programmatically in your script using the with xprof(): context manager. Live UI-triggered profiling (+ Capture new profile session) is supported on GKE clusters (covered in Part 2).

Managed XProf trace viewer

Clicking any profile session link opens the Managed XProf Trace Viewer (XProf v2.23.1) directly in your browser. Powered by a distributed worker-aggregator backend, Managed XProf loads multi-gigabyte multi-host TPU traces up to 10x faster than self-hosted TensorBoard servers (~1.7 minutes vs. 17 minutes for an 8-host 1.4GB profile) and enables instant link sharing across your team:

Inside Managed XProf, switch between all 4 TPU hosts (Hosts (4)) and 10 profiling tools (Tools (10)) to pinpoint hardware bottlenecks:

• Trace Viewer: Inspect a cycle-accurate timeline of host CPU PJRT dispatch threads (host:CPU), TPU TensorCore execution streams across chips (device:TPU:0 through device:TPU:3), XLA High-Level Optimizer (HLO) modules (XLA Modules, XLA Ops, XLA TraceMe), and framework operations.
• Roofline Analysis: Identify whether operations are memory-bound or compute-bound based on operational intensity (FLOPs/Byte) vs. achieved TFLOPs/s.
• Memory Viewer: Track High Bandwidth Memory (HBM) usage over time and inspect exact tensor buffer shapes at peak memory allocation to resolve Out-Of-Memory (OOM) errors.

Conclusion

In just a few minutes, you provisioned a Cloud TPU VM, instrumented a JAX workload with the Google Cloud ML Diagnostics SDK, streamed real-time hardware telemetry to Cloud Logging, and analyzed TPU execution traces in the zero-cost Managed XProf viewer.

Ready to try it on your own models? Replace the synthetic loop in test_workload.py with your actual JAX training step to start profiling your Cloud TPU workloads today. If you have questions or run into issues setting this up, drop a comment below!

What’s next: Scaling diagnostics to GKE clusters

While standalone Cloud TPU VMs are ideal for interactive prototyping and single-slice workloads, large-scale foundation model training often spans multi-slice TPU clusters orchestrated by Google Kubernetes Engine (GKE).

In Part 2 of this series, we will cover using ML Diagnostics with GKE clusters (including one-command cluster setup via xpk cluster create --managed-mldiagnostics), featuring:

• Live UI-triggered on-demand profiling: Trigger interactive multi-host XProf trace sessions on the fly directly from the Google Cloud console (+ Capture new profile session) without modifying training code or restarting jobs.
• Automated GKE workload correlation: Automatically map TPU slices and GKE pods to Kubernetes JobSet and LeaderWorkerSet hierarchies via the ML Diagnostics GKE webhook.
• Multi-slice observability: Stream cluster-wide TPU duty cycle, TensorCore utilization, and HBM telemetry across multi-host GKE workloads in the Run Diagnostics dashboard.

Resources and further reading

• ML Diagnostics Platform Overview: Cloud TPU ML Diagnostics Overview
• ML Diagnostics SDK Guide: Get started with the ML Diagnostics SDK
• Viewing ML Runs & Dashboard Guide: View machine learning runs with ML Diagnostics
• Cloud TPU v5e Architecture & Guide: Cloud TPU v5e Documentation
• Google I/O 2025 Technical Session: Supercharge ML performance on xPUs with XProf
• ML Diagnostics SDK GitHub Repository: AI-Hypercomputer/google-cloud-mldiagnostics
• ML Diagnostics SDK on PyPI: google-cloud-mldiagnostics
• OpenXLA XProf Documentation: openxla.org/xprof
• JAX Documentation: docs.jax.dev
• TensorBoard Profiler Guide: tensorflow.org/tensorboard

1 post - 2 participants

Read full topic

The Cloud Monitoring metrics documentation lists these metrics as BETA:

compute.googleapis.com/instance/tpu/accelerator/power_draw
compute.googleapis.com/instance/tpu/accelerator/hbm_power_draw
compute.googleapis.com/instance/tpu/accelerator/tray_power

I queried the Cloud Monitoring projects.timeSeries.list API for the past 48 hours, using a filter like this:

metric.type="compute.googleapis.com/instance/tpu/accelerator/power_draw"
AND resource.type="gce_instance"
AND resource.labels.instance_id="<INSTANCE_ID>"

The request succeeds with HTTP 200, but returns no time series or data points for any of these three metrics. For comparison, I can retrieve tpu.googleapis.com/accelerator/duty_cycle data for the same TPU.

Could someone clarify whether:

• There is a problem with my query or resource filter;
• These power metrics are not yet available for TPU VMs created through the Cloud TPU API;
• Additional configuration is required?

My goal is to measure TPU power consumption during inference. If these metrics are not currently supported on this deployment, is there another supported way to obtain that measurement?

Thank you!

1 post - 1 participant

Read full topic

Early Access Notice: In order to get early access to certain features and managed endpoints of the Gmail MCP server (`https://gmailmcp.googleapis.com/mcp/v1\`), apply to join the Google Workspace Developer Preview Program.

Introduction

Have you ever tried building an autonomous AI agent to triage customer inboxes, summarize urgent email threads, or draft responses, only to spend days writing brittle REST API wrappers and wrestling with token refresh edge cases?

Connecting Large Language Models (LLMs) to enterprise communication hubs like Google Workspace and Gmail is one of the highest-value opportunities for generative AI automation. However, enterprise developers often face two fundamental roadblocks:

1. Tool orchestration overhead: How do models discover and reliably call email actions without maintaining custom function schemas?

2. Enterprise identity and delegation: How to securely authenticate interactive user-facing assistants versus headless 24/7 background worker bots across one or many mailboxes?

In this post, we explore how to build production-grade Gmail agents using the Google Agent Development Kit (ADK) and the managed Model Context Protocol (MCP) endpoint. We’ll walk through two distinct authentication architectures—OAuth 2.0 User Consent and Service Account Domain-Wide Delegation (DWD)—and share a blueprint for deploying on Google Cloud Run.

The architectural shift: Why MCP and Google ADK?

Historically, integrating Gemini with Google Workspace APIs meant manually crafting OpenAPI specs or Python function-calling dictionaries for every Gmail endpoint, along with custom error handling, schema serialization, and pagination logic.

The Model Context Protocol (MCP) standardizes how AI applications connect to external tools and data sources. Google Workspace now provides managed, remote Streamable HTTP MCP endpoints such as https://gmailmcp.googleapis.com/mcp/v1.

Key benefits of this architecture

• Standardized JSON-RPC 2.0 discovery: The agent queries available tools (such as list_threads, get_message, send_message, and modify_labels) dynamically via tools/list.
• Zero custom tool wrappers: No need to hand-code parameter parsing or function schemas for the LLM.
• Native ADK toolset integration: Google ADK’s McpToolset class connects Gemini models to remote Streamable HTTP MCP endpoints with just a few lines of configuration.

Choosing between interactive user flow and service account Domain-wide Delegation (DWD)

The most critical architectural decision when designing an enterprise email agent is identity and credential delegation. Here we list two options that can be used for this purpose:

Authentication comparison matrix

[!NOTE]
Enterprise compliance note: Because the Service Account DWD pattern and internal OAuth client flows are scoped strictly to your organization’s Google Workspace domain, they are exempt from Google’s public OAuth app verification and external CASA Tier 2/3 assessments. The organization’s Workspace Super Admin maintains full sovereign control over which scopes are authorized via the Google Workspace Admin console.

Step-by-step implementation

Let’s walk through configuring both methods, solving the token refresh challenge, and implementing the agents in Python.

1. Required OAuth scopes

When connecting generative AI models to enterprise communication hubs, the fundamental security principle is least privilege. We intentionally configure our scopes to use gmail.compose rather than gmail.send for security concerns.

SCOPES = [
    "https://www.googleapis.com/auth/cloud-platform",
    "https://www.googleapis.com/auth/gmail.readonly",
    "https://www.googleapis.com/auth/gmail.modify",
    "https://www.googleapis.com/auth/gmail.compose", # Draft-only (Recommended)# 
    "https://www.googleapis.com/auth/gmail.labels",
]

Why you should avoid granting direct gmail.send to LLMs

1. Indirect prompt injection: Untrusted inbound emails could hijack the model into sending unauthorized outbound emails.
2. Irreversibility and brand liability: Sent emails cannot be recalled; hallucinations create instant damage.
3. Restricted OAuth compliance: Minimizes enterprise blast radius.

The recommended pattern is “agent drafts, human approves”, which implements a human-in-the-loop workflow using create_draft and review labels.

2. Setting up option 1: Interactive user flow

Option 1 is ideal when building interactive assistants where end users grant access to their individual accounts. Refer to Google Workspace MCP & ADK Codelab for a step-by-step guide on how to set up Option 1.

GCP Console setup:

1. In the Google Cloud Console Credentials Page, configure the OAuth consent screen (Internal or External).
2. Create an OAuth 2.0 Client ID (select Desktop App for local testing or Web Application for production).
3. Save the Client ID and Client Secret in .env.

GOOGLE_GENAI_USE_VERTEXAI=1
GOOGLE_CLOUD_PROJECT=your_project_id
GOOGLE_CLOUD_LOCATION=us-central1
OAUTH_CLIENT_ID=your_client_id.apps.googleusercontent.com
OAUTH_CLIENT_SECRET=your_client_secret

Running interactive authorization:

Run the following auth.py to trigger the local consent flow and store the authorized credentials in token.json and Application Default Credentials (ADC):

# auth.py
import json, os
from google_auth_oauthlib.flow import InstalledAppFlow

def run_oauth_flow():
    client_config = {
        "installed": {
            "client_id": os.environ["OAUTH_CLIENT_ID"],
            "client_secret": os.environ["OAUTH_CLIENT_SECRET"],
            "auth_uri": "https://accounts.google.com/o/oauth2/auth",
            "token_uri": "https://oauth2.googleapis.com/token",
        }
    }
    flow = InstalledAppFlow.from_client_config(client_config, scopes=SCOPES)
    creds = flow.run_local_server(port=0)

    cred_dict = json.loads(creds.to_json())
    cred_dict["type"] = "authorized_user"
    cred_dict["quota_project_id"] = os.environ.get("GOOGLE_CLOUD_PROJECT")

    with open("token.json", "w") as f:
        json.dump(cred_dict, f, indent=2)
    return creds

3. Setting up option 2: Service account domain-wide delegation (DWD)

Option 2 allows headless background workers to interact with inboxes on behalf of domain users without any interactive login prompts.

Step 1: Create a Service Account in GCP Console

1. Navigate to IAM & Admin > Service Accounts and create a Service Account (e.g., sa-workspace-agent).
2. Copy the 21-digit Numeric Client ID (OAuth2 Unique ID) from the service account details.
3. Grant your caller principal or runtime identity the Service Account Token Creator (roles/iam.serviceAccountTokenCreator) role.

Step 2: Authorize scopes in Google Workspace Admin Console

1. Sign in to admin.google.com as a Super Admin.
2. Go to Security > Access and data control > API controls > Manage Domain Wide Delegation.
3. Click Add new, paste the Numeric Client ID, and add the comma-delimited OAuth scopes.
4. Click Authorize.
   
   

Step 3: Configure environment (.env)

GOOGLE_GENAI_USE_VERTEXAI=1
GOOGLE_CLOUD_PROJECT=your_project_id
GOOGLE_CLOUD_LOCATION=us-central1
SERVICE_ACCOUNT_EMAIL=sa-workspace-agent@your_project.iam.gserviceaccount.com
TARGET_USER_EMAIL=target_employee@yourcompany.com

4. Resilient in-memory token caching (TokenRelay)

When an LLM agent executes complex, multi-step tool calls in rapid succession, refreshing tokens synchronously within tool execution hooks can trigger redundant requests and infinite retry loops. Refer to Google Workspace MCP & ADK Codelab for the detailed source code.

The TokenRelay cache solves this by managing proactive refreshes with thread-safe in-memory caching:

import datetime
from google.auth.transport.requests import Request

class TokenRelay:
    """Manages in-memory caching of Google OAuth access tokens for Gmail MCP.
    
    Proactively refreshes access tokens 5 minutes prior to expiration.
    """

    def __init__(self, credentials):
        self._creds = credentials
        self._cached_token: str | None = None
        self._expiry: datetime.datetime | None = None

    def get_token(self) -> str:
        """Returns a valid access token, proactively refreshing when needed."""
        now = datetime.datetime.now()
        if (
            self._cached_token is None
            or self._expiry is None
            or now >= (self._expiry - datetime.timedelta(minutes=5))
        ):
            try:
                self._creds.refresh(Request())
                self._cached_token = self._creds.token
                self._expiry = now + datetime.timedelta(minutes=50)
            except Exception as e:
                print(f"[TokenRelay] Token refresh failed: {e}")
        return self._cached_token or ""

5. Assembling the agents with Google ADK

Now, let’s assemble both agents using Google ADK’s LlmAgent and McpToolset.

Option 1: Interactive user agent

Refer to Google Workspace MCP & ADK Codelab for a step-by-step guide and source code on how to implement Option 1.

import os
from google.oauth2.credentials import Credentials
from google.adk.agents.llm_agent import LlmAgent
from google.adk.tools.mcp_tool.mcp_toolset import McpToolset
from google.adk.tools.mcp_tool.mcp_session_manager import StreamableHTTPConnectionParams

os.environ["ADK_ENABLE_MCP_GRACEFUL_ERROR_HANDLING"] = "1"

# 1. Load user credentials and wrap with TokenRelay
creds = Credentials.from_authorized_user_file("token.json", scopes=SCOPES)
token_relay = TokenRelay(creds)

# 2. Define dynamic header provider for quota & auth
def auth_header_provider(tool_context=None) -> dict[str, str]:
    return {
        "Authorization": f"Bearer {token_relay.get_token()}",
        "X-Goog-User-Project": os.environ.get("GOOGLE_CLOUD_PROJECT", "your-project-id"),
    }

# 3. Connect to managed Gmail MCP endpoint
gmail_mcp = McpToolset(
    connection_params=StreamableHTTPConnectionParams(
        url="https://gmailmcp.googleapis.com/mcp/v1",
        timeout=60,
    ),
    header_provider=auth_header_provider,
)

# 4. Initialize the ADK Agent
root_agent = LlmAgent(
    model="gemini-3.7-flash",
    name="gmail_oauth_agent",
    instruction="You are an AI email assistant. Help the user search, summarize, and draft emails using Gmail MCP tools.",
    tools=[gmail_mcp],
)

Option 2: Enterprise headless DWD agent

import os
import google.auth
from google.auth import impersonated_credentials
from google.adk.agents.llm_agent import LlmAgent
from google.adk.tools.mcp_tool.mcp_toolset import McpToolset
from google.adk.tools.mcp_tool.mcp_session_manager import StreamableHTTPConnectionParams

os.environ["ADK_ENABLE_MCP_GRACEFUL_ERROR_HANDLING"] = "1"

# 1. Obtain base Application Default Credentials (e.g., Cloud Run Workload Identity)
base_credentials, _ = google.auth.default(scopes=SCOPES)

# 2. Impersonate Service Account and delegate domain authority
creds = impersonated_credentials.Credentials(
    source_credentials=base_credentials,
    target_principal=os.environ["SERVICE_ACCOUNT_EMAIL"],
    target_scopes=SCOPES,
    subject=os.environ["TARGET_USER_EMAIL"],  # Target employee mailbox
)

token_relay = TokenRelay(creds)

def auth_header_provider(tool_context=None) -> dict[str, str]:
    return {
        "Authorization": f"Bearer {token_relay.get_token()}",
        "X-Goog-User-Project": os.environ.get("GOOGLE_CLOUD_PROJECT", "your-project-id"),
    }

gmail_mcp = McpToolset(
    connection_params=StreamableHTTPConnectionParams(
        url="https://gmailmcp.googleapis.com/mcp/v1",
        timeout=60,
    ),
    header_provider=auth_header_provider,
)

root_agent = LlmAgent(
    model="gemini-3.7-flash",
    name="gmail_dwd_agent",
    instruction="You are an autonomous enterprise operations assistant performing automated inbox triage on behalf of the designated mailbox.",
    tools=[gmail_mcp],
)

Production deployment blueprint on Google Cloud Run

To scale your agent in production, deploy the application container to Google Cloud Run utilizing Workload Identity. Pay attention to the following best practices to ensure security and reliability:

1. Never hardcode JSON keys: In Cloud Run, authenticate your service container using Workload Identity attached to the runtime service account.
2. Always supply X-Goog-User-Project: This header ensures API quotas and metrics route accurately to your GCP project.
3. Configure timeout and error resiliency: Remote Streamable HTTP MCP connections should specify timeout=60 and ADK_ENABLE_MCP_GRACEFUL_ERROR_HANDLING="1" to avoid dropping conversational context during network spikes.
4. Follow the principle of least privilege: Scope permissions to the minimal set required (e.g., gmail.readonly for analysis bots, gmail.modify for categorization and triage).

Testing and local verification with ADK web server

You can test the agents locally using the built-in ADK developer UI. To understand how this architecture operates in production, let’s ground it in an enterprise scenario: Automating cross-functional project updates and draft generation.

Real-world application scenario: autonomous inbox triage for project updates

An engineering operations team receives dozens of daily emails regarding an initiative called “Forge Project”. Manual triage requires someone to read each thread, identify blocking items, draft updates, and update labels.

In our test inbox, we have active emails regarding the Forge Project in the inbox:

Execution walkthrough with ADK

We ask our headless DWD agent:

“Search for recent emails about the Forge Project, summarize the current blockers, and draft a response to the latest update acknowledging the timeline.”

The execution unfolds across three coordinated phases:

1. Dynamic tool discovery: Gemini inspects the live Gmail MCP schema and executes list_threads(q=‘Forge Project’).
2. Context extraction & synthesis: The agent fetches message contents via get_thread, extracts timeline updates, and identifies pending deliverables.
3. Safe drafting (HITL): Instead of executing an irreversible send, the agent calls create_draft to prepare a response and applies the FORGE_TRIAGED label.

The screenshot above illustrates how the agent’s MCP tool output accurately matches the live Gmail inbox state. The agent was able to successfully retrieve and summarize emails related to the ‘Forge Project’.

Conclusion and next steps

Integrating Google Agent Development Kit (ADK) with the Google Workspace Gmail MCP Server unlocks a robust, standardized foundation for enterprise AI automation. By pairing MCP’s standard tool discovery with the appropriate authentication model:

• Interactive User Flow for multi-tenant and interactive assistants
• Service Account Domain-Wide Delegation for 24/7 autonomous background triage

You will be able to achieve both the security compliance required by enterprise IT and the seamless developer experience of modern agent frameworks.

Ready to build?

• Apply to join the Google Workspace Developer Preview Program to access managed MCP endpoints.
• Work through the hands-on Google Workspace MCP & ADK Codelab.

1 post - 2 participants

Read full topic

I am the super admin for a defunct company workspace domain (wesense.ai). Our subscription was cancelled, and the admin console now blocks us from reactivating the plan or accessing data export tools because it states the account is being wiped out.

However, the Admin Console Storage dashboard still displays 128GB of data (primarily Google Drive files) actively residing on Google’s servers. These files contain vital legal compliance and shareholder documents that we urgently need to recover.

Because the standard UI blocks subscription reactivation and gives a “Not available to purchase” error due to the deletion queue, is there is a way to manually request to pause the automated deletion/purge process so we can export this data?

I can provide our super admin email and verification details privately.

Any help is greatly appreciated!

2 posts - 3 participants

Read full topic

For the managed gemini-3.5-flash-lite publisher model on Vertex/Agent Platform v1 generateContent (not the Developer API, Live API, or a tuned endpoint), could you clarify the provider-enforced token limit for this configuration?

• generationConfig.maxOutputTokens = G
• thinkingConfig.thinkingLevel = MINIMAL
• Single-turn, non-streaming text/JSON response
• Input: silent video plus text; no tools, grounding, or explicit cache

1. Does this enforce usageMetadata.thoughtsTokenCount + usageMetadata.candidatesTokenCount <= G? Please cite the official specification applicable to this exact model and API.
2. If thinking tokens are outside that limit, what provider-enforced maximum bounds all chargeable thinking tokens per request under this configuration?
3. If the client times out or disconnects while generation continues server-side, does the same bound still apply, and how can the final billable usage be obtained?

We have not enabled inference to test this assumption. We need a documented per-request bound before enabling it, rather than relying on an observed sample or a client timeout as a spending limit. If any model or configuration identifier above is not applicable, please identify the supported equivalent explicitly.

Thank you.

1 post - 1 participant

Read full topic

I am troubleshooting a Google OAuth redirect_uri_mismatch issue in a production application.

The same OAuth 2.0 Web application client is used for three integrations:

• Google Drive
• Google Calendar
• Google Contacts

The problem occurs only with the Google Contacts redirect URI.

OAuth Client ID:
(PII Removed by Staff)

Project:
(PII Removed by Staff)

Redirect URIs:

Drive:
https://api.davidcomputer.online/api/integrations/google-drive/callback

Calendar:
https://api.davidcomputer.online/api/integrations/google-calendar/callback

Contacts:
https://api.davidcomputer.online/api/integrations/google-contacts/callback

Current findings:

1. The Contacts redirect URI is saved and persisted under Google Auth Platform → Clients for the exact Web application OAuth client.

2. The same URI remains present after closing and reopening the OAuth client.

3. The URI is also visible in the legacy OAuth Credentials UI.

4. The production ERP generates the Contacts redirect URI exactly as registered.

5. A byte-for-byte comparison found zero differences:

   • same length
   • no whitespace differences
   • no hidden/control characters
   • no trailing slash difference

6. Drive authorization request:
   MATCH / HTTP 200

7. Calendar authorization request:
   MATCH / HTTP 200

8. Contacts authorization request:
   HTTP 400 redirect_uri_mismatch

9. A manually constructed OAuth authorization request using the exact same Contacts redirect URI also returns:
   HTTP 400 redirect_uri_mismatch

10. All three integrations use the exact same OAuth Client ID.

11. The OAuth client is active and no duplicate client was found.

12. Google Auth Platform shows no blocking configuration warning.

Therefore, the registered Contacts URI and the URI sent in the authorization request are identical, but Google’s live OAuth authorization endpoint rejects the Contacts URI while accepting the Drive and Calendar URIs on the same OAuth client.

Has anyone encountered a situation where a redirect URI is visibly persisted in Google Auth Platform but the live OAuth authorization endpoint still returns redirect_uri_mismatch specifically for that URI?

I would appreciate guidance on how to diagnose or reconcile this Google-side OAuth configuration/runtime inconsistency.

No client secret, access token, refresh token, authorization code, cookies, or other credentials are included here.

Thank you.

1 post - 1 participant

Read full topic

This is my website. (URL Removed by Staff)

I use Mindmaps (from Google Console) for this project.

This is used for saving and loading files from Google Drive.

To achieve this, follow the below steps.

1. Goto (URL Removed by Staff)
2. You can see the ‘Document’ dropdown menu on the upper right.
3. Click ‘Document’->‘Open…’->‘Open’ beneath ‘Google Drive’

Recently I got a message from Google that, “[Action Advised] Manage your unused OAuth clients and protect your information”

It states that “the following projects that you manage have OAuth clients that have been inactive for at least 5 months, and will be deleted in 30 days unless you take action.”

But I get a lot of traffic to my website and many people use ‘Open/Save’ for Google Drive.

Please check the picture below about the traffic I get (it is retrieved from Google Console).

There you can see the visitor stats.

You can see that it is very much active.

Then, why did Google send me this message?

Do I need to do anything?

Thanks

1 post - 1 participant

Read full topic

I have 4 format rules on the Inventory table (Expired, Expiring Soon, Low Stock, New Lot - Calibration Required). Each rule has a condition, an icon, a text color and a highlight (background) color.

What works: the conditions evaluate correctly — the correct icon appears on each row.

What doesn’t work: the text color and highlight color never render. Rows show default black text on white background, and even the icons render in black instead of the rule’s text color. This happens in the editor’s app preview (mobile and desktop modes), the app opened directly in a browser tab, and in both Deck and Detail views.

What I tried:

• Verified “Format these columns and actions” has the display columns selected.
• Changed the highlight color to FF0000 and to the preset “red”, saved — still no color. Reverted afterwards.
• Created a brand-new rule from scratch with identical settings — same behavior (icon only, no colors). Deleted the test rule afterwards.
• Saved multiple times; no editor errors.

Is this a known web-client issue, or is something wrong with my rule setup? Screenshots attached.

2 posts - 2 participants

Read full topic

The problem

The Verification Center will not let me submit. “Data access status” says:

Your app’s data access is not verified. Verification is required because your app requests sensitive or restricted scopes.
You must verify and publish your branding before you can request verification.

and the Prepare for verification button is disabled (disabled: true in the DOM, not merely styled that way).

The documented next step is the Branding page. Per

that page should offer a Verify Branding button, and then a Publish branding button once the automated review finishes.

Neither button exists on my Branding page. I enumerated every button and a element on console.cloud.google.com/auth/branding? (PII Removed by Staff). The complete list of actionable controls is:

제거 (Remove logo) · 로고 변경 (Change logo) · 도메인 추가 (Add domain)
저장 (Save) · 변경사항 취소 (Cancel changes)

There is no Verify, no Publish, and no submit control of any kind. So the Verification Center points at a button that the Branding page does not render — the two pages disagree, and there is no way forward from either.

Configuration (all complete)

All three pages return HTTP 200, are publicly reachable with no login, describe the app, and link to each other. Scopes requested are calendar.events and calendar.readonly, with the usage justification filled in on the Data access page and saved.

Saving the Branding form works — I added the terms-of-service URL, saved, reloaded, and the value persisted. So the form itself is functional; it simply never offers a verification action.

Same report from other developers

This does not look project-specific:

• Prepare for verification disabled, no Verify Branding button — External + In production, branding complete (StrobeDesk)
• "Prepare for verification" disabled, no "Verify Branding" button — External + In production, branding complete
• Verification Center stuck: "Prepare for verification" disabled, Branding has no publish/verify button (Project: realtyworld-crm)
• https://discuss.google.dev/t/verification-center-stuck-prepare-for-verification-disabled-branding-page-has-no-publish-button-pii-removed-by-staff/393040

None of them has a posted resolution.

What I am asking for

1. Could the OAuth verification / Trust & Safety team either trigger the branding review for this project manually, or say what condition actually unlocks the Verify Branding button? If it is a rollout gate rather than a missing setting, saying so would save a lot of people a lot of guessing.
2. A secondary question while I am here: my home page is on *.workers.dev, a Cloudflare-provided subdomain on the Public Suffix List. Search Console accepted it as a URL-prefix property. Is a workers.dev subdomain acceptable for branding verification, or will review re-flag domain ownership and require a domain I have registered myself? I would rather buy a domain now than after a rejection.

Happy to share anything else needed. Thanks.

1 post - 1 participant

Read full topic

Could you add a setting to define the first visible hour in Calendar day and week views?

Our workday starts at 07:00, but the AppSheet Calendar often opens around 13:00/current time. Users must scroll up every time to see the morning schedule. A configurable “First visible hour” (for example 07:00) would make the calendar much more practical for field-service and work-scheduling apps.

The calendar events and start times are correct; this is only about the initial visible scroll position.

Thank you!

3 posts - 3 participants

Read full topic

The Verification Center instructs us to reply to the Trust & Safety email thread, but no email has arrived.

How can we contact the verification team privately to have the email resent and the logo finding reviewed? We can provide project details and supporting evidence through a private support channel.

1 post - 1 participant

Read full topic

I’m located in Iraq and want to configure Google Sign-In for my application, Al-Akidi Orders, using Supabase Auth.

The Google Cloud welcome dialog requires me to select a country, but Iraq is missing from the dropdown, so I cannot complete registration.

Is Google Cloud registration currently available for developers residing in Iraq? What is the supported way to create an OAuth client using my actual country information?

Thank you

1 post - 2 participants

Read full topic

I’m experiencing an intermittent issue with an AppSheet app using AppSheet Database as the data source.

Occasionally, while a user is editing a record through the standard AppSheet Detail → Edit form, values in unrelated Ref columns unexpectedly become dates such as:

02/14/1900
01/04/1900

For example, in one occurrence:

Referral Source → Ref to ReferralSourcesMaster
Referenced key: Source ID (Text)

Primary Insurance Name → Ref to InsuranceMaster
Referenced key: Insurance Master ID (Text)

Both Ref columns are physical columns. They have:

• No App Formula
• No Initial Value
• No Valid If
• No Suggested Values
• Reset on edit OFF
• Correct Text keys in their referenced tables

The issue is intermittent and has occurred in different Ref columns. It does not happen on every edit.

Audit History shows the 1900 date values in the Edit row Start payload, and the subsequent update fails with:

Unable to update row in table 'Referrals'. → Invalid request provided to AppSheet database

The stack trace includes:

AsdbProvider.AsdbTable.UpdateRowsAsync

The affected users are using the browser and editing through the normal AppSheet form. There are no Quick Edit columns involved.

Current sync settings:

• Server caching: ON
• Delta sync: OFF
• Quick sync: ON
• Sync on start: ON
• Delayed sync: OFF
• Automatic updates: ON

We have also cleared AppSheet browser/site data for an affected user, but the issue occurred again.

In one failed request, Audit History showed two unrelated Ref fields containing 1900 dates in the same submitted row.

Has anyone experienced Ref values intermittently being converted/interpreted as 1900-era dates when using AppSheet Database?

Could this be related to Quick Sync/client row state, or is there another AppSheet Database/Ref configuration I should investigate?

Any suggestions for additional Audit History information I should capture would be appreciated.

4 posts - 2 participants

Read full topic

We noticed that the new Explore UI is now available in Looker.

We understand that both the classic and new versions are currently available. Will both continue to be supported, or is the new UI expected to replace the classic version in the future?

If the classic version will eventually be retired, do you know when that change is planned?

Thank you!

1 post - 1 participant

Read full topic

• Plan: AppSheet Starter (paid, $5/user/month)
• App status: Deployed (not Prototype)
• Account: personal Google account using a public domain (gmail.com)

What I’m doing:
I have a bot that triggers on “Adds” to a table and runs a “Send an email” task. The To field is set to a column ([Email]) that gets populated by an external script writing rows into the connected Google Sheet via the Sheets API.

What I’m seeing:

• When the row’s email address is my own account address, the email is delivered successfully.
• When the row’s email address is any other address (including other real addresses I personally control and monitor), no email is ever received - no bounce, not in spam/junk.
• Automation Monitor always shows the bot run and the “Send an email” step as “Complete,” regardless of recipient.

I came across this help article (Sending out of domain email from automations not supported - AppSheet Help) stating that as of May 4, 2026, accounts using a public email domain can only send automation emails to their own address unless the account predates that date or belongs to an existing Workspace org.

Questions for the community:

1. Has anyone else hit this on a paid Starter plan? Does upgrading to Core or Enterprise actually lift this restriction, or is it tied strictly to using a public domain (gmail.com) account regardless of plan?
2. Is switching to a Google Workspace account (custom domain) the only real workaround, or is there a way to get a public-domain account added to an allowlist?
3. Was this limitation something existing Starter subscribers were notified about, or is it a case of the plan being sold without this restriction being clearly disclosed?

Any guidance from people who’ve run into and solved this would be much appreciated.

4 posts - 4 participants

Read full topic

I am requesting help escalating the OAuth verification review for FacelessShorts. We have been waiting for several weeks, and repeated email follow-ups from 25 August through 10 September 2026 have not received a response.

As of 13 September 2026, the Verification Centre shows homepage and branding complete, last reviewed on 8 September. Privacy policy requirements remains in progress and says it is currently under review. App functionality, Appropriate data access, and Request minimum scopes are also in progress.

The delay is blocking us from fully enabling YouTube scheduling and publishing for our customers. We need this resolved as soon as possible and are ready to address any specific outstanding requirement promptly.

Could a Google staff member or moderator please help route this to the OAuth Trust and Safety verification team for urgent manual review? Please confirm whether the submission is actively assigned, identify any exact policy issue or action required from us, and provide an estimated completion time. If all requirements are satisfied, we would appreciate approval as soon as possible.

The Cloud support portal says OAuth is ineligible under our current Basic support plan, so I am seeking an alternative to the unanswered review emails. I can provide the project ID, review correspondence, and other account details through an official private support channel.

Thank you.

1 post - 1 participant

Read full topic