chore(function): send hits to the r2 data lake - #8550

Summary

Moves models.hit ingestion from the retained legacy S3 Tables lake (anomalyco/opencode#52514) into the platform R2 lake. S3 data and Athena access remain available to the internal dashboards; anomalyco/opencode#52515 stays on hold.

• The worker sends each opencode/bun hit to the platform lake event stream as {source: "models", type: "hit", timestamp, payload}, with method, path, useragent, ip, and cf_country in the payload. anomalyco/anomaly#674 projects these into a models.hit table. PostHog tracking is unchanged.
• The mapping lives in packages/core/src/hit.ts, so live hits and the backfill produce the same rows.
• Adds packages/core/script/backfill-hits.ts, which exports the hits already recorded in the S3 lake and replays them into the stream with their original timestamps.

Rollout

1. Deploy anomalyco/anomaly#674 to production so the models.hit table exists before hits arrive.
2. Set the new secrets on the dev stage, which serves models.dev. The endpoint is in the lake output of the anomalyco/platform/production stack, and the token needs the Workers Pipelines Send permission. Missing secrets fail deployment; follow-up fix(function): reject empty lake secrets #8579 also rejects empty token values and invalid stream endpoints.

   bunx sst secret set LakeEndpoint <endpoint> --stage dev
   bunx sst secret set LakeToken <token> --stage dev

3. Merge, then remove the old secrets with bunx sst secret remove LakeUrl --stage dev --fallback and bunx sst secret remove LakeSecret --stage dev --fallback (these legacy values were fallback secrets).

Backfill

Run this after the deploy and before the S3 lake is ever deleted. Wait for the old Worker to stop and Firehose to drain, then export every S3 hit. Do not use the first R2 hit as a cutoff: old and new Worker versions can overlap.

For small exports:

bun packages/core/script/backfill-hits.ts sql --to s3://opencode-production-lake-athena-results/backfill/models-hit/ --no-path
# Run the printed UNLOAD, download its data files, then:
bun packages/core/script/backfill-hits.ts send ./hits --dry-run
LAKE_ENDPOINT=<endpoint> LAKE_TOKEN=<token> bun packages/core/script/backfill-hits.ts send ./hits

The HTTP replay measures UTF-8 bytes, rejects invalid timestamps, validates the checkpoint against the endpoint and export hashes, and resumes accepted batches. Retries or a crash before checkpointing can duplicate submissions; reconcile counts before declaring completion.

The production source has approximately 2.6 billion hits. For that volume, packages/core/script/backfill-hits-glue.py reuses the existing Console generation-backfill Glue job with an overridden script location and separate checkpoint prefix. It reads a pinned S3 Tables snapshot, applies the same field mapping, and appends to both models.hit and default.event through the R2 Iceberg catalog. Each table append is atomic; daily destination counts recover a committed append after interruption and verify every historical day. A mismatched partial destination fails. Start with a bounded dev sample, then use DRY_RUN=true against production before the complete migration. Historical methods are null; path is null when absent in the S3 schema. Keep live producers running.

Validation

• Worker tests: 11 pass, including live event mapping, transient retry, and permanent rejection.
• Backfill integration tests: 4 pass, exercising actual subprocess replay, a 25,004-row failure/resume, changed export and endpoint rejection, multibyte batches, invalid timestamps, and complete exports.
• Package typechecks: same pre-existing 75 core and 10 function errors; no errors in the new runtime code or backfill files.
• Platform typecheck passed; targeted previews showed only the two new lake resources. Both dev and production deployments created those resources successfully.
• A 32,071-hit dev sample passed in both tables, followed by a production dry run. The full Glue backfill succeeded and reconciled all 122 historical days: 2,601,853,358 rows in each production R2 table, spanning 2026-06-02 18:21:46.964 UTC through 2026-10-01 19:25:37.136 UTC. An independent R2 SQL count and original timestamp bounds matched S3.
• Real models.dev requests were verified in both models.hit and default.event. S3 models.hit ingestion remains frozen; old fallback secrets were removed. fix(function): reject empty lake secrets #8579 merged and its worker deployment succeeded.
• An initial secret-setting command saved empty values despite reporting success. After correction and redeployment, 366,638 missing cutover hits were recovered from PostHog; 27,084 already-recorded S3/R2 hits were excluded using request fields and nearby timestamps. Typed row growth and raw unique payload counts matched the recovery export exactly. Recovered rows use PostHog receipt timestamps; 2,352 lack a recorded HTTP method and remain null. Raw payloads retain recovery provenance. Exact original Worker request-start timestamps for these recovered hits were not stored.
• Platform hosted deployment CI has a pre-existing Pulumi OIDC 401 failure. Dev and production were deployed locally, targeting only the two new resources, and production preview subsequently showed no changes.