Repository navigation
chore(stats): retire the S3 data lake - #52514
Merged
Merged
Conversation
Stats sync reads from the R2 lake, so stop feeding and reading the old AWS S3 Tables lake ahead of removing it: - Stop the log processor posting Zen events to the lake ingest service. - Drop the legacy Athena link and IAM permissions from stats sync, along with the unused Athena client. - Make the remaining lake resources deletable in every stage (forceDestroy, and opt the S3 buckets out of SST's production retain policy). Pulumi reads forceDestroy and retainOnDelete from state at delete time, so this must deploy to dev and production before the lake resources are removed from code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stats sync has read from the R2 lake since 46a14e6, so this stops feeding and reading the old AWS S3 Tables lake ahead of deleting it.
StatsSyncService, plus the unused Athena client and@aws-sdk/client-athena.forceDestroy: trueon the table bucket, both S3 buckets, and the Athena workgroup, andretainOnDelete: falseon the two S3 buckets so production'sremoval: "retain"doesn't orphan them.The ingest service keeps running for now because models.dev still posts to it, until anomalyco/models.dev#8550 moves those hits to the R2 lake.
Why two PRs
Pulumi reads
forceDestroyandretainOnDeletefrom state when it deletes a resource. Both were off in production, so removing the lake in a single deploy would fail on the Athena workgroup (it still has query history) and leave the S3 buckets behind. These settings have to deploy to dev and production before #52515.Rollout
models.hitevents to the R2 lake and backfills the hits recorded here.inferenceandanalyticsdashboards inanomalyco/anomalystill read its history.Validation
r2-sql.test.ts("database is locked" when bun:sqlite disposes), a file this PR doesn't touch./healthand/readyreturn 200) andstat-sync.tsbundles.