Skip to content

Sync → Refuse a first sync that would orphan remote files - #130

Merged
revett merged 2 commits into
mainfrom
revett/fix/109
Jul 24, 2026
Merged

revett merged 2 commits into
mainfrom
revett/fix/109

Conversation

@revett

@revett revett commented Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

Resolves #109

Problem

  • A missing manifest is treated as a clean first sync, so the pass pushes local files and writes a fresh manifest built purely from them
  • If the manifest was deleted while file objects survived (a bucket lifecycle rule, manual cleanup, a partial restore), every surviving object becomes invisible: absent from the new manifest, never pulled, never listed, orphaned forever
  • listObjects existed on the storage client but sync never consulted it

Changes

  • On a first sync, list the bucket and refuse the pass when it holds objects with no local file at their key, exactly the set a fresh manifest would strand; the status bar names the refusal and the log names each stranded key
  • Objects that do match a local path still proceed, so an interrupted first sync (files pushed, manifest upload never landed) keeps self healing on retry
  • A failed listing refuses the pass too, never guessed at as an empty bucket
  • Give the sync integration tests their own geode-sync-test bucket with a full wipe between scenarios, since the first sync check lists the whole bucket

Why

  • Silence must mean everything is fine; a sync that quietly strands remote files is the exact failure mode this project exists to rule out
  • Refusing loudly leaves every recovery open (restore the manifest, empty the bucket, copy the files into the vault), while proceeding forecloses them all silently

Greptile Summary

This PR closes the silent-orphan hole introduced when a manifest is deleted while bucket objects survive: a first sync now lists the bucket and refuses the pass when any object has no local counterpart, naming each stranded key. Objects that do match local files still proceed, so an interrupted first sync (files pushed, manifest never landed) heals correctly on retry.

  • orphanedKeys in sync.ts drives the new guard; syncOnce calls it only on first-sync paths, and a failed listing refuses rather than proceeding.
  • fakeStorage.listObjects is fixed to reflect the in-memory store instead of always returning empty, making the three new unit tests meaningful.
  • Sync integration tests move to a dedicated geode-sync-test bucket with a full wipe between scenarios, since the whole-bucket listing would trip the orphan refusal if leftover objects from other test files were present.

Confidence Score: 5/5

The change is additive and conservative: it only adds a refusal on the first-sync code path that previously had no listing call, and every new failure path returns early without touching the bucket.

The orphan detection logic is correct, all new failure modes return without side effects, the fakeStorage fix makes the unit tests genuinely exercise the new path, and the integration test isolation strategy is sound. No pre-existing behaviour on non-first-sync paths is touched.

No files require special attention.

Important Files Changed

Filename Overview
src/sync/sync.ts Adds orphanedKeys helper and first-sync bucket listing guard in syncOnce; logic is correct, ordering of snapshot-then-list is sound, and all failure paths return without side effects.
src/sync/fake.ts fakeStorage.listObjects now iterates the in-memory store instead of always returning empty; prefix filtering mirrors the real S3 implementation correctly.
src/sync/sync.test.ts Four new unit tests cover the orphan refusal, interrupted-first-sync pass-through, failed-listing refusal, and the orphanedKeys function directly; all assertions match the implementation.
src/sync/sync.itest.ts Migrated to dedicated geode-sync-test bucket with full wipe between scenarios; resetRemote now asserts on the listing result and the new orphan-refusal integration test is correct.
docker-compose.yml Adds geode-sync-test bucket creation alongside existing geode-test; shell syntax is correct.
CONTRIBUTING.md Documentation updated to describe both test buckets and why they are separate; accurate.

Reviews (2): Last reviewed commit: "Address comment" | Re-trigger Greptile

Comment thread src/sync/sync.itest.ts
@revett
revett merged commit 3056007 into main Jul 24, 2026
10 checks passed
@revett
revett deleted the revett/fix/109 branch July 24, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Missing manifest with a non-empty bucket is treated as a clean first sync

1 participant