Skip to content

Storage → Strictly encode object keys for SigV4 - #129

Merged
revett merged 1 commit into
mainfrom
revett/fix/92
Jul 24, 2026
Merged

revett merged 1 commit into
mainfrom
revett/fix/92

Conversation

@revett

@revett revett commented Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

Resolves #92

Problem

  • Object keys were encoded with encodeURIComponent, which leaves ! ' ( ) * bare on the wire, while SigV4 signatures are computed over the strictly encoded RFC 3986 form
  • Whether that mismatch breaks depends on how each provider canonicalizes the path before verifying the signature, so a note named Don't forget!.md could fail to sync on some providers with an unexplained per file error

Changes

  • Percent encode ! ' ( ) * in object keys and in the prefix and continuation-token list params, so the wire path matches the signed canonical form byte for byte
  • Add unit tests for the encoder and an end to end round trip test covering all five characters through put, get, list, and delete

Why

  • Apostrophes and parentheses are extremely common in real note names, and sync must be boring for those too
  • Strict encoding removes any dependence on provider specific canonicalization, the same choice every AWS SDK makes; verified against MinIO before and after the change

Greptile Summary

This PR fixes a SigV4 signing mismatch by encoding the five characters (! ' ( ) *) that encodeURIComponent leaves bare but RFC 3986 / SigV4 requires to be percent-encoded, ensuring the bytes on the wire match the signed canonical form byte-for-byte across all providers.

  • Introduces encodeComponent (a thin wrapper over encodeURIComponent that also encodes the five SigV4-sensitive characters) and wires it into encodeKey and the prefix / continuation-token query parameters in s3ListObjects.
  • Adds unit tests for every encoding case and a full end-to-end integration test exercising all five characters through put, get, list, and delete.

Confidence Score: 5/5

Safe to merge — the change is a narrow, well-tested encoding fix with no behavioural regressions for callers.

The percentEncode helper produces the correct %XX form for all five target characters. The regex [!'()*] is a literal character class with no unintended matches. encodeComponent and encodeKey compose cleanly, and the switch in s3ListObjects is the only remaining call site that needed updating. Unit tests cover all encoding cases, and the integration test verifies the full round-trip through a real S3-compatible endpoint.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename Overview
src/storage/encode.ts Introduces encodeComponent (strict RFC 3986 / SigV4 encoding) and rewires encodeKey to use it; percentEncode helper is correct for all five target characters
src/storage/encode.test.ts New unit tests cover all five SigV4-sensitive characters, plain paths, non-ASCII, multi-segment keys, and the slash-vs-component distinction; good coverage
src/storage/storage.ts Switches prefix and continuation-token query parameters from encodeURIComponent to encodeComponent; change is minimal and consistent with the encoding fix
src/storage/storage.itest.ts Adds an end-to-end round-trip integration test covering all five characters through put, get, list (with prefix containing ' and /), and delete

Reviews (1): Last reviewed commit: "Fix" | Re-trigger Greptile

@revett
revett merged commit e2809e8 into main Jul 24, 2026
10 checks passed
@revett
revett deleted the revett/fix/92 branch July 24, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify object keys with special characters against real providers

1 participant