Skip to content

fix(pvc-resize): avoid deadlock on simultaneous storage and resource change - #10427

Merged
leonardoce merged 4 commits into
mainfrom
dev/9786
Apr 16, 2026
Merged

leonardoce merged 4 commits into
mainfrom
dev/9786

Conversation

@mnencia

@mnencia mnencia commented Apr 9, 2026 •

Copy link
Copy Markdown
Member

When both storage.size and resources are changed in a single patch, the operator resizes PVCs and then immediately starts a rolling update that deletes a pod. The orphaned PVC was classified as "resizing" instead of "dangling", preventing the pod from being recreated and leaving the cluster stuck.

Fix by always classifying podless resizing PVCs as "dangling" so a replacement pod is created. This is safe because volumes are usable during expansion and filesystem resize requires a mounted pod anyway.

Also remove the now-unused isFileSystemResizePending helper, which was only called from the removed conditional branch in classifyPVC.

Closes #9786

@mnencia
mnencia requested a review from a team April 9, 2026 16:39
@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Apr 9, 2026
@cnpg-bot cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.25 release-1.28 labels Apr 9, 2026
@github-actions

github-actions Bot commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

❗ By default, the pull request is configured to backport to all release branches.

  • To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
  • To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

@dosubot dosubot Bot added the bug 🐛 Something isn't working label Apr 9, 2026
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Apr 9, 2026
@mnencia

mnencia commented Apr 9, 2026

Copy link
Copy Markdown
Member Author

/test

@github-actions

github-actions Bot commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/24202483543

@cnpg-bot cnpg-bot added the ok to merge 👌 This PR can be merged label Apr 9, 2026
@dosubot dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Apr 12, 2026
@dosubot dosubot Bot added lgtm This PR has been approved by a maintainer size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Apr 14, 2026
mnencia and others added 3 commits April 16, 2026 13:54
…change

When both storage.size and resources are changed in a single PATCH,
the operator resizes PVCs and then immediately starts a rolling
update that deletes a pod. The orphaned PVC was classified as
"resizing" instead of "dangling", preventing the pod from being
recreated and leaving the cluster stuck.

Fix by always classifying podless resizing PVCs as "dangling" so a
replacement pod is created. This is safe because volumes are usable
during expansion and filesystem resize requires a mounted pod anyway.

Also remove the now-unused isFileSystemResizePending helper, which
was only called from the removed conditional branch in classifyPVC.

Closes #9786

Signed-off-by: Marco Nenciarini <[email protected]>
Move the "PVC detection" and "PVC classification with resizing PVCs"
test blocks to status_test.go, alongside the other tests for
functions defined in status.go (EnrichStatus, classifyPVC). Keep
only the BelongToInstance and tablespace tests in resources_test.go.

Signed-off-by: Marco Nenciarini <[email protected]>
Add integration test verifying that EnrichStatus classification flows
through findInstancePodToCreate to unblock pod recreation. Add unit
test for the resizing+Job ordering edge case in classifyPVC. Fix
inaccurate "running pod" comment and clarify existing test comments.

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
@leonardoce
leonardoce merged commit 9147871 into main Apr 16, 2026
43 checks passed
@leonardoce
leonardoce deleted the dev/9786 branch April 16, 2026 12:17
cnpg-bot pushed a commit that referenced this pull request Apr 16, 2026
…change (#10427)

When both storage.size and resources are changed in a single patch, the
operator resizes PVCs and then immediately starts a rolling update that
deletes a pod. The orphaned PVC was classified as "resizing" instead of
"dangling", preventing the pod from being recreated and leaving the
cluster stuck.

Fix by always classifying podless resizing PVCs as "dangling" so a
replacement pod is created. This is safe because volumes are usable
during expansion and filesystem resize requires a mounted pod anyway.

Also remove the now-unused isFileSystemResizePending helper, which was
only called from the removed conditional branch in classifyPVC.

Closes #9786

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Co-authored-by: Leonardo Cecchi <[email protected]>
(cherry picked from commit 9147871)
cnpg-bot pushed a commit that referenced this pull request Apr 16, 2026
…change (#10427)

When both storage.size and resources are changed in a single patch, the
operator resizes PVCs and then immediately starts a rolling update that
deletes a pod. The orphaned PVC was classified as "resizing" instead of
"dangling", preventing the pod from being recreated and leaving the
cluster stuck.

Fix by always classifying podless resizing PVCs as "dangling" so a
replacement pod is created. This is safe because volumes are usable
during expansion and filesystem resize requires a mounted pod anyway.

Also remove the now-unused isFileSystemResizePending helper, which was
only called from the removed conditional branch in classifyPVC.

Closes #9786

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Co-authored-by: Leonardo Cecchi <[email protected]>
(cherry picked from commit 9147871)
leonardoce added a commit that referenced this pull request Apr 16, 2026
…change (#10427)

When both storage.size and resources are changed in a single patch, the
operator resizes PVCs and then immediately starts a rolling update that
deletes a pod. The orphaned PVC was classified as "resizing" instead of
"dangling", preventing the pod from being recreated and leaving the
cluster stuck.

Fix by always classifying podless resizing PVCs as "dangling" so a
replacement pod is created. This is safe because volumes are usable
during expansion and filesystem resize requires a mounted pod anyway.

Also remove the now-unused isFileSystemResizePending helper, which was
only called from the removed conditional branch in classifyPVC.

Closes #9786

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Co-authored-by: Leonardo Cecchi <[email protected]>
(cherry picked from commit 9147871)
(cherry picked from commit 881706a)
sdwilsh pushed a commit to sdwilsh/ansible-playbooks that referenced this pull request May 11, 2026
##### [\`v1.29.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.29.1)

**Release date:** May 8, 2026

##### Security and Supply Chain

- **`CVE-2026-44477` / `GHSA-423p-g724-fr39`: metrics exporter privilege escalation**: the metrics exporter no longer authenticates as the `postgres` superuser. It now uses a dedicated `cnpg_metrics_exporter` role with `pg_monitor` privileges only, closing a chain that let a low-privilege database user gain PostgreSQL superuser. ([`GHSA-423p-g724-fr39`](GHSA-423p-g724-fr39)) <!-- 1.29 1.28 1.25 -->

  Upgrade impact: custom monitoring queries that read user-owned tables, or use `target_databases: '*'` against databases where `PUBLIC CONNECT` has been revoked, need explicit `GRANT` statements to `cnpg_metrics_exporter`. See ["Custom query privileges and safety"](../monitoring.md#custom-query-privileges-and-safety) and ["Manually creating the metrics exporter role"](../monitoring.md#manually-creating-the-metrics-exporter-role) in the monitoring documentation.

  For replica clusters, upgrade the source primary cluster before any replica clusters that consume from it. The `cnpg_metrics_exporter` role is created on the source primary and replicates downstream; a replica cluster upgraded first will scrape against a missing role until the source primary upgrades. The manual-recovery section linked above also covers replica clusters.

- **Schema-qualified catalog references in default monitoring queries**: hardened the shipped monitoring configuration and documentation samples by qualifying every `pg_catalog` object explicitly. Unqualified references resolve through `search_path`, which a database user can manipulate to shadow built-in objects. ([#10576](cloudnative-pg/cloudnative-pg#10576)) <!-- 1.29 1.28 1.25 -->

- **Discoverable SBOM and provenance attestations**: SBOM and SLSA provenance attached to operator container images now follow the OCI 1.1 Referrers spec, so standard registry tooling and supply-chain scanners can discover them automatically. ([#10601](cloudnative-pg/cloudnative-pg#10601)) <!-- 1.29 1.28 1.25 -->

- **CVE remediation in `github.com/jackc/pgx/v5`**: bumped to v5.9.2 to pick up upstream fixes for `CVE-2026-33816` (memory-safety in `pgproto3`) and `GHSA-j88v-2chj-qfwx` (SQL injection via simple-protocol dollar-quoted string handling). ([#10437](cloudnative-pg/cloudnative-pg#10437), [#10499](cloudnative-pg/cloudnative-pg#10499))

- **CVE remediation in the Go runtime**: built with Go 1.26.3 to pick up upstream fixes in `crypto/x509`, `crypto/tls`, `net/http`, and `net` (CVE-2026-32280, CVE-2026-32281, CVE-2026-33810, CVE-2026-33814, CVE-2026-33811, CVE-2026-39825). ([#10463](cloudnative-pg/cloudnative-pg#10463), [#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 -->

- **Build pipeline hardening**: the Go 1.26.3 bump also addresses CVE-2026-42501 (`cmd/go` module-checksum validation), reducing supply-chain exposure during release builds. The affected code paths are not reachable from the running operator. ([#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 -->

##### Changes

- Switched TLS peer verification from `VerifyPeerCertificate` to `VerifyConnection`, which runs on every completed handshake (the former is skipped on resumed TLS 1.3 sessions). Session resumption is not enabled in CloudNativePG today, so this has no observable effect, but it future-proofs verification if session caching is introduced later. ([#10478](cloudnative-pg/cloudnative-pg#10478)) <!-- 1.29 1.28 1.25 -->

##### Fixes

- Fixed a failover window where the former primary kept its primary label. If it returned during failover (for example, after a transient network partition), the `-rw` service kept routing to it, replicas could reconnect, and committed writes were lost to `pg_rewind`. The old primary is now labeled `unhealthy` to isolate it from service traffic during failover. ([#10409](cloudnative-pg/cloudnative-pg#10409)) <!-- 1.29 1.28 1.25 -->

- Fixed failover not being triggered when the node hosting the primary becomes unreachable. The operator now reads the pod's `Ready` condition (flipped to `False` by the node controller when the kubelet stops reporting) instead of `ContainersReady`, which stays stale as `True` in that scenario. Combined with the spurious-failover guard ([#10445](cloudnative-pg/cloudnative-pg#10445)), failover triggers only when Kubernetes itself marks the pod not Ready. ([#10448](cloudnative-pg/cloudnative-pg#10448)) <!-- 1.29 1.28 1.25 -->

- Fixed spurious failovers caused by transient failures on the primary's HTTP status endpoint. ([#10445](cloudnative-pg/cloudnative-pg#10445)) <!-- 1.29 1.28 1.25 -->

- Fixed escaping of backslashes and control characters in PostgreSQL configuration values. Previously, such characters in parameters like `log_line_prefix` could corrupt the configuration file or be silently stripped at runtime. ([#10515](cloudnative-pg/cloudnative-pg#10515)) <!-- 1.29 1.28 1.25 -->

- Fixed `restore_command` construction to shell-quote each argument. Values such as a `destinationPath` containing whitespace (for example, `s3://my bucket/wal`) were word-split by the POSIX shell and passed to the WAL restore tool as separate arguments. ([#10518](cloudnative-pg/cloudnative-pg#10518)) <!-- 1.29 1.28 1.25 -->

- Tightened `recoveryTarget` validation in the admission webhook: `targetXID` must now be a non-negative 32-bit integer, and `targetName` must be shorter than 64 bytes and free of ASCII control characters. Malformed values are rejected at admission instead of failing later during PostgreSQL recovery. ([#10565](cloudnative-pg/cloudnative-pg#10565)) <!-- 1.29 1.28 1.25 -->

- Fixed snapshot restores failing when leftover `pgsql_tmp*` directories were present in the data directory. ([#10447](cloudnative-pg/cloudnative-pg#10447)) <!-- 1.29 1.28 1.25 -->

- Fixed a deadlock occurring when PVC storage size and resource requests are changed simultaneously. ([#10427](cloudnative-pg/cloudnative-pg#10427)) <!-- 1.29 1.28 1.25 -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-requested ◀️ This pull request should be backported to all supported releases bug 🐛 Something isn't working lgtm This PR has been approved by a maintainer ok to merge 👌 This PR can be merged release-1.28 size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: invalid PATCH operation with storage and resource resize

4 participants