Repository navigation
fix(pvc-resize): avoid deadlock on simultaneous storage and resource change - #10427
Merged
Merged
Conversation
Contributor
|
❗ By default, the pull request is configured to backport to all release branches.
|
Member
Author
|
/test |
Contributor
|
@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/24202483543 |
armru
approved these changes
Apr 14, 2026
leonardoce
approved these changes
Apr 16, 2026
…change When both storage.size and resources are changed in a single PATCH, the operator resizes PVCs and then immediately starts a rolling update that deletes a pod. The orphaned PVC was classified as "resizing" instead of "dangling", preventing the pod from being recreated and leaving the cluster stuck. Fix by always classifying podless resizing PVCs as "dangling" so a replacement pod is created. This is safe because volumes are usable during expansion and filesystem resize requires a mounted pod anyway. Also remove the now-unused isFileSystemResizePending helper, which was only called from the removed conditional branch in classifyPVC. Closes #9786 Signed-off-by: Marco Nenciarini <[email protected]>
Move the "PVC detection" and "PVC classification with resizing PVCs" test blocks to status_test.go, alongside the other tests for functions defined in status.go (EnrichStatus, classifyPVC). Keep only the BelongToInstance and tablespace tests in resources_test.go. Signed-off-by: Marco Nenciarini <[email protected]>
Add integration test verifying that EnrichStatus classification flows through findInstancePodToCreate to unblock pod recreation. Add unit test for the resizing+Job ordering edge case in classifyPVC. Fix inaccurate "running pod" comment and clarify existing test comments. Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
cnpg-bot
pushed a commit
that referenced
this pull request
Apr 16, 2026
…change (#10427) When both storage.size and resources are changed in a single patch, the operator resizes PVCs and then immediately starts a rolling update that deletes a pod. The orphaned PVC was classified as "resizing" instead of "dangling", preventing the pod from being recreated and leaving the cluster stuck. Fix by always classifying podless resizing PVCs as "dangling" so a replacement pod is created. This is safe because volumes are usable during expansion and filesystem resize requires a mounted pod anyway. Also remove the now-unused isFileSystemResizePending helper, which was only called from the removed conditional branch in classifyPVC. Closes #9786 Signed-off-by: Marco Nenciarini <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Leonardo Cecchi <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> Co-authored-by: Leonardo Cecchi <[email protected]> (cherry picked from commit 9147871)
cnpg-bot
pushed a commit
that referenced
this pull request
Apr 16, 2026
…change (#10427) When both storage.size and resources are changed in a single patch, the operator resizes PVCs and then immediately starts a rolling update that deletes a pod. The orphaned PVC was classified as "resizing" instead of "dangling", preventing the pod from being recreated and leaving the cluster stuck. Fix by always classifying podless resizing PVCs as "dangling" so a replacement pod is created. This is safe because volumes are usable during expansion and filesystem resize requires a mounted pod anyway. Also remove the now-unused isFileSystemResizePending helper, which was only called from the removed conditional branch in classifyPVC. Closes #9786 Signed-off-by: Marco Nenciarini <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Leonardo Cecchi <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> Co-authored-by: Leonardo Cecchi <[email protected]> (cherry picked from commit 9147871)
leonardoce
added a commit
that referenced
this pull request
Apr 16, 2026
…change (#10427) When both storage.size and resources are changed in a single patch, the operator resizes PVCs and then immediately starts a rolling update that deletes a pod. The orphaned PVC was classified as "resizing" instead of "dangling", preventing the pod from being recreated and leaving the cluster stuck. Fix by always classifying podless resizing PVCs as "dangling" so a replacement pod is created. This is safe because volumes are usable during expansion and filesystem resize requires a mounted pod anyway. Also remove the now-unused isFileSystemResizePending helper, which was only called from the removed conditional branch in classifyPVC. Closes #9786 Signed-off-by: Marco Nenciarini <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Leonardo Cecchi <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> Co-authored-by: Leonardo Cecchi <[email protected]> (cherry picked from commit 9147871) (cherry picked from commit 881706a)
sdwilsh
pushed a commit
to sdwilsh/ansible-playbooks
that referenced
this pull request
May 11, 2026
##### [\`v1.29.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.29.1) **Release date:** May 8, 2026 ##### Security and Supply Chain - **`CVE-2026-44477` / `GHSA-423p-g724-fr39`: metrics exporter privilege escalation**: the metrics exporter no longer authenticates as the `postgres` superuser. It now uses a dedicated `cnpg_metrics_exporter` role with `pg_monitor` privileges only, closing a chain that let a low-privilege database user gain PostgreSQL superuser. ([`GHSA-423p-g724-fr39`](GHSA-423p-g724-fr39)) <!-- 1.29 1.28 1.25 --> Upgrade impact: custom monitoring queries that read user-owned tables, or use `target_databases: '*'` against databases where `PUBLIC CONNECT` has been revoked, need explicit `GRANT` statements to `cnpg_metrics_exporter`. See ["Custom query privileges and safety"](../monitoring.md#custom-query-privileges-and-safety) and ["Manually creating the metrics exporter role"](../monitoring.md#manually-creating-the-metrics-exporter-role) in the monitoring documentation. For replica clusters, upgrade the source primary cluster before any replica clusters that consume from it. The `cnpg_metrics_exporter` role is created on the source primary and replicates downstream; a replica cluster upgraded first will scrape against a missing role until the source primary upgrades. The manual-recovery section linked above also covers replica clusters. - **Schema-qualified catalog references in default monitoring queries**: hardened the shipped monitoring configuration and documentation samples by qualifying every `pg_catalog` object explicitly. Unqualified references resolve through `search_path`, which a database user can manipulate to shadow built-in objects. ([#10576](cloudnative-pg/cloudnative-pg#10576)) <!-- 1.29 1.28 1.25 --> - **Discoverable SBOM and provenance attestations**: SBOM and SLSA provenance attached to operator container images now follow the OCI 1.1 Referrers spec, so standard registry tooling and supply-chain scanners can discover them automatically. ([#10601](cloudnative-pg/cloudnative-pg#10601)) <!-- 1.29 1.28 1.25 --> - **CVE remediation in `github.com/jackc/pgx/v5`**: bumped to v5.9.2 to pick up upstream fixes for `CVE-2026-33816` (memory-safety in `pgproto3`) and `GHSA-j88v-2chj-qfwx` (SQL injection via simple-protocol dollar-quoted string handling). ([#10437](cloudnative-pg/cloudnative-pg#10437), [#10499](cloudnative-pg/cloudnative-pg#10499)) - **CVE remediation in the Go runtime**: built with Go 1.26.3 to pick up upstream fixes in `crypto/x509`, `crypto/tls`, `net/http`, and `net` (CVE-2026-32280, CVE-2026-32281, CVE-2026-33810, CVE-2026-33814, CVE-2026-33811, CVE-2026-39825). ([#10463](cloudnative-pg/cloudnative-pg#10463), [#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 --> - **Build pipeline hardening**: the Go 1.26.3 bump also addresses CVE-2026-42501 (`cmd/go` module-checksum validation), reducing supply-chain exposure during release builds. The affected code paths are not reachable from the running operator. ([#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 --> ##### Changes - Switched TLS peer verification from `VerifyPeerCertificate` to `VerifyConnection`, which runs on every completed handshake (the former is skipped on resumed TLS 1.3 sessions). Session resumption is not enabled in CloudNativePG today, so this has no observable effect, but it future-proofs verification if session caching is introduced later. ([#10478](cloudnative-pg/cloudnative-pg#10478)) <!-- 1.29 1.28 1.25 --> ##### Fixes - Fixed a failover window where the former primary kept its primary label. If it returned during failover (for example, after a transient network partition), the `-rw` service kept routing to it, replicas could reconnect, and committed writes were lost to `pg_rewind`. The old primary is now labeled `unhealthy` to isolate it from service traffic during failover. ([#10409](cloudnative-pg/cloudnative-pg#10409)) <!-- 1.29 1.28 1.25 --> - Fixed failover not being triggered when the node hosting the primary becomes unreachable. The operator now reads the pod's `Ready` condition (flipped to `False` by the node controller when the kubelet stops reporting) instead of `ContainersReady`, which stays stale as `True` in that scenario. Combined with the spurious-failover guard ([#10445](cloudnative-pg/cloudnative-pg#10445)), failover triggers only when Kubernetes itself marks the pod not Ready. ([#10448](cloudnative-pg/cloudnative-pg#10448)) <!-- 1.29 1.28 1.25 --> - Fixed spurious failovers caused by transient failures on the primary's HTTP status endpoint. ([#10445](cloudnative-pg/cloudnative-pg#10445)) <!-- 1.29 1.28 1.25 --> - Fixed escaping of backslashes and control characters in PostgreSQL configuration values. Previously, such characters in parameters like `log_line_prefix` could corrupt the configuration file or be silently stripped at runtime. ([#10515](cloudnative-pg/cloudnative-pg#10515)) <!-- 1.29 1.28 1.25 --> - Fixed `restore_command` construction to shell-quote each argument. Values such as a `destinationPath` containing whitespace (for example, `s3://my bucket/wal`) were word-split by the POSIX shell and passed to the WAL restore tool as separate arguments. ([#10518](cloudnative-pg/cloudnative-pg#10518)) <!-- 1.29 1.28 1.25 --> - Tightened `recoveryTarget` validation in the admission webhook: `targetXID` must now be a non-negative 32-bit integer, and `targetName` must be shorter than 64 bytes and free of ASCII control characters. Malformed values are rejected at admission instead of failing later during PostgreSQL recovery. ([#10565](cloudnative-pg/cloudnative-pg#10565)) <!-- 1.29 1.28 1.25 --> - Fixed snapshot restores failing when leftover `pgsql_tmp*` directories were present in the data directory. ([#10447](cloudnative-pg/cloudnative-pg#10447)) <!-- 1.29 1.28 1.25 --> - Fixed a deadlock occurring when PVC storage size and resource requests are changed simultaneously. ([#10427](cloudnative-pg/cloudnative-pg#10427)) <!-- 1.29 1.28 1.25 -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When both storage.size and resources are changed in a single patch, the operator resizes PVCs and then immediately starts a rolling update that deletes a pod. The orphaned PVC was classified as "resizing" instead of "dangling", preventing the pod from being recreated and leaving the cluster stuck.
Fix by always classifying podless resizing PVCs as "dangling" so a replacement pod is created. This is safe because volumes are usable during expansion and filesystem resize requires a mounted pod anyway.
Also remove the now-unused isFileSystemResizePending helper, which was only called from the removed conditional branch in classifyPVC.
Closes #9786