PrometheusMonitor: use oc exec instead of oc rsh to avoid 16KB truncation - #31705

Summary

compat_otp.Monitor's Prometheus-querying methods (InstantQuery/SimpleQuery, RangeQuery, queryRules/GetAllRules/GetAlertRules/GetRecordRules, and GetAlerts) all fetch their response by running curl inside the prometheus-k8s pod via RemoteShPod, which executes the command with oc rsh.

Direct experimentation (see discussion on openshift/openshift-tests-private#30147) found that oc rsh-captured output gets hard-truncated at 16,384 bytes, while oc exec returns the complete response regardless of size:

On a cluster with enough concurrently firing/pending alerts to push /api/v1/alerts past 16KB, this makes consumers of GetAlerts() fail with an unparseable/truncated JSON response, deterministically, on every call — not an intermittent flake.

Fix

• Added RemoteExecPod() in pods.go: same signature shape as RemoteShPod, but runs the command via oc exec -n <namespace> <pod> -- <cmd...> instead of oc rsh.
• Switched all four Monitor/PrometheusMonitor query methods (InstantQuery, RangeQuery, queryRules, GetAlerts) to use RemoteExecPod instead of RemoteShPod.
• RemoteShPod itself is untouched, so this doesn't change behavior for any other caller.

Test plan

• go build ./test/extended/util/compat_otp/... passes
• Re-run an MCO alert test (e.g. TC 63865/63866 in openshift-tests-private) on a cluster with multiple concurrent alerts firing, to confirm GetAlerts() no longer truncates

Summary by CodeRabbit

• Bug Fixes
  • Prometheus queries and alert retrieval now preserve complete command output, including results larger than 16 KB.