Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
ba83d54
refactor(ci): split PR triage into 4-job pipeline
yiliang114 Jun 8, 2026
99d2d0e
fix(ci): pass untrusted event data through env vars
yiliang114 Jun 8, 2026
cb1fa47
refactor(skills): update /triage to dispatch to 4 independent skills
yiliang114 Jun 8, 2026
23a9933
fix(ci): make tmux-testing depend on review (serial pipeline)
yiliang114 Jun 8, 2026
904a5de
fix(ci): implement file-based verdict passing between jobs
yiliang114 Jun 8, 2026
3b7e94b
fix(ci): review verdict gates tmux + add push delay
yiliang114 Jun 8, 2026
53ea4c3
fix(ci): address review findings (P1-P3)
yiliang114 Jun 9, 2026
7a04c8c
fix(ci): use env var for steps.run.outcome (style consistency)
yiliang114 Jun 9, 2026
6c492c7
fix(ci): resolve Copilot review comments
yiliang114 Jun 9, 2026
10ca42e
ci(triage): tune review timeouts and run product-decision on self-hos…
yiliang114 Jun 10, 2026
fa5b72b
ci(triage): render stream-json logs readably and front-load PR templa…
yiliang114 Jun 10, 2026
69c188f
fix(triage): harden self-hosted review findings
yiliang114 Jun 10, 2026
e2a36d9
fix(triage): close product gate bypass, real-time logs, per-job scrat…
yiliang114 Jun 10, 2026
880a306
ci(triage): use REVIEW_OPENAI_* credentials for product-decision
yiliang114 Jun 10, 2026
6e9cb7c
ci(triage): restore queued-acknowledgement for explicit comment triggers
yiliang114 Jun 10, 2026
30b0ca9
ci(triage): restore review-comment and review-body trigger surfaces
yiliang114 Jun 10, 2026
c45cf89
ci(triage): re-check PR state after the debounce wait
yiliang114 Jun 10, 2026
f37c8b7
ci(triage): restore internal-author gate and timeout_minutes input
yiliang114 Jun 10, 2026
07bd197
ci(triage): require open PR state for comment triggers in resolve
yiliang114 Jun 10, 2026
86e36d0
ci: extend qwen PR review timeout to 90min and queue delay to 30min
yiliang114 Jun 10, 2026
cad67c1
Merge remote-tracking branch 'origin/main' into worktree-triage-ci-re…
yiliang114 Jun 10, 2026
86dc442
Merge branch 'ci/extend-pr-review-timeouts' (PR #4962)
yiliang114 Jun 10, 2026
2453112
ci(triage): align debounce comments with the 30-minute wait timer
yiliang114 Jun 10, 2026
a1f6a03
ci(triage): single-quote inline if: conditions for yamllint
yiliang114 Jun 11, 2026
d75fcf5
fix(ci): harden triage pipeline gating and ECS network resilience
yiliang114 Jun 11, 2026
37ae8a5
Merge remote-tracking branch 'origin/main' into worktree-triage-ci-re…
yiliang114 Jun 11, 2026
eeb006a
fix(ci): raise approval-decision turn limit and tolerate exit-53
yiliang114 Jun 12, 2026
b5c25dd
fix(ci): remove checkout retries, add tmux PR state precheck
yiliang114 Jun 12, 2026
6c8464c
fix(ci): restore checkout retry with 3 attempts and stall detection
yiliang114 Jun 12, 2026
5e7e703
fix(ci): remove checkout retries and reduce fetch-depth
yiliang114 Jun 12, 2026
ab84bc4
fix(ci): fail-closed verdict defaults and cover tmux in fallback
yiliang114 Jun 12, 2026
b36ffae
fix(ci): prune stale review worktrees before checkout; simplify runne…
yiliang114 Jun 12, 2026
459f029
feat(ci): auto-trigger triage for all PRs including forks
yiliang114 Jun 12, 2026
df1613d
fix(ci): split agent analysis from privileged publish in triage
yiliang114 Jun 12, 2026
95b925f
fix(ci): use find instead of ls for review JSON fallback glob
yiliang114 Jun 12, 2026
a3c6e32
fix(ci): use review model secrets for approval stage
yiliang114 Jun 14, 2026
d050856
fix(ci): checkout workflow ref during PR triage dispatch
yiliang114 Jun 14, 2026
64a31af
fix(ci): fail triage on missing agent outputs
yiliang114 Jun 14, 2026
471e5a6
fix(ci): handle unavailable review results
yiliang114 Jun 14, 2026
55ad9e1
fix(ci): dedupe unavailable review notices
yiliang114 Jun 14, 2026
3e4a561
fix(review): prewrite emit-only fallback result
yiliang114 Jun 14, 2026
0bb5e25
fix(ci): harden review result publishing
yiliang114 Jun 14, 2026
e409461
Merge remote-tracking branch 'origin/main' into HEAD
yiliang114 Jun 14, 2026
b298f91
fix(ci): address triage review follow-ups
yiliang114 Jun 15, 2026
70dc200
fix(ci): harden triage pipeline triggers
yiliang114 Jun 15, 2026
89e2d94
fix(ci): tighten triage pipeline permissions
yiliang114 Jun 15, 2026
12070a3
fix(ci): require product decision comment output
yiliang114 Jun 15, 2026
0610d61
fix(ci): harden triage review gates
yiliang114 Jun 15, 2026
38e2527
fix(ci): harden approval publish and drop persisted checkout creds
yiliang114 Jun 15, 2026
c92d1ca
fix(ci): bypass proxy for qwen model calls
yiliang114 Jun 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(triage): close product gate bypass, real-time logs, per-job scrat…
…ch dirs

Local review round 2 findings:

- product-decision: propagate the qwen exit code at the end of the run step.
  Previously a timeout/crash left the step green, and when no verdict file was
  written the capture fallback defaulted to "pass" — silently waving the PR
  through the gate. Now the job fails and the verdict resolves to fail, matching
  the original action-based semantics.
- jq: add --unbuffered so progress lines stream into the CI log in real time
  instead of arriving in block-buffered chunks.
- Move runner scratch state from /tmp to runner.temp, which is emptied per job
  and is per runner instance: triage-results verdict dir (via TRIAGE_RESULTS_DIR
  env, skill falls back to /tmp locally), raw stream-json tee files, and the
  tmux-results artifact dir. /tmp persists across runs and is shared between
  runner instances on the same host, risking stale or cross-PR contamination.
  The approval-decision side stays on /tmp (ephemeral ubuntu-latest VM).
  • Loading branch information
yiliang114 committed Jun 10, 2026
commit e2a36d96b40760a2bd90e4782bd0adc761e9f071
41 changes: 24 additions & 17 deletions .github/workflows/qwen-pr-triage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,9 @@ env:
# jq program that renders qwen `--output-format stream-json` events into
# readable one-line CI log progress (tool calls + narration + final result)
# instead of raw JSON envelopes. Shared by the product-decision / review /
# tmux-testing steps via `qwen ... | tee raw.jsonl | jq -Rr "$STREAM_JSON_FMT"`.
# Each input line is one JSON message; `fromjson?` skips any non-JSON line.
# tmux-testing steps via `qwen ... | tee raw.jsonl | "${FMT_CMD[@]}"`, where
# FMT_CMD is `jq --unbuffered -Rr "$STREAM_JSON_FMT"` (or `cat` if jq is
# absent). Each input line is one JSON message; `fromjson?` skips non-JSON.
STREAM_JSON_FMT: |-
(fromjson?) as $m
| try (
Expand Down Expand Up @@ -155,6 +156,10 @@ jobs:
# direction reasoning. Empty if the runner var is unset (skill
# falls back to fetching the remote CHANGELOG).
CLAUDE_CODE_SRC: '${{ vars.CLAUDE_CODE_SRC_PATH }}'
# runner.temp is emptied per job and is per runner instance, unlike
# /tmp which persists across runs and is shared between runner
# instances on the same host (stale or cross-PR verdicts).
TRIAGE_RESULTS_DIR: '${{ runner.temp }}/triage-results'
run: |
set -euo pipefail

Expand All @@ -163,18 +168,13 @@ jobs:
exit 1
fi

# Self-hosted runner: /tmp persists across runs, so a stale verdict
# from a previous PR could be read by the capture step if this run's
# skill fails to write one. Clear it before the run.
rm -f /tmp/triage-results/product-decision.json

MODEL_ARGS=()
if [ -n "${OPENAI_MODEL:-}" ]; then
MODEL_ARGS=(--model "$OPENAI_MODEL")
fi

# Readable log formatter; fall back to raw passthrough if jq is absent.
FMT_CMD=(jq -Rr "$STREAM_JSON_FMT")
FMT_CMD=(jq --unbuffered -Rr "$STREAM_JSON_FMT")
command -v jq >/dev/null 2>&1 || FMT_CMD=(cat)

set +e
Expand All @@ -184,7 +184,7 @@ jobs:
"${MODEL_ARGS[@]}" \
--prompt "/product-decision ${PR_NUMBER} --repo ${REPOSITORY}" \
--output-format stream-json \
| tee /tmp/product-decision-raw.jsonl \
| tee "$RUNNER_TEMP/product-decision-raw.jsonl" \
| "${FMT_CMD[@]}"
EXIT_CODE=${PIPESTATUS[0]}
set -e
Expand All @@ -194,14 +194,19 @@ jobs:
elif [ "$EXIT_CODE" -ne 0 ]; then
echo "::warning::Product decision exited with code $EXIT_CODE"
fi
# Propagate: this stage is a gate. Without this, a timed-out or
# crashed run leaves the step green and the capture fallback would
# default the verdict to "pass", silently waving the PR through.
exit "$EXIT_CODE"

- name: 'Read verdict from skill output'
if: always()
id: 'capture'
env:
RUN_OUTCOME: '${{ steps.run.outcome }}'
TRIAGE_RESULTS_DIR: '${{ runner.temp }}/triage-results'
run: |
VERDICT_FILE="/tmp/triage-results/product-decision.json"
VERDICT_FILE="$TRIAGE_RESULTS_DIR/product-decision.json"
if [ -f "$VERDICT_FILE" ]; then
VERDICT=$(jq -r '.verdict // "pass"' "$VERDICT_FILE")
else
Expand Down Expand Up @@ -274,7 +279,7 @@ jobs:
fi

# Readable log formatter; fall back to raw passthrough if jq is absent.
FMT_CMD=(jq -Rr "$STREAM_JSON_FMT")
FMT_CMD=(jq --unbuffered -Rr "$STREAM_JSON_FMT")
command -v jq >/dev/null 2>&1 || FMT_CMD=(cat)

set +e
Expand All @@ -284,7 +289,7 @@ jobs:
"${MODEL_ARGS[@]}" \
--prompt "/review ${REVIEW_URL} --comment" \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Critical] The old review-pr job validated the qwen result stream event after process exit (~20 lines checking is_error, subtype == "success", and absence of [API Error in the result text). The new "Run review" step only checks exit code and whether the output JSON file exists with valid structure (type == "object" and .event is a string).

If the agent exits 0 but the model aborted mid-analysis (e.g., API connection dropped after writing a partial JSON), the file may carry event: "COMMENT" with an error-laced body or event: "APPROVE" based on incomplete analysis. The publish step submits it as-is. The PR receives a broken-looking review and the workflow shows green.

Suggested fix: After the qwen invocation, inspect the raw stream log for error indicators:

RESULT_LINE="$(grep '"type":"result"' "$RUNNER_TEMP/review-raw.jsonl" | tail -n1 || true)"
if [ -n "$RESULT_LINE" ]; then
  RESULT_IS_ERROR="$(printf '%s' "$RESULT_LINE" | jq -r '.is_error // false')"
  RESULT_SUBTYPE="$(printf '%s' "$RESULT_LINE" | jq -r '.subtype // ""')"
  if [ "$RESULT_IS_ERROR" = "true" ] || [ "$RESULT_SUBTYPE" != "success" ]; then
    write_review_unavailable "review agent ended in error state (subtype=${RESULT_SUBTYPE})"
    exit 0
  fi
fi

— qwen3.7-max via Qwen Code /review

--output-format stream-json \
| tee /tmp/review-raw.jsonl \
| tee "$RUNNER_TEMP/review-raw.jsonl" \
| "${FMT_CMD[@]}"
EXIT_CODE=${PIPESTATUS[0]}
set -e
Expand Down Expand Up @@ -362,19 +367,21 @@ jobs:
fi

# Readable log formatter; fall back to raw passthrough if jq is absent.
FMT_CMD=(jq -Rr "$STREAM_JSON_FMT")
FMT_CMD=(jq --unbuffered -Rr "$STREAM_JSON_FMT")
command -v jq >/dev/null 2>&1 || FMT_CMD=(cat)

# Run tmux testing — output goes to a file for artifact upload
mkdir -p /tmp/tmux-results
# Run tmux testing — output goes to a file for artifact upload.
# runner.temp (not /tmp): per-job and per-instance, so concurrent
# jobs on a shared host can't cross-contaminate the artifact.
mkdir -p "$RUNNER_TEMP/tmux-results"
set +e
timeout --kill-after=10s 15m qwen \
--auth-type openai \
--approval-mode yolo \
"${MODEL_ARGS[@]}" \
--prompt "/tmux-real-user-testing ${PR_NUMBER} --repo ${REPOSITORY}" \
--output-format stream-json \
| tee /tmp/tmux-results/output.jsonl \
| tee "$RUNNER_TEMP/tmux-results/output.jsonl" \
| "${FMT_CMD[@]}"
EXIT_CODE=${PIPESTATUS[0]}
set -e
Expand All @@ -394,7 +401,7 @@ jobs:
uses: 'actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02' # v4.6.2
with:
name: 'tmux-testing-results-${{ needs.resolve.outputs.pr_number }}'
path: '/tmp/tmux-results/'
path: '${{ runner.temp }}/tmux-results/'
retention-days: 7

# ─── Stage 3: Approval Decision ────────────────────────────────────
Expand Down
9 changes: 6 additions & 3 deletions .qwen/skills/product-decision/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,11 +142,14 @@ EXISTING=$(gh api "repos/$REPO/issues/$PR_NUMBER/comments" --jq '.[] | select(.b

### 7. Output Verdict

Write the verdict to a file so the CI workflow can read it:
Write the verdict to a file so the CI workflow can read it. In CI the
workflow sets `TRIAGE_RESULTS_DIR` (a per-job temp dir on the self-hosted
runner); fall back to `/tmp/triage-results` when running locally:

```bash
mkdir -p /tmp/triage-results
cat > /tmp/triage-results/product-decision.json << 'VERDICT_EOF'
RESULTS_DIR="${TRIAGE_RESULTS_DIR:-/tmp/triage-results}"
mkdir -p "$RESULTS_DIR"
cat > "$RESULTS_DIR/product-decision.json" << 'VERDICT_EOF'
{
"verdict": "pass",
"summary": "<one-line summary of decision>",
Expand Down