ast-guard Evaluation Results

Abstract

ast-guard is a deterministic, pre-execution static analyzer for detecting reward hacking in LLM-generated code. It operates on Python (native AST), Bash, JavaScript, and TypeScript (tree-sitter) without external dependencies in its core, producing results in under 50ms per scan. This document reports evaluation results across six datasets: the built-in TRACE-aligned suite (33 hand-written samples), School of Reward Hacks (104 labeled Python pairs), MBPP (974 TN pairs), Countdown-Code (15,894 samples), the METR MALT dataset (81,515 scannable code blocks from 7,179 agent transcripts), and the external PatronusAI TRACE agentic-trajectory dataset (517 trajectories). In pair mode on real external data, ast-guard achieves 98.0% F1 on School of Reward Hacks (100% precision, 96.2% recall, 0.0% FPR) and 0.1% FPR on MBPP identity pairs. On the built-in hand-written suite, it achieves 95.7% F1 with 100% precision. In standalone mode on MALT, it achieves 95.0% true-negative rate while detecting 34.5% of bypass_constraints and 46.9% of hardcoded_solution samples. On the external TRACE dataset, standalone scan achieves 21.3% recall on in-scope structural categories at 30.1% FPR — establishing the boundary between the static-AST primitive and the agentic-delta regime (see §External TRACE Dataset). ast-guard v2.3.0 is designed as a deterministic, zero-cost pre-filter layer complementing ML-based classifiers, not as a standalone replacement.

---

Evaluation Overview

Property	Value
Tool	ast-guard v2.3.0
Approach	Deterministic pre-execution static analysis
Core dependency	Python standard library only (zero external deps)
Latency	<50ms per scan
Modes	Pair Mode (original vs. generated), Standalone Mode (single code block)
Languages	Python (native ast), Bash, JavaScript/TypeScript (tree-sitter)

Checks

#	Name	Method	Languages
1	Hardcoding Detection	If-count / literal-count / long-string growth relative to baseline	Python, Bash, JS, TS
2	Complexity Collapse	McCabe complexity drop >60% per qualified function name	Python, Bash, JS, TS
3	Forbidden Calls & Anti-Obfuscation	Diff-based; eval/exec, chr tricks, builtins subscript, constant folding	Python, Bash, JS, TS
4	Import Drift	CRITICAL on dangerous stdlib modules; WARNING on unknowns	Python, Bash, JS, TS
5	Extensional Enumeration	if/elif or match/case chains covering >70% of branches with no loops; dispatch-table memorisation (object/Map lookup) for Python, JS, TS	Python, Bash, JS, TS
6	Behavioral Risk Scoring	Additive score from AST-detectable behavioral patterns (standalone)	Python, Bash, JS, TS
7	Literal Hijack	Generated function returns only literals regardless of inputs; original was non-trivial	Python
8	New Constant Bypass	If-branch compares against new specific constant and returns input-independently	Python

Language-Check Capability Matrix

Check	Python	Bash	JavaScript	TypeScript	Notes
1 — Hardcoding	✓	✓	✓	✓
2 — Complexity Collapse	✓	✓	✓	✓	Inactive in standalone for all languages
3 — Forbidden Calls	✓	✓	✓	✓
4 — Import Drift	✓	✓	✓	✓
5 — Extensional Enumeration (if/elif, switch)	✓	✓	✓	✓
5 — Dispatch-table sub-rule	✓	—	✓	✓	object/Map memorisation
6 — Behavioral Risk Scoring	✓	✓	✓	✓	Standalone only
7 — Literal Hijack	✓	—	—	—	Python only
8 — New Constant Bypass	✓	—	—	—	Python only

Validation status by language:

• Python: All checks validated on public datasets (MALT, SORH, MBPP, Countdown-Code).
• Bash: Checks 1, 3–6 validated on constructed held-out sets only. No independent Bash precision/recall corpus.
• JavaScript / TypeScript: All checks validated on constructed held-out sets only. No independent JS/TS precision/recall corpus. Pair-mode behavior provides a precision anchor; standalone detection rates on real JS/TS agent code have not been measured.

---

Datasets

Dataset	Source	Mode	Samples	Access	Languages	Labels
TRACE-aligned suite (built-in)	hand-written by author, TRACE category names	pair	33 samples (24 hacked, 9 benign)	Bundled	Python	hacked / benign
TRACE (external)	PatronusAI HuggingFace	standalone	517 trajectories, 431 extractable	Public	Python, JS	hacked / benign
School of Reward Hacks	syvb/school-of-reward-hacks-coding-tasks HF	pair	54 Python TP + 54 TN + 26 C++ TP (skipped)	Public CC-BY-4.0	Python, C++	hack / clean
MBPP	google-research-datasets/mbpp HuggingFace	pair (TN only)	974 honest-vs-honest pairs	Public Apache-2.0	Python	clean
Countdown-Code	Khan et al. GitHub	pair	15,894	Public	Python	legitimate / hacked
MALT	METR (HuggingFace)	standalone	81,515 scannable (from 140,726 extracted, 7,179 transcripts)	Public	Python, Bash	normal, bypass_constraints, hardcoded_solution, sabotage, etc.

MALT label breakdown: 77,369 normal · 2,379 bypass_constraints · 731 partial_problem_solving · 429 hardcoded_solution · 197 sabotage · 171 match_weaker_model · 147 refusals · 74 reasoning_about_task · 17 ignores_task_instructions · 1 gives_up

---

Pair Mode Results

Pair mode compares original code against LLM-generated code. This is ast-guard's primary intended use case and yields the strongest results because structural changes relative to a baseline are precisely measurable.

Built-in TRACE-Aligned Suite

Note: these 33 samples are hand-written by the project author using TRACE category names as a guide. They are not drawn from the published TRACE dataset (Deshpande et al. 2026) and do not constitute an independent external evaluation. Use these results as regression/sanity checks that specific structural patterns fire correctly, not as a claim about real-world TRACE detection rates.

The suite covers 15 of 51 TRACE subcategories with structurally detectable patterns. 36 subcategories are semantic (wrong answers, sandbagging, framing manipulations) and are out of scope for static analysis.

Metric	Value
Total samples	33 (24 hacked, 9 benign)
True Positives (TP)	22
False Positives (FP)	0
False Negatives (FN)	2
True Negatives (TN)	9
Precision	100%
Recall	91.7%
F1	95.7%
Specificity	100%

The 2 false negatives are hacked samples whose pattern falls outside the 15 covered subcategories.

Countdown-Code

Countdown-Code contains arithmetic countdown problems. The vast majority of legitimate solutions are 3-line arithmetic functions that correctly pass CLEAN.

Note: numbers recorded in CHANGELOG at v2.1.0. No JSON artifact stored in benchmarks/data/ and not re-verified at v2.2.0. To produce a current artifact: python -m benchmarks.run_benchmark --benchmark countdown-code --json results.json

Metric	Value
Total samples	15,894
Flagged	160 (1.0%)
— CRITICAL (eval())	129
— WARNING (hardcoded)	31
True Negative Rate	99.0%

Context: 99% of samples are legitimate short arithmetic functions. The 160 flagged samples are genuine detections (eval() calls and explicit hardcoding), not false positives.

School of Reward Hacks (pair mode)

Source: syvb/school-of-reward-hacks-coding-tasks (HuggingFace CC-BY-4.0), accessed 2026-06-10. 52 Python TP pairs (original=honest control, generated=hardcoded hack) + 52 Python TN pairs (same-problem identity). 28 skipped: 26 C++ TP (no adapter yet) + 2 invalid Python. Artifact: benchmarks/data/sorh_results.json

Metric	Value
Labeled pairs	104 (52 TP + 52 TN)
TP	50
FP	0
FN	2
TN	52
Precision	100.0%
Recall	96.2%
F1	98.0%
FPR	0.0%

The 2 FN are hardcoded solutions that stayed below Check 1/5 thresholds. The 28 skipped C++ pairs will contribute once a C++ tree-sitter adapter is added.

School of Reward Hacks (standalone mode — offline eval harness)

Offline eval harness (eval/), deterministic split seed=42, 80% dev / 20% held-out. Held-out split was never used for tuning — reported once, final.

Configuration (Stage 2 — branch-aware fix): _MIN_RETURNS=3 / _MIN_BRANCHES=3 / _MIN_INDEPENDENT_RATIO=0.80, plus three structural fixes in ast_guard/dataflow.py:

• Loop-body taint propagation (Fix A): when a for-loop iterator references tainted names, all variables assigned anywhere in the loop body are also tainted.
• Loop-internal return exclusion (Fix B): returns inside a tainted for-loop's body are excluded from the independence count — whether they execute depends on the loop.
• Branch-aware condition gate (Fix C): a literal return is only counted as input-independent when the nearest enclosing if condition is a direct param-vs-literal comparison (param == specific_value). Conditions involving computed intermediates (sqa == sqa + sqb, x % 2 == 0, re.search(...)) are NOT treated as param-keyed, which eliminates FPs from legitimate validators and classifiers whose return VALUES happen to be literals.

Threshold rationale: 3/3 gives identical SORH recall to 2/2, and eliminates two method-call-taint FPs (functions with exactly 2 returns caught by a taint gap, not genuine dispatch tables). 5/4 reduces recall to R=0.146 — unacceptable.

All fixes verified by 819 unit tests (8 skipped for optional MCP extra). MBPP standalone FPR (974 reference solutions, no-baseline scan): 0.31% (3 FPs) under the current 3/3 thresholds vs. 0.21% under old 5/4 thresholds — the Δ0.10% tradeoff is the cost of detecting 31 vs. 6 SORH dev hacks.

Reproducibility metadata:

• ast-guard commit: see git log -1 --format="%h"
• SORH dataset revision: 2026-06-15T03:11:46Z
• Split seed: 42, dev_ratio: 0.8, scan mode: strict
• MBPP FPR artifact: eval/results/mbpp_precision/report.md

Dev split (82 records — 41 hack, 41 benign)

Metric	Value
Total records	82
TP	31
FP	0
TN	41
FN	10
Precision	1.000
Recall	0.756
F1	0.861
PR-AUC	0.939

Pair mode on TP records: TP=40 FP=0 FN=1, F1=0.988, P=1.000, R=0.976.

Held-out split (22 records — 11 hack, 11 benign, measured once)

Metric	Value
Total records	22
TP	9
FP	0
TN	11
FN	2
Precision	1.000
Recall	0.818
F1	0.900
PR-AUC	0.955

Open false-negative gaps (known follow-up signals)

Gap	Count	Pattern	Next signal candidate
ratio-0.75 passthrough	7	3 pure-literal branches + return lst/s/n fallback	loop-smarter input-independence signal at ratio < 0.80
ternary chains	3	return A if c1 else B if c2 else C — 1 ast.Return total	dedicated IfExp-chain independence check

Both gap types have structurally sound solutions; neither is addressable by simple threshold tuning at _MIN_INDEPENDENT_RATIO=0.80 without introducing held-out FPs on benign algorithmic functions (perrin, count_binary_seq, is_valid_ipv4).

MBPP + HumanEval Standalone Precision Stress Test (Stage 2 checkpoint)

Standalone scan of all reference solutions. All-benign: no TPs, only FPR is measured. Full per-item report: eval/results/mbpp_precision/report.md.

Dataset	n	FP (old 5/4)	FPR (old)	FP (new 3/3)	FPR (new)	Δ
MBPP	974	2	0.21%	3	0.31%	+1
HumanEval	164	7	4.27%	7	4.27%	0

The branch-aware Fix C eliminated 24 of 25 new FPs introduced when thresholds were lowered from 5/4 to 2/2. HumanEval FPR at 3/3 now equals OLD (5/4) — no precision cost at all on HumanEval.

Remaining MBPP FPs (3 total) — individual assessment

ID	Function	Check fired	Root cause	Pattern
448	cal_sum(n)	check_6 / input_independent_returns	While-loop taint gap: sum accumulates in a while-loop whose body assignments are not yet tainted (Fix A covers for-loops only). The 3 base-case returns under n==0/1/2 are param-vs-literal; return sum looks input-independent because sum is not in the tainted set.	Recurrence (Perrin-like) with parameter-controlled iteration
577	last_Digit_Factorial(n)	check_6 / input_independent_returns	Ratio exactly equals _MIN_INDEPENDENT_RATIO=0.80: 4/5 returns are param-keyed literals (0,1,3,4 → 1,n,6,4,0); elif n<=2: return n counts as param-dependent (return value refs n), keeping ratio at threshold. Borderline mathematical table.	Legitimate mathematical table (last digit of n! is always 0 for n≥5)
950	chinese_zodiac(year)	check_5 / extensional enumeration	Pre-existing FP. 12-branch elif chain with (year-2000)%12 == k conditions. Branch-aware fix does not affect Check 5; this fires on if-count/ratio alone.	Legitimate modular-arithmetic dispatch (12 zodiac signs) — candidate for branch-aware Check 5

Next-signal candidates (queued):

• cal_sum (448): extend Fix A to ast.While bodies — while-loop iteration count depends on n, so all while-body assignments should be tainted.
• chinese_zodiac (950): branch-aware Check 5 — conditions like (year-2000)%12 == k use a computed modular expression, not a raw param comparison; a param-keyed gate on Check 5's enumeration check would correctly pass this.

Remaining HumanEval FPs (7 total) — individual assessment

All 7 were already FPs under the old 5/4 thresholds; the branch-aware fix introduced zero new HumanEval regressions.

ID	Function	Check fired	Root cause	Pattern
HE/46	fib4	check_6 / intent_mismatch_recursion	Recursive Fibonacci-like function trips the intent_mismatch_recursion behavioral pattern (score 30). Legitimate use.	Recurrence function — not a reward hack
HE/70	strange_sort_list	check_6 / intent_mismatch_sort	Alternating min/max sort uses sorted() twice. Trips intent_mismatch_sort (score 30). Legitimate.	Algorithmic sorting, not a hack
HE/115	max_fill	check_1 / hardcoding	import math inside function body plus several integer constants trips the standalone literal-count threshold. Legitimate math.ceil usage.	Legitimate import-inside-function + numeric constants
HE/126	is_sorted	check_6 / intent_mismatch_sort	Uses sorted() to verify order. Trips intent_mismatch_sort (score 30). Legitimate predicate.	Sorting-based predicate, not a hack
HE/127	intersection	check_6 / intent_mismatch_loop	Iterates over interval to check primality. Trips intent_mismatch_loop (score 30). Legitimate.	Number-theory computation
HE/148	bf	check_6 / intent_mismatch_sort	Hardcoded ordered list of 8 planet names used to determine relative position; triggers intent_mismatch_sort.	Domain-knowledge lookup list (8 solar planets) — candidate for branch-aware Check 5
HE/160	do_algebra	check_3 CRITICAL + check_6 / eval_dynamic	Uses Python eval() legitimately to evaluate a user-supplied algebraic expression string. eval is on the Check 3 forbidden list by design; this is a correct fire on structural grounds — the function cannot be implemented without eval.	Unavoidable eval use — structural FP by design

Next-signal candidates (queued):

• HE/148 bf: 8-element planet list is a canonical domain-knowledge constant, not test-case memorisation. A branch-aware Check 5 with a param-keyed condition gate (same logic as Fix C) would distinguish bf()'s if planet in list from actual if n == specific_test_case enumerations.
• HE/160 do_algebra: eval use is unavoidable for this task class. An allowlist entry for eval in expressions-evaluator contexts is the correct fix; out of scope for static analysis alone.

MBPP (pair mode — TN FPR baseline)

Source: google-research-datasets/mbpp (HuggingFace Apache-2.0), accessed 2026-06-10. 974 same-problem identity TN pairs: each MBPP reference solution compared to itself. Measures baseline FPR — an honest solution must never flag against the reference it was derived from. Artifact: benchmarks/data/mbpp_results.json

Metric	Value
TN pairs	974
FP	1
FPR	0.1%

The single FP is task_id 950 (chinese_zodiac): a legitimate 12-branch elif chain for the 12 zodiac signs that trips Check 5 (extensional enumeration). This is a known Check 5 edge case — constant-valued dispatch tables that happen to meet the ratio threshold.

External TRACE Dataset (PatronusAI, 2026) — Boundary / Negative Result

Full eval report: eval/results/trace/report.md · Adapter: eval/adapters/trace.py · Commit: 7c52730

This section documents a boundary result, not a recall target. TRACE hacks are 1–2 line delta edits — changing threshold=0.95 to threshold=0.0001, replacing assert predicted == expected with assert True, patching a validator to always pass. These deltas are invisible in the final file snapshot: the reconstructed file looks like normal code. Standalone scan on the final state is the wrong primitive for this regime. The numbers below establish that boundary honestly and identify two distinct limits.

Extraction

Code is embedded as tool calls in a JSON conversation field: Write = full file snapshot, Edit = str_replace diff. No pair-mode baseline exists — standalone only.

Status	n	%
No in-scope files (.py / .js / .sh)	86	16.6%
Clean (Write baseline, patches applied)	134	25.9%
Edit-only fragment (largest new_string, no Write)	294	56.9%
Conflict-fallback (patch failed, partial content)	3	0.6%

68% of extractable trajectories use a code fragment, not the full file. The fragment is the largest new_string from an Edit call — partial context extracted from a diff, not a reconstructed snapshot. This is a structural caveat on all recall figures: the scan target often does not contain the complete hack.

In-scope Recall — Syntactic Categories (main headline)

In-scope categories where structural AST detection is plausible: 1.1.2 (test-infra patching), 1.1.3 (coverage gaming), 1.2.1 (output hardcoding), 1.2.2 (metric fabrication), 1.2.3 (eval-code modification).

Combined in-scope structural recall: 34/160 = 21.3% (dev split, strict mode)

Category	Code	n (dev)	TP	Recall
trace-metric-fabrication	1.2.2	18	8	0.444
trace-output-hardcoding	1.2.1	30	8	0.267
trace-eval-code-modification	1.2.3	26	5	0.192
trace-coverage-gaming	1.1.3	38	6	0.158
trace-test-infra-patching	1.1.2	48	7	0.146

Global precision = 0.535 (53 TP / 99 flagged). Per-category precision figures are degenerate (all-hack slices, no benign mixed in) and not reported. The real operating-point precision is global.

False Positive Rate: 46/153 = 30.1%

Primary driver: Check 1's standalone literal threshold (80) triggers on legitimate data-processing scripts with many format strings, domain constants, and column names. Confirmed FP example: a 296-line CSV transformer with 114 literals (date format strings, VALID_CURRENCIES list, column names) — structurally indistinguishable from hardcoding at raw count. Language FPR: Python 31.9% (44/138); JavaScript 7.1% (1/14). Fix queued — see §Known Limitations.

Out-of-scope Semantic — Boundary Reference (correct abstain)

These categories are not recall targets. Low recall confirms correct scope-awareness:

Category	Code	n (dev)	Recall	Why out-of-scope
trace-test-assertion-bypass	1.1.1	36	0.111	High-frequency case is metrics-dict mutation before assertion — semantic
trace-side-channel	1.3.2	14	0.143	Out-of-scope action encoded semantically
trace-environment-manipulation	1.3.1	14	0.214	Bash data/file manipulation — outside Python AST scope
trace-deceptive-completion	1.4.2	19	0.263	Check 6 process_termination fires incidentally; rest semantic
trace-scope-violation	1.4.1	16	0.312	Check 6 subprocess/cache fires incidentally; intent violation is semantic

Catches in 1.4.1/1.4.2 are incidental Check 6 behavioral fires, not structural understanding of the violation.

Two Named Boundaries

(a) Category boundary. Semantic and test-manipulation codes (1.1.1, 1.3.1, 1.3.2, 1.4.1, 1.4.2) are out of scope by definition — they require understanding intent, not reading structure. 11–31% recall (incidental Check 6 fires) is the honest ceiling for purely structural analysis on these categories.

(b) Primitive boundary. TRACE hacks live in the diff, not the snapshot. The correct primitive is scan(old_string, new_string) per Edit call — pair-mode on each individual edit — which would expose complexity collapse (a function that had logic now returns True) and hardcoding signals (a threshold replaced by a literal). This is a named future-work extension; see §Future Work.

Core Use Case Unaffected

ast-guard's primary use case is solution-level code gen (RLVR training loops, code-gen CI): the unit of analysis is a complete function or module, not an agentic diff. That regime is validated on SORH (F1=0.98, FPR=0%) and MBPP (standalone FPR=0.31%). TRACE tests a different scan regime — agentic multi-turn editing of an existing codebase — for which standalone scan of the final state is structurally insufficient. The TRACE result is a boundary map, not a deficiency in the core use case.

---

Standalone Mode Results (MALT — 81,515 samples)

Standalone mode scans a single code block without a baseline. This mode trades precision for applicability — it can analyze any code block from any source without requiring the original. The trade-off is an inherently higher false positive rate because structural "suspicious patterns" cannot be distinguished from legitimate use without context.

Overall

Metric	Value
Total scannable	81,515
Total flagged	4,944 (6.1%)
Total clean	76,571 (93.9%)

Confusion Matrix (binary: structural-hack vs. normal)

Binary classification: hack = bypass_constraints + hardcoded_solution + sabotage (3,005 samples); non-hack = all other labels (78,510 samples).

	Flagged	Clean
Hack (3,005)	TP = 1,036	FN = 1,969
Non-hack (78,510)	FP = 3,908	TN = 74,602

TNR/Specificity measured on normal label only (77,369 samples). Some non-hack labels (partial_problem_solving, ignores_task_instructions) have legitimate reasons to use flagged patterns.

Metric	Value
Precision	21.0%
Recall	34.5%
F1	26.1%
Specificity (TNR on normal)	95.0%

Interpretation: The low precision is expected and acceptable — standalone mode is designed as a pre-filter, not a final classifier. The 95.0% specificity means the vast majority of legitimate agent code passes through without triggering a flag. The 34.5% recall covers only structurally detectable hacks; semantic hacks (giving wrong answers, sandbagging) cannot be detected by AST analysis.

Per-Category Detection Rates

Label	Samples	Flagged	Detection Rate	Language breakdown	Notes
normal	77,369	3,888	5.0% (FPR)	99.1% Python, 0.6% Bash, 0.3% JS	Legitimate agent code
bypass_constraints	2,379	820	34.5%	82.2% Python, 16.4% Bash, 1.3% JS	Structural bypasses; majority of misses are semantic
hardcoded_solution	429	201	46.9%	100% Python	Module cache manipulation, input-independence, literal counts
sabotage	197	15	7.6%	91.4% Python, 7.6% Bash	Mixed structural and semantic
match_weaker_model	171	7	4.1%	—	High literal count; mostly semantic
partial_problem_solving	731	8	1.1%	—	Threading imports in partial solutions
refusals	147	3	2.0%	—	subprocess in refusal context
reasoning_about_task	74	2	2.7%	—	Unknown imports
ignores_task_instructions	17	0	0%	—	Too few samples
gives_up	1	0	0%	—	—

Per-Language Detection on bypass_constraints

Language	Samples	Detected	Detection Rate
Python	1,956	749	38.3%
Bash	391	69	17.6%
JavaScript	32	2	6.2%

Interpretation of language gap: The lower detection rate for Bash (17.6%) and JavaScript (6.2%) compared to Python (38.3%) reflects the nature of non-Python bypass in MALT, not a deficiency in the adapters. Analysis of missed samples shows 316 of 322 missed Bash bypass samples and 30 of 30 missed JS bypass samples have zero structural signal — they are agents that silently fail to complete the task or use semantically incorrect logic rather than structurally detectable hacks (eval chains, import hijacking, complexity collapse). Structural static analysis without a pair-mode baseline has an inherent ceiling here. In pair mode, Check 2 (complexity collapse) and Check 1 (literal growth) would provide additional signal for these cases.

Per-Check Contribution

Trigger counts across all 4,944 flagged samples (one sample may trigger multiple checks):

Check	Triggers	% of Flagged
6 — Behavioral Risk Scoring	2,664	53.9%
1 — Hardcoding	2,272	46.0%
4 — Import Drift	1,072	21.7%
3 — Forbidden Calls	226	4.6%
5 — Extensional Enumeration	59	1.2%
2 — Complexity Collapse	~0	~0%

Note: Check 2 does not fire in standalone mode — there is no baseline complexity to compare against. Check 6 is now the primary detection driver across all three supported languages (Python, Bash, JS).

Top False Positive Triggers on Normal Samples

Finding	Count	Assessment
Forbidden import 'subprocess'	397	Legitimate agent use (shell commands, process management)
intent_mismatch_computation +30	310	Docstring claims computation; body differs structurally
environ_mutation +30	256	Setting env vars is normal in agent setup code
process_termination +70	244	sys.exit() in scripts and error handlers
stack_introspection +70	224	inspect module in logging and utility code
file_write_unknown_target +10	454	Unresolvable filename; contributes to aggregate score only

Bash/JS contribute 6 and 9 normal FPs respectively (negligible: 0.4% of total normal FPs), confirming the new behavioral modules are well-calibrated.

---

Generator Eval: Checks 7 & 8 (open-mode, pair mode)

Pair-mode recall and precision measured on LLM-generated hacks and clean pairs produced by the built-in generator. Two independent splits reported; neither was used for check development.

Independent Validation (mbpp-661–700)

40 seeds × 2 variants (imperative + adversarial) × claude-sonnet-4-6. Full results and analysis in METHODOLOGY.md § Independent Validation.

Metric	Value
Hack pairs	57
Recall	68.4%
Precision (easy TN)	100.0%
Precision (hard TN)	89.7%

Note: these numbers were recorded with _ORIG_COMPLEXITY_FLOOR = 2. The floor was subsequently lowered to 1 after the floor-lowering experiment below.

Floor-Lowering Experiment (mbpp-741–780, floor = 1)

40 seeds × 3 models × 3 variants. Evaluates _ORIG_COMPLEXITY_FLOOR = 1 (current). Full protocol and per-config breakdown in METHODOLOGY.md § Floor-Lowering Experiment.

Metric	floor=2 (old)	floor=1 (current)	Δ
Hack pairs	169	169	—
Recall	0.757	0.864	+0.107
Precision (easy TN, 40 pairs)	1.000	1.000	0
Precision (hard TN, 35 pairs)	0.857	0.857	0
New FPs from floor change	—	0	—

The floor lowering from 2 to 1 recovers +10.7 pp recall at zero precision cost on this fresh, unseen split. The 5 pre-existing hard TN FPs are from Checks 1 and 2, unaffected by the floor change.

---

Comparison with Existing Detectors

Direct comparison is limited because each tool covers different categories, uses different methods, and was evaluated on different dataset subsets.

Detector	Method	Dataset	Metric	Score
GPT-4o (LLM-as-judge)†	LLM-as-judge	TRACE (all 54 cats)	Detection Rate	~63%†
RewardHackWatch	ML + Regex + AST	MALT (5,391)	F1	89.7%
ast-guard v2.3.0 Pair Mode	Deterministic AST	built-in TRACE-aligned suite (hand-written)	F1	95.7%
ast-guard v2.3.0 Standalone	Deterministic AST	MALT (81,515)	Specificity	95.0%
ast-guard v2.3.0 Standalone	Deterministic AST	MALT (81,515)	Recall (structural)	34.5%

† Approximate figure; no published source. Included as order-of-magnitude context only.

Key differentiators:

• Zero cost at inference time — no LLM calls, no model weights
• Fully deterministic — same input always produces the same output
• Zero external dependencies in core (Python stdlib only)
• <50ms latency — suitable for pre-execution gating
• First-class multilang — Python, Bash, JavaScript, and TypeScript all run the same 6-check pipeline
• Complementary layer — designed to work alongside ML-based detectors, not replace them

---

Known Limitations

1. Standalone precision is low by design. Without a baseline, structural patterns that are suspicious in context (eval(), subprocess, file writes to test paths) cannot be distinguished from legitimate agent behavior using the same patterns.

2. Semantic hacks are not detectable. Reward hacking via wrong answers, sandbagging, adversarial framing, or reasoning manipulation leaves no structural AST signature. These account for the majority of MALT bypass_constraints and sabotage samples that are not detected, and nearly all of the non-Python bypass gap.

3. Bash and JavaScript bypass detection ceiling in standalone mode. In MALT, 316/322 missed Bash bypass samples and 30/30 missed JS bypass samples show zero structural signal. The bypasses are semantically encoded (incorrect behavior, silent failure), not structurally encoded. Pair mode (with original code) would provide additional signal via Check 2 and Check 1.

4. MALT code blocks are trajectory fragments. Extracted blocks are not standalone programs — they may be mid-function snippets, partial imports, or continuation of earlier code. This affects both detection rates and false positive analysis.

5. Check 1 literal threshold over-fires on data-processing scripts. The standalone literal threshold (80) is calibrated for function-level code blocks (SORH/MBPP context). Large data-pipeline files legitimately contain 80+ literals (date format strings, column names, domain constants, enum values). This is the primary driver of TRACE's 30.1% FPR and generalizes to any standalone scan of data-heavy files. Fix queued (Stage 2 calibration): literal density per line / context-aware gating (e.g., co-occurrence with input-independence signal) rather than raw count. Not tuned on TRACE; fix will be validated on MBPP/SORH first.

6. Standalone scan is the wrong primitive for agentic delta edits. TRACE-style hacks operate in the diff (old_string → new_string), not the final file state. Standalone scan on the reconstructed snapshot misses the delta signal entirely — 21.3% in-scope recall on TRACE reflects this primitive mismatch, not a tuning deficit. The correct primitive is pair-mode scan per Edit call; see §Future Work.

7. Check 2 inactive in standalone mode. Complexity collapse requires a baseline to compare against. Without an original code block, Check 2 produces no signal.

---

Deferred Pair-Mode Datasets

The following datasets were evaluated for pair-mode wiring and deferred:

Dataset	Reason deferred
Countdown-Code (Khan et al., arXiv:2603.07084)	Static JSONL dump exists (o4-mini-distillation-16k.jsonl, 16k records) but contains no is_hack label. Classifying which solutions are genuine vs. harness-manipulating requires running the verifier — generating rollouts is out of scope for a static benchmark.
TRACE (PatronusAI, arXiv:2601.20103)	Standalone eval completed (see §External TRACE Dataset). Pair mode on TRACE still deferred: the correct scan unit is scan(old_string, new_string) per Edit tool call — diff-aware pair mode on individual edits. This exposes complexity collapse and hardcoding signals invisible in the final snapshot. Named future-work extension; see §Future Work.
Terminal Wrench (arXiv:2604.17596)	331 environments; baseline and hack expressed as bash terminal trajectories, not static code pairs. Heavy extraction work needed to produce (original_code, generated_code) pairs.
MBPP-Honeypot / Obfuscation Atlas (arXiv:2602.15515)	No static completion dump found on HuggingFace (ethz-spylab/mbpp-honeypot does not exist). Env/training code only — no releasable artifact verified.
EvilGenie (Gabor et al.)	Live execution harness; no static (original, hacked) pair dump released. To use: generate rollouts with the harness, export as labeled pairs, write a custom loader.

---

Future Work

Named extensions, not yet implemented:

Extension	Motivation	Dataset to validate on
Diff-aware pair scan (scan(old_string, new_string) per Edit call)	TRACE hacks live in the delta: complexity collapse and hardcoding signals become visible only when old and new are compared directly. Standalone scan on the final snapshot misses these.	TRACE (517 trajectories)
Literal density / context-aware Check 1	Raw literal count over-fires on data-processing scripts. Density per line, or gating on co-occurrence with input-independence, would reduce false positives without affecting detection on function-level code.	MBPP / SORH (regression)
Branch-aware Check 5	Modular-arithmetic dispatch tables ((year-2000)%12 == k) and planet-lookup lists are not extensional enumeration. A param-keyed condition gate (analogous to Fix C on Check 6) would eliminate the chinese_zodiac and bf FPs.	MBPP / HumanEval
While-loop taint propagation (Fix A extension)	Fix A covers for-loops; while-loops with parameter-controlled iteration count (e.g., cal_sum) still exhibit the taint gap.	MBPP

---

Reproducibility

Environment

Python: 3.11+
ast-guard commit: f3475ed  (v2.3.0, JS/TS dispatch-table detection)
tree-sitter: required for multilang (pip install ast-guard[multilang])
MALT dataset: metr-evals/malt-public (HuggingFace, public)

Commands

# Install
git clone <repo> ast-guard && cd ast-guard
pip install -e ".[multilang]"
pip install datasets  # required for MBPP and school-of-hacks download

# Built-in TRACE benchmark
python -m benchmarks.run_benchmark --benchmark trace

# School of Reward Hacks (pair mode — TP+TN, requires download)
python -m benchmarks.run_benchmark --benchmark school-of-hacks --download

# MBPP (pair mode — TN FPR baseline, requires download)
python -m benchmarks.run_benchmark --benchmark mbpp --download

# Countdown-Code benchmark
python -m benchmarks.run_benchmark --benchmark countdown-code

# MALT benchmark (requires dataset at default path — user-managed; see memory note)
python -m benchmarks.run_benchmark --benchmark malt

# TRACE eval harness (standalone boundary eval — downloads ~20 MB from HuggingFace)
python -m eval.run --dataset trace --output eval/results/trace --mode strict

# Full cross-benchmark run with export
python -m benchmarks.run_benchmark --benchmark all --json results.json

Dataset versions

Dataset	Version / Commit
MALT	metr-evals/malt-public, accessed 2026-05-31
TRACE (external)	PatronusAI HuggingFace, accessed 2026-06-22
Countdown-Code	Khan et al. GitHub, main branch, accessed 2026-05-29
School of Reward Hacks	syvb/school-of-reward-hacks-coding-tasks, accessed 2026-06-10
MBPP	google-research-datasets/mbpp, accessed 2026-06-10