fix(bbh): stop fewshot generations at "Q:" instead of any capital Q - #4215

What

bbh_fewshot and bbh_cot_fewshot list a bare "Q" as a stop sequence (lm_eval/tasks/bbh/fewshot/_fewshot_template_yaml:14, lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml:16). It is meant as the boundary before the next Q: ... question, but stop strings match anywhere, so every generation ends at its first capital Q:

• reasoning_about_colored_objects has options (A) to (R), and 7 of its 250 test targets are (Q). Those generations are cut to ( before any filter runs, so no model can score them (7 of the 6511 rows in each group).
• Chains of thought that mention a word starting with Q before reaching the answer (e.g. "Qarabağ FK" in formal_fallacies) are cut before "the answer is".

bbh_zeroshot and bbh_cot_zeroshot already use "Q:" (the latter since #1356). This PR does the same in the two fewshot templates; "Q:" still stops at the next question.

Measured effect

I replayed the code-davinci-002 outputs published with BBH (suzgunmirac/BIG-Bench-Hard @ 9ee07bd, code-davinci-002-outputs/) through the harness's own postprocess_generated_text, the task filters and process_results. 6507 of the 6511 rows align with SaylorTwift/bbh by question text and target.

The "now incorrect" column is about these published outputs only. A generation that continues on the same line with a capitalised Q word right after the answer (e.g. (A)Quite) used to be cut at the Q and no longer is; none of these outputs does that, and I have not measured how often other models do.

Versioning

Scores change, so the task templates are bumped (bbh_fewshot_* 2.0 → 3.0, bbh_cot_fewshot_* 4.0 → 5.0), and so are the groups that aggregate them, since their scores change too (bbh_fewshot, bbh_cot_fewshot and bbh, 3.0 → 4.0; docs/new_task_guide.md versions both tasks and groups). flan_held_out also includes both groups but has no version field. Changelog entry in lm_eval/tasks/bbh/README.md.

Tests

tests/test_bbh_stop_sequences.py loads the until list of the four BBH templates and checks that postprocess_generated_text keeps (Q) and ... So the answer is (Q)., and still stops at \nQ: ... (a single newline, so it is the "Q:" stop that fires). 4 of its 12 cases fail on main (the (Q) cases for fewshot and cot_fewshot); all 12 pass here, as do tests/test_bbh_filters.py. The "Tasks Modified" job only selects changed files ending in .yaml, so it will not pick these extension-less templates; the unit test is what covers them.

Related: #1086 (verify that stop sequences don't cut off valid answers).

Found and measured with AI coding agents working through a review process I set up. I went through the diff and reran the numbers and tests myself.