sre_engine: resume the tail match at prefix_skip, not past the whole prefix - #8473

re.search returns wrong spans — and misses matches outright — for any pattern whose literal prefix extends past the INFO block's prefix_skip boundary.

>>> re.search(r'ab(cd)', 'xabcdcd')
<re.Match object; span=(1, 7), match='abcdcd'>   # CPython: span=(1, 5), match='abcd'
>>> re.search(r'a(bc)', 'xabc')
None                                             # CPython: span=(1, 4)

re.match / re.fullmatch are unaffected — only the search literal-prefix fast path is wrong. A non-capturing group is also fine, because -(?:ab) sets SRE_INFO_LITERAL and returns early.

Cause

search_info_literal's len > 1 arm advanced the tail-match cursor one character past the matched prefix:

let mut next_ctx = ctx;
if skip != 0 {
    next_ctx.advance_char::<S>();   // ctx.cursor is the LAST prefix char, so this is start + len
} else {
    next_ctx.cursor = state.cursor;
}

ctx.cursor sits on the last prefix character, so advance_char lands on start + prefix_len. The tail has to resume at start + prefix_skip; the two agree only when the prefix happens to end where the skip does.

_get_literal_prefix extends the prefix through a capturing group while pinning prefix_skip at the group boundary, so the two routinely differ. ab(cd) compiles to prefix_len=4, prefix_skip=2:

[INFO, 14, 1, 4, 4, prefix_len=4, prefix_skip=2, 97, 98, 99, 100, 0,0,0,0, LITERAL,97, LITERAL,98, MARK,0, ...]

so the tail resumed two characters too far and re-matched cd at 5..7.

The code position was already advanced correctly by ctx.skip_code(2 * skip) — only the string cursor was wrong.

state.cursor already holds exactly the right value: state.reset(req, req.start) followed by S::skip(&mut state.cursor, skip) makes it req.start + skip. That matches sre_lib.h SRE(search), which sets state->ptr = ptr - (prefix_len - prefix_skip - 1), and it is what the len == 1 arm a few lines above already does (next_ctx.skip_char::<S>(skip)). So the conditional can just go away.

Verification

Both the pristine and patched engine were linked against a corpus stamped with CPython's own re.search answers (30 patterns x 20 subjects = 600 cases), so each is judged against CPython rather than against the other:

Many of the 94 are want=(2,6) got=(-1,-1) — the wrong resume drops matches, it does not only widen them.

The added tests/tests.rs::search_literal_prefix_longer_than_skip fails on the unpatched engine (assertion left == right, right: 5) and passes with the fix. The crate's other 19 tests are unchanged and still pass, including the existing info_literal and info_single, which cover the same function's other arms.

— commented by Claude

Summary by CodeRabbit

• Bug Fixes

  • Fixed literal searches with longer prefixes so matching resumes at the correct position.
  • Corrected reported match start and cursor positions for affected searches.

• Tests

  • Added regression coverage for literal-prefix matching and cursor behavior.