regexp: REGEXP_SUBSTR and REGEXP_INSTR read past a fixed match array with ten or more capture groups (stack-buffer-overflow under AddressSanitizer) #2324

Summary

ora_setup_regexp_substr_matches() and ora_setup_regexp_instr_matches() in src/backend/utils/adt/regexp.c keep the match positions in a fixed regmatch_t pmatch[10] and pass 10 to the matcher, but matchctx->npatterns is re->re_nsub and the loop that records the subexpressions reads pmatch[1..npatterns]. A pattern with ten or more capture groups reads past the array. The same code is on master (lines 1566/1604 and 1738/1776 at 069766e) and on IVORY_REL_5_STABLE.

How it shows

An AddressSanitizer build (-fsanitize=address with --enable-cassert) of 5.6 running the Oracle regression suite of contrib/ivorysql_ora reports:

ERROR: AddressSanitizer: stack-buffer-overflow ... in ora_setup_regexp_substr_matches regexp.c

Without the sanitizer, REGEXP_SUBSTR(str, pattern, 1, 1, 'c', n) and REGEXP_INSTR(..., n) with a subexpression number above 9 return whatever lies beyond the array on the stack.

Fix

Allocate the array with one slot for the whole match and one per subexpression, and pass that count to the matcher:

	regmatch_t   *pmatch;
	int			nmatch;
	...
	nmatch = re->re_nsub + 1;
	pmatch = (regmatch_t *) palloc(nmatch * sizeof(regmatch_t));
	...
	while (RE_wchar_execute(re, data, data_len, search_start, nmatch, pmatch))

in both functions. I have the patch ready on a branch from master (24 lines, compiles; it is the fix running in my 5.6-based tree, where the sanitizer run of the extension suite is clean with it) and will open the pull request from it. A regression test needs a pattern with eleven groups and subexpr = 11; the expected row for it must come from a run on master, so the PR will carry the code change and the test separately if the maintainers prefer.