Skip to content

Utf8Proc::Analyze: check 32 bytes per iteration while the input is ASCII - #25721

Open
carlopi wants to merge 2 commits into
duckdb:v2.0-cyanopterafrom
carlopi:faster_utf8_analyze
Open

carlopi wants to merge 2 commits into
duckdb:v2.0-cyanopterafrom
carlopi:faster_utf8_analyze

Conversation

@carlopi

@carlopi carlopi commented Sep 15, 2026

Copy link
Copy Markdown
Member

The ASCII scan checked 8 bytes per iteration with a branch each. Folding four loads into one mask check per 32 bytes is neutral on short string, bails out at first problem (so neutral on long one), but consume faster when appending long ASCII-only strings.

The ASCII scan checked 8 bytes per iteration with a branch each. Folding
four loads into one mask check per 32 bytes is neutral on short strings
and about 9% faster when appending long ASCII strings.
@carlopi carlopi closed this Sep 15, 2026
The 32-byte scan cost every short string an extra loop test before the
original code, which showed up as +3% on TPC-H ingestion where almost all
strings are shorter than 32 bytes. Gate it behind a single length test so
short strings run exactly the previous code and only long inputs, where
the scan runs at least twice, take the wide path.

Locally: 40M 28-byte strings unchanged, 20M 208-byte strings -9%.
@carlopi carlopi reopened this Sep 15, 2026
@carlopi carlopi added the Draft label Sep 15, 2026
@Mytherin
Mytherin enabled auto-merge September 15, 2026 10:43
@Mytherin

Copy link
Copy Markdown
Collaborator

Thanks - good idea

@carlopi carlopi removed the Draft label Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants