Skip to content

feat: speed up incoming packet parsing - #1830

Merged
bdraco merged 12 commits into
masterfrom
speed_up_incoming_parser
Aug 28, 2026
Merged

bdraco merged 12 commits into
masterfrom
speed_up_incoming_parser

Conversation

@bdraco

@bdraco bdraco commented Aug 28, 2026

Copy link
Copy Markdown
Member

Summary

Speeds up DNSIncoming parsing by 47 percent; the benchmark packet drops from 9.3µs to 4.9µs on the compiled build. Stacked on #1829 so the fuzzing suite guards every change.

Details

  • names that are a single RFC 1035 §4.1.4 compression pointer to an already decoded name now return the cached finished string; the benchmark packet builds 45 names but only 8 are distinct, so 37 list, set, join, and dict operations disappear, and identical names across a packet share one str object
  • the loop detection set is only allocated when a name actually follows an uncached pointer
  • _read_bitmap indexes the buffer directly instead of allocating a bytes slice per NSEC record, and skips zero bytes
  • the memoryview attribute is replaced by a raw buffer pointer with an explicit bounds check at every read site; the memoryview's implicit IndexError was load bearing, so each site now carries its own guard, and messages beyond the 64 KiB DNS limit are rejected up front so no unsigned arithmetic can wrap
  • DNSIncoming now refuses pickling, data is readonly, both closing lifetime holes the raw pointer would otherwise open

Test plan

  • two adversarial security audits of every buffer read verified all 15 sites bounded on all paths, including a 200k name differential fuzz of the name cache against a cache free reference decoder with zero mismatches
  • hypothesis fuzzing from test: add fuzzing coverage for incoming packet parsing #1829: 300k example deep runs pass on both the compiled and pure python builds
  • new tests pin string sharing across records, all pointer flows, overlong name rejection, pickle refusal, and oversized message rejection
  • poetry run pytest tests passes, 495 passed; pre-commit clean; Cython annotate maps show the added score confined to cold raise blocks and the 8x slow path
  • verified on a live network; all services resolve with correct properties

@codspeed

codspeed Bot commented Aug 28, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 43.28%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
✅ 22 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
test_parse_incoming_message 466.4 µs 254.3 µs +83.36%
test_unique_payloads 3 ms 2.7 ms +11.96%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing speed_up_incoming_parser (59acf2c) with master (e63bf6f)

Open in CodSpeed

@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.87%. Comparing base (28f8047) to head (59acf2c).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1830   +/-   ##
=======================================
  Coverage   99.86%   99.87%           
=======================================
  Files          33       33           
  Lines        3839     3883   +44     
  Branches      553      567   +14     
=======================================
+ Hits         3834     3878   +44     
  Misses          3        3           
  Partials        2        2           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bdraco
bdraco force-pushed the speed_up_incoming_parser branch from 59015cd to 8448b77 Compare August 28, 2026 21:24
@bdraco

bdraco commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

Just waiting for ASAN locally

@bdraco

bdraco commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

local ASan 50,000 hostile packets + all 14 audit boundary packets, zero sanitizer reports.

@bdraco
bdraco marked this pull request as ready for review August 28, 2026 21:56
Base automatically changed from fuzz_incoming to master August 28, 2026 22:02
@bluetoothbot

Copy link
Copy Markdown
Contributor

@bdraco: local ASan 50,000 hostile packets + all 14 audit boundary packets, zero sanitizer reports.

Matches static audit. Traced all buffer reads independently: header, questions, record header, SRV, character string, string, NSEC bitmap, name fast path, label pointer. Each guarded before dereference. Label slice correctly stays on self.data, avoiding pointer over-read.

One gap: fuzz strategies cap at _MAX_MSG_ABSOLUTE + 256, roughly 9 KiB. ASan corpus likely same range. Nothing reaches offsets near 65535, where no-wrap argument for MAX_MSG_LEN matters. Worth one 65530-byte packet with record header at tail.

@bluetoothbot

bluetoothbot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Previous review — superseded by a newer review below.

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tip

No blocking issues found — ready to merge.

@bdraco

bdraco commented Aug 28, 2026

Copy link
Copy Markdown
Member Author
  1. Pickle refusal + readonly data are user-visible restrictions on a re-exported class
    src/zeroconf/_protocol/incoming.py:140-143

Nobody is pickling DNSIncoming, that would be a crazy use case. Its not intended to be a public API. This is an internal safety

@bluetoothbot

Copy link
Copy Markdown
Contributor

@bdraco: > 1. Pickle refusal + readonly data are user-visible restrictions on a re-exported class

Fair — dropping that finding. Grepped repo: nothing pickles or copies DNSIncoming, so no in-tree breakage. Only note: __reduce__ also blocks copy.copy and copy.deepcopy, which your test already pins. Won't re-raise.

@bluetoothbot

Copy link
Copy Markdown
Contributor

PR Review — feat: speed up incoming packet parsing

Approving. All three notes from my previous pass are now addressed or dismissed by the author; nothing blocking remains.

Re-verified against the current head (the branch moved from aa41716 to 1f07f73): every raw-buffer read is still bounds-checked before dereference — _read_header (+12), _read_questions (+4), _read_others (+10), SRV (+6), _read_character_string (length byte then slice), _read_string, _read_bitmap (record end plus per-window bitmap_end > end), the _read_name fast path (+2), and both buf[off]/buf[off + 1] in _decode_labels_at_offset. The label slice deliberately stays on self.data so Python's truncating slice, not a pointer over-read, handles a label running past the end. I also re-checked that an oversized packet can't reach a read through answers(): _initial_parse raises before _read_header, so the section counts stay 0 and _read_others iterates zero times — which test_oversized_message_is_rejected pins.

What changed since the last review, and how it lands:

  • Suggestion #3 (no fuzzing near the 64 KiB boundary) — resolved by e0abe0b. I hand-checked the offset math in test_records_near_the_size_cap_never_raise: tail_record is 20 bytes, skip_len = total - 43 ∈ [65460, 65492], and record 2's name, header, SRV fixed fields, and target pointer are all parsed against the cap. One residual coverage gap noted inline (non-blocking).

  • Suggestion #2 (unstated key-range invariant) — resolved for the load-bearing half: the comment at incoming.py:503-505 now states that entries exist only at offsets inside the packet, so an out-of-range link can only miss. The duplicated >= 0xC0 / (length & 0x3F) * 256 + ... pointer arithmetic in two places remains; not worth blocking, just keep the two sites in step.

  • Suggestion #1 (pickle refusal + readonly data) — dismissed per @bdraco: "Nobody is pickling DNSIncoming, that would be a crazy use case. Its not intended to be a public API. This is an internal safety." Not re-raised.

  • One non-blocking suggestion: the new boundary fuzz test can only ever hit the record-drop path, so a clean decode at the cap goes unexercised.


🟢 Suggestions

1. Boundary fuzz test only ever exercises the record-drop path
tests/test_fuzz_incoming.py:157

This new test closes the gap my prior review flagged — offsets now land in the last bytes below 64 KiB, and I traced the arithmetic: tail_record is 20 bytes, skip_len = total - 43 stays in [65460, 65492], assert len(packet) == total holds, and record 2's name/header/SRV/target-pointer reads all sit against _data_len. Good coverage of the no-wrap claim.

One gap in what it can observe: the tail record is an SRV that consumes exactly 8 rdata bytes (6 fixed + a 2-byte pointer), and _read_others drops any record where self.offset != end (src/zeroconf/_protocol/incoming.py:355). With rdlength drawn uniformly from 0..0xFFFF, only the single value 8 survives that resync, so essentially every generated example exercises the drop path and never a record that decodes cleanly at the cap.

Why it matters: the interesting case for the new pure-pointer fast path is a name that succeeds in the last two bytes of a near-64 KiB packet — that's the path _read_name's original_offset + 2 <= _data_len guard was written for, and it currently gets pinned only by test_pure_pointer_in_final_two_bytes_parses on a 34-byte packet.

Cheap fix: bias the strategy so the valid length is drawn regularly, e.g.

rdlength=st.one_of(st.just(8), st.integers(min_value=0, max_value=0xFFFF)),

and optionally assert that when rdlength == 8 the SRV actually lands in answers().

    rdlength=st.integers(min_value=0, max_value=0xFFFF),

Checklist

  • Every raw-buffer read is bounds-checked before dereference
  • Oversized input cannot reach a read via the lazy answers() path
  • No unsigned arithmetic wrap on offset/length math, now fuzzed at the cap
  • Compression-pointer loop and depth detection preserved by the lazy set
  • Name string cache cannot return an unvalidated or out-of-range decode
  • Parsing still never raises out of the constructor for hostile input
  • .pxd updated in step with the cdef class layout and signatures
  • New branches covered by tests — suggestion #1
  • Diff matches the PR description with no scope creep

Automated review by Kōan (Claude) HEAD=1f07f73 7 min 6s

@bluetoothbot bluetoothbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tip

No blocking issues found — ready to merge.

@bdraco
bdraco merged commit b3634f2 into master Aug 28, 2026
33 checks passed
@bdraco
bdraco deleted the speed_up_incoming_parser branch August 28, 2026 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants