perf(api): batch indexing status segment counts - #42191

Summary

• Batch-load completed_segments and total_segments for indexing-status responses with one grouped query keyed by document ownership.
• Apply the bounded query path to the console dataset status endpoint, console batch status endpoint, and Service API batch status endpoint.
• Preserve the response shape, RE_SEGMENT exclusion, completed-at semantics, and tenant/dataset ownership filtering.

Fixes #42188

Motivation and scope

The web UI polls batch indexing status every 2.5 seconds. The previous implementation executed two COUNT(DocumentSegment.id) queries inside the response loop for every document. This made the segment-count query count grow as 2N for a batch of N documents.

This change is limited to the indexing-status endpoints. It does not change the document-list endpoint addressed by #40123 and #40125, and it does not depend on that still-open PR.

Verification evidence

Environment: Linux, CPython 3.12.14, SQLAlchemy SQLite test engine, using real controller methods and database rows. No Docker or external service was required for this query-count verification. The measurement verifies query shape and returned values; it is not a production latency benchmark.

The same harness populated 1, 5, and 20 documents with three segments per document and counted emitted SELECT statements:

After-fix harness output was:

documents=1 dataset_indexing_status_selects=2 console_batch_indexing_status_selects=1 service_batch_indexing_status_selects=2
documents=5 dataset_indexing_status_selects=2 console_batch_indexing_status_selects=1 service_batch_indexing_status_selects=2
documents=20 dataset_indexing_status_selects=2 console_batch_indexing_status_selects=1 service_batch_indexing_status_selects=2

The Service API total includes its unchanged dataset lookup; the batch document lookup was isolated in the harness so the status phase could be measured separately. The console batch document lookup is likewise unchanged.

The regression tests also verify that each document keeps its completed and total counts, and that RE_SEGMENT rows and rows belonging to another tenant are excluded.

Screenshots

Not applicable (backend-only change).

Tests

• DEBUG=false uv run --project api python -m pytest -o addopts='' api/tests/unit_tests/controllers/console/datasets/test_datasets.py api/tests/unit_tests/controllers/console/datasets/test_datasets_document.py api/tests/unit_tests/controllers/service_api/dataset/test_document.py -q — 255 passed.
• make check — passed.
• uv run --project api --dev ruff format --check ./api — 3523 files already formatted.
• make api-contract-lint — 528 valid, 0 mismatch, 0 refactorable.
• uv run --directory api --dev lint-imports — 31 contracts kept, 0 broken.
• make type-check — no issues found in 1796 source files.

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.
• I ran the backend Ruff, response-contract, import, unit-test, and type-check validations listed above.