fix(rag): preserve CSV cell text in batch segment import - #42159

Summary

Fixes #42157

The batch segment import task now reads uploaded CSV files with dtype=str and keep_default_na=False. This preserves numeric-looking values and literal NA values as text for both paragraph and QA imports, including leading zeros in values such as 00123.

No new dependency is required.

Root cause and scope

The batch task used the default Pandas type inference. Values such as 00123 and NA could become NumPy numeric or missing-value scalars, and the subsequent len(content) call raised TypeError.

This change is limited to the batch segment import path. It is separate from #41922, which fixes the standard CSVExtractor path, and #38863, which addresses batch-task Redis error status handling.

Validation

Before

On the pre-fix upstream/main commit f0d69e7b39, the same CSV:

content,answer
00123,NA
NA,00456

was read as:

before: dtypes={'content': 'float64', 'answer': 'float64'}, values=[('float64', np.float64(123.0), 'float64', np.float64(nan)), ('float64', np.float64(nan), 'float64', np.float64(456.0))]

The task-level reproduction failed with:

TypeError: object of type 'numpy.float64' has no len()

After

The same task-level reproduction, using the real task function and real Pandas parsing with database, storage, Redis, and vector service boundaries mocked in memory, passed with:

observed=[('str', '00123', 'NA', 7), ('str', 'NA', '00456', 7)]
redis=[('segment_batch_import_job-id', 600, 'completed')]

The direct Pandas comparison now reports:

after: dtypes={'content': 'str', 'answer': 'str'}, values=[('str', '00123', 'str', 'NA'), ('str', 'NA', 'str', '00456')]

Other checks:

• pytest -q api/tests/unit_tests/core/rag/extractor/test_csv_extractor.py --no-cov: 15 passed
• Targeted ruff check: All checks passed
• Targeted ruff format --check: 2 files already formatted
• git diff --check: passed
• Added a task-level regression test covering numeric-looking content, literal NA, and QA answers.

The container integration test was not runnable to completion locally because the current user cannot access the Docker daemon socket (Permission denied on /var/run/docker.sock). The production fix itself does not require Docker; this limitation only affects the repository's Testcontainers-based PostgreSQL/Redis integration fixture. CI can run the integration test in its normal container-enabled environment.

Screenshots

Checklist

• This change requires a documentation update
• I understand that this PR may be closed in case there was no previous discussion or issues
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change
• I've updated the documentation accordingly
• I ran make lint && make type-check; targeted Ruff checks were run instead

From Codex.