Test/uv 16309 span flush race + fix - #1901

Agreed — added test_get_and_clear_execution_data_flushes_before_reading in 7552722. It builds the same trace_manager/span_exporter/span_collector/logs_exporter wiring UiPathEvalRuntime.__init__ sets up, then calls the real _get_and_clear_execution_data (unbound, via the class, against a duck-typed self). I verified it directly: temporarily removed the flush_spans() line locally, confirmed this new test fails ({'root'} == {'root', 'tool_call'}), then restored the line and confirmed it passes again.

Good catch. Fixed in 8dde043 — evaluate() now calls self._get_actual_output(workload_execution) instead of passing workload_execution.workload_output directly, so target_output_key extraction and job-attachment URI resolution both apply before the value reaches AgentRunHistory. Added a test (target_output_key="result", a multi-key output dict) that fails without this fix — it previously leaked the whole output object (including an unrelated key) into the prompt.

This flush cannot catch anything the read would otherwise miss, because production already flushes twice between the last span ending and this read:

1. UiPathTraceManager.start_execution_span flushes in its finally when the root span closes.
2. UiPathExecutionRuntime.execute() (uipath-runtime 0.13.5, uipath/runtime/base.py:186) flushes again in its own finally before returning.

The eval runtime then calls _get_and_clear_execution_data right after await execution_runtime.execute(...) returns (line ~1063 → ~1084) with no await in between, so no coroutine can end a span in that window. A span that has already ended is exported by flush (2); a span that has not ended is missed with or without this line. The only remaining gap is a span ending on another OS thread in the microseconds between those two statements, and this flush does not close that either.

The Jira data points the same way: none of the 34 cases in UV-16309 is "tool calls happened, history empty". The two tool-call cases had 7.4k-char histories that were missing the agent's text, which is the trace_to_str issue Part 2 fixes.

I'd drop this block and test_execution_span_race.py, and let the PR stand on Part 2. If you prefer to keep it as belt-and-braces, please reword the comment so it does not claim to fix UV-16309 or an observed race.

For conversational agents workload_output is {"uipath__agent_response_messages": [...]}, so f"...{final_output}" hands the judge a Python repr of nested dicts with the actual answer buried under contentParts[].data.inline, next to ids, timestamps and mime types. That is the exact case UV-16309 is about (the low-code conversational agent), so it is worth rendering the text properly. For non-conversational agents str(dict) is single-quoted repr rather than JSON, which is also harder for the model to read.

Suggest a small renderer in _helpers/evaluators_helpers.py and using it here and in the legacy evaluator (so both trajectory evaluators produce the same section):

# packages/uipath/src/uipath/eval/_helpers/evaluators_helpers.py
AGENT_RESPONSE_MESSAGES_KEY = "uipath__agent_response_messages"


def render_agent_output(output: dict[str, Any] | str) -> str:
    """Render the workload output for an LLM-judge prompt.

    Conversational agents return ``{"uipath__agent_response_messages": [...]}``
    where the text sits under ``contentParts[].data.inline``. Surface that text
    (and any tool calls) instead of the raw dict repr; anything else is emitted
    as indented JSON.
    """
    if isinstance(output, str):
        return output
    messages = output.get(AGENT_RESPONSE_MESSAGES_KEY)
    if not isinstance(messages, list):
        return json.dumps(output, indent=2, default=str)

    lines: list[str] = []
    for message in messages:
        if not isinstance(message, dict):
            continue
        role = message.get("role", "assistant")
        parts = message.get("contentParts") or message.get("content_parts") or []
        for part in parts:
            data = part.get("data") if isinstance(part, dict) else None
            text = data.get("inline") if isinstance(data, dict) else None
            if text:
                lines.append(f"[{role}] {text}")
        calls = message.get("toolCalls") or message.get("tool_calls") or []
        for call in calls:
            if isinstance(call, dict):
                args = json.dumps(call.get("input"), default=str)
                lines.append(f"[{role}] Tool call {call.get('name')}: {args}")

    rest = {k: v for k, v in output.items() if k != AGENT_RESPONSE_MESSAGES_KEY}
    if rest:
        lines.append(json.dumps(rest, indent=2, default=str))
    return "\n".join(lines) or json.dumps(output, indent=2, default=str)

Then at this call site:

(plus from .._helpers.evaluators_helpers import render_agent_output, trace_to_str at the top). Handles both camelCase aliases and snake_case field names since the runtime does model_dump() without by_alias for pydantic outputs but passes dicts through untouched. A test with a realistic uipath__agent_response_messages payload asserting the assistant text appears verbatim would pin this down.

Same rendering issue as in llm_judge_trajectory_evaluator.py (see that comment for the proposed render_agent_output helper). Using the same helper here keeps the two trajectory evaluators consistent, which they currently are not: this one resolves target_output_key and job-attachment URIs via _get_actual_output, the non-legacy one appends the raw field.

_batch_processor, _schedule_delay_millis and _schedule_delay are private attributes of the OpenTelemetry SDK's BatchSpanProcessor (the _batch_processor delegate only exists in newer SDK versions). An SDK bump will silently turn this into an AttributeError or, worse, a no-op that lets the worker thread export and makes the "not yet exported" assertions flaky again. If this file stays, prefer constructing the processor with schedule_delay_millis=3_600_000 via the public constructor argument (UiPathExecutionBatchTraceProcessor forwards to BatchSpanProcessor.__init__) instead of patching internals after the fact.

This test reproduces the missing span only because it omits the flush that production performs after the tool span. In UiPathExecutionRuntime.execute() the finally: self.trace_manager.flush_spans() runs after delegate.execute() returns, i.e. after any span that ended inside the awaited call, and only then does the eval runtime read. To model the real order, add a second trace_manager.flush_spans() after the tool_call span block below and before calling _get_and_clear_execution_data. With that, the test passes with or without the new flush line, which is the point I'm making on runtime.py: the line is not what closes the gap.