Skip to content

feat(openai): add async and streamed parity for hosted tool discovery - #9280

Open
shenjm (Jiaming07) wants to merge 16 commits into
crewAIInc:mainfrom
Jiaming07:feat/responses-stream-tool-discovery
Open

shenjm (Jiaming07) wants to merge 16 commits into
crewAIInc:mainfrom
Jiaming07:feat/responses-stream-tool-discovery

Conversation

@Jiaming07

@Jiaming07 shenjm (Jiaming07) commented Nov 8, 2026 •

Copy link
Copy Markdown

Related issue

Fixes #8126.

Summary

Extends hosted tool discovery on the OpenAI Responses API to the async and streamed paths, so discovered function calls behave the same way regardless of how the response arrives.

Streamed function calls. A _StreamedFunctionCallAccumulator assembles function calls from response.output_item.added, response.function_call_arguments.delta/done and response.output_item.done events, keyed by item id so interleaved fragments from parallel calls stay separate. Hosted tool_search_call / tool_search_output items run server-side and are ignored. Once the stream reaches response.completed, the function calls in that event's response are the source of truth (identical to _extract_function_calls_from_response on the non-streaming path, including call IDs); the accumulated fragments are the fallback when that output carries no function calls.

Terminal-state handling. Builds on #7972's response.completed / response.failed / response.incomplete handling. If any function call started but the stream did not reach response.completed, the call raises a RuntimeError that names the terminal status and incomplete_details.reason (or that no terminal event arrived), and no partially streamed call is executed. This check now runs before the parse_tool_outputs early return, so that mode cannot bypass it.

Async execution. The async Responses handlers previously called the sync _handle_tool_execution, so an async def tool returned a coroutine object that was stringified. BaseLLM gains _ahandle_tool_execution, which awaits awaitable results, with the started/finished/error event emission shared with the sync variant. Both async Responses handlers use it. The four Responses execution loops now share one helper per sync/async variant; malformed or non-object JSON arguments fall back to {} with an error log, matching the Chat Completions path.

Chat Completions and other providers are unchanged.

Verification

  • Tests added or updated for the changed behavior
  • Relevant tests and quality checks pass locally

New tests/llms/openai/test_responses_tool_search_streaming.py (31 cases, sync and async parametrized). Streams are built from OpenAI SDK event models and served by fake clients through call() / acall(); no network or API key. Covers:

  • two discovered calls with interleaved argument fragments, result equal to the non-streaming extraction with matching call IDs
  • arguments assembled from deltas alone (no arguments.done, empty output_item.done arguments, no calls in the completed response) — fails if delta assembly is removed
  • lost output_item.done event, call still taken from the completed response
  • only catalog functions run; async tools are awaited; ToolUsageStarted/Finished and TOOL_CALL completion events emitted
  • tool failure emits ToolUsageErrorEvent and LLMCallFailedEvent
  • response.failed, response.incomplete (mid-arguments and after a completed call) and a stream with no terminal event: no tool runs, one failed event, also with parse_tool_outputs=True
  • async non-streaming path awaits a discovered tool
  • crew round trip over streaming with call_ids matching between function_call and function_call_output items

Also ran tests/llms/openai and tests/llms/test_base_llm.py (360 passed). The six test_reasoning_effort_forwarding.py::TestLLMSurface failures also fail on unmodified main. The failure list for tests/llms + tests/agents is identical before and after this change. ruff check, ruff format and strict mypy are clean on the changed source files.

Additional context

Depends on #8040, #9085 and #7972, all still open. Their commits are included in this branch with original authorship; this contribution is the final commit (base_llm.py, completion.py, the new test file). The request building and configuration belong to #8040, the sync execution path and argument validation to #9085, and the streaming terminal-state fix to #7972, which is reused rather than reimplemented.

Two points worth a maintainer's call, left as is to stay in scope:

  • crew.kickoff_async() runs the agent executor in a thread and calls the sync llm.call, so the crew-level test is sync only; async parity is verified at the LLM layer with acall.
  • Multi-turn requests rebuild function_call items from the executor's tool-call history and do not carry the tool_search_call / tool_search_output items back into the next input. Whether the API needs them when previous_response_id is not used has not been verified against the live service.

tests/llms/openai/test_tools_reasoning_effort_retry.py::TestStreamedToolCalls (from #9035) streamed a function call without any terminal event, which no real stream does and which the terminal-state handling now rejects. Its fixture ends with response.completed carrying the same call. #7972 will meet the same three tests when it is rebased.

Prepared with AI assistance (Claude Code), per CONTRIBUTING. The llm-generated label needs a maintainer.

🤖 Generated with Claude Code


Note

Medium Risk
Changes core OpenAI Responses streaming, tool execution, and argument parsing paths; behavior shifts on incomplete streams and async tools, though coverage is extensive and defaults leave tool search off.

Overview
Adds hosted tool search on the OpenAI Responses API via opt-in tool_search=True, which injects a tool_search tool and sets defer_loading on function tools so models can load definitions on demand. Docs for the new parameter are updated in EN, AR, KO, and PT-BR.

Streaming and execution parity: _StreamedFunctionCallAccumulator rebuilds function calls from fragmented stream events (including parallel interleaved deltas), prefers calls from response.completed output, and raises on failed/incomplete streams or missing terminal events before running tools—including when parse_tool_outputs is enabled.

Async tools: BaseLLM refactors sync tool execution into shared start/finish/error helpers and adds _ahandle_tool_execution to await coroutine tool results; OpenAI Responses sync/async paths share _execute_responses_function_calls / _aexecute_responses_function_calls with stricter JSON-object argument validation (aligned with parse_tool_call_args).

Large new OpenAI test suites cover tool search configuration, streaming tool calls, hosted discovery, and crew round-trips.

Reviewed by Cursor Bugbot for commit 6a29db0. Bugbot is set up for automated code reviews on this repo. Configure here.

gaoanze and others added 11 commits October 9, 2026 16:40
Add a tool_search option to OpenAICompletion. When enabled on the
Responses API with two or more function tools, the hosted tool_search
tool is added and function tools are sent with defer_loading=True.
An explicit defer_loading flag on a tool is preserved during
conversion. Eager loading stays the default.

Refs crewAIInc#8008
OpenAI rejects a deferred function tool unless tool_search is in the
request. A single tool with defer_loading=True (or any explicitly
deferred tool without the tool_search option) was sent without it.
Tools without the flag are only deferred when tool_search is enabled
and there are two or more of them, as before.
Verify hosted discovery and eager calls through the existing authorized tool catalog, including matching call IDs and error results. Reject non-object JSON arguments before execution instead of falling back to tool defaults.

Refs crewAIInc#8125. Depends on crewAIInc#8040.
Streamed Responses calls now assemble function calls from fragmented
argument events and only act on them once the stream reaches
response.completed. The completed event's response is the source of
truth, so streamed calls match the non-streaming path and keep the same
call IDs; the accumulated fragments are the fallback. A stream that
fails, is incomplete, or ends without a terminal event never executes a
partially streamed call.

The async Responses handlers now await tool results through a new
BaseLLM._ahandle_tool_execution, so coroutine tools run instead of
returning a coroutine object. The four Responses execution loops share
one helper per sync/async variant.

Tests use OpenAI SDK stream event models with fake clients and need no
network: interleaved fragments across two discovered calls, async tool
awaiting with completion and failure events, failed/incomplete/truncated
streams, a lost output_item.done event, and a crew round trip with
matching call IDs.

Refs crewAIInc#8126. Depends on crewAIInc#8040, crewAIInc#9085 and crewAIInc#7972.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit ebbaf74.

Approve preview

@strix-security

strix-security Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Strix Security Review

Warning

This pull request has 105 commits after the last Strix review (04d77f9). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

✅ No open security findings.

PR summary: Extends hosted tool discovery on the OpenAI Responses API to the async and streamed paths: a new accumulator assembles fragmented function calls from stream events, terminal-state handling refuses to execute calls from a stream that never reached `response.completed`, and a new `BaseLLM._ahandle_tool_execution` awaits async tools instead of stringifying coroutines. It also adds an opt-in `tool_search` config that defers function-tool loading, and hardens `parse_tool_call_args` to reject non-object JSON arguments.

Review summary

Reviewed the full diff of PR #9280 (OpenAI Responses async/streamed hosted-tool-discovery parity) across completion.py, base_llm.py, agent_utils.py and the new tests, plus the lint items in or reached by the changed code. No security vulnerability was confirmed. The change is plumbing: it assembles streamed function calls into the same shape the non-streaming path produces, adds an async tool-execution variant that awaits awaitable results, and tightens argument validation. The tool-invocation gate is unchanged — available_functions is still a plain developer-supplied dict and the model-supplied name is still checked by membership before fn(**args), with the executor re-gating independently. The new terminal-state checks run before the parse_tool_outputs early return, so a failed or truncated stream can no longer execute a partially streamed call. Three suspected issues were raised and each was independently rejected: the incomplete-stream guard being keyed on observed stream events rather than terminal status, an empty function_call_arguments.done overwriting assembled arguments, and malformed/non-object arguments silently becoming {}. In every case the only party who can produce the triggering input is the configured model endpoint, which already controls tool arguments outright, so no trust boundary is crossed. The lint leads (hook-less provider-side tool loop, unbounded stream accumulation, new log sinks, async after_llm_call hook gap) were all traced and found to be pre-existing patterns the PR neither introduces nor widens.

Ruled out (8)
  • Can a function call from an incomplete Responses stream be executed when no function-call stream events were observed?
    The bypass is real but crosses no trust boundary: the stream is produced by the operator-configured endpoint, which can already return any tool call via a normal `response.completed`, and the non-streaming Responses and Chat Completions paths have never checked status before executing calls.
  • Can an empty `function_call_arguments.done` event wipe assembled arguments and run a tool with defaults?
    The unconditional overwrite at completion.py:386 is a genuine defect, but the stream producer is already the full authority over tool arguments and the accumulator is only a fallback behind the completed response's output, so no capability is gained.
  • Does mapping malformed or non-object tool arguments to `{}` let a tool run with dangerous defaults?
    The model already controls the arguments and could send `{}` directly, the tool must be in the developer-supplied `available_functions` allowlist, and tools with required parameters still raise and emit a `ToolUsageErrorEvent`.
  • Does the new async tool-execution path bypass the pre/post tool-call guardrail hooks?
    The hook-less provider-side loop predates the PR and is shared by every provider; the PR only replaces the same sync call with an awaiting copy, and `BaseTool.run` still enforces `max_usage_count`.
  • Does the new streamed argument accumulator create an unbounded-memory denial of service?
    The accumulator is a per-call local with replace-not-append on `output_item.done`, and the only actor able to stream indefinitely is the endpoint operator, who already has arbitrary tool invocation.
  • Do the new `logging.error` calls in `_parse_responses_function_args` allow log forging or content echo?
    `JSONDecodeError.__str__` omits the offending document and the other sink logs only a Python type name, so no model or gateway content reaches the log.
  • Does the PR widen the async `after_llm_call` guardrail gap?
    The gap is pre-existing and unchanged, the new early-return branch is symmetric with the sync handler, and `_invoke_after_llm_call_hooks` returns early for non-string responses so that data was never hook-processed.
  • Can `tool_search` / `defer_loading` request construction alter the tool catalog or inject keys into the outbound request?
    `_convert_tools_for_responses` builds a fresh dict from a fixed field set, `defer_loading` is gated by `isinstance(value, bool)`, no tool is added or removed, and `tool_search` is developer-only configuration not reachable from `additional_params`.

Review 1 on this PR · 04d77f9 · requested Nov 8 23:53 UTC when the PR was opened


Reviewed by Strix
Re-run review · Configure security review settings

shenjm (Jiaming07) and others added 5 commits November 8, 2026 15:58
…-tool-discovery

# Conflicts:
#	lib/crewai/src/crewai/llms/providers/openai/completion.py
The streamed Responses handlers now refuse to return function calls from
a stream that never reached response.completed. The fixture only had
response.created and output_item.done, which no real stream produces, so
it is given the terminal event with the completed response.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Add async and streamed parity for hosted tool discovery

5 participants