Repository navigation
feat(openai): add async and streamed parity for hosted tool discovery - #9280
shenjm (Jiaming07) wants to merge 16 commits into
Conversation
Add a tool_search option to OpenAICompletion. When enabled on the Responses API with two or more function tools, the hosted tool_search tool is added and function tools are sent with defer_loading=True. An explicit defer_loading flag on a tool is preserved during conversion. Eager loading stays the default. Refs crewAIInc#8008
OpenAI rejects a deferred function tool unless tool_search is in the request. A single tool with defer_loading=True (or any explicitly deferred tool without the tool_search option) was sent without it. Tools without the flag are only deferred when tool_search is enabled and there are two or more of them, as before.
Verify hosted discovery and eager calls through the existing authorized tool catalog, including matching call IDs and error results. Reject non-object JSON arguments before execution instead of falling back to tool defaults. Refs crewAIInc#8125. Depends on crewAIInc#8040.
Streamed Responses calls now assemble function calls from fragmented argument events and only act on them once the stream reaches response.completed. The completed event's response is the source of truth, so streamed calls match the non-streaming path and keep the same call IDs; the accumulated fragments are the fallback. A stream that fails, is incomplete, or ends without a terminal event never executes a partially streamed call. The async Responses handlers now await tool results through a new BaseLLM._ahandle_tool_execution, so coroutine tools run instead of returning a coroutine object. The four Responses execution loops share one helper per sync/async variant. Tests use OpenAI SDK stream event models with fake clients and need no network: interleaved fragments across two discovered calls, async tool awaiting with completion and failure events, failed/incomplete/truncated streams, a lost output_item.done event, and a crew round trip with matching call IDs. Refs crewAIInc#8126. Depends on crewAIInc#8040, crewAIInc#9085 and crewAIInc#7972. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
Strix Security ReviewWarning This pull request has 105 commits after the last Strix review ( ✅ No open security findings. PR summary: Extends hosted tool discovery on the OpenAI Responses API to the async and streamed paths: a new accumulator assembles fragmented function calls from stream events, terminal-state handling refuses to execute calls from a stream that never reached `response.completed`, and a new `BaseLLM._ahandle_tool_execution` awaits async tools instead of stringifying coroutines. It also adds an opt-in `tool_search` config that defers function-tool loading, and hardens `parse_tool_call_args` to reject non-object JSON arguments. Review summaryReviewed the full diff of PR #9280 (OpenAI Responses async/streamed hosted-tool-discovery parity) across Ruled out (8)
Review 1 on this PR · Reviewed by Strix |
…-tool-discovery # Conflicts: # lib/crewai/src/crewai/llms/providers/openai/completion.py
The streamed Responses handlers now refuse to return function calls from a stream that never reached response.completed. The fixture only had response.created and output_item.done, which no real stream produces, so it is given the terminal event with the completed response. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Related issue
Fixes #8126.
Summary
Extends hosted tool discovery on the OpenAI Responses API to the async and streamed paths, so discovered function calls behave the same way regardless of how the response arrives.
Streamed function calls. A
_StreamedFunctionCallAccumulatorassembles function calls fromresponse.output_item.added,response.function_call_arguments.delta/doneandresponse.output_item.doneevents, keyed by item id so interleaved fragments from parallel calls stay separate. Hostedtool_search_call/tool_search_outputitems run server-side and are ignored. Once the stream reachesresponse.completed, the function calls in that event's response are the source of truth (identical to_extract_function_calls_from_responseon the non-streaming path, including call IDs); the accumulated fragments are the fallback when that output carries no function calls.Terminal-state handling. Builds on #7972's
response.completed/response.failed/response.incompletehandling. If any function call started but the stream did not reachresponse.completed, the call raises aRuntimeErrorthat names the terminal status andincomplete_details.reason(or that no terminal event arrived), and no partially streamed call is executed. This check now runs before theparse_tool_outputsearly return, so that mode cannot bypass it.Async execution. The async Responses handlers previously called the sync
_handle_tool_execution, so anasync deftool returned a coroutine object that was stringified.BaseLLMgains_ahandle_tool_execution, which awaits awaitable results, with the started/finished/error event emission shared with the sync variant. Both async Responses handlers use it. The four Responses execution loops now share one helper per sync/async variant; malformed or non-object JSON arguments fall back to{}with an error log, matching the Chat Completions path.Chat Completions and other providers are unchanged.
Verification
New
tests/llms/openai/test_responses_tool_search_streaming.py(31 cases, sync and async parametrized). Streams are built from OpenAI SDK event models and served by fake clients throughcall()/acall(); no network or API key. Covers:arguments.done, emptyoutput_item.donearguments, no calls in the completed response) — fails if delta assembly is removedoutput_item.doneevent, call still taken from the completed responseToolUsageStarted/FinishedandTOOL_CALLcompletion events emittedToolUsageErrorEventandLLMCallFailedEventresponse.failed,response.incomplete(mid-arguments and after a completed call) and a stream with no terminal event: no tool runs, one failed event, also withparse_tool_outputs=Truecall_ids matching betweenfunction_callandfunction_call_outputitemsAlso ran
tests/llms/openaiandtests/llms/test_base_llm.py(360 passed). The sixtest_reasoning_effort_forwarding.py::TestLLMSurfacefailures also fail on unmodifiedmain. The failure list fortests/llms+tests/agentsis identical before and after this change.ruff check,ruff formatand strictmypyare clean on the changed source files.Additional context
Depends on #8040, #9085 and #7972, all still open. Their commits are included in this branch with original authorship; this contribution is the final commit (
base_llm.py,completion.py, the new test file). The request building and configuration belong to #8040, the sync execution path and argument validation to #9085, and the streaming terminal-state fix to #7972, which is reused rather than reimplemented.Two points worth a maintainer's call, left as is to stay in scope:
crew.kickoff_async()runs the agent executor in a thread and calls the syncllm.call, so the crew-level test is sync only; async parity is verified at the LLM layer withacall.function_callitems from the executor's tool-call history and do not carry thetool_search_call/tool_search_outputitems back into the nextinput. Whether the API needs them whenprevious_response_idis not used has not been verified against the live service.tests/llms/openai/test_tools_reasoning_effort_retry.py::TestStreamedToolCalls(from #9035) streamed a function call without any terminal event, which no real stream does and which the terminal-state handling now rejects. Its fixture ends withresponse.completedcarrying the same call. #7972 will meet the same three tests when it is rebased.Prepared with AI assistance (Claude Code), per CONTRIBUTING. The
llm-generatedlabel needs a maintainer.🤖 Generated with Claude Code
Note
Medium Risk
Changes core OpenAI Responses streaming, tool execution, and argument parsing paths; behavior shifts on incomplete streams and async tools, though coverage is extensive and defaults leave tool search off.
Overview
Adds hosted tool search on the OpenAI Responses API via opt-in
tool_search=True, which injects atool_searchtool and setsdefer_loadingon function tools so models can load definitions on demand. Docs for the new parameter are updated in EN, AR, KO, and PT-BR.Streaming and execution parity:
_StreamedFunctionCallAccumulatorrebuilds function calls from fragmented stream events (including parallel interleaved deltas), prefers calls fromresponse.completedoutput, and raises on failed/incomplete streams or missing terminal events before running tools—including whenparse_tool_outputsis enabled.Async tools:
BaseLLMrefactors sync tool execution into shared start/finish/error helpers and adds_ahandle_tool_executionto await coroutine tool results; OpenAI Responses sync/async paths share_execute_responses_function_calls/_aexecute_responses_function_callswith stricter JSON-object argument validation (aligned withparse_tool_call_args).Large new OpenAI test suites cover tool search configuration, streaming tool calls, hosted discovery, and crew round-trips.
Reviewed by Cursor Bugbot for commit 6a29db0. Bugbot is set up for automated code reviews on this repo. Configure here.