[abhishekraok]

Written by Claude Code session sglang-concurrent-agenerate

Fixes concurrent agenerate on the SGLang provider. Scaffolds that run a chunk's conversations at once (multi_turn_chat, and by the same pattern bfcl_multi_turn, stirrup, vanillux) failed on SGLang with RuntimeError: This event loop is already running. Until now the only workaround was batching.chunk_size=1, which runs one conversation at a time (a 5-item multi_if took 2,030 s).

Cause

agenerate was asyncio.to_thread(self.generate, ...), and generate called Engine.generate, which drives engine.loop with run_until_complete. SGLang binds its tokenizer manager to that one loop, so a second thread calling Engine.generate while the first is waiting re-enters the running loop. The docstring said "sequential engine calls", but nothing serialized them.

Fix

• One daemon thread (sglang-engine-loop) runs engine.loop for the provider's lifetime, started on the first request.
• generate and agenerate both submit Engine.async_generate to that loop. At the pinned SGLang commit it builds the same GenerateReqInput from the same arguments as Engine.generate and awaits the same tokenizer_manager.generate_request. So prompts, chat templating, sampling parameters, logprobs and finish reasons are unchanged: preparation and output conversion are the existing code, split into _prepare_batch / _finish_batch.
• agenerate awaits the request (asyncio.wrap_future) instead of holding a worker thread, so every concurrent conversation is in the engine at once and the scheduler batches them. Rendering and conversion still run in worker threads, so the harness loop is not blocked.
• The capacity hint can no longer call Engine.get_server_info once the loop thread runs (that method also drives the loop itself). It now reads tokenizer_manager.get_internal_state() on the loop, plus server_args.dp_size: the only fields it used. The startup capacity read in __init__ runs before the thread starts and is unchanged.
• close() stops the loop thread after engine.shutdown().

No lock: nothing is serialized.

Unchanged

• Request hashes do not read the provider or its trace. describe_request still reports endpoint: engine.generate, since the engine receives the same request; old and new traces stay identical.
• Signal handling: SGLang only installs its SIGTERM handler on the main thread. Before, agenerate already drove the loop from worker threads, so a background loop thread changes nothing there.

Tests

• New test_concurrent_agenerate_calls_are_batched_on_one_engine_loop uses a fake engine with SGLang's loop contract. Its generate drives loop.run_until_complete. Its async_generate must run on that loop and returns only once all 8 requests are in flight together, so a serializing fix would time out. On main the test fails with RuntimeError: This event loop is already running; on this branch it passes, with peak concurrency 8. It also checks that the sync path works while the loop thread is running, and that close() stops the thread.
• The shared fixture's fake engine gains a real loop and an async_generate that records through the existing engine.generate mock, so the existing assertions keep reading one mock. The capacity-hint tests answer through get_internal_state instead of get_server_info.
• ruff check, ruff format --check, ty check src/: pass. tests/providers/test_sglang_*.py: 161 passed.

GPU smoke

5-item multi_if on holmes at the default batching (chunk_size: "auto", resolved to 64), with the same flags as the run that failed on main (01M73QVRRCVF159KS3N45CVGV2).

run	batching	loop error	processed	wall	saved
main, workaround (01M73RN8TFANNA0XGK8XPEETTV)	chunk_size=1, serial	none	5/5	2,030 s	5/5, cap-hit 1/5
this branch, smoke1 (01M792EADE9JERKPTPK84A14NP)	auto, concurrent	none	5/5	1,685 s	4/5
this branch, smoke2 (01M794KT0SAZF35R5ECMGK73E6)	auto, concurrent	none	5/5	226 s	5/5, cap-hit 0/5, exit 0

Smoke2 passes. Its wall time is not directly comparable with the serial run's, because smoke2 drew no runaway conversation. Without the runaway item, the serial run spent about 390 s on its other four conversations; smoke2 ran all five in 226 s.

Smoke1 ran all five conversations concurrently, with no loop error. Its wall time is close to the slowest item alone (1,641 s in the serial run). It still exited 1 on the failure budget. In one conversation (instance 1), a turn sampled at T=0.6 filled the whole 262,144-token context (bare multi_if sets no max_tokens; the suites' _t10_32k variants cap it at 32,768). The next turn's input was then 28 tokens over the limit, and SGLang rejected it. The serial run on main hit the same item's length cap (cap-hit 1/5) but drew a shorter history. Sampling is unseeded, so smoke2 is a fresh draw. That overflow is a property of the item and of unbounded multi-turn generation, not of this change; see Follow-ups.

The Task was destroyed but it is pending! lines at exit come from SGLang's tokenizer-manager tasks and appear in the main run too.

Review

Other-family review (Codex, exec review --base origin/main): no actionable regressions; 100 provider tests passed in its checkout. It could not exercise a real GPU engine; the smoke above covers that.

Follow-ups (not in this PR)

• The vLLM offline provider's agenerate uses the same asyncio.to_thread(self.generate, ...) pattern.
• multi_turn_chat sends a turn whose history already exceeds the context, so one runaway turn fails the whole conversation instead of ending it as capped.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-12T23:49:24.487009Z	e5a989f	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.