Inside Marie's LLM Dispatch Runtime

Published July 7, 2026

LLM calls inside document workflows do not all behave the same.

A short operator chat turn, a page-level extraction call, a vision-heavy document review, and a background backfill job can all target the same model backend. They should not all compete as one invisible stream of HTTP requests.

That is the problem Marie's LLM Dispatch Runtime solves.

It gives Marie a control plane for executor-originated model work. Requests enter a Runtime Fabric group, move through configured dispatch pools, run through an OpenAI-compatible backend, and return to the producer that submitted them. Operators can see what is waiting, what is already in flight, which pool owns the work, which gateway is dispatching it, and how recent executions behaved.

The important design choice is what the runtime does not try to own.

Marie Dispatch is not a replacement for LiteLLM, OpenRouter, vLLM, or a hosted provider gateway. Marie owns executor ingress, queue lifecycle, fairness, liveness, backpressure, and observability. The configured OpenAI-compatible endpoint still owns provider routing, model fallback, budgets, and provider-specific policy.

That split keeps the system practical. Marie can protect workflow execution from noisy LLM traffic without becoming another provider router.

Why Direct Calls Break Down

The simplest design is to let every executor call an LLM backend directly.

That works until the system needs operational control.

Direct calls make it hard to answer basic production questions:

• Which workload is filling the model backend?
• Are requests waiting in a queue or stuck during execution?
• Is latency coming from queue wait, backend execution, or reply delivery?
• Can interactive work stay responsive while background document jobs keep draining?
• Did a request vanish because the producer disappeared, the dispatcher failed, or the provider rejected it?

Marie already has Runtime Fabric as the operational boundary for gateway-owned execution. LLM Dispatch extends that boundary to model calls. Executors still submit normal completion work. The dispatch runtime decides when that work leaves the queue, which pool it belongs to, and which OpenAI-compatible endpoint receives it.

This is the same broad lesson described in TNG's Hugging Face article on request queueing: once requests have entered a backend inference engine's FIFO queue, the upstream service has mostly lost control over ordering. Fairness has to happen before requests are handed to the backend. Marie applies that idea at the Runtime Fabric boundary with operational pools instead of per-user queues.

The Runtime Shape

The runtime path is deliberately small:

Executor
  -> Marie Gateway / Runtime Fabric ingress
  -> Valkey request queue
  -> LLM Dispatch Runtime
  -> OpenAI-compatible backend URL
  -> provider gateway or model backend
  -> completion response
  -> producer reply queue
  -> executor reply pump

The implementation uses a canonical completion contract before the request enters the queue. That matters because the direct path and queued path should execute the same model request shape, not drift into two separate APIs.

At enqueue time, the producer creates a queued completion envelope with:

• a request_id for reply correlation
• a producer_id for reply routing and liveness
• a pool_id for dispatch lane selection
• the normalized OpenAI-compatible completion call
• trace context for downstream span correlation
• optional metadata such as image count, page count, or estimated cost units

The Valkey transport is simple and explicit:

Key	Purpose
list:llm:requests:{pool_id}	Pending requests for one dispatch pool
list:llm:replies:{producer_id}	Replies waiting for one producer
key:llm:producer:{producer_id}:alive	Short-lived producer liveness marker

The producer keeps its liveness key fresh while it can receive replies. A local reply pump demultiplexes replies by request_id, so one producer can safely have many concurrent queued requests.

What The Gateway Owns

The gateway starts the dispatch runtime as a background service. If queueing is disabled, nothing starts. If queueing is enabled, the runtime requires a Valkey URL and an OpenAI-compatible backend configuration.

At startup, the gateway loads scheduler configuration for its Runtime Fabric group. If the policy is fifo, the runtime reads one pool in arrival order. If the policy is drr, the runtime builds a lane scheduler from the configured pools and starts a pool-aware dispatcher.

Each dispatcher owns these runtime responsibilities:

• pop eligible work from Valkey
• check producer liveness before execution
• execute the completion call through the OpenAI-compatible adapter
• check producer liveness again before publishing the reply
• track in-flight requests after they leave Valkey
• record drops, failures, queue wait, execution time, and total latency
• expose live health snapshots for Studio

That ownership boundary is intentionally narrower than provider routing. A pool may point at a different OpenAI-compatible endpoint URL, but provider fallback chains, hosted provider limits, and cost routing remain downstream.

FIFO First, Then Fair Pools

The first useful version of a dispatch layer is a single default pool. It proves the request/reply path, producer liveness, gateway execution, and observability without adding scheduler complexity.

The production shape needs more than one lane.

Marie uses Deficit Round Robin, or DRR, for pool-aware scheduling. Operators configure a finite set of pools inside a Runtime Fabric group. Each pool has a pool_id, a relative weight, and optional capacity controls.

The important fields are:

Field	Meaning
pool_id	Stable lane identifier selected by producers
quantum	DRR weight, shown to operators as Weight
min_concurrent	Protected running slots while the pool has backlog
max_concurrent	Hard per-pool in-flight cap
max_burst_per_visit	Limit for how many requests a pool can launch in one scheduler visit
endpoint_url	Optional OpenAI-compatible endpoint for that pool

Weight is not a concurrency limit. It is relative dispatch credit.

If an interactive pool has weight 4 and a backfill pool has weight 1, and both always have work, interactive work gets roughly four dispatch opportunities for every one backfill opportunity.

DRR also accounts for request cost. A tiny text prompt and an image-heavy document extraction should not consume the same scheduling share. Marie estimates cost from explicit metadata when provided, otherwise from image count and page count, then clamps the result into a bounded range. A pool must accumulate enough credit before the request at the head of its queue can launch.

The scheduler peeks before it pops. If a pool does not have enough credit or capacity, the request stays in Valkey. Once a dispatcher pops a request, the runtime owns execution and the accepted Phase 1 loss semantics.

The Failure Model Is Explicit

Valkey LISTs are the live transport, not a durable audit ledger.

That distinction shapes the runtime's failure model:

• A request still in list:llm:requests:{pool_id} is pending and visible as queued work.
• A request popped by a dispatcher is no longer durable in Valkey.
• If a dispatcher crashes after pop and before reply, that request may be lost.
• If the producer liveness key is gone before execution, the dispatcher drops the request.
• If the producer disappears after execution but before reply publish, the dispatcher drops the reply.
• Scheduler-level retry or resubmission belongs above the dispatch runtime.

This looks conservative because it is. The dispatch layer should make loss modes visible before pretending to be a durable workflow system. Completed history comes from traces, not from the queue.

What Operators See

The runtime exposes two different kinds of state.

Live state comes from the dispatch runtime and Valkey:

• pending requests still waiting in a pool queue
• in-flight requests already owned by a dispatcher
• running dispatcher count
• pool queue depth
• lane deficit, in-flight count, head request cost, and skip counts
• malformed request drops
• offline producer request and reply drops

Completed history comes from OpenTelemetry spans stored in ClickHouse:

• request, producer, pool, fabric group, gateway, and dispatcher identity
• backend address with credentials stripped
• model and message count
• queue wait, execution time, and total latency
• status, error type, and error message
• token usage when present in the completion response

This split is important. Valkey answers "what is waiting or currently executing?" ClickHouse answers "what completed, how long did it wait, how long did execution take, how many tokens did it use, and did it fail?"

The runtime also publishes snapshot events. A UI can subscribe to those events for a live fabric visualization instead of polling continuously. When the snapshot changes, the operator surface can update the pool map, active requests, and runtime health.

Reading The Signals

A dispatch layer is only useful if the signals map to operational decisions.

Symptom	First place to inspect
Pending count grows	Pool weights, capacity limits, Valkey depth, and backend saturation
In-flight count sticks	Dispatcher health and OpenAI-compatible backend latency
Requests disappear without completion	Producer liveness drops or dispatcher crash-after-pop
Backend connection errors	Gateway dispatcher logs and backend URL configuration
Provider fallback did not happen	LiteLLM, OpenRouter, vLLM, or provider gateway configuration
Completed item missing from history	OTel exporter, ClickHouse, fabric group identity, and gateway identity

The clean responsibility split matters here. If the issue is queue wait, producer liveness, dispatch capacity, or lost popped work, it belongs to Marie Dispatch. If the issue is provider fallback, model routing, provider budgets, or provider-specific rate limits, it belongs to the downstream provider gateway.

Why Runtime Fabric Scope Matters

The scheduler is scoped to a Runtime Fabric group.

That gives operators a natural boundary:

• one group can run interactive and document-processing pools for a production environment
• another group can test a different backend route
• gateway identity and fabric group identity show up in dispatch spans
• Studio can resolve a readable gateway for live state and query ClickHouse for completed spans in the same group

The pool model then becomes a lightweight contract between producers and operators. Producers choose a pool_id; the runtime decides how that pool behaves.

A typical setup might include:

• interactive for chat, operator actions, and short agent turns
• document-extract for heavier extraction prompts
• backfill for replay or enrichment work
• default for legacy, manual, or unclassified work

Idle pools do not waste capacity. Other pools can use available slots. When a protected pool gets backlog again, protected running slots and DRR credit bring it back into rotation.

This is the main operational payoff: large document jobs can keep moving without turning every operator-facing LLM call into collateral damage.

What Comes Next

The current runtime draws a clean line between the Phase 1 transport and later reliability features.

Useful next steps are already visible:

• dispatch-layer circuit breaker state around backend URLs
• explicit retry attempts and retry outcomes in history
• durable replay mode for crash-after-pop recovery
• clearer admission and backpressure UX before queues become large
• a separate streaming contract for token streaming, distinct from terminal batch_generate() replies

Those features should extend the dispatch layer without moving provider fallback, cost routing, or provider budgets into Marie. The split is the design: Marie controls workflow-facing dispatch; the provider gateway controls provider-facing policy.

One especially useful future extension is backend feedback gating. The TNG article describes fetching inference-backend metrics, such as backend queue depth or token timing, and only sending more work when the backend is below a target threshold. Marie's analysis docs reserve that as a dispatch-gating slice: start with local adapter inflight counters, then add backend queue-depth or token-speed polling when an adapter exposes those metrics cleanly.

Takeaway

LLM Dispatch turns model calls from isolated executor-side HTTP requests into Runtime Fabric work.

The result is not just better utilization. It is a clearer operating model:

• producers submit normalized completion work
• pools express workload intent
• the gateway enforces dispatch fairness and liveness
• operators see both live state and completed history
• provider gateways keep owning provider-specific routing

That boundary is what makes the system useful. Marie can protect workflows from noisy LLM traffic without becoming yet another provider router.

Related Reading

• LLM Dispatch Runtime
• LLM Scheduler
• Runtime Fabric
• LLM Observability
• Efficient Request Queueing – Optimizing LLM Performance