Adventures of a Software Engineer

Blog

  • Inside Marie LLM Dispatch Runtime

    Inside Marie's LLM Dispatch Runtime

    Published July 7, 2026

    LLM calls inside document workflows do not all behave the same.

    A short operator chat turn, a page-level extraction call, a vision-heavy document review, and a background backfill job can all target the same model backend. They should not all compete as one invisible stream of HTTP requests.

    That is the problem Marie's LLM Dispatch Runtime solves.

    It gives Marie a control plane for executor-originated model work. Requests enter a Runtime Fabric group, move through configured dispatch pools, run through an OpenAI-compatible backend, and return to the producer that submitted them. Operators can see what is waiting, what is already in flight, which pool owns the work, which gateway is dispatching it, and how recent executions behaved.

    The important design choice is what the runtime does not try to own.

    Marie Dispatch is not a replacement for LiteLLM, OpenRouter, vLLM, or a hosted provider gateway. Marie owns executor ingress, queue lifecycle, fairness, liveness, backpressure, and observability. The configured OpenAI-compatible endpoint still owns provider routing, model fallback, budgets, and provider-specific policy.

    That split keeps the system practical. Marie can protect workflow execution from noisy LLM traffic without becoming another provider router.

    Marie LLM Dispatch Runtime architecture

    Why Direct Calls Break Down

    The simplest design is to let every executor call an LLM backend directly.

    That works until the system needs operational control.

    Direct calls make it hard to answer basic production questions:

    • Which workload is filling the model backend?
    • Are requests waiting in a queue or stuck during execution?
    • Is latency coming from queue wait, backend execution, or reply delivery?
    • Can interactive work stay responsive while background document jobs keep draining?
    • Did a request vanish because the producer disappeared, the dispatcher failed, or the provider rejected it?

    Marie already has Runtime Fabric as the operational boundary for gateway-owned execution. LLM Dispatch extends that boundary to model calls. Executors still submit normal completion work. The dispatch runtime decides when that work leaves the queue, which pool it belongs to, and which OpenAI-compatible endpoint receives it.

    This is the same broad lesson described in TNG's Hugging Face article on request queueing: once requests have entered a backend inference engine's FIFO queue, the upstream service has mostly lost control over ordering. Fairness has to happen before requests are handed to the backend. Marie applies that idea at the Runtime Fabric boundary with operational pools instead of per-user queues.

    The Runtime Shape

    The runtime path is deliberately small:

    Executor
  -> Marie Gateway / Runtime Fabric ingress
  -> Valkey request queue
  -> LLM Dispatch Runtime
  -> OpenAI-compatible backend URL
  -> provider gateway or model backend
  -> completion response
  -> producer reply queue
  -> executor reply pump

    The implementation uses a canonical completion contract before the request enters the queue. That matters because the direct path and queued path should execute the same model request shape, not drift into two separate APIs.

    At enqueue time, the producer creates a queued completion envelope with:

    • a request_id for reply correlation
    • a producer_id for reply routing and liveness
    • a pool_id for dispatch lane selection
    • the normalized OpenAI-compatible completion call
    • trace context for downstream span correlation
    • optional metadata such as image count, page count, or estimated cost units

    The Valkey transport is simple and explicit:

    Key Purpose
    list:llm:requests:{pool_id} Pending requests for one dispatch pool
    list:llm:replies:{producer_id} Replies waiting for one producer
    key:llm:producer:{producer_id}:alive Short-lived producer liveness marker

    The producer keeps its liveness key fresh while it can receive replies. A local reply pump demultiplexes replies by request_id, so one producer can safely have many concurrent queued requests.

    What The Gateway Owns

    The gateway starts the dispatch runtime as a background service. If queueing is disabled, nothing starts. If queueing is enabled, the runtime requires a Valkey URL and an OpenAI-compatible backend configuration.

    At startup, the gateway loads scheduler configuration for its Runtime Fabric group. If the policy is fifo, the runtime reads one pool in arrival order. If the policy is drr, the runtime builds a lane scheduler from the configured pools and starts a pool-aware dispatcher.

    Each dispatcher owns these runtime responsibilities:

    • pop eligible work from Valkey
    • check producer liveness before execution
    • execute the completion call through the OpenAI-compatible adapter
    • check producer liveness again before publishing the reply
    • track in-flight requests after they leave Valkey
    • record drops, failures, queue wait, execution time, and total latency
    • expose live health snapshots for Studio

    That ownership boundary is intentionally narrower than provider routing. A pool may point at a different OpenAI-compatible endpoint URL, but provider fallback chains, hosted provider limits, and cost routing remain downstream.

    FIFO First, Then Fair Pools

    The first useful version of a dispatch layer is a single default pool. It proves the request/reply path, producer liveness, gateway execution, and observability without adding scheduler complexity.

    The production shape needs more than one lane.

    Marie uses Deficit Round Robin, or DRR, for pool-aware scheduling. Operators configure a finite set of pools inside a Runtime Fabric group. Each pool has a pool_id, a relative weight, and optional capacity controls.

    The important fields are:

    Field Meaning
    pool_id Stable lane identifier selected by producers
    quantum DRR weight, shown to operators as Weight
    min_concurrent Protected running slots while the pool has backlog
    max_concurrent Hard per-pool in-flight cap
    max_burst_per_visit Limit for how many requests a pool can launch in one scheduler visit
    endpoint_url Optional OpenAI-compatible endpoint for that pool

    Weight is not a concurrency limit. It is relative dispatch credit.

    If an interactive pool has weight 4 and a backfill pool has weight 1, and both always have work, interactive work gets roughly four dispatch opportunities for every one backfill opportunity.

    DRR also accounts for request cost. A tiny text prompt and an image-heavy document extraction should not consume the same scheduling share. Marie estimates cost from explicit metadata when provided, otherwise from image count and page count, then clamps the result into a bounded range. A pool must accumulate enough credit before the request at the head of its queue can launch.

    The scheduler peeks before it pops. If a pool does not have enough credit or capacity, the request stays in Valkey. Once a dispatcher pops a request, the runtime owns execution and the accepted Phase 1 loss semantics.

    The Failure Model Is Explicit

    Valkey LISTs are the live transport, not a durable audit ledger.

    That distinction shapes the runtime's failure model:

    • A request still in list:llm:requests:{pool_id} is pending and visible as queued work.
    • A request popped by a dispatcher is no longer durable in Valkey.
    • If a dispatcher crashes after pop and before reply, that request may be lost.
    • If the producer liveness key is gone before execution, the dispatcher drops the request.
    • If the producer disappears after execution but before reply publish, the dispatcher drops the reply.
    • Scheduler-level retry or resubmission belongs above the dispatch runtime.

    This looks conservative because it is. The dispatch layer should make loss modes visible before pretending to be a durable workflow system. Completed history comes from traces, not from the queue.

    What Operators See

    The runtime exposes two different kinds of state.

    Live state comes from the dispatch runtime and Valkey:

    • pending requests still waiting in a pool queue
    • in-flight requests already owned by a dispatcher
    • running dispatcher count
    • pool queue depth
    • lane deficit, in-flight count, head request cost, and skip counts
    • malformed request drops
    • offline producer request and reply drops

    Completed history comes from OpenTelemetry spans stored in ClickHouse:

    • request, producer, pool, fabric group, gateway, and dispatcher identity
    • backend address with credentials stripped
    • model and message count
    • queue wait, execution time, and total latency
    • status, error type, and error message
    • token usage when present in the completion response

    This split is important. Valkey answers "what is waiting or currently executing?" ClickHouse answers "what completed, how long did it wait, how long did execution take, how many tokens did it use, and did it fail?"

    The runtime also publishes snapshot events. A UI can subscribe to those events for a live fabric visualization instead of polling continuously. When the snapshot changes, the operator surface can update the pool map, active requests, and runtime health.

    Reading The Signals

    A dispatch layer is only useful if the signals map to operational decisions.

    Symptom First place to inspect
    Pending count grows Pool weights, capacity limits, Valkey depth, and backend saturation
    In-flight count sticks Dispatcher health and OpenAI-compatible backend latency
    Requests disappear without completion Producer liveness drops or dispatcher crash-after-pop
    Backend connection errors Gateway dispatcher logs and backend URL configuration
    Provider fallback did not happen LiteLLM, OpenRouter, vLLM, or provider gateway configuration
    Completed item missing from history OTel exporter, ClickHouse, fabric group identity, and gateway identity

    The clean responsibility split matters here. If the issue is queue wait, producer liveness, dispatch capacity, or lost popped work, it belongs to Marie Dispatch. If the issue is provider fallback, model routing, provider budgets, or provider-specific rate limits, it belongs to the downstream provider gateway.

    Why Runtime Fabric Scope Matters

    The scheduler is scoped to a Runtime Fabric group.

    That gives operators a natural boundary:

    • one group can run interactive and document-processing pools for a production environment
    • another group can test a different backend route
    • gateway identity and fabric group identity show up in dispatch spans
    • Studio can resolve a readable gateway for live state and query ClickHouse for completed spans in the same group

    The pool model then becomes a lightweight contract between producers and operators. Producers choose a pool_id; the runtime decides how that pool behaves.

    A typical setup might include:

    • interactive for chat, operator actions, and short agent turns
    • document-extract for heavier extraction prompts
    • backfill for replay or enrichment work
    • default for legacy, manual, or unclassified work

    Idle pools do not waste capacity. Other pools can use available slots. When a protected pool gets backlog again, protected running slots and DRR credit bring it back into rotation.

    This is the main operational payoff: large document jobs can keep moving without turning every operator-facing LLM call into collateral damage.

    What Comes Next

    The current runtime draws a clean line between the Phase 1 transport and later reliability features.

    Useful next steps are already visible:

    • dispatch-layer circuit breaker state around backend URLs
    • explicit retry attempts and retry outcomes in history
    • durable replay mode for crash-after-pop recovery
    • clearer admission and backpressure UX before queues become large
    • a separate streaming contract for token streaming, distinct from terminal batch_generate() replies

    Those features should extend the dispatch layer without moving provider fallback, cost routing, or provider budgets into Marie. The split is the design: Marie controls workflow-facing dispatch; the provider gateway controls provider-facing policy.

    One especially useful future extension is backend feedback gating. The TNG article describes fetching inference-backend metrics, such as backend queue depth or token timing, and only sending more work when the backend is below a target threshold. Marie's analysis docs reserve that as a dispatch-gating slice: start with local adapter inflight counters, then add backend queue-depth or token-speed polling when an adapter exposes those metrics cleanly.

    Takeaway

    LLM Dispatch turns model calls from isolated executor-side HTTP requests into Runtime Fabric work.

    The result is not just better utilization. It is a clearer operating model:

    • producers submit normalized completion work
    • pools express workload intent
    • the gateway enforces dispatch fairness and liveness
    • operators see both live state and completed history
    • provider gateways keep owning provider-specific routing

    That boundary is what makes the system useful. Marie can protect workflows from noisy LLM traffic without becoming yet another provider router.

  • Reveres engineering Alienware keyboard

    USBPCAP
    Starting USBPCap with following command as per instructions from here ;

    USBPcapCMD.exe -d \\.\USBPcap1 -o - | "C:\Program Files\Wireshark\Wireshark.exe" -k -i -
Selected capture options result in empty capture.

    Browsing the source we can see that this is caused when all these conditions are true
    https://github.com/desowin/usbpcap/blob/master/USBPcapCMD/cmd.c

      /* Sanity check capture configuration. */
    if ((data->capture_all == FALSE) &&
        (data->capture_new == FALSE) &&
        (data->address_list == NULL))
    {
        fprintf(stderr, "Selected capture options result in empty capture.\n");
        return;
    }

  • Death by QDS_LOADDB Wait type

    As the title implies ‘Death by QDS_LOADDB Wait type’

  • Fast query, but slow from Stored Procedure / Function- How query optimizer can screw your day

    How parameter sniffing can destroy your query performance

  • ScheduledExecutorService in Python with asyncio

    Running scheduled tasks with ease in Python. This implementation API is based on Java ScheduledExecutorService but it has been pythonized. There are other methods of running scheduled tasks in python but this seems like a natural choice for asyncio.

    The source code for the scheduler can be found here https://github.com/gregbugaj/marie-ai/blob/main/marie/concur/ScheduledExecutorService.py

    The schedule_with_fixed_delay is the primary method of the scheduler with the following signature.

        def schedule_with_fixed_delay(
        self,
        func: Callable,
        initial_delay: int,
        delay: int,
        unit: TimeUnit,
        *args,
        **kwargs,
    ) -> Any:
        """
        Creates and executes a periodic action that becomes enabled first after the given initial delay,
        and subsequently with the given period.
        Args:
            func:  the task to execute
            initial_delay:  the time to delay first execution
            delay: the period between successive executions
            unit:  the time unit of the initial_delay and period parameters
        Return:
            a ScheduledTask representing pending completion of the task
        """

    Creating a new scheduler and creating a task in just a few lines of code.

    # Task to execute
async def async_task(name: str):
    print(f"{name} : {current_thread().name} {time.time()}  Start")
    await asyncio.sleep(1)
    print(f"{name} : {current_thread().name} {time.time()} Complete")


# Create scheduler and schedule a task
scheduler = ScheduledExecutorService.new_scheduled_asyncio_pool()
t1 = scheduler.schedule_at_fixed_rate(async_task, 1, TimeUnit.MILLISECONDS, name="T1")

    Sample Usage

    import asyncio
import threading
import time

from marie.concur import ScheduledExecutorService

from threading import current_thread

from marie.concur.ScheduledExecutorService import TimeUnit


async def async_task(name: str):
    print(f"{name} : {current_thread().name} {time.time()}  Start")
    await asyncio.sleep(1)
    print(f"{name} : {current_thread().name} {time.time()} Complete")


async def cancel_callback(task):
    print(f"canceling task : {task}")
    await task.stop()


async def main():
    scheduler = ScheduledExecutorService.new_scheduled_asyncio_pool()

    t1 = scheduler.schedule_at_fixed_rate(
        async_task, 1, TimeUnit.MILLISECONDS, name="T1"
    )

    t2 = scheduler.schedule_at_fixed_rate(
        async_task, 2, TimeUnit.MILLISECONDS, name="T2"
    )

    asyncio.get_event_loop().call_later(3, asyncio.create_task, cancel_callback(t1))


async def main_single():
    scheduler = ScheduledExecutorService.new_scheduled_asyncio_pool()
    t1 = scheduler.schedule_at_fixed_rate(
        async_task, 1, TimeUnit.MILLISECONDS, name="T1"
    )

    # call_later() only supports callbacks (regular functions); you  can’t pass in a coroutine.
    asyncio.get_event_loop().call_later(3, asyncio.create_task, cancel_callback(t1))
    # await t1.task


async def main_delay():
    scheduler = ScheduledExecutorService.new_scheduled_asyncio_pool()
    t1 = scheduler.schedule_with_fixed_delay(
        async_task, 2, 1, TimeUnit.MILLISECONDS, name="T1"
    )

    await t1.start()
    print(t1.task)

    asyncio.get_event_loop().call_later(3, cancel_callback, t1)

    await t1.task


if __name__ == "__main__":
    print("Main")
    loop = asyncio.get_event_loop()
    try:
        # asyncio.ensure_future(main_single())
        asyncio.ensure_future(main())
        loop.run_forever()
    except KeyboardInterrupt:
        pass
    finally:
        print("Closing Loop")
        loop.close()

    I am utilizing this in the Marie-AI project for my watchdog service.

  • Delven is a Domain Specific Language (DSL) designed for mining content

    Delven is a Domain Specific Language (DSL) designed for mining content from static and dynamic sources, It closely resembles SQL with features borrowed from other popular languages.

    This documentation is your guide to an advanced new world of real-time data connectivity.

    • To get an idea of what Delven is and how it can benefit your organization, visit the Introduction Section page.
    • New users and experienced users alike may refer to the Syntax section for all the information you need to create robust queries.
    • For new users, the Tutorial provides an introduction to basic query writing skills and a page of sample queries to get you started.
    • Programmers interested in embedding Delven within their application should visit the API Reference.

  • After a year of hiatus, I am back

    Not much more to share here. I will be publishing a bi-weekly post on technology-related things.

    This was due to the loss of my domain name after 14 years and it was used as a spam aggregator. Companies like Godaddy and Afernic should be held accountable for the spread of malware and spyware.

  • Machine Learning Datasets

    Collection of some useful datasets for machine learning. Some are free some require registration

    Machine Learning Datasets

    TC-11 Online Resources (Text, OCR, Signatures)

    http://iapr-tc11.org/mediawiki/index.php/Datasets_List
    http://tc11.cvc.uab.es/datasets/type/

    Center for Machine Learning and Intelligent Systems

  • Metrics in Hoplin.io

    Hoplin does not have any dependencies on any existing metrics libraries but rather it provides a way to hook into the underlying metrics via MetricsPublisher interface. Metrics expose a number of key/value pairs that are updated and send to metrics consumers.

    Depending on which client we use the metrics key will be different and it is up to the consumer to normalize the name. Data is packed into a Map<String, Map<String,String>> structure, this allows us to add metrics easily without breaking any API.

    Sample Payload
    {exchange.rpc.logs-rpc.request.log={received.size=211, sent.size=204, received.count=1, sent.count=1}}

    Metrics Key = exchange.rpc.logs-rpc.request.log
    received.size = Amount of data received by this client
    received.count = Number of messages received

    sent.size = Amount of data sent by this client
    sent.count = Number of messages sent

    Instantiating metrics consumer

    FunctionMetricsPublisher
	.consumer(EmitLogTopic::metrics)
	.withInterval(1, TimeUnit.SECONDS)
	.withResetOnReporting(false)
	.build()
	.start();

private static void metrics(final Map<String, Map<String, String>> stat)
{
  System.out.println("Metrics Info : " + stat);
}

  • x

    Inside Marie LLM Dispatch Runtime