Rethinking the SGLang HiCache Boundary: From L2 Offload to a Cache Runtime

中文版：重新思考 SGLang HiCache 的边界：从 L2 卸载到 Cache Runtime

This post is a record of an ongoing design exploration, not a finalized SGLang architecture proposal.

It started from #37372: Out-of-process HiCache data plane with device-memory IPC. The original question was straightforward: could HiCache's GPU ↔ Host transfers, Host KV pool, and L3 I/O move out of the scheduler process, with a standalone daemon operating on GPU KV through CUDA IPC?

After tracing the Unified Cache linker, Rust TreeCore, ReqToTokenPool, CUDA Graph, and shared-replica paths together, the more interesting question changed.

The question I care about now is:

How should SGLang separate logical cache control from physical GPU/cache execution without putting IPC on the scheduler hot path or sacrificing compute/transfer overlap?

This note reflects SGLang main around 2026-09-03 and the following discussions:

• #37372 — out-of-process HiCache data plane
• #35687 — Unified Cache external linker
• #37381 — external linker end-to-end integration
• #32710 — Rust TreeCore backend
• #35648 — same-GPU replicas and a shared KV pool

Names such as CacheRuntime, LocalGPURuntime, RequestRuntimeHandle, CacheExtentRef, and BatchPlan below are still working design names rather than accepted SGLang abstractions.

The original hypothesis: move the HiCache L2 data plane into a daemon

The first version of the design looked roughly like this:

Scheduler process
  ├── scheduling
  ├── TreeCore
  ├── prefix matching
  ├── GPU allocator
  └── backup / restore intent
              │
              │ CUDA IPC + control messages
              ▼
HiCache daemon
  ├── imported GPU KV mappings
  ├── H2D / D2H
  ├── Host KV pool
  ├── transfer queues
  └── L3 clients

That boundary has two obvious attractions:

1. storage and transfer work can leave the scheduler process;
2. a daemon can independently own pinned Host memory, transfer queues, batching, throttling, and L3 connections.

After looking deeper into the CUDA and forward paths, however, I now think this process split was being chosen too early.

The harder question is not whether another process can map GPU KV through CUDA IPC. It is:

Why should the control process own or understand physical GPU state at all?

With the right ownership boundary, it may not need to.

Today the Scheduler process is already both control plane and GPU runtime

SGLang's Scheduler is not only a lightweight request-policy loop.

It creates a TpModelWorker, which owns a ModelRunner, and the same ownership domain can reach:

ReqToTokenPool
TokenToKVPoolAllocator
ModelRunner
forward_stream
CUDA Graph state
HiCache controller / transfer state

The decode path is a useful example:

Scheduler
  ↓
ScheduleBatch.prepare_for_decode()
  ↓
alloc_for_decode()
  ↓
KV allocator
  ↓
ReqToTokenPool.write()
  ↓
ModelRunner forward

For a single engine this is direct and efficient.

The current architecture is therefore not obviously “too slow.” The architectural issue is narrower:

scheduling policy and physical GPU/cache mechanisms do not have a clean ownership boundary.

That is not necessarily a problem for single-instance serving. It becomes more important once we care about shared replicas, runtime lifecycle, stronger storage isolation, or explicit GPU-resource scheduling.

The existing TreeCore split already provides a useful seam

The current cache_action.py describes the relationship clearly:

A TreeCore emits CacheActions through the TreeCoreInterface
to guide Controller behavior.

TreeCore owns tree structure, matching, LRU state, lock/ref bookkeeping, and related logical operations. It emits CacheActions that the controller/pool side applies.

Conceptually:

TreeCore
   │
   │ decision / CacheAction
   ▼
Controller
   │
   ▼
physical pools

The limitation is that current actions can still carry raw torch.Tensor physical indices, for example:

FreeDeviceKV(indices)
FreeComponentDeviceSlot(indices)

That is a convenient in-process representation, but it becomes a representation leak across a real runtime boundary.

So rather than inventing an unrelated HiCacheDaemonProtocol, a more natural direction is to keep extending the existing seam while reducing control-side dependence on raw physical placement.

ReqToTokenPool is the harder ownership boundary

ReqToTokenPool looks like bookkeeping, but it combines three different responsibilities:

request lifetime
+
request-row allocation
+
GPU-resident request -> KV location table

The req_to_token table itself is a device tensor consumed directly by attention backends. Prefill/extend and decode allocation paths also update it directly.

That makes one point increasingly clear:

the device-side ReqToToken table, KV allocator, and model execution should remain inside the same GPU execution domain.

This is also why mechanically converting existing methods into RPCs would be a bad design.

If decode became:

allocate RPC
→ ReqToToken write RPC
→ forward RPC

IPC would immediately enter the per-token hot path.

The first hard invariant is therefore:

No per-token IPC.

Design update — 2026-09-03: ownership is architecture; process topology is implementation

After tracing the current Scheduler, ModelRunner, HiCache transfer path, and CUDA execution model, I no longer think a standalone HiCache transfer daemon should be the first implementation.

The architectural question should first be:

What belongs to control?
What belongs to GPU execution?

rather than:

Which process should own L2?

The boundary I currently prefer is:

The most important conclusion from this round of analysis is:

Process topology is an implementation choice. Ownership boundary is the architecture.

The same runtime contract could eventually have multiple implementations:

LocalGPURuntime     # same process
ProcessGPURuntime   # separate process
SharedGPURuntime    # future shared authority

The first implementation does not need to assume that the final answer is a daemon.

Why compute, H2D, and D2H should stay in one GPU execution domain

Compute/transfer overlap does not come from having another process. It comes from GPU execution resources:

compute stream
H2D stream
D2H stream
CUDA events
copy engines

The real restore dependency is something like:

H2D stream
    │
    │ restore KV
    ▼
KV_READY event
    │
    ▼
compute stream
    │
    ▼
attention / forward

Backup is the reverse:

compute produces KV
    │
    ▼
CUDA event
    │
    ▼
D2H stream
    │
    ▼
pinned Host buffer

When H2D/D2H and model compute share the same GPU execution domain, these dependencies can be expressed directly with local streams and events.

Moving transfer execution into a different CUDA context only to gain process isolation adds complexity instead:

CUDA IPC mapping
cross-process event lifetime
context scheduling
failure cleanup
extra completion protocol

So my current preference is:

move cache-transfer control out of the Scheduler, but keep H2D/D2H execution next to ModelRunner.

That does not imply using one CUDA stream. The GPU Runtime should still independently schedule:

compute stream
H2D stream
D2H stream

and account for decode latency, PCIe/HBM pressure, and transfer priority when deciding how aggressively to overlap work.

Four runtime invariants

At this point it is more useful to lock down invariants than an RPC library.

1. No per-token IPC

These operations should stay runtime-local:

KV alloc/free
ReqToToken write
attention metadata
CUDA Graph replay
kernel launch
physical free-list manipulation

A boundary crossing should represent at least one scheduling/batch decision, not one memory operation.

2. CUDA-dependent ordering stays runtime-local

The control side should not own or manipulate:

cudaStream_t
cudaEvent_t
GPU pointer
CUDA Graph execution state

It describes dependencies and intent instead.

3. Control does not own physical GPU addresses

The control side may hold identifiers such as:

RequestRuntimeHandle
RuntimePageId / PageRun
CacheExtentRef
generation

but not:

torch.Tensor device_indices
raw GPU pointer
allocator free list

4. Compute/transfer overlap is a runtime responsibility

The Scheduler may say:

Batch N+1 needs extent X

but the runtime decides:

which H2D stream
which event
when compute waits
whether D2H write-back should be throttled

PrepareBatch / ExecuteBatch gives overlap a better window than one large forward RPC

If the only API is:

execute_batch(N+1)

then the runtime learns about missing cache state only when N+1 is ready to run. That leaves little room to hide H2D behind the current batch.

A two-stage API is more useful:

prepare_batch(plan)
    -> PreparedBatchHandle

execute_batch(handle)
    -> BatchCompletion

While the GPU is computing Batch N, the Scheduler can already prepare N+1:

The target timeline is:

time ─────────────────────────────────────>

CPU scheduler    prepare N+1
                 ███████

GPU compute N    ███████████████████

H2D for N+1             ███████

GPU compute N+1                       █████████████

This fits naturally with SGLang's overlap-scheduling direction: while the CPU prepares the next batch, it can also expose cache-prefetch intent early enough for the GPU runtime to act on it.

The runtime does not necessarily need separate Compute and I/O GPU workers

GPU concurrency primarily comes from CUDA streams, not from the number of CPU workers.

A runtime dispatcher can enqueue work onto multiple streams:

GPU Runtime process
│
├── runtime dispatcher / execution loop
│
├── compute stream
├── H2D stream
├── D2H stream
└── CUDA events

CPU-thread separation is more valuable for blocking or host-side work such as:

Mooncake RPC
RDMA completion
NVMe / filesystem
storage retry
Host buffer management

Those tasks can live in storage workers or a separate storage service without also moving CUDA-transfer ownership away from ModelRunner.

RequestRuntimeHandle and CacheExtentRef

If the Scheduler no longer owns the physical allocator, raw req_pool_idx and physical page tensors should not remain long-lived contracts.

A request handle might look conceptually like:

RequestRuntimeHandle {
    runtime_id
    slot_id
    generation
}

The Scheduler keeps the handle, while row allocation and generation authority stay with the runtime.

For prefix cache state, I also would not jump immediately to a heavyweight distributed-object model.

A first phase could use typed page IDs or page runs:

CacheExtentRef {
    runtime_id
    generation
    page_runs[]
}

PageRun {
    first_page
    page_count
}

The important property is not that these IDs become perfectly opaque on day one. It is that:

their lifecycle is defined by the runtime, and control does not treat them as physical addresses it owns.

If shared replicas or migration later require a stronger abstraction, the representation can evolve into a more opaque extent model.

What TreeCore nodes should store remains the most important open question

Today TreeCore / MatchResult still has a relationship with physical device_indices.

If the long-term goal is a clean control/runtime split, TreeCore values may need to evolve along a path like:

today
raw physical tensors / indices

        ↓

phase 1
runtime page IDs / page runs

        ↓

future
opaque CacheExtentRef

I would not move TreeCore into a remote metadata service in the first phase.

TreeCore, prefix matching, LRU, and lock/ref should remain control-local. The thing to reduce is how much physical placement the control side needs to understand.

The first implementation should be LocalGPURuntime, not a daemon

This is the most concrete engineering conclusion from the current design update.

The first step should look like:

Scheduler
    │
    │ RuntimeInterface
    ▼
LocalGPURuntime
    ├── ReqToToken
    ├── KV allocator
    ├── physical pools
    ├── ModelRunner
    ├── CUDA Graph
    └── H2D / D2H

and still be:

same process

That isolates the ownership-boundary experiment from a second set of problems:

IPC transport
CUDA IPC
cross-process CUDA events
daemon lifecycle
crash cleanup

If the RuntimeInterface cannot be made clean and cheap in the local implementation, moving it out of process would only freeze the wrong seam into a protocol.

When ProcessGPURuntime becomes worth considering

Out-of-process should be triggered by a measurable or architectural benefit rather than being a goal itself.

I would seriously consider it when at least one of these becomes concrete:

1. scheduler/cache CPU interference is measurable in benchmarks;
2. scheduler and GPU runtime need independent lifecycles;
3. shared replicas need one KV-allocation authority;
4. multiple control clients need to drive one GPU execution runtime;
5. stronger fault isolation is required.

At that point, the more natural split is:

Scheduler process
      │
      │ RuntimeInterface over IPC
      ▼
GPU Runtime process
      ├── ModelRunner
      ├── ReqToToken
      ├── allocator
      ├── KV pool
      └── H2D / D2H

rather than the original RFC shape:

Scheduler + ModelRunner
      │ CUDA IPC
      ▼
HiCache transfer daemon

In other words:

if the design does become cross-process, it is more natural for physical GPU execution to become one ownership domain than for transfer execution alone to be cut away from ModelRunner.

L1 ↔ L2 is still the best cache-level validation surface

Even though the first phase is no longer daemon-first, L1 ↔ L2 is still a useful test of the runtime boundary.

A minimal scope could remain:

single scheduler
single GPU / TP-rank semantics
Full Attention first
LocalGPURuntime
TreeCore control-local
L1 ↔ L2 backup / restore

The useful questions are now:

Can prepare_batch(N+1)
start H2D while compute(N) is running?

Can D2H write-back
make progress without hurting decode ITL?

Can Scheduler stop directly depending on
allocator / ReqToToken physical mutation?

That is more meaningful than simply proving that CUDA IPC works.

buffer_only is another reason not to model the API as an L2 service

With buffer_only, Host memory is not really long-lived L2 residency at all. It is staging between GPU memory and storage:

GPU KV
  │ D2H
  ▼
pinned Host staging
  │ storage write
  ▼
L3

or:

L3
  │ storage read
  ▼
pinned Host staging
  │ H2D
  ▼
GPU KV

That suggests PinnedHostBufferPool, H2D/D2H streams, and CUDA-event ordering belong near the GPU Runtime, while storage networking, filesystems, and Mooncake clients can be isolated separately.

A cleaner split is:

GPU Runtime
    ├── GPU-facing Host staging
    └── CUDA transfer ordering

Storage side
    ├── Mooncake / RDMA / network
    ├── NVMe
    ├── retry / replication
    └── storage metadata

The external linker remains orthogonal

I still would not merge UnifiedCacheLinker and the runtime boundary into one abstraction.

UnifiedCacheLinker
  direct L1 <-> external/global cache path

RuntimeInterface
  control <-> physical GPU/cache execution boundary

A linker backend may eventually execute inside a GPU Runtime, but the two abstractions answer different questions.

Shared replicas remain a useful test of whether the abstraction is general enough

#35648 is still a valuable design test because it exposes:

allocation authority
publication
reference lifetime
eviction
generation
failure recovery

A future topology could look like:

Scheduler A ─┐
Scheduler B ─┼── SharedGPURuntime
Scheduler C ─┘       ├ allocator
                      └ shared KV pool

One authoritative allocator is much easier to reason about than each scheduler keeping private raw physical indices.

But this should remain a design test, not v1 scope. The first implementation does not need multi-scheduler TreeCore, distributed refcounts, or MPS lifecycle management.

Updated evolution sequence

I would now stage the work like this:

Phase 0 — define ownership

  Make Scheduler / TreeCore / ReqToToken /
  allocator / pools / streams authority explicit.

Phase 1 — LocalGPURuntime, same process

  Introduce RuntimeInterface.
  Scheduler no longer performs physical KV allocation
  or ReqToToken mutation directly.
  No daemon and no CUDA IPC yet.

Phase 2 — L1 ↔ L2 under RuntimeInterface

  Add prepare_batch / execute_batch.
  Validate compute(N) with H2D(N+1)
  and background D2H write-back.

Phase 3 — benchmark the boundary

  TTFT / ITL / scheduler CPU / GPU bubbles /
  transfer throughput / HBM & PCIe contention.

Phase 4 — optional ProcessGPURuntime

  Replace the local RuntimeInterface with IPC only when
  isolation, shared authority, or measurable interference
  makes the process split worthwhile.

Phase 5 — shared replicas / broader cache state

  Then consider multi-scheduler authority, shared KV,
  and multi-component state such as SWA / Mamba / DSA.

This is still not a roadmap. The goal is to separate the correctness, performance, and process-isolation risks so they can be validated independently.

The question now

The original question was:

Can HiCache L2 run out of process?

It then became:

Can SGLang separate logical cache control from the physical memory/execution runtime?

I think the next useful version is even more precise:

Can SGLang define a GPU-runtime ownership boundary that keeps per-token physical cache work and CUDA ordering local, while allowing the scheduler to prepare future cache dependencies early enough to preserve or improve compute/transfer overlap?

If the answer is yes, a daemon is one deployment of RuntimeInterface, not the architecture itself.

If the answer is no, we should learn exactly what requires tighter coupling: ReqToTokenPool, CUDA Graph, attention metadata, TreeCore physical representation, or speculative/hybrid cache paths.

That is the part of the design I now think is worth validating first.

This post will continue to evolve with #37372. Its purpose is to record how the ownership boundary changes as the code paths and hardware constraints become clearer, not to freeze an RFC too early.