Projects and code walkthroughs

This page collects the projects I maintain and code walkthroughs based on my study of SGLang, Mooncake, and HiCache. Each project includes its current status and known gaps.

Rust serving prototype

SGLang Rust Runtime

Implements and tests request handling, scheduling, caching, protocols, and PD transfer in Rust.

KV cache platform

NexusKV

Separates cache reuse, state indexing, data movement, and inference-engine integration.

Code walkthroughs

Mooncake and HiCache

Follows real read and write paths through cache lifecycle, Store semantics, and Transfer Engine integration.

Experimental
SGLang Rust Runtime
Repository · github.com/imReese/sglang-rs
An independent Rust prototype based on SGLang's request handling, scheduling, caching, and PD transfer concepts. It is useful for architecture and integration experiments, but it does not yet provide complete model execution, accelerator support, or production deployment.
Current status and limitations
Request lifecycle and several integration paths are implemented and tested. Production model execution, support for accelerator backends, and deployment tooling are still incomplete.
What is implemented
- A Rust implementation of the request lifecycle, prefill/decode scheduling, bounded KV page allocation, and RadixCache-style prefix reuse.
- Tonic gRPC and an OpenAI-compatible HTTP interface, plus a process-level CPU PD smoke test.
- CPU reference execution with runtime detection of CUDA/cuBLAS support and clear errors when no backend is available.
- PD transfer plans, readiness polling, descriptor checks, and memory registration through the Mooncake ABI.
RustSGLangScheduler
Implementation references
Current scope and limitations Request lifecycle tests PD transfer-plan tests Local PD smoke
Request flow
SGLang Rust Runtime — implemented request flow
This diagram shows the flow covered by the current code and tests. Production model execution and complete transport backends are not implemented yet.
1. 1
  Protocol handling
  HTTP / gRPC / router
  Validate text and token requests while preserving the bootstrap information required for PD.
  Typed protocolOpenAI-compatible API
2. 2
  Scheduler
  waiting → prefill → decode
  Form batches within token budgets and wait for transferred KV before decode.
  Request lifecycleReadiness checks
3. 3
  Cache and workers
  RadixCache + KV pages
  Match reusable prefixes, allocate pages from a bounded KV pool, and dispatch work.
  Page layoutPrefix reuse
4. 4
  PD transfer
  in-memory transport / snapshots / Mooncake ABI
  Build and monitor transfer plans; production Mooncake transport is not implemented yet.
  Descriptor checksRegistered memory
Core platform in development
NexusKV
Repository · github.com/imReese/NexusKV
NexusKV separates KV cache reuse planning, state indexing, data movement, and inference-engine integration into independent components. The repository includes runnable baseline implementations and tests; remote transport is still in development.
Current status and limitations
Data structures, reuse planning, engine integration, and baseline backends are implemented. RDMA, GPU Direct, distributed coordination, and performance work are not complete.
What is implemented
- Uses versioned attention-state definitions and a Rust nxradixtree core to plan exact matches and prefix reuse.
- Provides a PyO3 bridge that connects the Rust planner to SGLang and vLLM.
- Includes a baseline executor, backend registry, fallback handling, data handles, and an in-memory store.
- Supports Go control-plane configuration and policy reloads while preserving the last valid configuration on failure.
GoRustPython
Implementation references
Target architecture Migration status Planner bridge Connector lifecycle Execution boundary
Module layout
NexusKV — implemented modules and planned work
Control, reuse planning, data movement, and engine integration live in separate components instead of one connector layer.
Control plane
Configuration and policy reloads are implemented.
Go configurationHealth and administrationExecution policy
Status
Keeps the last valid configuration
State and index
Versioned state formats and a Rust reuse planner are implemented.
DescriptorsnxradixtreeExact and prefix matching
Execution and data transfer
A baseline executor and in-memory backends exist; remote transport is still in development.
Backend registryData handlesFallback handling
Status
Remote transport is not implemented
Engine integration
Request lifecycle integration is implemented for SGLang and vLLM.
Prefill and extend phasesRequest startPlanner integration

Upstream projects I follow

These repositories track upstream projects; the related notes cover the implementation and integration paths I study.

SGLang
Focuses on the scheduler, HiCache, PD transfer, and how they fit into the serving flow.
ServingSchedulerHiCache
Upstream project Related article
Mooncake
Focuses on Store object lifecycle, Transfer Engine memory and data movement, and integration with SGLang.
StoreTransfer EngineKV cache
Upstream project Related article
xgrammar
Used to study how tokenization, grammar compilation, and inference systems work together for structured generation.
Structured generationInference runtimeGrammar compilation
Upstream project

Code walkthroughs

Articles that follow concrete code paths through inference, caching, and data transfer.

SGLang to Mooncake Store KV path
A walkthrough of a complete write and read cycle, from page keys and host-memory KV buffer pointers to Store objects and the shared Transfer Engine.
SGLangMooncakeKV cache
HiCache read and write paths
A closer look at storage hits, loading data back into host memory, GPU reuse, backup, and batch persistence in HiCache.
HiCacheRead and write pathsCorrectness checks
Write path
How Mooncake manages KV cache
How runtime state, Store metadata, replica lifecycle, and Transfer Engine responsibilities fit together.
Cache architectureStoreData transfer

GitHub activity

Recent commits and project updates from GitHub.

View GitHub profile