跳转至

Migration Status

PR 2: Legacy Root Go Baseline

What was broken

  • missing Prometheus dependency in the root Go module
  • incomplete pkg/health implementation
  • incomplete pkg/wal implementation and broken WAL tests
  • nonexistent legacy Raft symbols referenced by cmd/server
  • duplicate-registration risk in pkg/metrics

What was fixed

  • root go test ./... now passes under Go 1.25.9
  • pkg/health now returns a minimal serving response
  • pkg/wal now has a minimal working segmented-log baseline sufficient for the legacy tests
  • pkg/raft now has explicit no-op legacy placeholders so the old server binary compiles
  • pkg/metrics registration is guarded with sync.Once

What remains intentionally legacy

  • the root Go tree is still a legacy prototype, not the future architecture
  • Raft remains a compile-time placeholder baseline, not a production implementation
  • the legacy storage engine remains stubby and is not being evolved into the new data plane

PR 3: Shared Cross-Language Contract

Completed:

  • versioned schema source of truth
  • generated Python and Rust bindings
  • first-class hardware, transfer, tier, and materialization abstractions

PR 4: nxradixtree Planning Boundary

Completed:

  • stable identity model for tree scope and entries
  • rich match result for exact and partial reuse
  • explicit partial-hit planner boundary
  • structured lineage/version, location, and policy placeholders
  • deterministic matching tests across identity boundaries and related prefixes

PR 5: Engine Lifecycle Integration

Completed:

  • explicit SGLang lifecycle surface: prefill, extend, decode
  • explicit vLLM lifecycle surface: request_start, block_table_extend, decode_step
  • connector operation semantics for lookup, materialize, store, prefetch, fallback, and unsupported-path handling
  • connector mapping onto generated QueryKey / ReuseKey / MatchResult / PartialHitPlan
  • clean degrade-to-recompute or simpler-transfer fallback behavior

Current Repo Status

  • legacy Go baseline is stable enough for comparison
  • shared schema now covers descriptor and planner boundary types
  • nxradixtree is a real planning/index subsystem
  • Python connectors are lifecycle-aware and planner-driven
  • Python planner protocol is now backed by a real Rust nxradixtree execution path through a thin extension bridge
  • Python connectors now route planner results through a tier-aware execution boundary instead of keeping materialization policy in connector-local branches
  • execution decisions now flow through a concrete backend protocol with observable materialize, prefetch, store, skip, and recompute calls
  • backend selection is now deterministic through a transport catalog, and the baseline backend has a minimal stateful store model for exact store/materialize testing
  • execution results now carry explicit payload-handle and transfer-session metadata, creating a stable seam for future data-movement implementations
  • Go control-plane and Python execution now share a versioned execution-policy contract for backend enablement, backend priority, allowed tiers/devices/buffers, and fallback behavior
  • execution policy now has a file-based handoff/reload path and backend capability overlays that affect future backend selections without connector changes
  • the versioned Locus bridge now serves lookup, estimate, and materialize across a real process boundary with Rust matcher integration, source/option capability binding, strict errors, and shared conformance fixtures
  • Locus cross-repository E2E validates planning plus prepare/materialize/commit and cold fallback; the receipt remains protocol-only with zero verified physical bytes

Next Planned Step

PR 13 should introduce the first admission/quota policy hook behind the execution policy, keeping quota enforcement explicit and testable before staged-buffer reuse work.

PR 12.5: Go Toolchain Baseline

Completed:

  • updated the root Go module baseline to Go 1.25.9
  • updated the nested go/ control-plane module baseline to Go 1.25.9
  • updated test documentation to use GOTOOLCHAIN=go1.25.9
  • kept this separate from quota/admission work

PR 6: Rust-Backed Planner Bridge

Completed:

  • added a thin Python-to-Rust planner bridge using PyO3
  • extended the shared contract to include planner boundary types
  • kept the Python ReusePlanner protocol stable while replacing fake planner behavior with Rust-backed execution
  • added Python integration tests that exercise connectors against the real Rust planner
  • documented the binding mechanism and its current limitations

PR 7: Tier-Aware Materialization Execution Boundary

Completed:

  • introduced a narrow execution-layer contract for materialize, prefetch, store, recompute, and skip decisions
  • centralized tier-aware and backend-aware execution semantics in a baseline runner
  • integrated SGLang and vLLM connectors with the execution boundary while preserving their lifecycle differences
  • added deterministic tests for exact hits, partial hits, backend downgrade, unsupported prefetch, and recompute fallback
  • documented what is implemented now versus what remains deferred to future transport and tiering work

PR 8: Execution Backend Protocol

Completed:

  • introduced a narrow execution backend protocol for materialize, prefetch, store, skip, and recompute actions
  • added a baseline backend that records calls and returns structured outcomes
  • wired the execution runner to dispatch decisions into backend calls and handle deterministic fallback on backend rejection
  • added focused backend-aware execution tests and connector integration tests against the real Rust planner
  • documented how future RDMA, host-staged, remote shared-store, and file-backed backends will plug into the same boundary

PR 9: Backend Catalog And Minimal Store State

Completed:

  • introduced a deterministic transport backend catalog and registration model
  • added minimal backend implementations for baseline in-memory, staged-copy stub, and remote-shared-store stub paths
  • added a small in-memory store model behind the baseline backend for exact store/materialize tests
  • updated the runner to select backends through the catalog and preserve degraded-path behavior
  • added tests for backend selection, missing backend fallback, minimal store semantics, and connector stability through the real Rust planner

PR 10: Payload Handles And Transfer Sessions

Completed:

  • introduced explicit payload handle, descriptor, location, ownership, and slice types
  • added a narrow transfer request/result/session contract across the execution layer
  • updated backend actions so store receives payload metadata and materialize/prefetch return payload and transfer-session metadata
  • kept transport fake/stubbed while exposing staged-copy intermediate handles and remote-store payload handles
  • added tests proving payload metadata is visible through execution outcomes and end-to-end connector flows

PR 11: Control-Plane Execution Policy Contract

Completed:

  • added a separate versioned execution-policy JSON schema for operator-authored control-plane configuration
  • added Go control-plane execution policy types, validation, load helpers, and render helpers
  • added Python execution policy loading and validation
  • wired backend catalog registration and selection through policy-controlled backend enablement and priority
  • wired runner target/source tier and device/buffer enforcement through policy
  • made recompute versus skip fallback behavior configurable through policy
  • kept SGLang and vLLM connectors policy-agnostic while proving connector behavior remains stable under policy changes

PR 12: Policy Distribution And Backend Overlays

Completed:

  • added file-based Go-to-Python execution policy distribution support
  • added Go-side ExportExecutionPolicy rendering with validation before write
  • added Python FileExecutionPolicyProvider with explicit path and NEXUSKV_EXECUTION_POLICY_PATH support
  • added last-known-good behavior for invalid Python reloads
  • added backend capability overlays for backend enabled state, priority, tiers, devices, buffers, materialization capabilities, and degraded-selection allowance
  • updated the backend catalog to apply overlays deterministically during selection
  • updated the execution runner to consume an active policy provider on every execution call
  • kept SGLang and vLLM connectors policy-agnostic while proving policy reload changes connector-observable execution behavior

PR 13: Quota Admission Policy & Cost Estimator

Completed:

  • updated execution policy schema and Go control-plane validation to support enforced quota mode, max concurrent transfers, and max pinned DRAM bytes
  • added thread-safe QuotaTracker for tracking active runtime bytes, entries, transfers, and pinned memory
  • added CostEstimator with Effective Gain model ($G = T_{compute} - T_{cache}$)
  • integrated quota and cost evaluation directly into BaselineExecutionRunner

PR 14: Benchmark Evidence Pipeline

Completed:

  • added synthetic and replay trace generator (python/nexuskv/benchmarks/trace.py)
  • added three-way strategy comparison evaluator (python/nexuskv/benchmarks/runner.py) for Pure Recompute, Hit-Driven, and NexusKV Cost-Based strategies
  • added metric collector and JSON report generator (python/nexuskv/benchmarks/metrics.py)

PR 15: Multi-Attention Taxonomy (DeepSeek MLA, DSA & Kimi KDA)

Completed:

  • expanded StateSemanticType contract schema with DSA_STATE and KDA_CHECKPOINT
  • regenerated Python and Rust contract bindings
  • added descriptor creation and validation helpers for DeepSeek MLA, DeepSeek DSA, and Kimi KDA architectures in python/nexuskv/adapters/state.py

PR 16: Control-Plane Distributed Fabric & Multi-Tenant Isolation

Completed:

  • added Go control-plane fabric package (go/controlplane/fabric/fabric.go) with thread-safe LeaseManager, EpochTracker, and GarbageCollector
  • added Python tenant security authorizer (python/nexuskv/security/tenant.py) with tenant authorization checks and HMAC-SHA256 keyed hash computation for cache key isolation

PR 17: Native Zero-Copy & Pinned Memory Transport Manager

Completed:

  • added NativePinnedMemoryHandle and ZeroCopyRegistration structs to rust/crates/nexus-transfer
  • added NativeTransportManager (python/nexuskv/execution/native_transport.py) for zero-copy handle registrations and Host DRAM pinned memory management

PR 18: Native Engine Hook Interceptor & Cluster Stress Suite

Completed:

  • added NativeEngineHookInterceptor (python/nexuskv/connectors/native_hooks.py) providing sub-millisecond fail-open fallback guarantees
  • added ClusterStressTestRunner (python/nexuskv/benchmarks/stress.py) for high-concurrency stress testing and RSS memory leak detection

PR 19: Prefill-Decode (PD) Disaggregation Handshake Hook

Completed:

  • added PDDisaggregateContext and on_pd_disaggregate_handshake lifecycle hook method to EngineConnector in python/nexuskv/connectors/base.py

PR 22: Control Plane Discovery, Maturin Wheels, Prometheus Metrics & POSIX SHM

Completed:

  • added NodeDiscoveryService and WorkerHeartbeatMonitor in Go (go/controlplane/fabric/discovery.go) for worker node heartbeat tracking and lease revocation
  • added Maturin PyO3 configuration in rust/crates/bindings-py/pyproject.toml
  • added PrometheusMetricsExporter in Go (go/controlplane/fabric/metrics.go) for Prometheus metric endpoints
  • added PosixShmAllocator in Rust (rust/crates/nexus-transfer/src/shm.rs) for POSIX shared memory zero-copy handles