MemoryFlow Lab

Decision under test

Where should long-context KV cache live?

Compare HBM-only placement, a page-aware remote tier, and near-memory partial-state attention under one synthetic 7B GQA workload. Every result comes from versioned inputs.

Context: 8,192 tokens
Batch: 16 sequences
Weights: FP16
KV cache: FP16 GQA

Feasibility Loading Checking capacity constraints

Mean decode - milliseconds per decode step

Throughput - batched tokens per second

Remote-media reads - page-rounded GiB across decode

Interconnect reads - payload + protocol GiB across decode

Policy comparison

Four scenarios, distinct service paths

Deterministic sensitivity

Break-even and counterexample, not a universal win

64 GB/s link break-even - synthetic peak near-memory throughput

One-at-a-time speedup range - declared input multipliers only

Compute counterexample - winner loading

Peak link bandwidth	Break-even peak compute	Status

Separate PyTorch MPS aggregate

Decode attention: equation check, not system validation

Loading device metadata

Apple M4/MPS median and p95 summaries only; no raw iterations. This is not CUDA, HBM/CXL, near-memory/PIM, or end-to-end validation and does not calibrate the synthetic profile.

Independent roofline MAPE - copy + GEMM calibration

Attention-calibrated MAPE - held-out context lengths

Maximum validation error - calibrated model

Effective stream rate - kernel-specific fit

Validation context	Measured	Independent roofline	Attention-calibrated	Calibrated error

Supporting calibration

Device-copy transfer equation

Bandwidth-only MAPE - held-out sizes

Affine MAPE - base latency + bytes / bandwidth

Fitted bandwidth - decimal GB/s

Fitted base latency - microseconds

Validation size	Measured	Bandwidth-only	Affine	Affine error

Reproducible evidence

Inputs, assumptions, and failure states stay visible

Repository

Policy	Feasible	Mean decode	Throughput	Remote-media read	Interconnect read	Page read amp.	Bottleneck