Interactive AI Systems Engineering Lab

AI Systems Engineering Lab

Explore the three quantitative boundaries governing production AI reliability and cloud unit economics: accelerator HBM and KV-cache scaling, two-stage hybrid retrieval and reranking latency, and stateful multi-turn agent memory consolidation.

Architecture Preset

Select an open-weight model backbone to inspect resident weights, KV-cache scaling, and GPU fit.

VRAM & KV-Cache Controls

Adjust quantization, sequence window, and active batch concurrency.

Weight Precision
KV-Cache Precision
16,384 tokens (16K)
16 active seqs
1 seq 16 seqs 32 seqs 64 seqs

Serving Cost Inputs

Compare 24/7 dedicated GPU hosting against pay-per-token APIs.

50,000 / day

Total HBM Needed

35.5 GB

Single-node target

KV Rate / Token

26.3 KB

420.0 MB / seq

Active Params

3.8B

of 25.2B resident

Smallest Fit

1x A100/H100 80G

$4.50/hr reference

1. GPU Memory Breakdown

30 layers · Hybrid 5:1
Model Weights 26.46 GB

25.2B × 1.05 B/param (74.5%)

KV Cache 6.56 GB

16 seqs × 420.0 MB (18.5%)

CUDA & Activations 2.49 GB

Graphs + prefill buffer (7.0%)

2. Attention Mechanism KV-Cache Comparison

KV cache across 16 seqs @ 16,384 tokens (FP8) if this backbone used each layout:

3. Hardware Target Fit Matrix

Includes tensor-parallel NCCL collective buffers on multi-GPU nodes and 0.92 vLLM allocator ceiling.

Hardware Target Status & Headroom Max Seqs Est. $/hr

4. Monthly Self-Hosted vs Managed Token API Break-Even

Pay-per-Token Favored

Managed Token API

$2,038 / mo

$1.34 per 1K calls

24/7 Self-Hosted GPU

$3,285 / mo

1x A100 / H100 80GB

Break-Even Volume

80,597 / day

170.9M tok/day

Analytical Model

Serving, Retrieval & Context-Scaling Equations

Each simulator combines measured empirical anchors from the companion benchmark suites with analytical scaling equations for accelerator memory, two-stage candidate scoring, and multi-turn token accumulation.

01 · Weight Residency

Total vs Active Parameters

Sparse Mixture-of-Experts models decouple per-token FLOPs from HBM capacity. Even when Gemma 4 activates 3.8B parameters per token (or DeepSeek-V3 activates 37B), every routed expert stays resident in GPU HBM:

M_weights = P_total * b_weight

BF16 : 2.00 bytes/param
FP8  : 1.05 bytes/param (E4M3 + scales)
INT4 : 0.56 bytes/param (4-bit + group-128)

02 · Sequence State

KV-Cache Topology Scaling

For batch concurrency B, sequence length S, and element width b_kv, per-layer KV state drops sharply across successive attention layouts:

MHA    = L * 2 * H_q  * d_h * S * b_kv
GQA    = L * 2 * H_kv * d_h * S * b_kv
Hybrid = (L_loc*min(S,W) + L_glob*S)
         * 2 * H_kv * d_h * b_kv
MLA    = L * (d_c + d_rope) * S * b_kv

03 · Roofline & Unit Cost

Prefill Compute vs Decode Bandwidth

Prompt prefill processes input matrices in parallel on Tensor Cores, while autoregressive decode reads active weights plus the active KV cache from HBM at every step:

t_decode >= (P_active*b_w + B*M_kv_seq)
            / BW_HBM

Q_breakeven = (Cost_GPU_hr * 24)
              / Cost_API_per_req

04 · Candidate Fusion

Reciprocal Rank Fusion (RRF)

Fusing Okapi BM25 lexical postings with dense bi-encoder vectors via ordinal ranks avoids brittle linear score calibration across exact SKU and paraphrase queries:

RRF(d) = sum_m  1 / (k + rank_m(d))

k = 60 : balanced lexical + dense recall
k = 20 : sharpens top-1 exact-SKU boost

05 · Precision Gate

Cross-Encoder Stage-2 Budget

Parallel Stage-1 retrieval hides the faster index branch, while Stage-2 joint query-document self-attention scales linearly with rerank depth K:

T_p95 = max(T_bm25, T_embed + T_hnsw)
        + T_rrf + T_cross_enc(K)

score(q,d) = CE([CLS] q [SEP] d [SEP])

06 · Stateful Context

Transcript Growth vs Active Slots

Appending raw conversation history across T turns scales cumulative input tokens quadratically, whereas active-slot consolidation bounds every turn to O(1) width:

Tok_full(T) = T*S_sys + u*T*(T + 1)/2

Tok_slot(T) = sum_t (S_sys + M_slot + w*u)
              + N_compact * Tok_extract

Runnable Code & Benchmarks

Companion Engineering Guides

Explore Concept Constellation →