Open-Weight Agent Serving: LoRA-Tuning Gemma 4 26B-A4B MoE on Vertex AI with vLLM

Size Gemma 4 26B-A4B MoE memory and hybrid KV caches, scope LoRA adapters around multimodal vision layers, and connect Google ADK 2.14 directly to vLLM on Vertex AI.

Deep Dive 17 min read November 11, 2026 Code on GitHub
0
Open-Weight Agent Serving: LoRA-Tuning Gemma 4 26B-A4B MoE on Vertex AI with vLLM

Series · Part 3 of 3

Production Agents on Vertex AI

View full track →
  1. 01 Agent Orchestration in Practice: Google ADK vs LangGraph vs AWS Strands
  2. 02 Stateful Agent Memory: Rolling Sessions vs Vector RAG vs Vertex AI Memory Bank
  3. 03 Open-Weight Agent Serving: LoRA-Tuning Gemma 4 26B-A4B MoE on Vertex AI with vLLM
On this page (9 sections)

In Part 1 of this series, we built typed state-graph orchestrators in Google ADK 2.14 and LangGraph 1.2 over live NOAA, Open-Meteo, and National Weather Service telemetry. In Part 2, we replaced append-only chat RAG with slot-reconciling memory (VertexAiMemoryBankService and SQLite) so mutating boat drafts and crew safety limits stay accurate across months of trips.

Once an agent handles both multi-source tool routing and background memory extraction at steady volume, model serving becomes your primary architectural lever. Teams move high-frequency tool dispatch and structured JSON extraction onto open-weight models for three concrete reasons: Apache 2.0 weight ownership, predictable dedicated endpoint latency without shared per-minute token quotas, and compact Low-Rank Adaptation (LoRA) cartridges tuned to domain schemas.

This guide completes our three-part engineering series on building, remembering, and serving production agents on Google Cloud Vertex AI:

  1. Part 1: Agent Orchestration in Practice: Typed state graphs, parallel marine API fan-out, and LLM round-trip budgets across Google ADK, LangGraph, and AWS Strands.
  2. Part 2: Stateful Agent Memory: Rolling windows, keyword recall, SQLite vector RAG, and consolidated slot memory across an 8-session coastal history.
  3. Part 3 (this guide): Sizing GPU memory for Gemma 4 (26B-A4B Mixture-of-Experts), LoRA fine-tuning its language backbone without hitting multimodal layer traps, serving multi-LoRA adapters on Vertex AI with vLLM, and connecting Google ADK 2.14 directly over OpenAI-compatible HTTP.

Before Setting Sail

  • Python 3.11+ with a virtual environment
  • Familiarity with GPU memory fundamentals (see GPU Sizing for ML Workloads) and Google ADK BaseLlm from Parts 1 and 2
  • A personal Google Cloud project with Vertex AI and Cloud Storage enabled for custom training or endpoint deployment (optional; all sizing math, LoRA tree audits, and ADK loopback tests run offline)
  • Basic familiarity with Hugging Face PEFT, TRL SFTTrainer, and vLLM serving flags

Complete Runnable Companion Package

Every calculator, SFT curation validator, PEFT regex auditor, Vertex AI vLLM container builder, and ADK VllmOpenAiLlm adapter in this guide lives in github.com/steve-largo/largo-tutorials/tree/main/agents/gemma4-vertex-vllm. Clone it to run the 6-test hermetic pytest suite and capacity scorecard offline in under two seconds before provisioning a single cloud GPU.

Where Standard Gemma Recipes Break on 26B-A4B

Moving a tool-calling agent from a managed Gemini endpoint onto open-weight google/gemma-4-26B-A4B-it exposes three failure modes that rarely appear in quickstart notebooks:

1. Sizing GPUs by 3.8B active params ──▶ OOM at container startup (25.2B weights stay resident)
2. Unscoped PEFT target_modules      ──▶ ValueError on vision_tower Gemma4ClippableLinear
3. ADK Gemma wrapper or base ID call ──▶ Clears config.tools=[] or skips LoRA weights
  1. Active FLOPs do not reduce resident VRAM: Gemma 4 26B-A4B activates only 3.8B parameters per token (8 routed experts out of 128 plus 1 shared expert), giving you the decode speed of a 4B dense model. Every single one of those 128 experts per layer still has to sit in GPU HBM (25.2B total parameters) because the router can pick a different subset of 8 experts on the very next token.
  2. Multimodal linear wrappers break naive LoRA configs: Gemma 4 ships with a multimodal vision_tower and projector built from Gemma4ClippableLinear modules rather than standard torch.nn.Linear. Passing a flat PEFT list such as target_modules=["q_proj", "k_proj", "v_proj", "o_proj"] matches 163 vision projection layers and raises an unsupported-module ValueError before step 1.
  3. Framework and alias mismatches silently degrade tool calling: Inside google-adk==2.14.0, the built-in google.adk.models.gemma_llm.Gemma class is explicitly written for Gemma 3 on Google AI Studio; its request preprocessor moves system_instruction into a user turn and clears llm_request.config.tools = []. Even after replacing that wrapper, sending model="google/gemma-4-26B-A4B-it" to vLLM instead of your --lora-modules alias serves the unadapted base weights with HTTP 200 and zero warnings.

Let’s walk through each piece in code.

Step 1: Sizing MoE Weights and the Hybrid KV Cache

In our earlier GPU Sizing for ML Workloads and DeepSeek V3 Architecture guides, we separated static weight memory from per-sequence KV-cache growth. google/gemma-4-26B-A4B-it combines two architectural choices that directly shape its serving footprint:

  • Sparse Mixture-of-Experts (25.2B resident, 3.8B active): Across 30 transformer blocks (hidden_size=2560), each MoE block holds 1 shared SwiGLU expert and 128 routed experts (top-8 active per token), alongside a ~550M vision tower and 262K vocabulary.
  • Hybrid Sliding-Window + Global Unified-KV Attention (5:1 ratio): Out of 30 layers, 25 are local sliding-window attention layers capped at 1,024 tokens (head_dim=256, 8 KV heads). Only 5 layers maintain global full-context caches (head_dim=512, 2 KV heads with unified key-value sharing; allocated in uniform (8, 256) paged KV blocks in vLLM).

Let’s codify exact architectural specs and compute both resident weights and hybrid KV-cache budgets across Vertex AI g2 (NVIDIA L4) and a2 (NVIDIA A100) instances in gemma4_serving/sizing.py:

# gemma4_serving/sizing.py
from typing import Literal
from gemma4_serving.schemas import GpuSkuSpec, ModelArchSpec, PrecisionFormat, VramBreakdown

_BYTES_PER_PARAM: dict[PrecisionFormat, float] = {
    PrecisionFormat.BF16: 2.0,
    PrecisionFormat.FP8: 1.05,     # 1 byte + per-channel FP16 scale metadata
    PrecisionFormat.AWQ_INT4: 0.56,  # 0.5 byte + group-128 scales and zero-points
}

GEMMA4_26B_A4B_SPEC = ModelArchSpec(
    model_id="google/gemma-4-26B-A4B-it",
    total_params_billions=25.2,
    active_params_billions=3.8,
    num_layers=30,
    sliding_window_layers=25,
    global_attention_layers=5,
    sliding_window_tokens=1024,
    num_kv_heads=8,
    head_dim=256,
    max_context_tokens=262144,
)


def compute_hybrid_kv_cache_mb_per_seq(
    arch: ModelArchSpec,
    sequence_length: int,
    kv_cache_dtype: Literal["bfloat16", "fp8"] = "fp8",
) -> float:
    """Compute per-sequence KV-cache memory (MB) under vLLM's uniform (8, 256) paged allocator.

    Note: Custom kernels with native heterogeneous K=V sharing (2 heads x 512 dim x 1 buffer)
    further reduce the 5 global layers at 16K FP8 from 320.00 MB to 80.00 MB (180.00 MB total).
    """
    bytes_per_elem = 2.0 if kv_cache_dtype == "bfloat16" else 1.0
    # 2 tensors (K and V) * num_kv_heads * head_dim * bytes_per_elem
    bytes_per_layer_token = 2.0 * arch.num_kv_heads * arch.head_dim * bytes_per_elem

    local_tokens = min(sequence_length, arch.sliding_window_tokens)
    local_bytes = arch.sliding_window_layers * local_tokens * bytes_per_layer_token
    global_bytes = arch.global_attention_layers * sequence_length * bytes_per_layer_token
    return round((local_bytes + global_bytes) / (1024.0 * 1024.0), 2)

Subtracting weights, CUDA graph workspace (1.8 to 2.6 GB), and a rank-16 LoRA adapter pool (0.25 GB) from vLLM’s 0.92 gpu_memory_utilization ceiling gives us the analytical concurrency matrix for 16,384-token agent sessions (--kv-cache-dtype=fp8):

SKUGPUs (VRAM)PrecisionWeightsUsable VRAMMax 16K Seqs
g2-standard-121x L4 (24 GB)fp826.5 GB22.1 GB0 (OOM)
g2-standard-121x L4 (24 GB)awq_int414.1 GB22.1 GB14 seqs
g2-standard-242x L4 (48 GB)bfloat1650.4 GB44.2 GB0 (OOM)
g2-standard-242x L4 (48 GB)fp826.5 GB44.2 GB29 seqs
a2-highgpu-1g1x A100 (40 GB)fp826.5 GB36.8 GB20 seqs
a2-ultragpu-1g1x A100 (80 GB)bfloat1650.4 GB73.6 GB51 seqs

Note: All VRAM and concurrency figures above are analytical calculations from gemma4_serving.sizing using google/gemma-4-26B-A4B-it layer dimensions and 0.92 vLLM HBM utilization. You can adjust context length from 2K to 128K interactively in the GPU VRAM, KV-Cache & Serving Cost Lab.

Look closely at two outcomes in that table:

  1. Hybrid attention cuts 16K KV-cache memory by 78.1%: Caching 16,384 tokens across all 30 layers at full (8, 256) head width would consume 1,920.00 MB per sequence. Because 25 of Gemma 4’s 30 layers wrap a 1,024-token ring buffer (100.00 MB), vLLM’s uniform (8, 256) paged KV allocator needs 420.00 MB per 16K agent session in FP8 (320.00 MB across the 5 global layers, or 180.00 MB total on runtimes with native heterogeneous K=V kernels).
  2. g2-standard-24 (2x L4 in FP8) is the price-performance sweet spot: NVIDIA L4 GPUs include Ada Lovelace fourth-generation Tensor Cores with native hardware FP8 execution. Tensor-parallel FP8 across two L4s (48 GB total HBM) leaves 15.1 GB of KV-cache headroom after weights and CUDA graphs, supporting 29 concurrent 16K-token sessions without paying A100 80GB node rates.

Step 2: Chronological SFT Curation Without Data Leakage

Before launching LoRA fine-tuning, we need clean supervised fine-tuning (SFT) trajectories covering our two highest-volume agent tasks: dispatching typed NOAA/NWS/Open-Meteo tool calls (Part 1) and emitting consolidated MemoryReconcileOutput JSON patches (Part 2).

Three data hygiene invariants matter when curating production agent logs (gemma4_serving/sft_data.py):

  1. Validate every assistant target against strict Pydantic v2 contracts before writing training rows: Training LoRA weights on even 5% malformed historical tool calls teaches the adapter to emit invalid station IDs or unparseable JSON arguments.
  2. Strip <thought>...</thought> scratchpads from historical turns: Leaving verbose internal reasoning traces inside assistant message targets inflates output token counts at serving time and pollutes structured JSON extraction turns.
  3. Split chronologically (train before val), never via random shuffle(): Coastal weather patterns, seasonal runs, and multi-session user updates are time-correlated. Random row shuffling leaks Tuesday’s tropical storm advisory into training while testing on Monday’s precursor turns, overstating validation accuracy.
# gemma4_serving/sft_data.py (core curation pipeline)
import re
from pydantic import ValidationError
from gemma4_serving.schemas import (
    CurationAuditReport,
    MarineToolCall,
    MemoryReconcileOutput,
    SftConversationRecord,
    SftMessage,
)

_THOUGHT_BLOCK_RE = re.compile(r"<thought>[\s\S]*?</thought>\s*", flags=re.IGNORECASE)


def strip_thought_traces(text: str) -> tuple[str, int]:
    """Remove `<thought>...</thought>` blocks and return `(cleaned_text, count_removed)`."""
    matches = list(_THOUGHT_BLOCK_RE.finditer(text))
    if not matches:
        return text.strip(), 0
    return _THOUGHT_BLOCK_RE.sub("", text).strip(), len(matches)


def curate_and_split_sft_dataset(
    raw_candidates: list[dict],
    train_fraction: float = 0.8,
) -> tuple[list[SftConversationRecord], list[SftConversationRecord], CurationAuditReport]:
    """Validate schemas, strip reasoning traces, and split strictly by chronological_index."""
    ordered = sorted(raw_candidates, key=lambda item: int(item["chronological_index"]))
    accepted: list[SftConversationRecord] = []
    rejected_malformed = 0
    stripped_thoughts = 0

    for item in ordered:
        task_type = item["task_type"]
        valid = True
        cleaned_messages: list[SftMessage] = []

        for msg in item["messages"]:
            cleaned_text, n_removed = strip_thought_traces(msg.get("content", ""))
            stripped_thoughts += n_removed

            if msg["role"] == "assistant" and msg.get("tool_calls"):
                for tc in msg["tool_calls"]:
                    try:
                        MarineToolCall.model_validate(tc)
                    except ValidationError:
                        valid = False
                        break
            elif msg["role"] == "assistant" and task_type == "memory_reconciliation":
                try:
                    MemoryReconcileOutput.model_validate_json(cleaned_text)
                except ValidationError:
                    valid = False

            if not valid:
                break
            cleaned_messages.append(
                SftMessage(
                    role=msg["role"],
                    content=cleaned_text,
                    tool_calls=msg.get("tool_calls"),
                    tool_call_id=msg.get("tool_call_id"),
                )
            )

        if not valid:
            rejected_malformed += 1
            continue

        accepted.append(
            SftConversationRecord(
                record_id=item["record_id"],
                chronological_index=int(item["chronological_index"]),
                task_type=task_type,
                messages=cleaned_messages,
            )
        )

    split_idx = max(1, min(len(accepted) - 1, int(len(accepted) * train_fraction)))
    train_split, val_split = accepted[:split_idx], accepted[split_idx:]
    report = CurationAuditReport(
        raw_candidates=len(raw_candidates),
        accepted_records=len(accepted),
        rejected_malformed_args=rejected_malformed,
        stripped_thought_blocks=stripped_thoughts,
        train_count=len(train_split),
        val_count=len(val_split),
    )
    return train_split, val_split, report

When writing curated splits to disk (export_sft_jsonl), gemma4_serving/config.py enforces validate_artifact_jsonl_path, rejecting .. path traversal segments and confining exports to an explicit artifact directory.

Step 3: Scoping PEFT LoRA Around Gemma4ClippableLinear and Routers

With clean JSONL splits ready, we configure peft.LoraConfig and trl.SFTTrainer (transformers>=5.5.2, trl>=1.1.0). Here is where Gemma 4’s multimodal MoE parameter tree requires surgical regex scoping (gemma4_serving/train_lora.py):

  1. Exclude vision_tower and multi_modal_projector: Gemma 4 wraps its 163 vision encoder and projector linear projections in Gemma4ClippableLinear to bound activation ranges. Standard PEFT LoRA injection only supports nn.Linear, nn.Embedding, nn.Conv2d, and Conv1D, so matching vision_tower.*.q_proj aborts initialization.
  2. Freeze the 30 mlp.router top-8 gating layers: On domain datasets of 1,000 to 20,000 examples, adapting MoE router logits shifts token dispatch away from pre-trained expert specializations and causes expert collapse. Keep mlp.router frozen.
  3. Choose deliberately between Attention + Shared Expert vs All 128 Routed Experts:
    • Adapting self_attn.{q,k,v,o}_proj plus shared_expert.{gate,up,down}_proj touches 210 Linear layers (16.71M parameters at r=16, or 0.0663% of the model), yielding a ~33 MB BF16 safetensors adapter that loads instantaneously into vLLM’s multi-LoRA cache.
    • Adapting all 128 routed experts (experts.\d+.{gate,up,down}_proj) touches 11,730 Linear layers (630.13M parameters at r=16, or 2.50%), inflating the adapter file nearly 38x (~1.26 GB) while spreading gradient updates thinly across 128 sparse branches.
# gemma4_serving/train_lora.py
from gemma4_serving.config import ServingRuntimePolicy, ServingSettings

# Compact default: adapts language attention + always-active shared expert (210 Linear layers)
GEMMA4_ATTN_SHARED_ONLY_REGEX = (
    r"^model\.language_model\.layers\.\d+\."
    r"(self_attn\.(q_proj|k_proj|v_proj|o_proj)|"
    r"shared_expert\.(gate_proj|up_proj|down_proj))$"
)

# Full MoE option: includes all 128 routed experts per layer (11,730 Linear layers)
GEMMA4_LANGUAGE_LORA_REGEX = (
    r"^model\.language_model\.layers\.\d+\."
    r"(self_attn\.(q_proj|k_proj|v_proj|o_proj)|"
    r"(mlp|shared_expert|experts\.\d+)\.(gate_proj|up_proj|down_proj))$"
)


def build_peft_lora_config_dict(
    policy: ServingRuntimePolicy,
    include_routed_experts: bool = False,
) -> dict:
    """Return validated kwargs for `peft.LoraConfig` on Gemma 4 26B-A4B."""
    return {
        "r": policy.lora_rank,
        "lora_alpha": policy.lora_alpha,
        "lora_dropout": policy.lora_dropout,
        "bias": "none",
        "task_type": "CAUSAL_LM",
        "target_modules": (
            GEMMA4_LANGUAGE_LORA_REGEX
            if include_routed_experts
            else GEMMA4_ATTN_SHARED_ONLY_REGEX
        ),
    }

Do Not Use target_modules='all-linear' on Multimodal MoE Checkpoints

Passing target_modules="all-linear" on google/gemma-4-26B-A4B-it attaches adapters to all 30 mlp.router gating matrices and every 128-expert MLP tensor simultaneously, producing a multi-gigabyte adapter that degrades expert routing balance and exceeds default vLLM --max-cpu-loras cache budgets. Anchor target_modules with ^model\.language_model\.layers\.\d+\. instead.

In gemma4_serving/train_lora.py, run_trl_sft_lora_job(...) loads AutoModelForCausalLM in torch.bfloat16 (with optional 4-bit NF4 QLoRA for single-A100 training jobs), enables gradient_checkpointing_kwargs={"use_reentrant": False}, trains via trl.SFTTrainer, and writes only the compact adapter_model.safetensors + adapter_config.json artifacts for upload to Cloud Storage (gs://your-bucket/adapters/marine-v1).

Step 4: Connecting Google ADK 2.14 Directly to vLLM (VllmOpenAiLlm)

Once vLLM serves our base model and LoRA adapter, how should our Google ADK LlmAgent from Parts 1 and 2 talk to it?

Inspecting google-adk==2.14.0 reveals why both built-in shortcuts fall short for Gemma 4 on custom vLLM endpoints:

  • google.adk.models.gemma_llm.Gemma is hardcoded for Gemma 3 on the Gemini API. Its _move_function_calls_into_system_instruction hook converts tool declarations into raw markdown prompt text, moves system_instruction into a user message, and sets llm_request.config.tools = []. Gemma 4 natively supports both role="system" turns and structured function-calling tokens (--tool-call-parser=gemma4 in vLLM), so stripping config.tools throws away the exact behavior we just fine-tuned.
  • google.adk.models.lite_llm.LiteLlm requires installing google-adk[extensions] plus litellm’s broad transitive dependency graph just to POST JSON to a single OpenAI-compatible /v1/chat/completions URL.

Because ADK’s BaseLlm abstract class requires implementing just one async generator (generate_content_async), we can write a zero-bloat VllmOpenAiLlm adapter directly over httpx.AsyncClient in ~150 lines (gemma4_serving/vllm_adk_adapter.py):

# gemma4_serving/vllm_adk_adapter.py (excerpt)
import json
from collections.abc import AsyncGenerator
import httpx
from google.adk.models.base_llm import BaseLlm
from google.adk.models.llm_request import LlmRequest
from google.adk.models.llm_response import LlmResponse
from google.genai import types
from pydantic import ConfigDict, Field, PrivateAttr
from typing_extensions import override

from gemma4_serving.config import ServingRuntimePolicy, ServingSettings


class VllmOpenAiLlm(BaseLlm):
    """Direct Google ADK BaseLlm client for vLLM OpenAI-compatible endpoints."""

    model_config = ConfigDict(arbitrary_types_allowed=True)

    settings: ServingSettings
    policy: ServingRuntimePolicy = Field(default_factory=ServingRuntimePolicy)
    _http_client: httpx.AsyncClient | None = PrivateAttr(default=None)

    def __init__(
        self,
        settings: ServingSettings,
        policy: ServingRuntimePolicy | None = None,
        http_client: httpx.AsyncClient | None = None,
    ) -> None:
        resolved_policy = policy or ServingRuntimePolicy()
        super().__init__(
            model=settings.served_model_name,
            settings=settings,
            policy=resolved_policy,
        )
        self._http_client = http_client

    @override
    async def generate_content_async(
        self,
        llm_request: LlmRequest,
        stream: bool = False,
    ) -> AsyncGenerator[LlmResponse, None]:
        payload = translate_adk_request_to_openai_payload(
            llm_request=llm_request,
            served_model_name=self.settings.served_model_name,
            policy=self.policy,
        )
        headers = {"Content-Type": "application/json"}
        if self.settings.vllm_api_key:
            headers["Authorization"] = f"Bearer {self.settings.vllm_api_key}"

        url = f"{self.settings.vllm_api_base}/chat/completions"
        owns_client = self._http_client is None
        client = self._http_client or httpx.AsyncClient(
            timeout=self.settings.request_timeout_seconds
        )
        try:
            resp = await client.post(url, json=payload, headers=headers)
            resp.raise_for_status()
            body = resp.json()
        finally:
            if owns_client:
                await client.aclose()

        choice = body["choices"][0]
        msg = choice["message"]
        parts: list[types.Part] = []

        if msg.get("content"):
            parts.append(types.Part.from_text(text=str(msg["content"])))

        for tc in msg.get("tool_calls") or []:
            fn = tc["function"]
            raw_args = fn.get("arguments", "{}")
            parsed_args = json.loads(raw_args) if isinstance(raw_args, str) else dict(raw_args)
            parts.append(
                types.Part(
                    function_call=types.FunctionCall(
                        id=tc.get("id"),
                        name=fn["name"],
                        args=parsed_args,
                    )
                )
            )

        usage = body.get("usage") or {}
        yield LlmResponse(
            model_version=body.get("model", self.settings.served_model_name),
            content=types.Content(role="model", parts=parts),
            turn_complete=True,
            finish_reason=types.FinishReason.STOP,
            usage_metadata=types.GenerateContentResponseUsageMetadata(
                prompt_token_count=int(usage.get("prompt_tokens", 0)),
                candidates_token_count=int(usage.get("completion_tokens", 0)),
                total_token_count=int(usage.get("total_tokens", 0)),
            ),
        )

Two boundary checks in ServingSettings (gemma4_serving/config.py) harden VllmOpenAiLlm before a single packet leaves the process:

  1. SSRF and cleartext token guard: VLLM_API_BASE accepts https:// for remote Vertex AI or Cloud Run endpoints and restricts cleartext http:// strictly to 127.0.0.1 or localhost tunnels, raising a ValidationError if misconfigured toward an unencrypted internal IP (http://10.0.0.42:8000/v1).
  2. LoRA alias bypass guard: When lora_adapter_alias is configured (marine-triage-gemma4-lora), ServingSettings rejects SERVED_MODEL_NAME == BASE_MODEL_ID (google/gemma-4-26B-A4B-it) at startup so you never benchmark or serve unadapted base weights by mistake.

Step 5: Deploying Multi-LoRA vLLM on Vertex AI and Tearing Down Cleanly

Just as we containerized FastAPI and TorchInductor inference in Low-Latency Model Serving, Vertex AI Model Garden provides pre-built pytorch-vllm-serve containers that mount Cloud Storage gs:// adapter URIs directly into vLLM’s dynamic LoRA manager (gemma4_serving/vertex_deploy.py):

# gemma4_serving/vertex_deploy.py (excerpt)
from gemma4_serving.config import ServingRuntimePolicy, ServingSettings
from gemma4_serving.schemas import GpuSkuSpec, PrecisionFormat

VLLM_VERTEX_CONTAINER_URI = (
    "us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/"
    "pytorch-vllm-serve:20260815_0916_RC01"
)


def build_vllm_serving_container_spec(
    settings: ServingSettings,
    policy: ServingRuntimePolicy,
    sku: GpuSkuSpec,
    precision: PrecisionFormat = PrecisionFormat.FP8,
) -> dict:
    """Construct allowlisted pytorch-vllm-serve container arguments for Vertex AI."""
    args = [
        f"--model={settings.base_model_id}",
        f"--tensor-parallel-size={sku.accelerator_count}",
        f"--gpu-memory-utilization={policy.gpu_memory_utilization}",
        f"--max-model-len={policy.max_model_len}",
        "--kv-cache-dtype=fp8",
        "--enable-prefix-caching",
        "--enable-chunked-prefill",
        "--enable-auto-tool-choice",
        "--tool-call-parser=gemma4",
        "--enable-lora",
        f"--max-lora-rank={policy.lora_rank}",
        f"--lora-modules={settings.lora_adapter_alias}={settings.gcs_adapter_uri}",
    ]
    if precision == PrecisionFormat.FP8:
        args.append("--quantization=fp8")
    elif precision == PrecisionFormat.AWQ_INT4:
        args.append("--quantization=awq")
    else:
        args.append("--dtype=bfloat16")

    return {
        "display_name": f"gemma4-26b-a4b-{settings.lora_adapter_alias}",
        "serving_container_image_uri": VLLM_VERTEX_CONTAINER_URI,
        "serving_container_command": ["python3", "-m", "vllm.entrypoints.openai.api_server"],
        "serving_container_args": args,
        "serving_container_ports": [8000],
        "serving_container_health_route": "/health",
        "serving_container_predict_route": "/v1/chat/completions",
        "dedicated_endpoint_enabled": True,
        "machine_type": sku.machine_type,
        "accelerator_type": sku.accelerator_type,
        "accelerator_count": sku.accelerator_count,
    }

Notice how --enable-prefix-caching and --enable-lora reinforce Parts 1 and 2 of our architecture:

  • Prefix caching amortizes ADK system instructions and tool JSON schemas: Every turn in an ADK session shares an identical system prompt and OpenAI tools block (~900 to 1,400 tokens). Automatic prefix caching reuses those KV blocks across requests, trimming time-to-first-token (TTFT).
  • Multi-LoRA serves both agent roles on one GPU pool: Instead of spinning up separate g2-standard-24 endpoints for marine tool dispatch (--lora-modules=marine-tools=gs://...) and background memory consolidation (memory-reconciler=gs://...), vLLM keeps one 26.5 GB FP8 base checkpoint in HBM and swaps 33 MB rank-16 adapters per request based on the model field.

Dedicated Vertex AI Endpoints Bill Per Node-Hour Until Undeployed

Unlike pay-per-token Gemini API calls, a deployed Vertex AI GPU Endpoint bills continuously ($2.55/hr on g2-standard-24, or ~$1,862/month) even at zero QPS. Always call teardown_vertex_endpoint(endpoint) (endpoint.undeploy_all(sync=True)) as soon as a benchmark or staging run finishes.

Step 6: Running the Scorecard, Hermetic Tests, and Break-Even Math

Clone the companion repository and run both the 6-test hermetic pytest suite and the CLI scorecard offline:

git clone https://github.com/steve-largo/largo-tutorials.git
cd largo-tutorials/agents/gemma4-vertex-vllm

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

.venv/bin/pytest -v
.venv/bin/python -m gemma4_serving.cli --seq-len 16384
Output
============================= test session starts ==============================
platform linux -- Python 3.13.15, pytest-9.2.0, pluggy-1.7.0
collected 6 items

tests/test_gemma4_serving.py::test_security_and_lora_alias_boundaries PASSED [ 16%]
tests/test_gemma4_serving.py::test_moe_vram_and_hybrid_kv_cache_invariants PASSED [ 33%]
tests/test_gemma4_serving.py::test_lora_regex_excludes_vision_clippable_linear_and_router PASSED [ 50%]
tests/test_gemma4_serving.py::test_chronological_sft_curation_and_jsonl_export PASSED [ 66%]
tests/test_gemma4_serving.py::test_adk_gemma3_wrapper_vs_gemma4_vllm_request_translation PASSED [ 83%]
tests/test_gemma4_serving.py::test_end_to_end_adk_runner_with_vllm_openai_adapter PASSED [100%]

============================== 6 passed in 1.43s ===============================
=== 1. Analytical VRAM & Concurrency Matrix (seq_len=16384 tokens) ===
Hybrid KV cache / seq (FP8): 420.00 MB (vs 1920.00 MB if all 30 layers cached full global context -> 78.1% KV reduction)

Vertex AI SKU        | GPUs           | Precision | Weights  | Usable   | Max Seqs | Fits
-----------------------------------------------------------------------------------------
g2-standard-12       | 1x L4          | fp8       |  26.5 GB |  22.1 GB |        0 | NO
g2-standard-12       | 1x L4          | awq_int4  |  14.1 GB |  22.1 GB |       14 | YES
g2-standard-24       | 2x L4          | bfloat16  |  50.4 GB |  44.2 GB |        0 | NO
g2-standard-24       | 2x L4          | fp8       |  26.5 GB |  44.2 GB |       29 | YES
a2-highgpu-1g        | 1x TESLA_A100  | fp8       |  26.5 GB |  36.8 GB |       20 | YES
a2-ultragpu-1g       | 1x A100_80GB   | bfloat16  |  50.4 GB |  73.6 GB |       51 | YES

=== 2. Measured Gemma 4 Parameter-Tree LoRA Scoping Audit ===
Attention + Shared Expert (r=16): matched=210 Linear layers, trainable=16.71M params (0.0663%), vision_excluded=163, router_excluded=30
All 128 Routed Experts + Attn (r=16): matched=11730 Linear layers, trainable=630.13M params (2.5005%), vision_excluded=163

=== 3. Measured SFT Trajectory Curation & Schema Audit ===
{'raw_candidates': 12, 'accepted_records': 10, 'rejected_malformed_args': 2, 'stripped_thought_blocks': 5, 'train_count': 8, 'val_count': 2}

=== 4. Analytical Dedicated Endpoint vs Managed Gemini 2.5 Flash Break-Even ===
g2-standard-12 (1x NVIDIA_L4)        | $ 934.40/mo (24/7) | Break-even: 22,925 calls/day | At 50k/day: $934 vs $2,038 managed
g2-standard-24 (2x NVIDIA_L4)        | $1861.50/mo (24/7) | Break-even: 45,672 calls/day | At 50k/day: $1,862 vs $2,038 managed
a2-ultragpu-1g (1x A100_80GB)        | $4270.50/mo (24/7) | Break-even: 104,776 calls/day | At 50k/day: $4,270 vs $2,038 managed

=== 5. Measured ADK 2.14 Runner + VllmOpenAiLlm Loopback Wire Overhead ===
10 two-turn tool-calling sessions (20 OpenAI /v1/chat/completions calls): mean ADK + SQLite + JSON schema translation time = 53.63 ms/session

Dedicated vLLM GPU vs Managed Token API Break-Even

Using an agent turn profile of 1,800 input tokens (system instruction + tool declarations + retrieved slot memory) and 320 output tokens ($1.34 per 1,000 calls at public Gemini 2.5 Flash reference rates of $0.30/1M input and $2.50/1M output), here is where 24/7 Vertex AI on-demand GPU hosting crosses managed per-token billing:

Vertex AI DeploymentPrecision24/7 Monthly CostDaily Break-EvenCost at 50K Calls/Day
Managed Gemini 2.5 FlashManagedPay-per-tokenBaseline$2,038 / mo
g2-standard-12 (1x L4)awq_int4$934.40 / mo22,925 calls/day$934 / mo
g2-standard-24 (2x L4)fp8$1,861.50 / mo45,672 calls/day$1,862 / mo
a2-ultragpu-1g (1x A100 80GB)bfloat16$4,270.50 / mo104,776 calls/day$4,271 / mo

Note: Cost figures are analytical estimates based on published us-central1 on-demand Vertex AI node-hour rates (730 hrs/mo) and public Gemini 2.5 Flash token pricing as of November 2026. Meanwhile, the ADK 2.14 Runner + SqliteSessionService + VllmOpenAiLlm orchestration overhead (53.63 ms per two-turn tool session, or ~26.8 ms per round-trip) was measured locally across 20 loopback /v1/chat/completions invocations.

Putting the 3-Part Architecture Together

Across all three guides in this series, a pragmatic blueprint emerges for production agents on Google Cloud:

  1. Control Flow (Part 1): Enforce deterministic input validation and parallel data fan-out in code (ADK Workflow with JoinNode or LangGraph StateGraph), reserving LLM calls strictly for ambiguity resolution and final synthesis.
  2. Long-Term State (Part 2): Store explicit scalars in SessionService user: state, keep immutable manuals in Vector RAG, and route evolving user facts through slot-reconciling memory (VertexAiMemoryBankService) so stale drafts and transient weather alerts never poison future prompts.
  3. Model Serving (Part 3): Stay on managed pay-per-token Gemini 2.5 Flash when traffic is bursty or below ~23,000 to 45,000 agent calls per day. Cross over to open-weight Gemma 4 26B-A4B MoE with FP8 vLLM on g2-standard-24 (2x L4) when sustained volume justifies dedicated GPUs, when strict weight/version pinning is required, or when compact 33 MB LoRA cartridges let one base checkpoint serve multiple specialized agent skills simultaneously.
Found this walkthrough useful?
0
Mile Marker 106 emblem

Steven W. White · Software engineer and AI systems builder writing from the Florida coast

Independent engineering notes, measured hardware benchmarks, and runnable code from Mile Marker 106.

Keep Exploring

Related Guides

Concept Map →

Discussion

Loading comments...