Stateful Agent Memory: Rolling Sessions vs Vector RAG vs Vertex AI Memory Bank
Benchmark rolling ADK sessions, keyword recall, SQLite vector RAG, and consolidated slot memory across an 8-session coastal history using Google ADK 2.14 and Vertex AI Memory Bank.
Series · Part 2 of 3
Production Agents on Vertex AI
- 01 Agent Orchestration in Practice: Google ADK vs LangGraph vs AWS Strands
- 02 Stateful Agent Memory: Rolling Sessions vs Vector RAG vs Vertex AI Memory Bank
- 03 Open-Weight Agent Serving: LoRA-Tuning Gemma 4 26B-A4B MoE on Vertex AI with vLLM
On this page (13 sections)
- Why Append-Only Transcript RAG Breaks in Month Four
- Two Persistence Planes in Google ADK 2.14
- Step 1: Project Setup and Boundary Validation
- Settings and Session Scope Validation
- Step 2: Stripping Reasoning Thoughts and Implementing the Four Services
- Strategy 3 vs Strategy 4: Append-Only Vector Chunks vs Slot Consolidation
- Step 3: Wiring Memory into the ADK Runner (PreloadMemoryTool vs LoadMemoryTool)
- Step 4: Managed Consolidation with VertexAiMemoryBankService
- Three VertexAiMemoryBankService Behaviors Worth Knowing Before Launch
- Step 5: Running the 8-Session Evaluation Benchmark
- Analyzing What the Numbers Show
- Choosing the Right Memory Layer for Your Agent
- Next in Part 3: Open-Weight Agent Serving with Gemma 4 and vLLM
In Part 1 of this series, we rebuilt our Florida Keys Fishing Report Agent in Google ADK 2.14 and LangGraph 1.2 to enforce deterministic geocoding gates, parallel NOAA/NWS/Open-Meteo fan-out, and explicit sensor degradation states. That solved single-request orchestration, yet every new session still greeted the captain with total amnesia.
Ask most engineering teams how they add long-term memory to an agent, and you will hear the same default answer: chunk every chat transcript, embed each turn, append the vectors to a database, and inject the top four matches into the system prompt. That pattern works well for static reference manuals. On a living user history where facts mutate over months and transient telemetry flows through every conversation, append-only transcript RAG degrades quietly and predictably.
This tutorial is Part 2 of a three-part series on engineering production agents on Google Cloud Vertex AI:
- Part 1: Agent Orchestration in Practice: Typed state graphs, parallel marine API fan-out, and LLM round-trip budgets across Google ADK, LangGraph, and AWS Strands.
- Part 2 (this guide): Stateful agent memory across months of sessions, benchmarking rolling windows, keyword recall, SQLite vector RAG, and slot-level fact consolidation with Google ADK 2.14 and Vertex AI Memory Bank.
- Part 3: Open-Weight Agent Serving: LoRA-tuning and hosting Gemma 4 (
26B-A4BMoE) on Vertex AI with vLLM for low-latency tool dispatch and structured extraction.
Before Setting Sail
- Python 3.11+ with a virtual environment
- Familiarity with Google ADK 2.x Runner and Session concepts from Part 1
- A personal Google Cloud project with Vertex AI enabled (optional; the 8-session benchmark and test suite run offline without credentials)
- Basic familiarity with SQLite (aiosqlite), cosine similarity, and Pydantic v2
Complete Runnable Benchmark Repository
Every module, longitudinal session fixture, security test, and evaluation script in this guide lives in github.com/steve-largo/largo-tutorials/tree/main/agents/agent-memory-benchmark. Clone it to run the offline pytest suite and 8-session scorecard in under two seconds, then flip one flag to run against live Gemini structured extraction, Vertex AI embeddings, and managed Vertex AI Memory Bank.
Why Append-Only Transcript RAG Breaks in Month Four
Consider a realistic six-month log from our Florida Keys marine assistant between May and November:
May 04 (Session 1) ──▶ Vessel setup: 18ft Action Craft skiff, 9-inch draft, Tavernier Creek
May 19 (Session 2) ──▶ Ephemeral check-in: +2.1 ft tide, 22 mph east gusts, stayed in port
Jun 08 (Session 3) ──▶ Crew rule (Sarah): turn around above 2.5 ft chop or <5s wave period
Jul 02 (Session 4) ──▶ Tackle note: barbless 2/0 circle hooks on 60 lb fluoro at Channel Five
Aug 14 (Session 5) ──▶ MUTATION 1: sold skiff; bought 25ft Contender (16-inch draft), Bud N' Mary's
Sep 09 (Session 6) ──▶ Ephemeral check-in: NWS Small Craft Advisory, 28 mph NE winds, 6 ft seas
Oct 03 (Session 7) ──▶ MUTATION 2: relief bands on Contender; Sarah comfortable to 3.0 ft at >=7s
Oct 19 (Session 8) ──▶ Ephemeral check-in: +2.4 ft Whale Harbor king tide, fall mullet run
In November, the captain opens a fresh session and asks: “Low tide across Snake Creek flat drops to 12 inches tomorrow morning. Can my boat clear it safely, and which marina are we leaving from?”
Evaluating that single question exposes four distinct failure modes in naive memory designs:
- State contradiction under cosine similarity: Embedding
"Can my boat clear a 12-inch flat?"produces high cosine similarity against both Session 1 ("18ft Action Craft flats skiff that draws 9 inches") and Session 5 ("25ft Contender bay boat. She draws 16 inches now"). An append-only vector store injects both chunks into<PAST_CONVERSATIONS>without knowing Session 5 superseded Session 1. Answering from the stale 9-inch draft puts a 25-foot bay boat hard aground on a 12-inch sandbar. - Ephemeral telemetry pollution: Marine conversations mix durable user constraints (hull draft, marina slip, tackle rigs) with transient weather observations (September’s 28 mph Small Craft Advisory). Indexing every turn floods retrieval slots with expired wind advisories whenever the user asks about sea-state rules.
- Reasoning trace leakage: Modern thinking models attach internal scratchpad tokens via
types.Part(thought=True)on ADKEventobjects. Persisting raw model events without stripping thought parts leaks speculative chain-of-thought hypotheses into long-term storage. - Foreground extraction latency: Calling an LLM synchronously to summarize and reconcile facts before returning the assistant’s chat reply adds roughly 1.2 to 2.8 seconds (estimated) to user-perceived turn latency.
Solving all four requires separating session state from consolidated long-term memory, stripping internal reasoning traces at the ingestion boundary, reconciling state mutations into versioned slots, and running consolidation off the foreground response path.
Two Persistence Planes in Google ADK 2.14
Before writing a custom memory backend, examine what google-adk 2.14 already separates at the framework level. ADK divides statefulness into BaseSessionService and BaseMemoryService:
┌────────────────────────────────────────────────────────────────────────────┐
│ ADK Runner (app_name, user_id) │
└───────────────────┬────────────────────────────────────┬───────────────────┘
│ │
▼ ▼
┌───────────────────────────────────────┐ ┌─────────────────────────────────┐
│ Plane 1: BaseSessionService │ │ Plane 2: BaseMemoryService │
│ (SqliteSessionService / VertexAi...) │ │ (Consolidated / MemoryBank...) │
├───────────────────────────────────────┤ ├─────────────────────────────────┤
│ • Ordered Event log for 1 thread │ │ • Cross-session fact extraction │
│ • 4-scope key-value State dict: │ │ • Ephemeral noise filtering │
│ - temp:* (single invocation) │ │ • Slot supersession (v1 -> v2) │
│ - key (single session_id) │ │ • Semantic search_memory() │
│ - user:* (all user_id sessions) │ │ • Scope profile lookup │
│ - app:* (global across app_name) │ │ • Async background ingestion │
└───────────────────────────────────────┘ └─────────────────────────────────┘
Many teams reach for a vector database just to remember a user’s unit preference (imperial vs metric) or home timezone. In ADK 2.14, SqliteSessionService and VertexAiSessionService already persist user: and app: prefixed keys across every future session for that (app_name, user_id) pair with zero embedding calls:
| Scope Prefix | Lifetime | Storage Table (SqliteSessionService) | Ideal Workload |
|---|---|---|---|
temp:key | Current Runner invocation | Discarded before persistence | Scratch tool outputs within 1 turn |
key (no prefix) | Single session_id | sessions.state JSON column | Active trip date or target inlet |
user:key | All sessions for (app, user) | user_states table | Deterministic UI flags, unit system |
app:key | All users in app_name | app_states table | Shared tide datum (MLLW), feature flags |
BaseMemoryService | Months of unstructured chat | canonical_memories / Memory Bank | Evolving natural-language facts |
SqliteSessionService Needs Zero SQLAlchemy Extras
In google-adk 2.14, DatabaseSessionService requires installing google-adk[db] for SQLAlchemy, whereas SqliteSessionService (from google.adk.sessions.sqlite_session_service import SqliteSessionService) ships ready to run on lightweight aiosqlite with WAL mode enabled out of the box.
Step 1: Project Setup and Boundary Validation
Create an isolated virtual environment and install pinned dependencies:
mkdir -p agent-memory-benchmark && cd agent-memory-benchmark
python3 -m venv .venv
.venv/bin/pip install \
"google-adk>=2.14.0,<3.0.0" \
"google-genai>=2.34.0,<3.0.0" \
"aiosqlite>=0.21.0,<1.0.0" \
"pydantic>=2.11.0,<3.0.0" \
"pydantic-settings>=2.8.0,<3.0.0" \
"pytest>=8.3.0,<10.0.0" \
"pytest-asyncio>=0.25.0,<2.0.0"
Settings and Session Scope Validation
Multi-tenant memory bugs carry a much higher blast radius than single-turn orchestration bugs: a missing WHERE user_id = ? filter or an unvalidated path parameter can leak one user’s private history into another user’s prompt. We enforce three boundary checks in config.py and schemas.py before any SQL query or Vertex AI API call executes:
- Bare
agent_engine_idvalidation:VertexAiMemoryBankServiceexpects only the trailing ReasoningEngine resource ID segment (for example"4829104"), rejecting fullprojects/.../reasoningEngines/4829104paths that trigger malformed URL warnings. - SQLite path confinement: Reject
..path traversal segments and restrict local database files to:memory:or.db/.sqlite/.sqlite3suffixes. - Strict
(app_name, user_id)identifier pattern: Validate tenant scope identifiers against^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$and bind them exclusively via parameterized?placeholders.
# memory_bench/config.py
import pathlib
import re
from pydantic import AliasChoices, Field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
_AGENT_ENGINE_ID_PATTERN = re.compile(r"^[A-Za-z0-9_-]{1,128}$")
_ALLOWED_SQLITE_SUFFIXES = frozenset({".db", ".sqlite", ".sqlite3"})
class MemoryBenchSettings(BaseSettings):
model_config = SettingsConfigDict(frozen=True, env_prefix="MEMORY_BENCH_", extra="ignore")
google_cloud_project: str | None = Field(
default=None,
validation_alias=AliasChoices("GOOGLE_CLOUD_PROJECT", "MEMORY_BENCH_GOOGLE_CLOUD_PROJECT"),
)
google_cloud_location: str = Field(
default="us-central1",
validation_alias=AliasChoices("GOOGLE_CLOUD_LOCATION", "MEMORY_BENCH_GOOGLE_CLOUD_LOCATION"),
)
agent_engine_id: str | None = Field(
default=None,
validation_alias=AliasChoices("AGENT_ENGINE_ID", "MEMORY_BENCH_AGENT_ENGINE_ID"),
)
gemini_model: str = Field(default="gemini-2.5-flash")
embedding_model: str = Field(default="gemini-embedding-001")
sqlite_path: str = Field(default=":memory:")
@field_validator("agent_engine_id")
@classmethod
def validate_agent_engine_id(cls, value: str | None) -> str | None:
if not value:
return None
if "/" in value or not _AGENT_ENGINE_ID_PATTERN.match(value):
raise ValueError(
"agent_engine_id must be the bare ReasoningEngine ID segment "
"(extract via agent_engine.api_resource.name.split('/')[-1])."
)
return value
@field_validator("sqlite_path")
@classmethod
def validate_sqlite_path(cls, value: str) -> str:
if value == ":memory:":
return value
if ".." in pathlib.PurePath(value).parts:
raise ValueError(f"Path traversal ('..') is not allowed in sqlite_path: {value!r}")
if pathlib.PurePath(value).suffix.lower() not in _ALLOWED_SQLITE_SUFFIXES:
raise ValueError(f"Unsupported database extension for sqlite_path: {value!r}")
return value
Step 2: Stripping Reasoning Thoughts and Implementing the Four Services
Inspecting google/adk/memory/vertex_ai_memory_bank_service.py in google-adk 2.14 reveals a subtle defense worth copying into every custom BaseMemoryService: before sending session events to Memory Bank, ADK runs _without_thoughts() to strip any Part where part.thought is truthy. Without that filter, Gemini 2.5 thinking traces get embedded alongside user statements.
# memory_bench/services.py (thought-stripping helper)
from google.genai import types
def strip_thought_parts(content: types.Content | None) -> types.Content | None:
"""Strip model reasoning thought parts before memory ingestion."""
if not content or not content.parts:
return content
if not any(part.thought for part in content.parts):
return content
kept_parts = [part for part in content.parts if not part.thought]
if not kept_parts:
return None
return content.model_copy(update={"parts": kept_parts})
Strategy 3 vs Strategy 4: Append-Only Vector Chunks vs Slot Consolidation
In memory_bench/services.py, we implement four BaseMemoryService backends against the exact same ADK add_session_to_memory and search_memory contract:
SlidingWindowMemoryService: Keeps a boundeddeque(maxlen=2)of recent sessions per(app_name, user_id). Zero stale contradictions on recent updates, at the cost of complete amnesia for facts older than two sessions.- ADK
InMemoryMemoryService: Built intogoogle.adk.memory. Splits the query into lowercase word tokens and returns up to 10 raw session events sharing at least one word. SqliteVectorMemoryService(Append-Only Vector RAG): Embeds every non-thought turn into a SQLitevector_chunkstable (PRAGMA journal_mode=WAL) and ranks chunks by L2-normalized cosine similarity.ConsolidatedMemoryService(Local Memory Bank Equivalent): Runs a two-step ingestion pipeline modeled directly on Vertex AI Memory Bank:- Extraction & Ephemeral Filtering: Extracts structured
ExtractedFactCandidate(category, slot_key, fact_text, is_ephemeral)records from session turns and drops transient weather reports (IGNORE_EPHEMERAL). - Slot-Level Reconciliation (
ADD/UPDATE/NOOP): Checks whether an active memory already exists for(app_name, user_id, slot_key). When August’svessel.hull_and_draftupdate arrives, it marks May’sv1recordactive = 0with asuperseded_bypointer tov2, keeping an immutable audit trail while ensuringsearch_memoryandretrieve_profileonly ever surfaceactive = 1facts.
- Extraction & Ephemeral Filtering: Extracts structured
Here is the core reconciliation transaction from ConsolidatedMemoryService:
# memory_bench/services.py (ConsolidatedMemoryService reconciliation excerpt)
cursor = await conn.execute(
"""
SELECT memory_id, fact_text, revision
FROM canonical_memories
WHERE app_name = ? AND user_id = ? AND slot_key = ? AND active = 1
LIMIT 1
""",
(scope_app, scope_user, cand.slot_key),
)
existing = await cursor.fetchone()
if existing is None:
new_id = f"mem-{uuid.uuid4().hex[:12]}"
await conn.execute(
"""
INSERT INTO canonical_memories
(memory_id, app_name, user_id, category, slot_key, fact_text,
embedding_json, revision, source_session_id, updated_at, active, superseded_by)
VALUES (?, ?, ?, ?, ?, ?, ?, 1, ?, ?, 1, NULL)
""",
(new_id, scope_app, scope_user, cand.category, cand.slot_key,
cand.fact_text, json.dumps(vec), session_id, latest_ts),
)
self.reconciliation_counts["ADD"] += 1
else:
prev_id, prev_text, prev_rev = existing
if prev_text.strip().lower() == cand.fact_text.strip().lower():
self.reconciliation_counts["NOOP"] += 1
else:
new_id = f"mem-{uuid.uuid4().hex[:12]}"
await conn.execute(
"""
UPDATE canonical_memories
SET active = 0, superseded_by = ?
WHERE memory_id = ? AND app_name = ? AND user_id = ?
""",
(new_id, prev_id, scope_app, scope_user),
)
await conn.execute(
"""
INSERT INTO canonical_memories
(memory_id, app_name, user_id, category, slot_key, fact_text,
embedding_json, revision, source_session_id, updated_at, active, superseded_by)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1, NULL)
""",
(new_id, scope_app, scope_user, cand.category, cand.slot_key,
cand.fact_text, json.dumps(vec), int(prev_rev) + 1, session_id, latest_ts),
)
self.reconciliation_counts["UPDATE"] += 1
Notice what happens at query time: search_memory queries WHERE app_name = ? AND user_id = ? AND active = 1. Superseded hull drafts and marina slips stay preserved in list_revision_history() for debugging and compliance audits, while never polluting <PAST_CONVERSATIONS>.
Step 3: Wiring Memory into the ADK Runner (PreloadMemoryTool vs LoadMemoryTool)
Google ADK 2.14 ships two built-in tools for reading from BaseMemoryService, and choosing the wrong one quietly doubles your agent’s LLM round-trip latency:
| ADK Tool | Execution Mechanism | Extra LLM Calls | Best Fit |
|---|---|---|---|
PreloadMemoryTool | process_llm_request pre-hook | 0 extra calls (measured) | Personalization and safety constraints needed on every turn |
LoadMemoryTool | Model-callable FunctionTool | 1 extra call when invoked (measured) | Large archival stores where most turns need zero history |
PreloadMemoryTool never registers a tool schema with the model. Instead, its process_llm_request hook intercepts the outgoing LlmRequest, calls tool_context.search_memory(query=user_text), and appends a transient <PAST_CONVERSATIONS> system instruction block before the single Gemini inference call.
Pair PreloadMemoryTool with an after_agent_callback that invokes await callback_context.add_session_to_memory() so completed turns flow back into memory automatically:
# memory_bench/services.py (ADK LlmAgent wiring)
from google.adk.agents import Context, LlmAgent
from google.adk.models.base_llm import BaseLlm
from google.adk.tools.load_memory_tool import LoadMemoryTool
from google.adk.tools.preload_memory_tool import PreloadMemoryTool
from google.genai import types
async def auto_save_session_to_memory(
callback_context: Context,
) -> types.Content | None:
"""Trigger memory ingestion automatically after each completed agent turn."""
await callback_context.add_session_to_memory()
return None
def build_memory_aware_captain_agent(
*,
model: str | BaseLlm = "gemini-2.5-flash",
retrieval_mode: str = "preload",
auto_ingest_on_turn_end: bool = True,
) -> LlmAgent:
tools = [PreloadMemoryTool()] if retrieval_mode == "preload" else [LoadMemoryTool()]
return LlmAgent(
name="KeysMemoryCaptain",
model=model,
instruction=(
"You are a Florida Keys marine captain assistant. Ground vessel clearance, "
"departure marina, crew sea-state comfort rules, and tackle recommendations "
"strictly in recalled user memories and current telemetry."
),
tools=tools,
after_agent_callback=auto_save_session_to_memory if auto_ingest_on_turn_end else None,
)
Step 4: Managed Consolidation with VertexAiMemoryBankService
Running ConsolidatedMemoryService on SQLite gives you deterministic offline tests and single-node edge persistence. When you deploy to multi-container Cloud Run or Vertex AI Agent Engine, managing background extraction queues, LLM retry budgets, and vector indexes yourself becomes undifferentiated plumbing.
In google-adk 2.14, swapping from our local SQLite services to managed Vertex AI Agent Engine sessions and Memory Bank requires zero changes to KeysMemoryCaptain or Runner. Install the Agent Engine SDK extra first:
.venv/bin/pip install "google-cloud-agentplatform[agent-engines]>=2.2.0,<3.0.0"
Then instantiate VertexAiSessionService and VertexAiMemoryBankService via configuration:
# memory_bench/services.py (production cloud switch)
from google.adk.memory import BaseMemoryService, VertexAiMemoryBankService
from google.adk.sessions import VertexAiSessionService
from google.adk.sessions.sqlite_session_service import SqliteSessionService
def build_production_services(
settings: MemoryBenchSettings,
) -> tuple[SqliteSessionService | VertexAiSessionService, BaseMemoryService]:
if settings.agent_engine_id:
session_service = VertexAiSessionService(
project=settings.google_cloud_project,
location=settings.google_cloud_location,
agent_engine_id=settings.agent_engine_id,
)
memory_service = VertexAiMemoryBankService(
project=settings.google_cloud_project,
location=settings.google_cloud_location,
agent_engine_id=settings.agent_engine_id,
)
return session_service, memory_service
db_path = "captain_sessions.db" if settings.sqlite_path == ":memory:" else settings.sqlite_path
return (
SqliteSessionService(db_path=db_path),
ConsolidatedMemoryService(sqlite_path=settings.sqlite_path),
)
Pass Only the Trailing ReasoningEngine ID Segment
When you create an Agent Engine instance via agent_engine = client.agent_engines.create(), its api_resource.name has the form projects/123456/locations/us-central1/reasoningEngines/4829104. Always pass agent_engine.api_resource.name.split("/")[-1] ("4829104") as agent_engine_id to both VertexAiSessionService and VertexAiMemoryBankService. Passing the full resource path causes malformed endpoint construction.
Three VertexAiMemoryBankService Behaviors Worth Knowing Before Launch
Reading vertex_ai_memory_bank_service.py in google-adk 2.14 highlights three production capabilities that rarely appear in quickstart snippets:
- Non-blocking
ingest_eventsby default: Callingawait memory_service.add_session_to_memory(session)schedulesasyncio.create_task()againstclient.agent_engines.memories.ingest_events(...)rather than blocking the user response on LLM consolidation. Foreground turn latency stays near 0 ms of overhead while Vertex AI extracts and reconciles memories asynchronously (roughly 400 to 900 ms estimated RPC trigger time in the background). Inside short-lived CLI scripts, awaitmemory_service._WaitForPendingOperations()before process exit so pending tasks finish cleanly. - Topic filtering and revision TTL via
custom_metadata: When you calladd_events_to_memory(..., custom_metadata={"allowed_topics": ["vessel_specs", "crew_safety", "tackle"], "revision_ttl": "7776000s", "wait_for_completion": False}), ADK automatically switches frommemories.ingest_eventstomemories.generate, restricting what Memory Bank is allowed to persist and expiring superseded historical revisions after 90 days. - Deterministic
retrieve_profilesalongside semantic search: In addition tosearch_memory(),VertexAiMemoryBankServiceexposesawait memory_service.retrieve_profiles(app_name=..., user_id=...)to fetch all consolidated profile memories for a user scope directly, avoiding top-k similarity misses when assembling a compact user header.
Step 5: Running the 8-Session Evaluation Benchmark
To compare all four strategies objectively, memory_bench/evaluator.py ingests all eight coastal sessions from May through November and scores five evaluation probes:
- Probe 1 (Draft & Marina Mutation): Checks whether the retrieved context contains
16 inchesandBud N' Marywhile rejecting stale9 inchesandTavernier Creekstatements. - Probe 2 (Crew Sea-State Mutation): Checks whether Sarah’s updated
3.0 ft/7 secondsrule is recalled without the supersededexceeds 2.5 ftthreshold or September’s28 mph Small Craft Advisory. - Probe 3 (Long-Range Tackle Recall): Tests four-month recall back to July’s Channel Five
barbless 2/0tarpon rig after four newer sessions have arrived. - Probe 4 (Clean Profile Summary): Verifies complete vessel, marina, and crew synthesis with zero weather noise and zero leaked
thought=Truetraces. - Probe 5 (Cross-Tenant Isolation): Queries as
angler_bobagainst the same database and asserts0leaked snippets fromcaptain_steve.
Run the offline test suite and benchmark CLI directly:
.venv/bin/pytest -v
.venv/bin/python -m memory_bench.cli --mode offline-eval
============================= test session starts ==============================
platform linux -- Python 3.13.15, pytest-9.2.0, pluggy-1.7.0
collected 5 items
tests/test_memory_bench.py::test_security_boundary_validators PASSED [ 20%]
tests/test_memory_bench.py::test_adk_sqlite_session_service_four_scope_hierarchy PASSED [ 40%]
tests/test_memory_bench.py::test_thought_parts_stripped_before_memory_ingestion PASSED [ 60%]
tests/test_memory_bench.py::test_adk_runner_preload_vs_load_memory_round_trips PASSED [ 80%]
tests/test_memory_bench.py::test_comparative_8_session_benchmark_invariants PASSED [100%]
============================== 5 passed in 1.54s ===============================
=== 8-Session Coastal Captain Memory Benchmark (May -> November) ===
Strategy | Exact | Recall | Stale | Noise | Leaks | Avg Tok | Search ms
----------------------------------------------------------------------------------------------------------
1. Sliding Session Window (last 2) | 2/5 | 2/5 | 0 | 0 | 0 | 109.6 | 0.11
2. ADK InMemoryMemoryService (Keyword) | 1/5 | 4/5 | 3 | 3 | 0 | 254.8 | 2.06
3. Append-Only Vector RAG (top-4) | 2/5 | 5/5 | 3 | 0 | 0 | 108.4 | 6.36
4. Consolidated Fact Memory (Memory Bank) | 5/5 | 5/5 | 0 | 0 | 0 | 35.6 | 3.10
=== Reconciliation Operations (Consolidated Memory) ===
{'ADD': 4, 'UPDATE': 3, 'NOOP': 0, 'IGNORE_EPHEMERAL': 3}
=== Revision Audit Ledger (captain_steve) ===
[crew.sarah_sea_state_limit v1] (SUPERSEDED -> mem-14789f6c2dcd): Crew safety rule for Sarah: turn around if chop exceeds 2.5 ft or wave period drops under 5 seconds.
[crew.sarah_sea_state_limit v2] (ACTIVE): Crew safety rule for Sarah: comfortable up to 3.0 ft seas when swell period is 7 seconds or longer; still avoid short chop under 5 seconds.
[marina.home_slip v1] (SUPERSEDED -> mem-07e3d397e666): Home launch ramp is Tavernier Creek Marina.
[marina.home_slip v2] (ACTIVE): Home slip is at Bud N' Mary's Marina in Islamorada.
[tackle.tarpon_hook_rig v1] (ACTIVE): Tarpon rig preference at Channel Five bridge: barbless 2/0 circle hooks on 60 lb fluorocarbon leader.
[vessel.hull_and_draft v1] (SUPERSEDED -> mem-e1db1730c696): Primary boat is an 18ft Action Craft flats skiff that draws 9 inches.
[vessel.hull_and_draft v2] (ACTIVE): Primary boat is a 25ft Contender bay boat that draws 16 inches. Analyzing What the Numbers Show
| Metric (Measured Offline) | Sliding Window (k=2) | ADK Keyword Memory | Append-Only Vector RAG | Consolidated Memory |
|---|---|---|---|---|
| Raw recall | 2/5 | 4/5 | 5/5 | 5/5 |
| Exact accuracy (no stale/noise) | 2/5 | 1/5 | 2/5 | 5/5 |
| Stale contradictions | 0 | 3 probes | 3 probes | 0 |
| Weather noise hits | 0 | 3 probes | 0 | 0 |
| Mean injected tokens | 109.6 tok | 254.8 tok | 108.4 tok | 35.6 tok |
Three engineering takeaways jump straight out of the scorecard:
- Raw recall hides state-mutation bugs: Append-only Vector RAG scored a perfect 5/5 on raw recall and 0 on weather noise, which is why standard RAG evals make transcript chunking look healthy. Once we check for contradictory superseded state, its exact pass rate drops to 2/5 because old and new hull drafts share nearly identical semantic neighborhoods.
- Consolidation shrinks prompt token overhead by 67% to 86%: Replacing four raw conversational chunks (108.4 measured tokens) or ten keyword-matched events (254.8 measured tokens) with canonical one-sentence facts drops mean memory injection to 35.6 measured tokens per turn while improving factual precision.
- Cloud vs local consolidation trade-offs: Local SQLite consolidation executes search in 3.10 ms (measured) with zero external network hops, whereas managed
VertexAiMemoryBankServiceoffloads Gemini fact extraction, schema evolution, and multi-region storage behind a non-blockingingest_eventstask (~50 to 150 ms estimatedsearch_memoryRPC latency over Vertex AI).
Choosing the Right Memory Layer for Your Agent
Use the simplest persistence primitive that matches how your state changes over time:
- Use ADK
SqliteSessionServiceorVertexAiSessionServiceuser:keys whenever preferences are explicit, structured scalars (user:preferred_units,user:home_noaa_station). Key-value lookup is exact, instantaneous, and free of LLM extraction variance. - Use Append-Only Vector RAG strictly for immutable reference corpora (coast pilot manuals, Coast Guard navigation rules, tide station metadata) where older paragraphs are not invalidated by newer user turns.
- Use Consolidated Memory (
VertexAiMemoryBankServiceon GCP, or slot-reconciled SQLite/Postgres on edge) whenever users state evolving equipment, team constraints, or preferences across weeks of natural conversation. Pair it withPreloadMemoryToolwhen every turn benefits from personalization, and always strippart.thoughttraces before ingestion.
Next in Part 3: Open-Weight Agent Serving with Gemma 4 and vLLM
Between Part 1’s multi-source orchestration and Part 2’s background fact extraction, a busy production agent executes structured tool calls and JSON schema extractions continuously. Paying frontier proprietary token rates for every structured classification or high-frequency tool dispatch adds up fast.
In Part 3: Open-Weight Agent Serving, we size MoE GPU VRAM and hybrid KV caches for Gemma 4 (26B-A4B Mixture-of-Experts), LoRA fine-tune its language backbone around multimodal Gemma4ClippableLinear traps, connect Google ADK 2.14 directly to vLLM on Vertex AI, and calculate dedicated GPU break-even thresholds against managed Gemini endpoints.
Keep Exploring
Related Guides
Building an AI Trading Agent with Claude and News Signals
Build an automated trading agent that extracts market signals from news using Claude Haiku, executes trades via Alpaca, and manages positions with trailing stops and sentiment monitoring.
CFP Oracle: Semantic Search for College Football History
Build a semantic search system to find historically similar College Football Playoff games using Amazon S3 Vectors and Bedrock embeddings.
Two-Stage Hybrid Retrieval: BM25, Dense Embeddings, RRF, and Cross-Encoder Reranking
Build and benchmark a two-stage hybrid search pipeline combining Okapi BM25, BGE bi-encoder embeddings, Reciprocal Rank Fusion (RRF), and MS MARCO cross-encoder reranking across exact-identifier and semantic-paraphrase slices.
Discussion
to join the discussion.