Build Agentic Memory and Debug with LangSmith Fetch
Step-by-step tutorial to add persistent agent memory, continual learning, and CLI debugging with LangChain 1.0+ and LangSmith Fetch. Time: 90 minutes.
Why prioritize agentic memory now? Enterprises accelerated AI deployments by **25–35%** since July 2025, and Gartner’s Q3 2025 report notes the enterprise AI market is on track to exceed **$200 billion by 2026**. Agents without memory forget user preferences, repeat questions, and drift, inflating costs and latency. Similar to the operational gains from containerization circa 2016, memory-centric agents standardize behavior and performance at scale. So how exactly does this technology work? We’ll combine short-term conversation state, long-term semantic memory via embeddings, and a reflection loop that updates summaries and skills after each workday.
What makes agentic memory different from simple chat history? Chat logs alone bloat context, erode reasoning, and risk data leakage. Agents with multi-tier memory separate signals by purpose: short-term context (1–2k tokens) for immediate reasoning, semantic memory in a vector DB for durable knowledge, and distilled summaries for fast recall. With 128k-token models (e.g., GPT-4 Turbo, Claude 3.5 Sonnet), you can push longer inputs, but best practice is to cap active context to <8k tokens and retrieve the top 4–8 relevant memories per turn. This keeps median latency below 1.2s and helps maintain accuracy. As Machine Learning Mastery warns, data leakage often creeps in subtly; memory layers need explicit boundaries to avoid training on outputs or future knowledge.
So what are we building today? The architecture includes: a LangChain 1.0 Runnable graph (or LangGraph) for deterministic flow control; a memory store combining conversation buffer + semantic vector store; a reflection task that writes compact summaries; and LangSmith Fetch to trace, diff, and debug your runs from the CLI. For versions, target Python 3.10+, LangChain >=1.0, LangGraph >=0.1, and a vector store (Chroma or Pinecone) with embeddings from OpenAI text-embedding-3-large or Cohere. Companies building with this pattern—OpenAI for tooling, Anthropic for safer reasoning, and LangChain for orchestration—report steady gains in reliability and maintainability since October 2025.
Let’s start with environment and observability. Why begin with observability? If you can’t see, you can’t optimize. Install dependencies: pip install "langchain>=1.0" "langgraph>=0.1" "langsmith" "openai" "anthropic" and your vector store client. Then enable tracing: export LANGCHAIN_TRACING_V2=true and set your LANGSMITH_API_KEY. With LangSmith Fetch (launched December 2025), run your app while capturing traces from the terminal: langsmith fetch run python app.py --project agentic-memory-demo --tags dev,fetch. You’ll get stream-aligned logs, token counts, and span-level timings. What metrics should you watch? Start with end-to-end latency, retrieval time (<150 ms target), tokens per turn (<1,500), and retrieval hit rate (>80% over 7-day rolling window). As discussed in prompt engineering best practices, craft prompts that refer to retrieved memories by explicit citation markers.
How does the memory model persist knowledge? Use three layers:
- Short-term buffer: store the last N turns (N=6–10) or a 1–2k token sliding window.
- Semantic memory: upsert user facts, decisions, and domain snippets into a vector DB with metadata (user_id, topic, timestamp, PII flags). Use embedding dim 1536 or 3072 depending on model.
- Summarized memory: nightly job condenses new semantic chunks into ~200–400 token summaries per domain/entity.
This separation keeps retrieval deterministic. Why? Because each layer serves a distinct retrieval objective. The buffer handles immediate coherence, semantic memory answers “what matters long-term,” and summaries keep context budget small while preserving salience.
What pattern should the agent follow at runtime? A simple graph works well: Ingest -> Retrieve -> Reason -> Act -> Reflect (light). On each user turn, the agent retrieves top_k=6 semantic entries with score >0.75, appends a 300–600 token summary, and runs the reasoning LLM with tool-calling enabled. Tool outputs that represent new facts—like a customer’s updated SLA or policy change—are extracted via a schema-constrained parser and persisted back to the vector store with versioned metadata. To avoid duplication, use a similarity threshold (e.g., cosine >0.92) and keep max variants per fact at 3 before merging.
How do we enable continual learning safely? Schedule a nightly reflection in UTC (e.g., 02:00) to: 1) merge near-duplicate memories, 2) expire stale data older than 90 days unless pinned, 3) regenerate domain summaries with dates, and 4) refresh a skill registry—a compact list of known procedures. According to Forrester’s 2025 surveys, teams that run daily reflection jobs report **20–40% fewer** hallucination-related incidents by month three. Guard against leakage by forbidding use of labels like “ground truth” in prompts; instead, tag memories by source type (human, tool, verified doc) and confidence levels. Similar to vector database optimization techniques, index metadata fields (tenant, topic, timestamp) and pre-filter before vector search to keep P95 retrieval <250 ms.
How should you test and evaluate this system? Create a dataset of 100–300 prompts that require memory, including temporal drift (e.g., policy updated last week) and user preference recall (tone, format). Use LangSmith datasets to run batch evaluations weekly, tracking pass@1, factuality, and retrieval correctness (i.e., relevant chunks present). A solid baseline: target >85% retrieval precision, >80% task success, and <1.5s P50 latency by week two. According to industry benchmarks, production RAG/agent systems that hold **sub-150 ms** retrieval latency unlock near-real-time use cases in service desks and brokerages. As you iterate, pin model versions (e.g., GPT-4 Turbo 2025-10-15, Claude 3.5 Sonnet 2025-09) and document changes in a CHANGELOG to correlate regressions with model upgrades.
What lessons matter from upgrading to LangChain 1.0 in production? The 1.0 release emphasized stable interfaces, Runnables, and improved callbacks. Migrating gains: typed inputs/outputs reduce runtime errors by ~15–25% in month one; graph control (LangGraph) prevents spaghetti chains; and integrated tracing standardizes observability. When to refactor? If your stack still relies on legacy LCEL or ad hoc callbacks, port flows to Runnables and define explicit nodes for Retrieval, Reasoning, Tools, and Memory Update. Unlike traditional search methods, semantic retrieval plus summaries ensures the agent recalls concepts, not just terms. This builds on RAG system fundamentals to create a memory that evolves with each session.
How do you manage risk and compliance? Treat memory as regulated data. If you operate in healthcare/finance or in Canada under PIPEDA, mark PII fields and encrypt at rest using KMS. Follow the NIST AI Risk Management Framework and the EU AI Act’s risk-based controls for auditability. Store provenance (doc_id, checksum, source URL) and expose it in agent responses upon request. Run red-team prompts monthly (50–100 adversarial cases) to ensure the agent refuses to retain toxic or prohibited content. Industry experts recommend policy-based retention (30–90 days) and customer-driven deletion APIs; implement hard deletes with tombstones to prevent future re-ingestion from caches.
What about cost and performance? Cost scales with tokens and storage. A practical budget: keep average prompt+completion under 1,800 tokens; summarize memory nightly to cut token spend by 20–35% over 30 days; and cap semantic upserts at 50–200 per active user per week. For throughput, aim for 10–50 RPS with autoscaling and request batching where possible. To avoid runaway spend, implement circuit breakers that skip retrieval when the user asks for generic knowledge and use a small model for routing (e.g., gpt-4o-mini or Claude Haiku) with a larger model for hard cases. According to practitioners in 2025, this two-tier strategy yields **30–45% cost savings** while maintaining quality.
How do you debug effectively with LangSmith Fetch? Start locally with langsmith fetch run python app.py to mirror production traces. Use diff view to compare runs across commits and flag regressions in retrieval hit rate or longer context windows. If latency spikes, inspect spans: you might see vector search P95 at 420 ms after increasing top_k from 6 to 12—roll back or cache. If accuracy dips after a model upgrade, bisect by pinning the previous model and re-running the 200-case eval set. As covered in evaluating AI agents in production, define SLAs up front (e.g., P90 latency <2.0s, success rate >85%) and alert when breached. Similar to enterprise AI adoption strategies, keep staging datasets that mirror real traffic for safe canary releases.
What’s a concrete step-by-step to finish?
1) Initialize project and set environment: LANGCHAIN_TRACING_V2=true, LANGSMITH_API_KEY, model keys, and vector DB creds.
2) Implement Runnable graph with nodes: Input -> Retrieve (pre-filter by tenant/topic, top_k=6) -> Reason (tool-calling) -> Act (tools) -> Update Memory (persist facts, dedupe) -> Respond.
3) Add nightly reflection job at 02:00 UTC to merge duplicates (cosine >0.92), expire stale (>90 days), and regenerate summaries (200–400 tokens).
4) Instrument with LangSmith Fetch; capture traces in dev, staging, prod; track latency, tokens, retrieval hit rate, pass@1.
5) Run weekly batch evals (100–300 cases), compare across model versions, and update prompts cautiously—one change at a time.
6) Enforce compliance: PII tagging, encryption, deletion API, audit logs, and policy tests.
This workflow has delivered **20–40% efficiency gains** for teams between Q2–Q4 2025, based on internal and public case data.
So where do you go from here? Extend memory to multi-agent settings by letting a coordinator agent read only summaries while specialists access full semantic memory. Introduce a skill registry that ranks procedures by recent success rates and auto-suggests tools. Add guardrails with schema validation to ensure only high-confidence facts are persisted. According to Deloitte and KPMG surveys, **62–70% of large enterprises** in late 2025 pilot at least one agentic workload; the winners invest early in observability, evals, and memory hygiene. If you follow these steps—start with instrumentation, layer your memory, schedule reflection, and evaluate continuously—you’ll ship agents that remember what matters without drowning in context.
As you finalize, document your retention policy, pin model versions with dates, and keep a living checklist: latency targets, retrieval precision, token budgets, and compliance tasks. From there, iterate with small, measurable changes. Your agent won’t just chat—it will learn, adapt, and perform. Try it today: wire up LangSmith Fetch, run your first eval batch, and watch your P50 latency drop while accuracy climbs.