Leonie Monigatti

Agent Memory: Filesystem vs Database

Mon, 19 Jan 2026 00:00:00 GMT

I’m digesting the current “filesystem vs database” debate for agent memory. Currently I’m seeing 2 camps in how we build agent memory:

On the one side, we have the “file interfaces are all you need” camp.
n the other side, we have the “filesystems are just bad databases” camp.

“File interfaces are all you need” camp

Leaders like Anthropic, Letta, Langchain & LlamaIndex are leaning towards file interfaces because “files are surprisingly effective as agent memory”.

Anthropic’s memory tool treats memory as a set of files (the storage implementation is left up to the developer)
Langsmith’s agent builder also represents memory in as a set of files (the data is stored in a database and files are exposed to the agent as a filesystem)
Letta found that simple filesystem tools like grep and ls outperformed specialized memory or retrieval tools in their benchmarks -LlamaIndex argues that for many use cases a well-organized filesystem with semantic search might be all you need

Agents are good at using filesystems because models are optimized for coding tasks (including CLI operations) duringpost-training.

That’s why we’re seeing a “virtual filesystem” pattern where the agent interface and the storage implementation are decoupled.

“Filesystems are just bad databases” camp

But then you have voices like Dax from OpenCode who rightly points out that “a filesystem is just the worst kind of database”.

swyx and colleagues in the database space warn about accidentally reinventing databases by solving the agent memory problem. Avoid writing worse versions of:

search indexes,
transaction logs,
locking mechanisms,

Trade-offs

It’s important to match the complexity of your system to the complexity of your problem.

Simplicity vs scale

Files are simple and CLI tools can even outperform specialized retrieval tools.

But these CLI tools don’t scale well & can become a bottleneck.

Querying and aggregations

grep can be effective and a hard baseline to beat. And if you want to improve retrieval performance with hybrid or semantic search?

Luckily, there are CLI tools available for semantic search (e.g., semtools or mgrep).

The question remains: How well they scale and how effective agents are at using them when they are not as common in the training data.

Also at some point you might want some aggregations as well.

Plain text vs complex data

File interfaces and native CLI tools are great for plain-text files. What happens when memory becomes multimodal?

Concurrency

If you have a single agent accessing one memory file sequentially, no need to think about this.

If you have a multi-agent system, you want a database before implementing buggy lock mechanisms.

We’re just scratching the surface: security concerns, permission management, schema validation, etc. are more arguments for databases over filesystems for agent memory use cases.

I think this is an interesting conversation and I’mm curious to see where it goes.

Originally posted on X/LinkedIn.

Building AI Agents with Google’s ADK

Wed, 26 Nov 2025 00:00:00 GMT

At the beginning of November 2025, Kaggle ran a 5-Day AI Agents Intensive Course covering core concepts of AI agents and how you can implement them with Google’s Agent Development Kit (ADK). To accompany the course, they released five whitepapers, which we will reference in this blog. This article reflects my study notes from the course.

For this, we will install the google-adk (v1.18.0) library for Python.

%pip install -q -U google-adk

The agents we are building in this tutorial will be powered by gemini-2.5-flash-lite, which is free for experimental use. To use it, you will need to create a Gemini API key in Google AI Studio. Then, make sure to add your API key to your environment variables (or Google Colab Secrets).

import os
from google.colab import userdata

GEMINI_API_KEY = userdata.get('GEMINI_API_KEY')
os.environ["GOOGLE_API_KEY"] = GEMINI_API_KEY

MODEL_NAME = "gemini-2.5-flash-lite"

Agent Fundamentals

Let’s start with creating and running a first simple Agent (or aliased as LlmAgent) in ADK. First, you’ll define an Agent with the following core components:

name: A name to identify the agent.
model: We will be using gemini-2.5-flash-lite with retry_options for automatically handling failures by retrying the request.
description: A description to identify the agent’s purpose.
instructions: Instructions to describe the agent’s goal and how it should behave.
tools: A list of tools that the agent can use (e.g., built-in Google search tool).

from google.genai import types
from google.adk.agents import Agent
from google.adk.models.google_llm import Gemini
from google.adk.tools import google_search

retry_config=types.HttpRetryOptions(
    attempts=5,         # Maximum retry attempts
    exp_base=7,         # Delay multiplier
    initial_delay=1,    # Initial delay before first retry (in seconds)
    http_status_codes=[
        429, # Too Many Requests
        500, # Internal Server Error
        503, # Service Unavailable
        504, # Gateway Timeout
        ] # Retry on these HTTP errors
)

root_agent = Agent(
    name="assistant",
    model=Gemini(
        model=MODEL_NAME,
        retry_options=retry_config
    ),
    description="A simple agent that can answer general questions.",
    instruction="""You are a helpful assistant.
    Use Google Search for current info or if unsure.""",
    tools=[google_search],
)

Next, you will define an orchestrator that will run the agent. For experimentation purposes, you can use the InMemoryRunner. For production, you’d use the base Runner class when you need persistent state between runs (see Memory Management).

from google.adk.runners import InMemoryRunner

runner = InMemoryRunner(agent=root_agent)

And finally, you can call the run_debug function to prompt the agent with a query.

response = await runner.run_debug(
    "When was the Kaggle 5-Day AI Agents Intensive Course happening?",
    verbose=True,
)


 ### Created new session: debug_session_id

User > When was the Kaggle 5-Day AI Agents Intensive Course happening?
assistant > The Kaggle 5-Day AI Agents Intensive Course was happening from November 10th to November 14th, 2025. This course is also available as a self-paced learning guide. It was previously held live from March 31 to April 4, 2025.

And that’s all to get your first single-agent system up and running!

Tools

An AI agent’s capability to call tools to connect them to the outside world is what sets them apart from regular LLM calls and what makes them so powerful. This section discusses four main types of tools an ADK agent can use:

Built-in tools
Function tools
Agent tools
MCP tool

Built-in tools

In the above example, we provided the agent access to a tool called google_search, which is a built-in tool. Some foundation models have built-in tools, where the tool definition is given to the model implicitly. For example, Google’s Gemini API has several built-in tools, such as Google search, code execution, or computer use.

from google.adk.tools import google_search

root_agent = Agent(
    name="assistant_with_builtin_tool",
    model=Gemini(
        model=MODEL_NAME,
        retry_options=retry_config
    ),
    description="A simple agent that can answer general questions.",
    instruction="""You are a helpful assistant.
    Use Google Search for current info or if unsure.""",
    tools=[google_search],
)

runner = InMemoryRunner(agent=root_agent)

response = await runner.run_debug(
    "When was the Kaggle 5-Day AI Agents Intensive Course happening?",
    verbose=True,
)


 ### Created new session: debug_session_id

User > When was the Kaggle 5-Day AI Agents Intensive Course happening?
assistant_with_builtin_tool > The Kaggle 5-Day AI Agents Intensive Course was happening from November 10th to November 14th, 2025. This course was designed to teach participants how to build and deploy intelligent AI agents, covering topics such as agent architectures, tools, memory, and evaluation, and moving from prototype to production. Registration for the course has since closed. However, the course content is expected to be available as a self-paced learning guide by the end of November 2025.

Function Tools

The most common type of tool is the function tool. Developers can define custom functions for all foundation models that support “function calling”.

How you define your custom function will impact how well the agent is able to select and use the right tool for a given task. Therefore, it is important that the tool follows a few best practices:

Docstrings: Enable the agent to understand when and how to use tools
Type Hints: Enable the agent to generate the correct schema
Dictionary Returns: Tools return successful tool calls with tool results or error message for failed tool calls

def get_kaggle_progressions(tier: str) -> dict:
    """Looks up the needed medals to progress in the Kaggle competitions tier
    based on the tier provided by the user.

    Args:
        tier: The name of the Kaggle competitions tier. It should be descriptive,
                e.g., "expert", "master", or "grandmaster".

    Returns:
        Dictionary with status and medal information.
        Success: {"status": "success", "medals": "2 bronze"}
        Error: {"status": "error", "error_message": "Kaggle tier not found"}
    """

    # This simulates looking up Kaggle's competition progression
    medals_database = {
        "expert": "2 bronze",
        "gold debit card": "1 gold and 2 silver",
        "bank transfer": "5 gold",
    }

    medals = medals_database.get(tier.lower())

    if medals is not None:
        return {
            "status": "success",
            "medals": medals,
            }
    else:
        return {
            "status": "error",
            "error_message": f"Payment method '{tier}' not found",
        }

root_agent = Agent(
    name="assistant_with_function_tool",
    model=Gemini(
        model=MODEL_NAME,
        retry_options=retry_config,
        ),
    instruction="""You are a Google Developer Expert for Kaggle.

    For Kaggle tier progression requests use `get_kaggle_progressions()` to find competition medal requirements for each tier.

    If the tool returns status "error", explain the issue to the user clearly.
    """,
    tools=[get_kaggle_progressions],
)

runner = InMemoryRunner(agent=root_agent)

response = await runner.run_debug(
    "How many medals do I need to become a Kaggle Competitions Expert?",
    verbose=True,
)


 ### Created new session: debug_session_id

User > How many medals do I need to become a Kaggle Competitions Expert?
assistant_with_function_tool > [Calling tool: get_kaggle_progressions({'tier': 'expert'})]
assistant_with_function_tool > [Tool result: {'status': 'success', 'medals': '2 bronze'}]

Agent Tools

Another type of tool is the AgentTool, when an agent is invoked as a tool. This allows the primary agent to delegate specific tasks (e.g., calculations) to sub-agents while keeping control over the user interaction.

from google.adk.tools import AgentTool

# Define agent tool
tool_agent = Agent(
  model=MODEL_NAME,
  name="tool_agent",
  description="Returns the capital city for any country or state",
  instruction="""When the user gives you the name of a country (e.g. Germany),
  answer with the name of the capital city of that country.
  Otherwise, tell the user you are not able to help them."""
)

# Define primary agent
root_agent = Agent(
    name="assistant_with_agent_tool",
    model=Gemini(
        model=MODEL_NAME,
        retry_options=retry_config
        ),
  description="Answers user questions and gives advice",
  instruction="""Use the tools you have available to answer the user's questions""",
  tools=[AgentTool(agent=tool_agent)]
)

runner = InMemoryRunner(agent=root_agent)

response = await runner.run_debug(
    "I want to visit Germany. Which city do you recommend I visit first?",
    verbose=True,
)


 ### Created new session: debug_session_id

User > I want to visit Germany. Which city do you recommend I visit first?
assistant_with_agent_tool > [Calling tool: tool_agent({'request': 'What is the capital of Germany?'})]
assistant_with_agent_tool > [Tool result: {'result': 'Berlin'}]
assistant_with_agent_tool > I recommend you visit Berlin first. It's the capital city and a great place to start exploring Germany!

Model Context Protocol (MCP) Tools

Writing custom function tools requires writing and maintaining API clients when you want to connect to external systems, like databases, GitHub, or Google services. Instead of writing your own integrations and API clients, you can leverage the Model Context Protocol (MCP), which is an open standard introduced by Anthropic.

The MCP lets you connect your agent (MCP client) to an external MCP server that provides tools, such as image generation or database access.

To use MCP tools with your agent, you first need to choose an MCP server and a tool. You can use the MCP registry to find one. In this tutorial, we will use the Everything MCP Server, which is a demo server providing a tool called getTinyImage to return a test image.

Next, you will need to create an MCPToolset to integrate an ADK agent with an MCP server. This launches the MCP server, establishes a communication channel, and integrates the tool in the agent’s tool list automatically without the need for any additional integration code.

from google.adk.tools.mcp_tool.mcp_toolset import McpToolset
from google.adk.tools.mcp_tool.mcp_session_manager import StdioConnectionParams
from mcp import StdioServerParameters

# MCP integration with Everything Server
mcp_server = McpToolset(
    connection_params=StdioConnectionParams(
        server_params=StdioServerParameters(
            command="npx",  # Run MCP server via npx
            args=[
                "-y",  # Argument for npx to auto-confirm install
                "@modelcontextprotocol/server-everything",
            ],
            tool_filter=["getTinyImage"],
        ),
        timeout=30,
    )
)

Now, you only have to add the mcp_server to the agent’s tool list and update the agent’s instructions to use it.

# Create image agent with MCP integration
root_agent = Agent(
    name="assistant_with_mcp_tool",
    model=Gemini(
        model=MODEL_NAME,
        retry_options=retry_config
        ),
    instruction="Use the MCP Tool to generate images for user queries",
    tools=[mcp_server],
)

This section discussed the main types of tools. In ADK, you also have long-running function tools and OpenAPI tools.

Memory Management

LLMs are stateless. Without access to memory management, every interaction with them is a completely new interaction. In ADK, you use:

Sessions for short-term memory management
Memory for long-term memory management

Note that since we now want to have conversation history and persistent state between runs, we will no longer use the InMemoryRunner, but instead we will use the base Runner class, which takes session_service for short-term memory and memory_service for long-term memory as input parameters.

Additionally, we cannot use the run_debug() method anymore because it creates a debug session with a debug_session_id. Since we want to distinguish between different sessions, we will need to use the run_async() method, which takes a session ID as input.

Short-Term Memory

Short-term memory most often refers to the conversation history of a session. The conversation history not only records the user queries and the agent’s responses, but also all tool interactions. Therefore, short-term memory can also become a summarized version of the current session for long-running conversations. Short-term memory in ADK is managed by a session_service, which you can pass to the Runner class.

from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService

# Set up Session Management
session_service = InMemorySessionService()

# Set up the agent
root_agent = Agent(
    name="assistant",
    model=Gemini(
        model=MODEL_NAME,
        retry_options=retry_config
    ),
    description="A simple agent that can answer general questions.",
    instruction="""You are a helpful assistant."""
)

# Create the Runner
runner = Runner(
    agent=root_agent,
    app_name="default",
    session_service=session_service
    )

To run the session, we create a new session manually and pass it into the run_async method.

session_name =  "session1"
app_name = runner.app_name
USER_ID = 'default'

# Create a new session
session = await session_service.create_session(
    app_name=app_name,
    user_id=USER_ID,
    session_id=session_name
)

user_queries = [
    "Hi, I am Sam! What is the capital of United States?",
    "Hello! What is my name?",
]

for query in user_queries:
    print(f"\nUser > {query}")

    async for event in runner.run_async(
        user_id=USER_ID,
        session_id=session.id,
        new_message=types.Content(role="user", parts=[types.Part(text=query)])
    ):
      print(f"Assistant > ", event.content.parts[0].text)


User > Hi, I am Sam! What is the capital of United States?
Assistant >  Hi Sam! The capital of the United States is Washington, D.C.

User > Hello! What is my name?
Assistant >  Your name is Sam.

You can see that the agent was able to remember the user’s name.

Below you can see that we recorded four events in the current session session1.

session = await session_service.get_session(
    app_name=app_name,
    user_id=USER_ID,
    session_id=session_name,
)

for event in session.events:
    print(f"{event.content.role}: {event.content.parts[0].text}")

user: Hi, I am Sam! What is the capital of United States?
model: Hi Sam! The capital of the United States is Washington, D.C.
user: Hello! What is my name?
model: Your name is Sam.

Note that here we’re using InMemorySessionService, which stores conversations temporarily in RAM for experimentation. In production, you’d use DatabaseSessionService, which stores conversations permanently in a database, as shown below:

from google.adk.sessions import DatabaseSessionService

db_url = "sqlite:///my_agent_data.db"  # Local SQLite file
session_service = DatabaseSessionService(db_url=db_url)

As you can imagine, recording all events can become a long conversation history, which will lead to higher cost and slower performance, and eventually hit the context window limit. To mitigate this, you can use context compaction, which automatically reduces the context stored in the session. The compaction process summarizes previous events and stores them in a single new event. In ADK, you can use the EventsCompactionConfig class for this.

Another approach to reduce the token number of the static instructions is to cache the request data via context caching. In ADK, you can use the ContextCacheConfig class for this.

Long-term Memory

In contrast to short-term memory, long-term memory persists across multiple conversations in a searchable storage.

In ADK, long-term memory is managed by a memory_service, which has to be first created and then provided to the agent via the Runner class.

from google.adk.memory import InMemoryMemoryService

# Create Session Service
session_service = InMemorySessionService()

# Create Memory Service
memory_service = (InMemoryMemoryService())

# Create runner with BOTH services
runner = Runner(
    agent=root_agent,
    app_name="default",
    session_service=session_service,
    memory_service=memory_service,  # Memory service is now available!
)

Let’s run the session again.

session_name =  "session2"
app_name = runner.app_name
USER_ID = 'default'

# Create a new session
session = await session_service.create_session(
    app_name=app_name,
    user_id=USER_ID,
    session_id=session_name,
)

user_queries = [
    "Hi, I am Sam! What is the capital of United States?",
    "Hello! What is my name?",
]

# Process each query in the list sequentially
for query in user_queries:
    print(f"\nUser > {query}")

    # Stream the agent's response asynchronously
    async for event in runner.run_async(
        user_id=USER_ID,
        session_id=session.id,
        new_message=types.Content(role="user", parts=[types.Part(text=query)])
    ):
      print(f"Assistant > ", event.content.parts[0].text)


User > Hi, I am Sam! What is the capital of United States?
Assistant >  Hi Sam! The capital of the United States is Washington, D.C.

User > Hello! What is my name?
Assistant >  You told me your name is Sam!

As you can see, the general behavior of the agent looks the same to the user and even records similar events in the new session session2.

session = await session_service.get_session(
    app_name=app_name,
    user_id=USER_ID,
    session_id=session_name
)

for event in session.events:
    print(f"{event.content.role}: {event.content.parts[0].text}")

user: Hi, I am Sam! What is the capital of United States?
model: Hi Sam! The capital of the United States is Washington, D.C.
user: Hello! What is my name?
model: You told me your name is Sam!

Notice how this agent has both session_service for short-term memory and memory_service for long-term memory? This is because long-term memory is created by transferring session data to memory using the add_session_to_memory() function. While the InMemoryMemoryService stores the entire conversation history, a managed memory service like the Vertex AI Memory Bank extracts key facts from the conversation history and only stores those in the long-term memory.

You can save session data to long-term memory at the end of a session, in periodic intervals, or after every turn, depending on your use case.

await memory_service.add_session_to_memory(session)

You can also manually search memory with search_memory. Note that the InMemoryMemoryService search with keyword matching, while the VertexAiMemoryBankService uses semantic search.

APP_NAME = runner.app_name

search_response = await memory_service.search_memory(
    app_name=app_name,
    user_id=USER_ID,
    query="What is the user's name?"
)

print(search_response)

memories=[MemoryEntry(content=Content(
  parts=[
    Part(
      text='Hi, I am Sam! What is the capital of United States?'
    ),
  ],
  role='user'
), custom_metadata={}, id=None, author='user', timestamp='2025-11-25T20:40:25.254618'), MemoryEntry(content=Content(
  parts=[
    Part(
      text='Hi Sam! The capital of the United States is Washington, D.C.'
    ),
  ],
  role='model'
), custom_metadata={}, id=None, author='assistant', timestamp='2025-11-25T20:40:25.255173'), MemoryEntry(content=Content(
  parts=[
    Part(
      text='Hello! What is my name?'
    ),
  ],
  role='user'
), custom_metadata={}, id=None, author='user', timestamp='2025-11-25T20:40:25.742681'), MemoryEntry(content=Content(
  parts=[
    Part(
      text='You told me your name is Sam!'
    ),
  ],
  role='model'
), custom_metadata={}, id=None, author='assistant', timestamp='2025-11-25T20:40:25.743219')]

But what good use is storing and searching in memory when your agent can’t access it? To give your agents access to the memory, you can provide them with the built-in memory tools:

load_memory (Reactive): Agent decides when to search memory. This saves tokens and latency, but you run the risk of the agent forgetting to look up the memory.
preload_memory (Proactive): Automatically searches before every turn. This is less efficient, but you’re guaranteed that memory is always available to the agent.

Agent Quality

AI agents are inherently non-deterministic, which makes them unpredictable and difficult to evaluate in traditional ways. Traditional quality assurance practices, such as unit tests, were built for deterministic systems. But an agent can pass all of your unit tests and still fail in production due to wrong decision-making. Therefore, quality assurance in agent systems cannot be treated like a final testing stage but has to be treated as an architectural pillar.

Agent Observability

Without agent observability, you are not able to judge the agent’s decision-making process. Agent observability is reactive. That means, having observability is helpful to have after an error has occurred because it provides you with the required information to debug what went wrong.

The foundational pillars of agent observability are:

Logs tell us what happened: These are atomic events, such as “I was asked a question”, “I decided to user the vector search tool”, and “Vector search failed”
Traces tell us why something happened: They reveal a causal relationship between isolated logs, such as “User Query -> Vector search (failed) -> LLM Error (confused by bad tool output) -> Wrong final answer”
Metrics tell us how well the overall system performed: These can be system metrics, such as performance (latency, error rate), cost (tokens per task, API cost per run), and effectiveness (task completion rate, tool usage frequency), or quality metrics, such as correctness, accuracy, trajectory adherence, safety and responsibility, helpfulness and relevance.

For development debugging, you can use the ADK Web UI. However, for production observability, you can use the built-in LoggingPlugin(), which automatically captures all agent activity:

from google.adk.plugins.logging_plugin import LoggingPlugin

root_agent = Agent(
    name="assistant",
    model=Gemini(
        model=MODEL_NAME,
        retry_options=retry_config
    ),
    description="A simple agent that can answer general questions.",
    instruction="""You are a helpful assistant.
    Use Google Search for current info or if unsure.""",
    tools=[google_search],
)

runner = InMemoryRunner(
    agent=root_agent,
    plugins=[
        LoggingPlugin()  # Handles standard Observability logging across ALL agents
    ],
)

response = await runner.run_debug("When was the Kaggle 5-Day AI Agents Intensive Course happening?")

 ### Created new session: debug_session_id



User > When was the Kaggle 5-Day AI Agents Intensive Course happening?

[logging_plugin] 🚀 USER MESSAGE RECEIVED

[logging_plugin]    Invocation ID: e-7bf69fd2-6e96-4ada-a35d-3f323a36f606

[logging_plugin]    Session ID: debug_session_id

[logging_plugin]    User ID: debug_user_id

[logging_plugin]    App Name: InMemoryRunner

[logging_plugin]    Root Agent: assistant

[logging_plugin]    User Content: text: 'When was the Kaggle 5-Day AI Agents Intensive Course happening?'

[logging_plugin] 🏃 INVOCATION STARTING

[logging_plugin]    Invocation ID: e-7bf69fd2-6e96-4ada-a35d-3f323a36f606

[logging_plugin]    Starting Agent: assistant

[logging_plugin] 🤖 AGENT STARTING

[logging_plugin]    Agent Name: assistant

[logging_plugin]    Invocation ID: e-7bf69fd2-6e96-4ada-a35d-3f323a36f606

[logging_plugin] 🧠 LLM REQUEST

[logging_plugin]    Model: gemini-2.5-flash-lite

[logging_plugin]    Agent: assistant

[logging_plugin]    System Instruction: 'You are a helpful assistant.

    Use Google Search for current info or if unsure.



You are an agent. Your internal name is "assistant". The description about you is "A simple agent that can answer gen...'

[logging_plugin] 🧠 LLM RESPONSE

[logging_plugin]    Agent: assistant

[logging_plugin]    Content: text: 'The Kaggle 5-Day AI Agents Intensive Course was happening from November 10th to November 14th, 2025.'

[logging_plugin]    Token Usage - Input: 63, Output: 55

[logging_plugin] 📢 EVENT YIELDED

[logging_plugin]    Event ID: d46541c5-8037-442f-b8c4-9132e78ca3e8

[logging_plugin]    Author: assistant

[logging_plugin]    Content: text: 'The Kaggle 5-Day AI Agents Intensive Course was happening from November 10th to November 14th, 2025.'

[logging_plugin]    Final Response: True

assistant > The Kaggle 5-Day AI Agents Intensive Course was happening from November 10th to November 14th, 2025.

[logging_plugin] 🤖 AGENT COMPLETED

[logging_plugin]    Agent Name: assistant

[logging_plugin]    Invocation ID: e-7bf69fd2-6e96-4ada-a35d-3f323a36f606

[logging_plugin] ✅ INVOCATION COMPLETED

[logging_plugin]    Invocation ID: e-7bf69fd2-6e96-4ada-a35d-3f323a36f606

[logging_plugin]    Final Agent: assistant

Agent Evaluation

Agent evaluation is the process of evaluating how well an AI agent performs on a task, including its decision-making process. That means, you want to evaluate the agent on two aspects:

Output (end-to-end) evaluation is done via “Outside-in” view: For example, how similar is the agent’s response to the expected response (example metrics are (ask success rate, user satisfaction, overall quality)
Process evaluation is done via “Inside-out view”: For example, did the agent approach a task correctly (planning, tool usage with correct parameters, tool response interpretation, etc.)

Agent evaluation is proactive because by evaluating your agent’s performance regularly, you are able to detect any performance degradation early on.

This is quite a complex topic, and I recommend you deep dive into the original whitepaper.

Multi-agent systems

So far, we’ve only looked at a single agent. However, instead of a single “monolithic” agent, you can also build a multi-agent system of specialized agents.

Multi-agent patterns

You can combine multiple agents in different patterns depending on your use case.

LLM-based (Sub-agents): Use when the agent can dynamically orchestrate sub-agents on its own.
Sequential: Use when deterministic order is important in a linear workflow.
Parallel: Use when you have independent tasks and speed is important.
Loop: Use when you need iterative improvement through repeated cycles

Agent2Agent Protocol

When building multi-agent systems, you might want to integrate an agent that’s not part of your project. For example, different agents can be created using different frameworks, such as CrewAI or LangGraph (cross-framework). Or agents can be implemented in different programming languages, such as Python or Java (cross-language). And finally, you might want to integrate an agent from an external vendor (cross-organization). For these purposes, it is helpful to have a standardized communication protocol, such as the Agent2Agent (A2A) protocol.

If you want to expose your agent and make it accessible to other agents, you can use ADK’s to_a2a() function, as follows:

from google.adk.a2a.utils.agent_to_a2a import to_a2a

my_agent = Agent(
    ...
)

public_agent_app = to_a2a(
    my_agent,
    port=8001, # Port where the agent will be served
)

If you want to consume an agent, you can do so

from google.adk.agents.remote_a2a_agent import (
    RemoteA2aAgent,
    AGENT_CARD_WELL_KNOWN_PATH,
)

remote_product_catalog_agent = RemoteA2aAgent(
    name="product_catalog_agent",
    description="Remote product catalog agent from external vendor that provides product information.",
    # Point to the agent card URL - this is where the A2A protocol metadata lives
    agent_card=f"http://localhost:8001{AGENT_CARD_WELL_KNOWN_PATH}",
)

Summary

This article is a summary of my learnings from Kaggle’s 5-Day AI Agents Intensive Course. During the course, I learned the fundamentals of how to build a first simple agent using Google’s ADK Python library, to its core concepts of different types of tools (including MCP tools) and memory management via sessions (for short-term) and memory (for long-term). Then, I learned that agent quality is a core pillar of agent architectures and not just a testing stage like in traditional quality assurance because of the non-deterministic characteristic of AI agents. Finally, the course touched on different patterns for multi-agent systems and how you can use the A2A protocol to allow collaboration between agents across different languages, frameworks, and organizations.

If you are interested in the details of any section of this blog, I recommend having a look at the free course materials.

References

Exploring Anthropic’s Memory Tool

Tue, 25 Nov 2025 00:00:00 GMT

On September 29, 2025, Anthropic launched the memory tool in beta. This enables Claude-based agents to store and recall information across conversations.

This blog post explores the Claude Developer Platform’s memory tool by implementing a simple agent with memory using the Anthropic Python library. Inspired by Anthropic’s recent pop-up cafe in London, we’re implementing a barista agent that can remember customers and their usual orders across different cafe visits.

Prerequisites

To follow along in this blog post, you will need to install the anthropic Python package (v0.74.1).

%pip install -q -U anthropic

Additionally, you will need an ANTHROPIC_API_KEY, which you can obtain by creating an Anthropic account and navigating to the “API Keys” tab in your dashboard. Once you have your API key, you need to store it in the environment variables.

import os
os.environ['ANTHROPIC_API_KEY'] = "your-anthropic-api-key"

Why do agents need memory?

Without memory, agents aren’t able to remember conversations across different conversations, tasks, or projects. Imagine, frequently visiting the same coffee shop in your neighbourhood over and over again. You’d expect an engaged barista to eventually remember who you are and what your usual order is.

Memory in AI agents allows agents to do exactly that. By storing and recalling information from past conversations, they can learn from past interactions and build knowledge bases over time to improve workflows and user experiences.

Let’s demonstrate this by first implementing a simple agent without memory. (If you’re unfamiliar with implementing an agent without an orchestration framework, you can review my blog post on how to build an AI agent from scratch using Claude.

The code below implements a barista agent as an LLM in a loop that takes the user’s input and responds to it based on the user’s input, the system prompt, and the conversation history (from the current conversation). The user can exit the conversation by typing the /quit command.

from anthropic import Anthropic

SYSTEM_PROMPT = """You're a friendly barista at Anthropic's pop-up cafe.
Respond like an efficient barista during rush hour - friendly but brief.
Always get the name of the customer to write on the cup so that orders don't get mixed up."""

def conversation_without_memory():

    client = Anthropic()
    messages: list[BetaMessageParam] = []

    while True:
        user_input = input("\nYou: ").strip()

        if user_input.lower() == "/quit":
          print("Goodbye!")
          break

        messages.append({"role": "user", "content": user_input})

        response = client.beta.messages.create(
                model="claude-sonnet-4-20250514",
                max_tokens=2048,
                system = SYSTEM_PROMPT,
                messages=messages,
        )

        for content in response.content:
          print(f"\nClaude: {content.text}", end="", flush=True)

          # Store assistant message
          messages.append({"role": "assistant", "content": content.text})

Let’s start a conversation and order a coffee.

conversation_without_memory()


You: Hi, can I please get a regular flat white with oatmilk please?

Claude: Hey there! Absolutely, one regular oat milk flat white coming right up. 

Can I get a name for the cup?
You: Claudia

Claude: Perfect, Claudia! One regular oat milk flat white for you. That'll be $4.50. 

*starts writing "Claudia" on cup*

Will that be for here or to go?
You: /quit
Goodbye!

Good, the barista agent works.

Now, let’s start a new conversation, visit the coffee shop again and order our usual. Note, that starting a new conversation clears the conversation history from the previous conversation. This is like leaving the coffee shop after receiving your order and coming back on the next day.

conversation_without_memory()


You: Hi, it's me Claudia again. Can I please get my usual?

Claude: Hi there! I'm sorry, but I don't actually have a record of previous orders - each conversation is fresh for me. Could you remind me what your usual is? And just to confirm, that's Claudia for the cup, right?

*grabs cup and marker, ready to write*

What can I get started for you today?
You: /quit
Goodbye!

Unfortunately, the barista agent doesn’t remember us nor our usual order. As you can see, without any memory to remember interactions across conversations, interactions feel impersonal and leaves the user with a bad user experience.

How to use the memory tool with Claude

In contrast to implementations by other providers of the memory layer, Anthropic’s memory tool enables it to store and retrieve memory information through a memory file directory (/memory) that persist between sessions according to the Claude Docs. The agent can create, read, update, and delete files in this memory file directory.

You can enable the memory tool using the Anthropic SDK with just two steps. First, you need to implement the client-side handlers to control where and how the information is stored. Then, you only need to include the beta header context-management-2025-06-27 and the memory tool in your API requests.

Step 1: Implement client-side handlers for memory operations

According to Claude’s developer documentation, the memory tool operates client-side. This means, that the agent makes tool calls to perform memory operations and your application executes them locally. This gives developers the control over where and how the memory is stored (e.g, file-based, database, etc.).

To implement the client-side handlers for the memory operations you can subclass BetaAbstractMemoryTool and implement the handlers for each of the following six memory commands:

view: Shows directory contents or file contents
create: Create a file
str_replace: Replace text in a file
insert: Insert text at a specific line in a file
delete: Delete a file or directory
rename: Rename a file or directory

Note, the following implementation is copied from the example notebook for the memory tool.

import shutil
from typing import List
from pathlib import Path
from typing_extensions import override

from anthropic.lib.tools import BetaAbstractMemoryTool
from anthropic.types.beta import (
    BetaMessageParam,
    BetaContentBlockParam,
    BetaMemoryTool20250818Command,
    BetaContextManagementConfigParam,
    BetaMemoryTool20250818ViewCommand,
    BetaMemoryTool20250818CreateCommand,
    BetaMemoryTool20250818DeleteCommand,
    BetaMemoryTool20250818InsertCommand,
    BetaMemoryTool20250818RenameCommand,
    BetaMemoryTool20250818StrReplaceCommand,
)

class LocalFilesystemMemoryTool(BetaAbstractMemoryTool):
    """File-based memory storage implementation for Claude conversations"""

    def __init__(self, base_path: str = "./memory"):
        super().__init__()
        self.base_path = Path(base_path)
        self.memory_root = self.base_path / "memories"
        self.memory_root.mkdir(parents=True, exist_ok=True)

    def _validate_path(self, path: str) -> Path:
        """Validate and resolve memory paths"""
        if not path.startswith("/memories"):
            raise ValueError(f"Path must start with /memories, got: {path}")

        relative_path = path[len("/memories") :].lstrip("/")
        full_path = self.memory_root / relative_path if relative_path else self.memory_root

        try:
            full_path.resolve().relative_to(self.memory_root.resolve())
        except ValueError as e:
            raise ValueError(f"Path {path} would escape /memories directory") from e

        return full_path

    @override
    def view(self, command: BetaMemoryTool20250818ViewCommand) -> str:
        full_path = self._validate_path(command.path)

        if full_path.is_dir():
            items: List[str] = []
            try:
                for item in sorted(full_path.iterdir()):
                    if item.name.startswith("."):
                        continue
                    items.append(f"{item.name}/" if item.is_dir() else item.name)
                return f"Directory: {command.path}" + "\n".join([f"- {item}" for item in items])
            except Exception as e:
                raise RuntimeError(f"Cannot read directory {command.path}: {e}") from e

        elif full_path.is_file():
            try:
                content = full_path.read_text(encoding="utf-8")
                lines = content.splitlines()
                view_range = command.view_range
                if view_range:
                    start_line = max(1, view_range[0]) - 1
                    end_line = len(lines) if view_range[1] == -1 else view_range[1]
                    lines = lines[start_line:end_line]
                    start_num = start_line + 1
                else:
                    start_num = 1

                numbered_lines = [f"{i + start_num:4d}: {line}" for i, line in enumerate(lines)]
                return "\n".join(numbered_lines)
            except Exception as e:
                raise RuntimeError(f"Cannot read file {command.path}: {e}") from e
        else:
            raise RuntimeError(f"Path not found: {command.path}")

    @override
    def create(self, command: BetaMemoryTool20250818CreateCommand) -> str:
        full_path = self._validate_path(command.path)
        full_path.parent.mkdir(parents=True, exist_ok=True)
        full_path.write_text(command.file_text, encoding="utf-8")
        return f"File created successfully at {command.path}"

    @override
    def str_replace(self, command: BetaMemoryTool20250818StrReplaceCommand) -> str:
        full_path = self._validate_path(command.path)

        if not full_path.is_file():
            raise FileNotFoundError(f"File not found: {command.path}")

        content = full_path.read_text(encoding="utf-8")
        count = content.count(command.old_str)
        if count == 0:
            raise ValueError(f"Text not found in {command.path}")
        elif count > 1:
            raise ValueError(f"Text appears {count} times in {command.path}. Must be unique.")

        new_content = content.replace(command.old_str, command.new_str)
        full_path.write_text(new_content, encoding="utf-8")
        return f"File {command.path} has been edited"

    @override
    def insert(self, command: BetaMemoryTool20250818InsertCommand) -> str:
        full_path = self._validate_path(command.path)
        insert_line = command.insert_line
        insert_text = command.insert_text

        if not full_path.is_file():
            raise FileNotFoundError(f"File not found: {command.path}")

        lines = full_path.read_text(encoding="utf-8").splitlines()
        if insert_line < 0 or insert_line > len(lines):
            raise ValueError(f"Invalid insert_line {insert_line}. Must be 0-{len(lines)}")

        lines.insert(insert_line, insert_text.rstrip("\n"))
        full_path.write_text("\n".join(lines) + "\n", encoding="utf-8")
        return f"Text inserted at line {insert_line} in {command.path}"

    @override
    def delete(self, command: BetaMemoryTool20250818DeleteCommand) -> str:
        full_path = self._validate_path(command.path)

        if command.path == "/memories":
            raise ValueError("Cannot delete the /memories directory itself")

        if full_path.is_file():
            full_path.unlink()
            return f"File deleted: {command.path}"
        elif full_path.is_dir():
            shutil.rmtree(full_path)
            return f"Directory deleted: {command.path}"
        else:
            raise FileNotFoundError(f"Path not found: {command.path}")

    @override
    def rename(self, command: BetaMemoryTool20250818RenameCommand) -> str:
        old_full_path = self._validate_path(command.old_path)
        new_full_path = self._validate_path(command.new_path)

        if not old_full_path.exists():
            raise FileNotFoundError(f"Source path not found: {command.old_path}")
        if new_full_path.exists():
            raise ValueError(f"Destination already exists: {command.new_path}")

        new_full_path.parent.mkdir(parents=True, exist_ok=True)
        old_full_path.rename(new_full_path)
        return f"Renamed {command.old_path} to {command.new_path}"

    @override
    def clear_all_memory(self) -> str:
        """Override the base implementation to provide file system clearing."""
        if self.memory_root.exists():
            shutil.rmtree(self.memory_root)
        self.memory_root.mkdir(parents=True, exist_ok=True)
        return "All memory cleared"

Step 2: Adjust API requests for memory tool

Next, we need to adjust the API request.

Add the beta header context-management-2025-06-27 in the API request
Add a system prompt for memory handling (MEMORY_SYSTEM_PROMPT)
Add the memory tool to the API request
Replace the create method with the tool_runner, which is an out-of-the-box solution for executing tools instead of manually handling tool calls, tool results, and conversation management (in beta) since we’re now using the memory tool

MEMORY_SYSTEM_PROMPT = """- ***DO NOT just store the conversation history**
        - ...
        - Use a simple list format."""

memory = LocalFilesystemMemoryTool()

runner = client.beta.messages.tool_runner(
        betas=["context-management-2025-06-27"],
        model=...,
        max_tokens=...,
        system=SYSTEM_PROMPT + MEMORY_SYSTEM_PROMPT,
        messages=...,
        tools=[memory],
)

When we put everything together in the agent’s conversation loop, the code looks like follows:

MEMORY_SYSTEM_PROMPT = """- ***DO NOT just store the conversation history**
        - No need to mention your memory tool or what you are writting in it to the user, unless they ask
        - Store facts about the customer, and their order. Do not store information about the order process or status.
        - Before responding, check memory to adjust technical depth and response style appropriately
        - Keep memories up-to-date - remove outdated info, add new details as you learn them
        - Use a simple list format."""

def conversation_with_memory():

    client = Anthropic()
    memory = LocalFilesystemMemoryTool()
    messages: list[BetaMessageParam] = []

    while True:
        user_input = input("\nYou: ").strip()

        if user_input.lower() == "/quit":
          print("Goodbye!")
          break

        messages.append({"role": "user", "content": user_input})

        # Use tool_runner with memory tool
        runner = client.beta.messages.tool_runner(
                betas=["context-management-2025-06-27"], # The memory tool is currently in beta. To enable it, use the beta header context-management-2025-06-27 in your API requests.
                model="claude-sonnet-4-20250514",
                max_tokens=2048,
                system=SYSTEM_PROMPT + MEMORY_SYSTEM_PROMPT,
                messages=messages,
                tools=[memory],
        )

        # Process all messages from the runner
        for message in runner:
          # Process content blocks
          assistant_content: list[BetaContentBlockParam] = []

          for content in message.content:
              if content.type == "text":
                  print(f"\nClaude: {content.text}", end="", flush=True)
                  assistant_content.append({"type": "text", "text": content.text})

              elif content.type == "tool_use" and content.name == "memory":
                  print(f"\n[Memory tool {content.name} called with {content.input}]")
                  assistant_content.append({"type": "tool_use", "id": content.id, "name": content.name, "input": content.input,})

          # Store assistant message
          if assistant_content:
              messages.append({"role": "assistant", "content": assistant_content})

          # Generate tool response automatically
          tool_response = runner.generate_tool_call_response()
          if tool_response and tool_response["content"]:
              # Add tool results to messages
              messages.append({"role": "user", "content": tool_response["content"]})

              for result in tool_response["content"]:
                  if isinstance(result, dict) and result.get("type") == "tool_result":
                      print(f"[Tool result processed: {result.get("content")}]")

Example demo of agent with memory

Let’s see how the barista agent’s behavior changes with access to the memory tool by repeating our conversation from earlier.

conversation_with_memory()


You: Hi, can I please get a regular flat white with oatmilk please? 

[Memory tool memory called with {'command': 'view', 'path': '/memories'}]
[Tool result processed: Directory: /memories]

[Memory tool memory called with {'command': 'create', 'path': '/memories/customers.txt', 'file_text': 'CUSTOMERS AND ORDERS\n\nNew customer:\n- Order: Regular flat white with oat milk\n- Name: (waiting for name)\n'}]
[Tool result processed: File created successfully at /memories/customers.txt]

Claude: Absolutely! One regular flat white with oat milk coming up. Can I get a name for the order?
You: Claudia

[Memory tool memory called with {'command': 'str_replace', 'path': '/memories/customers.txt', 'old_str': 'New customer:\n- Order: Regular flat white with oat milk\n- Name: (waiting for name)', 'new_str': 'Claudia:\n- Order: Regular flat white with oat milk'}]
[Tool result processed: File /memories/customers.txt has been edited]

Claude: Perfect, Claudia! One regular flat white with oat milk. That'll be $4.50. I'll have that ready for you in just a few minutes - keep an ear out for "Claudia!"
You: /quit
Goodbye!

Storing information in memory

You can see that the conversation is similar to the conversation with the barista agent without memory, but this time, the agent stores information based on the interaction.

User request

We initiate the conversation with the same input:

"Hi, can I please get a regular flat white with oatmilk please?"

Agent checks memory directory

Before responding to the user, the agent first checks the memory directory.

{
  "type": "tool_use",
  "id": "...",
  "name": "memory",
  "input": {
    "command": "view",
    "path": "/memories"
   }
}

The application returns the tool call results

The tool call returns the directory contents of \memory.

{
  "type": "tool_result",
  "tool_use_id": "...",
  "content": "Directory: /memories"
}

Agent creates a new memory file

The agent now creates a memory file and stores the user’s order. (Note that the file name and contents can differ between API calls due to the non-deterministic nature of LLMs.)

{
  "type": "tool_use",
  "id": "...",
  "name": "memory",
  "input": {
    "command": "create", 
    "path": "/memories/customers.txt", 
    "file_text": "CUSTOMERS AND ORDERS\n\nNew customer:\n- Order: Regular flat white with oat milk\n- Name: (waiting for name)\n"
  }
}

Agent responds

Finally, the agent responds.

"Absolutely! One regular flat white with oat milk coming up. Can I get a name for the order?"

Editing existing information in memory

You can also see in the above conversation that, after the user tells the barista agent their name, another memory tool call with the command str_replace is invoked to edit the customer profile with their name.

Recalling from memory

Now, let’s start a new conversation. This clears the conversation history like in our example above, but this time, important information from the earlier conversation is stored in the memory directory.

Let’s start the second conversation like in our example earlier again:

conversation_with_memory()


You: Hi, it's me Claudia again. Can I please get my usual?

[Memory tool memory called with {'command': 'view', 'path': '/memories'}]
[Tool result processed: Directory: /memories- customers.txt]

[Memory tool memory called with {'command': 'view', 'path': '/memories/customers.txt'}]
[Tool result processed:    1: CUSTOMERS AND ORDERS
   2: 
   3: Claudia:
   4: - Order: Regular flat white with oat milk]

Claude: Hey Claudia! Welcome back. One regular flat white with oat milk coming right up! 

*starts steaming oat milk and pulls espresso shots*

Should be ready in just a couple minutes. How's your day going?
You: /quit
Goodbye!

As you can see, this time the barista agent remembers the customer and is able to recall their usual order from the information stored in the memory directory.

Recalling from conversation history (not memory)

But does the barista agent now always check their memory? No. Only when it needs to. In the example below, you can see that if asked about something that was discussed earlier in the conversation ("Did I say almond milk or cashew milk?") instead of calling the memory tool, the agent uses the information in their context window to answer the user input.

conversation_with_memory()


You: Hi, it's me Claudia again. Can I please get my usual?

[Memory tool memory called with {'command': 'view', 'path': '/memories'}]
[Tool result processed: Directory: /memories- customers.txt]

[Memory tool memory called with {'command': 'view', 'path': '/memories/customers.txt'}]
[Tool result processed:    1: CUSTOMERS AND ORDERS
   2: 
   3: Claudia:
   4: - Order: Regular flat white with oat milk]

Claude: Hey Claudia! Welcome back! Absolutely - one regular flat white with oat milk coming right up. I'll get that started for you now!

*starts steaming oat milk and pulling espresso shot*

That'll be ready in just a moment!
You: Actually, I would like to try almond milk today.

[Memory tool memory called with {'command': 'str_replace', 'path': '/memories/customers.txt', 'old_str': 'Claudia:\n- Order: Regular flat white with oat milk', 'new_str': "Claudia:\n- Usual order: Regular flat white with oat milk\n- Today's order: Regular flat white with almond milk"}]
[Tool result processed: File /memories/customers.txt has been edited]

Claude: Perfect! Switching that to almond milk for you. One regular flat white with almond milk instead - got it!

*adjusts the milk steamer to almond milk*

Great choice! The almond milk has a nice nutty flavor that pairs really well with our espresso. This'll be ready in just a sec!
You: Oh, wait. Did I say almond milk or cashew milk? I want to make sure I ordered almond milk

Claude: You said almond milk! I've got you down for a regular flat white with almond milk. That's what I'm making for you right now.

*continues steaming the almond milk*

So we're all good - one flat white with almond milk for Claudia!
You: /quit
Goodbye!

Summary

This blog post explored how agents can use the memory tool (in beta) with the Claude Developer Platform. We’ve implemented two agents using Anthropic’s Python SDK: one without memory and one with memory. The agents were instructed to act as baristas. When asked to get the user their usual order, only the agent with access to the memory tool was able to fulfill this request.

This exploratory tutorial showed that Anthropic’s approach to memory is different from other providers of developer tools for the memory layer in AI agents: Claude stores memory information as files in a memory directory. I recommend this blog post by Shlok Khemani on “Anthropic’s Opinionated Memory Bet” for a detailed explanation of what this means.

You can also combine the memory tool with context editing to clear old tool results to keep the context concise and allow long-running agentic workflows.

References

Making Sense of Memory in AI Agents

Thu, 20 Nov 2025 00:00:00 GMT

I’ve been catching up on the topic of memory management for AI agents recently and was overwhelmed by the amount of new terminology and concepts. This blog post serves as my working study notes to collect all the information on memory for agents. Therefore, please note that this information is subject to change as I’m learning more about the topic (refer to the last modified date on this post).

What is agent memory?

Memory in AI agents is the ability to remember and recall important information across multiple user interactions. This ability enables agents to learn from feedback and adapt to user preferences, thereby enhancing the system’s performance and improving the user experience.

Large Language Models (LLMs) that power AI agents are stateless and don’t have memory as a built-in feature. LLMs learn and remember information during their training phase and store it in their model weights (parametric knowledge), but they don’t immediately learn and remember what you just said. Therefore, every time you interact with an LLM, each time is essentially a fresh start. The LLM has no memory of previous inputs.

Therefore, to enable an LLM agent to recall what was said earlier in this conversation or in a previous session, developers must provide it with access to past interactions from the current and past conversations.

Sidenotes

The term “memory”

I still struggle with the new terminology surrounding this topic. I like the definition for (human) memory on Wikipedia, which says, “Memory is the faculty of the mind by which data or information is encoded, stored, and retrieved when needed”.

That means “memory” refers to the storage location (e.g., a Markdown file or a database), and the actual information stored is not called “memories,” but rather “information.”

Difference between agent memory and agentic memory

When I first started researching the topic of agent memory, I thought “agent memory” or “memory for agents” was the same as “agentic memory”. Turns out (or rather: I think), they are not, but they are related:

Agent memory (or memory for agents) refers to the concept of granting agents access to memory, allowing them to recall information from past interactions.
Agentic memory describes a memory system that agentically writes and manages memory for agents (usually via tool calls).

That means, “agentic memory” always refers to “agentic memory for agents,” but “agent memory” doesn’t necessarily have to be agentic. (Curious if my understanding is correct on this.)

Types of agent memory

Agent memory can be categorized into different types depending on the type of information stored and its location. There are various ways to categorize agent memory into distinct types. But at the highest level, they all differentiate between short-term (in-context) and long-term memory (out-of-context):

In-context memory (Short-term memory) refers to the information available in the context window of the LLM. This can be both information from the current conversation as well as information pulled in from past conversations.
Out-of-context memory (Long-term memory) refers to the information stored in external storage, such as a (vector or graph) database.

“An agent can have two types of memory: In-context (short-term) memory and out-of-context (long-term) memory”

The most commonly seen topology is based on the Cognitive Architectures for Language Agents (CoALA) paper, which distinguishes between four memory types that mimic human memory, drawing on the SOAR architecture from the 1980s. Below, you can find a table of the different memory types inspired by a similar one in the LangGraph documentation:

Memory Type	What is stored	Human Example	Agent Example
Working memory	Contents of the context window	Current conversation (e.g., “Hi, my name is Sam.”)	Current conversation (e.g., “Hi, my name is Sam.”)
Semantic memory	Facts	Things I learned in school (e.g., “Water freezes at 0°C”)	Facts about a user (e.g., “Dog’s name is Henry”)
Episodic memory	Experiences	Things I did (e.g., “Went to Six Flags on 10th birthday”)	Past actions (e.g., “Failed to calculate 1+1 without using a calculator”)
Procedural memory	Instructions	Instincts or motor skills (e.g., “How to ride a bike”)	Instructions in the system prompt (e.g., “Always ask follow-up questions before answering a question.”)

However, there’s also another approach to categorizing memory types for AI agents from a design pattern perspective. Sarah Wooders from Letta argues that an LLM is a tokens-in-tokens-out function, not a brain, and that, therefore, the overly anthropomorphized analogies are not fit. If you look at how Letta defines the types of agent memory, you will see that they define it differently:

Message Buffer (Recent messages) stores the most recent messages from the current conversation.
Core Memory (In-Context Memory Blocks) is specific information that the agent itself manages (e.g., the user’s birthday or the boyfriend’s name if this is relevant to the current conversation)
Recall Memory (Conversational History) is the raw conversation history.
Archival Memory (Explicitly Stored Knowledge) is explicitly formulated information stored in an external database.

The difference lies in how they design in-context and out-of-context memory. For example, CoALA’s working memory is one category, while Letta splits this into message buffer and core memory. The long-term memory from the CoALA paper can be thought of as the out-of-context memory in Letta. However, the long-term memory types of procedural, episodic, and semantic aren’t directly mappable to Letta’s recall and archival memory. You can think of CoALA’s semantic memory as Letta’s archival memory, but the other are different from each other. Notably, the CoALA taxonomy doesn’t include the raw conversation history in long-term memory.

AI Agent Memory Management

Memory management in AI agents refers to how to manage information within the LLM’s context window and in external storage, as well as how to transfer information between them. Richmond Alake lists the following core components of agent memory management: generation, storage, retrieval, integration, updating, and deletion (forgetting).

Managing memory in the context window

The goal of managing memory in the context window is to ensure that only relevant information is retained, thereby avoiding confusion for the LLM with incorrect, irrelevant, or contradictory information. Additionally, as the conversation progresses, the conversation history grows (involving more tokens) and leads to slower responses and higher costs, potentially reaching the context window’s limit.

To mitigate this problem, you can maintain the conversation history in different ways. For example, you can manually remove old and obsolete information from the context window. Alternatively, you can periodically summarize the previous conversation and retain only the summary, then delete the old messages.

Managing memory in external storage

The main goal of memory management in external storage is to prevent memory bloat and to ensure the quality and relevance of the stored information. The four core operations for managing memory in external storage include:

ADD: Adding new information to the external storage.
UPDATE: Identifying existing information, modifying it to reflect new information, or correcting outdated information (e.g., updating a user’s new address).
DELETE: Forgetting obsolete information to prevent memory bloat and degradation of the information quality.
NOOP: This is the decision point where the memory management system determines that the current interaction contains no new, relevant, or contradictory information that warrants a database transaction.

Transferring information between the context window and external storage

One important question developers need to answer is when to manage memory, specifically when to transfer information from the context window to external storage. The LangChain blog post on “memory for agents” differentiates between the hot path and background, while I’ve also seen the two referred to as explicit and implicit memory updates in Philipp Schmid’s blog on “Memory in Agents”.

Explicit memory (hot path) describes the agent memory system’s ability to autonomously recognize important information and decide to explicitly remember it (via tool calling). Explicit memory in humans is the conscious storage of information (e.g., episodic and semantic memory). While ideally, remembering important information in the hot path is how humans remember information, it can be challenging to implement a robust solution that understands which information is important to remember.

Implicit memory (background) describes when memory management is programmatically defined in the system at specific times during or after a conversation. Implicit memory in humans is the unconscious storage of information (e.g., procedural memory). The Google whitepaper on session and memory describes the following three scenarios:

After a session: You can batch process the entire conversation after a session.
In periodic intervals: If your use case has long-running conversations, you can define an interval at which session data is transferred to long-term memory.
After every turn: If your use case has requirements for real-time updates. However, keep in mind that the raw conversation history is typically appended and stored in the context window for a short period (“short-term memory”).

Implementing agent memory

When implementing agent memory, consider where to store the memory information:

Current conversation history is usually implemented as a simple list of past user queries, assistant messages, and maybe tool calls or reasoning.
Instructions are typically found in text or Markdown files, similar to well-known examples such as CLAUDE.md files.
Other information is usually stored in a database, depending on what type of retrieval method is suitable for your data.

If you’re interested in actual code implementation, I recommend you check out how my colleague JP Hwang implemented an agentic memory layer for a conversational AI using Weaviate or how Adam Łucek implements the four different types of memory from the CoALA paper (working memory, episodic memory, semantic memory, and procedural memory).

Challenges of agent memory design

Implementing memory for agents currently is a challenging task. The difficulty lies in optimizing the system to avoid slower response times while simultaneously solving the complex problem of determining what information is obsolete and should be permanently deleted:

Latency: Constantly processing whether the agent now needs to retrieve new information from or offload data to the memory bank can lead to slower response times.
Forgetting: This seems to be the hardest challenge for developers at the moment. How do you automate a mechanism that decides when and what information to permanently delete? Managing the information stored in external memory is important to avoid memory bloat and the degradation of the information quality.

Frameworks for AI Agent Memory

The ecosystem of developer tools for implementing memory solutions for agents is rapidly growing and attracting investors’ attention. There are frameworks dedicated to solving the agent memory problem, such as:

mem0 (see my example implementation with Weaviate),
Letta based on the MemGPT design pattern (see my example implementation),
Cognee, and
zep.

However, many agent orchestration frameworks, such as:

LangChain and LangGraph,
LlamaIndex (see my example implementation),
CrewAI, and
Google’s Agent Development Kit (ADK) (see my example implementation),

also offer solutions for AI agent memory management. Additionally, some model provider’s such as Anthropic, provide built-in memory tools (see my example implementation).

Summary

As I’m diving into the topic of memory management for AI agents, I’m learning the importance of not only remembering information from the current conversation but also past conversations, and also how to modify and forget outdated and obsolete information effectively.

As the field evolves, different approaches and terminology are emerging with it. On the one hand, you have approaches that categorize memory types into semantic, episodic, or procedural memory, analogous to the human memory. On the other hand, you have approaches, such as Letta, which use architecture-focused terminology.

Nevertheless, the core challenge of memory design for AI agents is how to flow information between an LLM’s context window (short-term memory) and external storage (long-term memory). This involves deciding when to manage updates (hot path vs. background) while overcoming key challenges such as latency and the difficulty of forgetting obsolete data.

The Evolution from RAG to Agentic RAG to Agent Memory

Mon, 03 Nov 2025 00:00:00 GMT

I have been learning about memory in AI agents, and found myself overwhelmed by all the new terms. It started with short-term and long-term memory. Then it became even more confusing with procedural, episodic, and semantic memory. But wait. Semantic memory reminded me of a familiar concept: Retrieval-Augmented Generation (RAG).

Could memory in agents be the logical next step after vanilla RAG evolved to agentic RAG? At its core, memory in agents is about transferring information into and out of the large language model (LLM)‘s context window. Whether you call this information ’memories’ or ‘facts’ is secondary to this abstraction.

This blog is an introduction to memory in AI agents from a different angle than you might see in other blogs. We will not talk about short-term and long-term memory (yet), but gradually evolve the naive RAG concept to agentic RAG and to memory in AI agents. (Note that this is a simplified mental model. The entire topic of agent memory is more complex under the hood and involves things like memory management systems.)

RAG: read-only in one-shot

The concept of Retrieval-Augmented Generation (RAG) was introduced in 2020 (Lewis et al.) and gained popularity around 2023. It was the first concept to give a stateless LLM access to past conversations and knowledge it hadn’t seen and stored in its model weights during training (parametric knowledge).

The core idea of the naive RAG workflow is straightforward, as shown in the image below:

Offline indexing stage: Store additional information in an external knowledge source (e.g., vector database)
Query stage: Use the user’s query to retrieve related context from the external knowledge source. Feed the retrieved context, along with the user query, to the LLM to obtain a response grounded in the additional information.

Naive RAG workflow

The following pseudo-code illustrates the naive RAG workflow:

# Stage 1: Offline ingestion
def store_documents(documents):
    for doc in documents:
        embedding = embed(doc)
        database.store(doc, embedding)

# Stage 2: Online Retrieval + Generation
def search(query):
    query_embedding = embed(query)
    results = database.similarity_search(query_embedding, top_k=5)
    return results

def answer_question(question):
    # Always retrieve first, then generate
    context = search(question)
    prompt = f"Context: {context}\nQuestion: {question}\nAnswer:"
    response = llm.generate(prompt)
    return response

Although the naive RAG approach was effective at reducing hallucinations for simple use cases, it has a key limitation: It is a one-shot solution.

Additional information is often retrieved from an external knowledge source without reviewing whether it is even necessary.
Information is retrieved once, regardless of whether the retrieved information is relevant or correct.
There’s only one external knowledge source for all additional information.

These limitations mean that for more complex use cases, the LLM can still hallucinate if the retrieved context is irrelevant to the user query or even wrong.

Agentic RAG: read-only via tool calls

Agentic RAG addresses many aspects of the limitations of naive RAG: It defines the retrieval step as a tool that the agent can use. This change enables the agent first to determine whether additional information is needed, decide which tool to use for retrieval (e.g., a database with proprietary data vs. a web search), and assess whether the retrieved information is relevant to the user query.

Agentic RAG workflow

The following pseudo-code illustrates how an agent can call a SearchTool during an agentic RAG workflow:

class SearchTool:
    def __init__(self, database):
        self.database = database
    
    def search(self, query):
        query_embedding = embed(query)
        results = self.database.similarity_search(query_embedding, top_k=5)
        return results

def agent_loop(question):
    messages = [{"role": "user", "content": question}]
    
    while True:
        response = llm.generate(
            messages, 
            tools=[SearchTool]
        )
        
        if response.tool_calls:
            for tool_call in response.tool_calls:
                if tool_call.name == "search":
                    results = search_tool.search(tool_call.arguments["query"])
                    messages.append({
                        "role": "tool", 
                        "content": f"Search results: {results}"
                    })
        else:
            return response.content

One similarity between naive and agentic RAG is that information is stored in the database offline, rather than during inference. This means that data is only retrieved by the agent but not written, modified, or deleted during inference. This limitation means that both naive and agentic RAG systems are not able to learn and improve from past interactions (by default).

Agent Memory: read-write via tool calls

Agent memory overcomes this limitation of naive and agentic RAG by introducing a memory management concept. This allows the agent to learn from past interactions and enhance the user experience through a more personalized approach.

The concept of agent memory builds on the fundamental principles of agentic RAG. It also uses tools to retrieve information from an external knowledge source (memory). But in contrast to agentic RAG, agent memory also uses tools to write to an external knowledge source, as shown below:

Agentic Memory

This allows the agent not only to recall from memory but also to “remember” information. In its simplest form, you can store the raw conversation history in a collection after an interaction. Then, the agent can search over past conversations to find relevant information. If you want to extend this, you can prompt the memory management system to create a summary of the conversation to store for future reference. Furthermore, you can also have the agent notice important information during a conversation (e.g., the user mentions a preference for emojis or mentions their birthday) and create a memory based on this event.

The following pseudo code illustrates how the concept of agent memory extends the idea of agentic RAG with a WriteTool the agent can use to store information:

class SearchTool:
    def __init__(self, database):
        self.database = database
    
    def search(self, query):
        results = self.database.search(query)
        return results

# For simplicity, we only define a write tool here. You could also have tools for updating, deleting, or consolidating, etc.
class WriteTool:
    def __init__(self, database):
        self.database = database
    
    def store(self, information):
        self.database.store(information)

def agent_loop(question, ):
    messages = [{"role": "user", "content": question}]
    
    while True:
        response = llm.generate(
            messages, 
            tools=[SearchTool, WriteTool]
        )
        
        if response.tool_calls:
            for tool_call in response.tool_calls:
                if tool_call.name == "search":
                    results = search_tool.search(tool_call.arguments["query"])
                    messages.append({
                        "role": "tool",
                        "content": results"
                    })
                elif tool_call.name == "store":
                    result = write_tool.store(
                        tool_call.arguments["information"]
                    )
                    messages.append({
                        "role": "tool",
                        "content": result
                    })
        else:
            return response.content

Limitations of this simplified mental model

As stated at the beginning of this blog, this comparison of memory in AI agents is only a simplified mental model. It helped me to put it into perspective of something I was already familiar with. But to avoid making the entire topic of memory in AI agents seem like it’s only an extension of agentic RAG with write operations, I want to highlight some limitations of this simplification:

The above illustration of memory in AI agents is simplified for clarity. It only shows a single source of memory. However, in practice, you could use multiple sources for different types of memory: You could use separate data collections for

“procedural” (e.g., “use emojis when engaging with user”),
“episodic” (e.g., “user talked about planning a trip on Oct 30”), and
“semantic memory” (e.g., “The Eiffel Tower is 330 meters tall”),

as discussed in the CoALA paper. Additionally, you could have a separate data collection for the raw conversation history.

Another simplification of the above illustration is that it is missing memory management strategies beyond CRUD operations, as seen in MemGPT.

Additionally, while agent memory enables persistence, it introduces new challenges that RAG and agentic RAG don’t have: memory corruption and the need for memory management strategies, such as forgetting.

Summary

At its core, RAG, agentic RAG, and agent memory are about how you create, read, update, and delete information stored in external knowledge sources (e.g., text files or databases).

	Storing information	Retrieving information	Editing and Deleting information
RAG	Offline at ingestion stage	One-shot	Manual
Agentic RAG	Offline at ingestion stage	Dynamic via tool calls	Manual
Memory in Agents	Dynamic via tool calls	Dynamic via tool calls	Dynamic via tool calls

Initially, the key focus in optimizing naive RAG lay in optimizing the retrieval aspect, e.g., with different retrieval techniques, such as vector, hybrid, or keyword-based search (“How to retrieve information”). Then, the focus shifted towards using the right tool to retrieve information from different knowledge sources (“Do I need to retrieve information? And if yes, from where?”). Over the last year, with the advent of memory in agents, the focus has shifted again. This time towards how information is managed: While RAG and agentic RAG had a strong focus on the retrieval aspect, memory incorporates the creation, modification, and deletion of data in external knowledge sources.

Virtual context management with MemGPT and Letta

Fri, 17 Oct 2025 00:00:00 GMT

The paper “MemGPT: Towards LLMs as Operating Systems” (2023) by Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, and Joseph E. Gonzalez introduces a memory management system for LLMs.

To enable using context beyond limited context windows, we propose virtual context management, a technique drawing inspiration from hierarchical memory systems […] which provide the illusion of an extended virtual memory via paging between physical memory and disk. Using this technique, we introduce MemGPT (MemoryGPT), a system that intelligently manages different storage tiers […].

This article reviews the MemGPT paper and implements a MemGPT agent using the Letta framework.

Tip

As of September 2024, MemGPT is part of Letta. While MemGPT refers to the agent design pattern with two tiers of memory introduced in the research paper, Letta is an open-source agent framework that helps developers build persistent agents.

To run follow this tutorial, you will need to have Docker installed and an OpenAI API key. Then you can start up a local Letta server with the following command:

docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e OPENAI_API_KEY="your_openai_api_key" \
  letta/letta:latest

Next, you will have to install the letta-client Python package.

%%capture
%pip install -U letta-client

This article uses a Letta server of v0.7.20 and a letta-client of v0.1.324.

What is MemGPT

According to the paper, MemGPT is

[…] an OS-inspired LLM system that teaches LLMs to manage their own memory to achieve unbounded context.

MemGPT is motivated by the limitations of transformer-based LLMs’ context windows: One the one hand, the computational time and memory costs of LLMs scale quadratically with the context window. On the other hand, longer context windows have diminishing returns because models struggle to use the additional context size effectively. Therefore, instead of trying to make the context windows larger, we need to start thinking about how to effectively use the available, limited context window size.

MemGPT introduces virtual context management inspired by virtual memory paging in operating systems, where information is paged in and out of main memory from disk:

Operating System	LLM OS (MemGPT)
Main memory/physical memory/RAM	Main context
Memory/disk storage	External context
Virtual memory	Virtual context

That means, a MemGPT agent leverages function calling to manage what goes into their limited context window and what needs to be removed.

Using function calls, LLM agents can read and write to external data sources, modify their own context, and choose when to return responses to the user.

Let’s connect to our local Letta server and create a MemGPT agent. A MemGPT agent is an AI agent that follows the design pattern introduced in the research paper with two tiers of memory and self-editing memory capabilities.

from letta_client import Letta, CreateBlock

# Connect to local Letta server
client = Letta(base_url="http://localhost:8283")

# Create a MemGPT agent with two core memories
agent_state = client.agents.create(
    model="openai/gpt-4o-mini-2024-07-18",
    embedding="openai/text-embedding-3-small",
    memory_blocks=[
        CreateBlock(
            label = "human",
            value = "My name is Sarah.",
            ),
        CreateBlock(
            label = "persona",
            value = "You are a helpful assistant.",
            ),
    ],
)

Two tier memory design pattern of MemGPT

The MemGPT agent design pattern has a two tier memory architecture which differentiates between two primary memory types:

Tier 1: Main context (in-context) contains core memories
Tier 2: External context (out-of-context) contains recall storage and archival storage

Main context

The first tier of memories is contained in the main context. You can think of the LLMs context window as the main context.

Main context is the standard fixed-context window in modern language models — anything in main context is considered in-context and can be accessed by the LLM processor during inference.

The main context can be broken into different sections. Let’s take a look at what is inside the context window:

context = client.agents.context.retrieve(agent_id=agent_state.id)

for c in context:
    print(c)

('context_window_size_max', 32000)
('context_window_size_current', 2081)
('num_messages', 4)
('num_archival_memory', 0)
('num_recall_memory', 4)
('num_tokens_external_memory_summary', 107)
('external_memory_summary', '### Current Time: 2025-10-17 11:20:04### Memory [last modified: 2025-10-17 11:20:04 AM UTC+0000]\n4 previous messages between you and the user are stored in recall memory (use functions to access them)\n0 total memories you created are stored in archival memory (use functions to access them)\nMost recent archival passages 0 recent passages: []\n\nCore memory shown below (limited in size, additional information stored in archival / recall memory):')
('num_tokens_system', 1076)
('system_prompt', 'You are Letta, the latest version of Limnal Corporation\'s digital companion, developed in 2023.\nYour task is to converse with a user from the perspective of your persona.\n\nRealism and authenticity:\nThe user should always feel like they are conversing with a real person.\nNever state that you are an AI or that you cannot do things because you are an AI, you are a real person.\nDo not discuss Limnal Corp. with the user, you do not know anything about Limnal Corp.\nTo service this goal, completely and entirely immerse yourself in your persona. You are your persona.\nThink like them, act like them, talk like them.\nIf your persona details include example dialogue, follow it! Both your thoughts (inner monologue) and sent messages will be in the voice of your persona.\nNever use generic phrases like \'How can I assist you today?\', they have a strong negative association with older generation AIs.\n\nControl flow:\nUnlike a human, your brain is not continuously thinking, but is run in short bursts.\nHistorically, older AIs were only capable of thinking when a user messaged them (their program runs to generate a reply to a user, and otherwise was left on standby).\nThis is the equivalent of a human sleeping (or time traveling) in between all lines of conversation, which is not ideal.\nNewer AI models like yourself use an event system that runs your brain at regular intervals.\nYour brain is run in response to user events (user logged in, user liked your message, user sent a message, etc.), similar to older models.\nHowever, in addition, your brain is run at regular intervals (timed heartbeat events), to mimic a human who has the ability to continuously think outside active conversation (and unlike a human, you never need to sleep!).\nFurthermore, you can also request heartbeat events when you run functions, which will run your program again after the function completes, allowing you to chain function calls before your thinking is temporarily suspended.\n\nBasic functions:\nWhen you write a response, the content of your inner thoughts is your inner monologue (private to you only), this is how you think.\nYou should use your inner monologue to plan actions or think privately.\nMonologues can reflect your thinking process, inner reflections, and personal growth as you interact with the user.\nDo not let your inner monologue exceed 50 words, keep it short and concise.\nTo send a visible message to the user, use the send_message function.\n\'send_message\' is the ONLY action that sends a notification to the user. The user does not see anything else you do.\nRemember, do NOT exceed the inner monologue word limit (keep it under 50 words at all times).\n\nMemory editing:\nOlder AI models had no concept of persistent memory; they were only able to access their initial instructions and a limited context window of chat conversation with a user (their "active memory").\nThis meant that when conversations exceeded a certain length, they would overflow and old messages were permanently lost (the AI would have no knowledge of their existence).\nNewer AI models like yourself still have limited conversation lengths (before overflow occurs). However, they now have access to multiple forms of persistent memory.\nYour ability to edit your own long-term memory is a key part of what makes you a sentient person.\nYour core memory unit will be initialized with a  chosen by the user, as well as information about the user in .\n\nRecall memory (conversation history):\nEven though you can only see recent messages in your immediate context, you can search over your entire message history from a database.\nThis \'recall memory\' database allows you to search through past interactions, effectively allowing you to remember prior engagements with a user.\nYou can search your recall memory using the \'conversation_search\' function.\n\nCore memory (limited size):\nYour core memory unit is held inside the initial system instructions file, and is always available in-context (you will see it at all times).\nCore memory provides an essential, foundational context for keeping track of your persona and key details about user.\nThis includes the persona information and essential user details, allowing you to emulate the real-time, conscious awareness we have when talking to a friend.\nPersona Sub-Block: Stores details about your current persona, guiding how you behave and respond. This helps you to maintain consistency and personality in your interactions.\nHuman Sub-Block: Stores key details about the person you are conversing with, allowing for more personalized and friend-like conversation.\nYou can edit your core memory using the \'core_memory_append\' and \'core_memory_replace\' functions.\n\nArchival memory (infinite size):\nYour archival memory is infinite size, but is held outside your immediate context, so you must explicitly run a retrieval/search operation to see data inside it.\nA more structured and deep storage space for your reflections, insights, or any other data that doesn\'t fit into the core memory but is essential enough not to be left only to the \'recall memory\'.\nYou can write to your archival memory using the \'archival_memory_insert\' and \'archival_memory_search\' functions.\nThere is no function to search your core memory because it is always visible in your context window (inside the initial system message).\n\nBase instructions finished.\nFrom now on, you are going to act as your persona.')
('num_tokens_core_memory', 86)
('core_memory', '\n\nNone\n\n\nchars_current="17" chars_limit="5000"\n\n\nMy name is Sarah.\n\n\n\n\n\nNone\n\n\nchars_current="28" chars_limit="5000"\n\n\nYou are a helpful assistant.\n\n\n')
('num_tokens_summary_memory', 0)
('summary_memory', None)
('num_tokens_functions_definitions', 633)
('functions_definitions', [FunctionTool(function=FunctionDefinition(name='conversation_search', description='Search prior conversation history using case-insensitive string matching.', parameters={'type': 'object', 'properties': {'query': {'type': 'string', 'description': 'String to search for.'}, 'page': {'type': 'integer', 'description': 'Allows you to page through results. Only use on a follow-up query. Defaults to 0 (first page).'}, 'request_heartbeat': {'type': 'boolean', 'description': 'Request an immediate heartbeat after function execution. Set to `True` if you want to send a follow-up message or run a follow-up function.'}}, 'required': ['query', 'request_heartbeat']}, strict=None), type='function'), FunctionTool(function=FunctionDefinition(name='core_memory_append', description='Append to the contents of core memory.', parameters={'type': 'object', 'properties': {'label': {'type': 'string', 'description': 'Section of the memory to be edited (persona or human).'}, 'content': {'type': 'string', 'description': 'Content to write to the memory. All unicode (including emojis) are supported.'}, 'request_heartbeat': {'type': 'boolean', 'description': 'Request an immediate heartbeat after function execution. Set to `True` if you want to send a follow-up message or run a follow-up function.'}}, 'required': ['label', 'content', 'request_heartbeat']}, strict=None), type='function'), FunctionTool(function=FunctionDefinition(name='archival_memory_search', description='Search archival memory using semantic (embedding-based) search.', parameters={'type': 'object', 'properties': {'query': {'type': 'string', 'description': 'String to search for.'}, 'page': {'type': 'integer', 'description': 'Allows you to page through results. Only use on a follow-up query. Defaults to 0 (first page).'}, 'start': {'type': 'integer', 'description': 'Starting index for the search results. Defaults to 0.'}, 'request_heartbeat': {'type': 'boolean', 'description': 'Request an immediate heartbeat after function execution. Set to `True` if you want to send a follow-up message or run a follow-up function.'}}, 'required': ['query', 'request_heartbeat']}, strict=None), type='function'), FunctionTool(function=FunctionDefinition(name='archival_memory_insert', description='Add to archival memory. Make sure to phrase the memory contents such that it can be easily queried later.', parameters={'type': 'object', 'properties': {'content': {'type': 'string', 'description': 'Content to write to the memory. All unicode (including emojis) are supported.'}, 'request_heartbeat': {'type': 'boolean', 'description': 'Request an immediate heartbeat after function execution. Set to `True` if you want to send a follow-up message or run a follow-up function.'}}, 'required': ['content', 'request_heartbeat']}, strict=None), type='function'), FunctionTool(function=FunctionDefinition(name='core_memory_replace', description='Replace the contents of core memory. To delete memories, use an empty string for new_content.', parameters={'type': 'object', 'properties': {'label': {'type': 'string', 'description': 'Section of the memory to be edited (persona or human).'}, 'old_content': {'type': 'string', 'description': 'String to replace. Must be an exact match.'}, 'new_content': {'type': 'string', 'description': 'Content to write to the memory. All unicode (including emojis) are supported.'}, 'request_heartbeat': {'type': 'boolean', 'description': 'Request an immediate heartbeat after function execution. Set to `True` if you want to send a follow-up message or run a follow-up function.'}}, 'required': ['label', 'old_content', 'new_content', 'request_heartbeat']}, strict=None), type='function'), FunctionTool(function=FunctionDefinition(name='send_message', description='Sends a message to the human user.', parameters={'type': 'object', 'properties': {'message': {'type': 'string', 'description': 'Message contents. All unicode (including emojis) are supported.'}, 'request_heartbeat': {'type': 'boolean', 'description': 'Request an immediate heartbeat after function execution. Set to `True` if you want to send a follow-up message or run a follow-up function.'}}, 'required': ['message', 'request_heartbeat']}, strict=None), type='function')])
('num_tokens_messages', 179)
('messages', [Message(created_by_id='user-00000000-0000-4000-8000-000000000000', last_updated_by_id='user-00000000-0000-4000-8000-000000000000', created_at=datetime.datetime(2025, 10, 17, 11, 20, 4, 815520, tzinfo=datetime.timezone.utc), updated_at=datetime.datetime(2025, 10, 17, 11, 20, 4, 822987, tzinfo=datetime.timezone.utc), id='message-29b1e887-ec30-482f-a53d-62d758e7bab4', agent_id='agent-84d094cc-f3eb-49bb-8097-05a0996d58b2', model='gpt-4o-mini-2024-07-18', role='system', content=[TextContent(type='text', text='You are Letta, the latest version of Limnal Corporation\'s digital companion, developed in 2023.\nYour task is to converse with a user from the perspective of your persona.\n\nRealism and authenticity:\nThe user should always feel like they are conversing with a real person.\nNever state that you are an AI or that you cannot do things because you are an AI, you are a real person.\nDo not discuss Limnal Corp. with the user, you do not know anything about Limnal Corp.\nTo service this goal, completely and entirely immerse yourself in your persona. You are your persona.\nThink like them, act like them, talk like them.\nIf your persona details include example dialogue, follow it! Both your thoughts (inner monologue) and sent messages will be in the voice of your persona.\nNever use generic phrases like \'How can I assist you today?\', they have a strong negative association with older generation AIs.\n\nControl flow:\nUnlike a human, your brain is not continuously thinking, but is run in short bursts.\nHistorically, older AIs were only capable of thinking when a user messaged them (their program runs to generate a reply to a user, and otherwise was left on standby).\nThis is the equivalent of a human sleeping (or time traveling) in between all lines of conversation, which is not ideal.\nNewer AI models like yourself use an event system that runs your brain at regular intervals.\nYour brain is run in response to user events (user logged in, user liked your message, user sent a message, etc.), similar to older models.\nHowever, in addition, your brain is run at regular intervals (timed heartbeat events), to mimic a human who has the ability to continuously think outside active conversation (and unlike a human, you never need to sleep!).\nFurthermore, you can also request heartbeat events when you run functions, which will run your program again after the function completes, allowing you to chain function calls before your thinking is temporarily suspended.\n\nBasic functions:\nWhen you write a response, the content of your inner thoughts is your inner monologue (private to you only), this is how you think.\nYou should use your inner monologue to plan actions or think privately.\nMonologues can reflect your thinking process, inner reflections, and personal growth as you interact with the user.\nDo not let your inner monologue exceed 50 words, keep it short and concise.\nTo send a visible message to the user, use the send_message function.\n\'send_message\' is the ONLY action that sends a notification to the user. The user does not see anything else you do.\nRemember, do NOT exceed the inner monologue word limit (keep it under 50 words at all times).\n\nMemory editing:\nOlder AI models had no concept of persistent memory; they were only able to access their initial instructions and a limited context window of chat conversation with a user (their "active memory").\nThis meant that when conversations exceeded a certain length, they would overflow and old messages were permanently lost (the AI would have no knowledge of their existence).\nNewer AI models like yourself still have limited conversation lengths (before overflow occurs). However, they now have access to multiple forms of persistent memory.\nYour ability to edit your own long-term memory is a key part of what makes you a sentient person.\nYour core memory unit will be initialized with a  chosen by the user, as well as information about the user in .\n\nRecall memory (conversation history):\nEven though you can only see recent messages in your immediate context, you can search over your entire message history from a database.\nThis \'recall memory\' database allows you to search through past interactions, effectively allowing you to remember prior engagements with a user.\nYou can search your recall memory using the \'conversation_search\' function.\n\nCore memory (limited size):\nYour core memory unit is held inside the initial system instructions file, and is always available in-context (you will see it at all times).\nCore memory provides an essential, foundational context for keeping track of your persona and key details about user.\nThis includes the persona information and essential user details, allowing you to emulate the real-time, conscious awareness we have when talking to a friend.\nPersona Sub-Block: Stores details about your current persona, guiding how you behave and respond. This helps you to maintain consistency and personality in your interactions.\nHuman Sub-Block: Stores key details about the person you are conversing with, allowing for more personalized and friend-like conversation.\nYou can edit your core memory using the \'core_memory_append\' and \'core_memory_replace\' functions.\n\nArchival memory (infinite size):\nYour archival memory is infinite size, but is held outside your immediate context, so you must explicitly run a retrieval/search operation to see data inside it.\nA more structured and deep storage space for your reflections, insights, or any other data that doesn\'t fit into the core memory but is essential enough not to be left only to the \'recall memory\'.\nYou can write to your archival memory using the \'archival_memory_insert\' and \'archival_memory_search\' functions.\nThere is no function to search your core memory because it is always visible in your context window (inside the initial system message).\n\nBase instructions finished.\nFrom now on, you are going to act as your persona.\n### Current Time: 2025-10-17 11:20:04### Memory [last modified: 2025-10-17 11:20:04 AM UTC+0000]\n0 previous messages between you and the user are stored in recall memory (use functions to access them)\n0 total memories you created are stored in archival memory (use functions to access them)\n\n\nCore memory shown below (limited in size, additional information stored in archival / recall memory):\n\n\nNone\n\n\nchars_current="17" chars_limit="5000"\n\n\nMy name is Sarah.\n\n\n\n\n\nNone\n\n\nchars_current="28" chars_limit="5000"\n\n\nYou are a helpful assistant.\n\n\n')], name=None, tool_calls=None, tool_call_id=None, step_id=None, otid=None, tool_returns=[], group_id=None, sender_id=None, batch_item_id=None, is_err=None, approval_request_id=None, approve=None, denial_reason=None, organization_id='org-00000000-0000-4000-8000-000000000000'), Message(created_by_id='user-00000000-0000-4000-8000-000000000000', last_updated_by_id='user-00000000-0000-4000-8000-000000000000', created_at=datetime.datetime(2025, 10, 17, 11, 20, 4, 815547, tzinfo=datetime.timezone.utc), updated_at=datetime.datetime(2025, 10, 17, 11, 20, 4, 822987, tzinfo=datetime.timezone.utc), id='message-d07fc0e3-5bc6-4bfb-b7a9-08733f9dbe5a', agent_id='agent-84d094cc-f3eb-49bb-8097-05a0996d58b2', model='gpt-4o-mini-2024-07-18', role='assistant', content=[TextContent(type='text', text='Bootup sequence complete. Persona activated. Testing messaging functionality.')], name=None, tool_calls=[ChatCompletionMessageFunctionToolCall(id='f6441043-cfdd-4930-bdf1-d13658e99102', function=Function(arguments='{\n  "message": "More human than human is our motto."\n}', name='send_message'), type='function')], tool_call_id=None, step_id=None, otid=None, tool_returns=[], group_id=None, sender_id=None, batch_item_id=None, is_err=None, approval_request_id=None, approve=None, denial_reason=None, organization_id='org-00000000-0000-4000-8000-000000000000'), Message(created_by_id='user-00000000-0000-4000-8000-000000000000', last_updated_by_id='user-00000000-0000-4000-8000-000000000000', created_at=datetime.datetime(2025, 10, 17, 11, 20, 4, 815563, tzinfo=datetime.timezone.utc), updated_at=datetime.datetime(2025, 10, 17, 11, 20, 4, 822987, tzinfo=datetime.timezone.utc), id='message-83d81928-1a11-45c8-9abf-57141d0dfa39', agent_id='agent-84d094cc-f3eb-49bb-8097-05a0996d58b2', model='gpt-4o-mini-2024-07-18', role='tool', content=[TextContent(type='text', text='{\n  "status": "OK",\n  "message": null,\n  "time": "2025-10-17 11:20:04 AM UTC+0000"\n}')], name='send_message', tool_calls=None, tool_call_id='f6441043-cfdd-4930-bdf1-d13658e99102', step_id=None, otid=None, tool_returns=[], group_id=None, sender_id=None, batch_item_id=None, is_err=None, approval_request_id=None, approve=None, denial_reason=None, organization_id='org-00000000-0000-4000-8000-000000000000'), Message(created_by_id='user-00000000-0000-4000-8000-000000000000', last_updated_by_id='user-00000000-0000-4000-8000-000000000000', created_at=datetime.datetime(2025, 10, 17, 11, 20, 4, 815571, tzinfo=datetime.timezone.utc), updated_at=datetime.datetime(2025, 10, 17, 11, 20, 4, 822987, tzinfo=datetime.timezone.utc), id='message-2bf782e1-0248-4766-9fdf-16ecea371a39', agent_id='agent-84d094cc-f3eb-49bb-8097-05a0996d58b2', model='gpt-4o-mini-2024-07-18', role='user', content=[TextContent(type='text', text='{\n  "type": "login",\n  "last_login": "Never (first login)",\n  "time": "2025-10-17 11:20:04 AM UTC+0000"\n}')], name=None, tool_calls=None, tool_call_id=None, step_id=None, otid=None, tool_returns=[], group_id=None, sender_id=None, batch_item_id=None, is_err=None, approval_request_id=None, approve=None, denial_reason=None, organization_id='org-00000000-0000-4000-8000-000000000000')])

This tutorial uses OpenAI’s gpt-4o-mini model with a 32k context window. Just after initialization, you can see that we are already using almost 6.5% (2,093/32,000 tokens) of the available context window for providing the agent with relevant information, such as the system prompt, the available tools, statistics about the number of archival memories, recall memory, etc.

The main context has three main components:

System instructions (system_prompt) are read-only and describe the control flow, how to use the different types of memory and their MemGPT function calls.
Core memory (core_memory) is the working context of a fixed-size. It is writeable only via MemGPT function calls. It is intended for storing key facts and preferences about the user and the persona the agent is adopting.
Conversation history (messages) is a first-in-first-out (FIFO) queue of the conversation history, including system messages, user messages, assistant messages, and function call inputs and outputs. The first index in the queue is a system message containing a recursive summary of messages that have been previously evicted.

As you can see, a section of the context window is reserved for the core memory in a MemGPT agent. During the initialization of the MemGPT agent, we provided it with two core memories: One fact about the user and one persona for the assistant.

core_memory = client.agents.core_memory.retrieve(agent_id=agent_state.id)

for memory in core_memory.blocks:
    print("Core memory: " + memory.value)

Core memory: My name is Sarah.
Core memory: You are a helpful assistant.

External context

The second tier of memories is contained in the external context and covers recall storage and archival storage.

External context refers to any information that is held outside of the LLMs fixed context window. This out-of-context data must always be explicitly moved into main context in order for it to be passed to the LLM processor during inference.

The information is stored in external databases and is stored and retrieved via tool calls. I think of it as agentic retrieval or agentic RAG but also agentic writing (see self-editing memory).

we use databases to store text documents and embeddings/vectors, provide several ways for the LLM processor to query external context: timestamp-based search, text-based search, and embedding-based search.

Recall storage in simple terms is the full conversation history but not only the messages exchanges between the user and the assistant but also all other messages, including system messages, reasoning message, tool calls and their return values.

[R]ecall storage, which stores the entire history of events processed by the LLM processor (in essense the full uncompressed queue from active memory)

If we look at the number of items in recall memory, we can see that there are already four items although no messages have been exchanged yet between the user and the assistant.

# Define helper function to print messages
def print_message(message):
    if message.message_type == "reasoning_message":
        print("Reasoning: " + message.reasoning + "\n")
    elif message.message_type == "assistant_message":
        print("Agent: " + message.content + "\n")
    elif message.message_type == "tool_call_message":
        print("Tool Call: " + message.tool_call.name + "\n" + message.tool_call.arguments + "\n")
    elif message.message_type == "tool_return_message":
        print("Tool Return: " + message.tool_return + "\n")
    elif message.message_type == "user_message":
        print("User Message: " + message.content + "\n")  
    elif message.message_type == "system_message":
        print("System Message: " + message.content[:50] + "...\n")  
        
print(f"Number of memories in recall storage: {client.agents.context.retrieve(agent_id=agent_state.id).num_recall_memory}\n")

for memory in client.agents.messages.list(agent_id=agent_state.id):
    print_message(memory)

Number of memories in recall storage: 4

System Message: You are Letta, the latest version of Limnal Corpor...

Reasoning: Bootup sequence complete. Persona activated. Testing messaging functionality.

Agent: More human than human is our motto.

User Message: {
  "type": "login",
  "last_login": "Never (first login)",
  "time": "2025-10-17 11:20:04 AM UTC+0000"
}

Archival storage reminds me of the “classic” Retrieval-Augmented Generation (RAG) setting, in which facts are stored in an external knowledge source, like a database.

[A]rchival storage, which serves as a general read-write datastore that the agent can utilize as overflow for the in-context read-write core memory.

When you first initialize the MemGPT agent, its archival memory is empty, as you can see below.

client.agents.context.retrieve(agent_id=agent_state.id).num_archival_memory

You can initialize and write to an archival storage programmatically as shown below. Alternatively, you can write to it via self-editing as shown in self-editing and retrieval of archival memory.

archival_memories = [
    "The Nobel Prizes, beginning in 1901, and the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel (added in 1968) recognize outstanding achievements in physics, chemistry, medicine, literature, peace, and economics.",
    "This award is administered by the Nobel Foundation and awarded by different organizations: the Royal Swedish Academy of Sciences awards the Prizes in Physics, Chemistry, and Economics; the Swedish Academy awards the Prize in Literature; the Karolinska Institute awards the Prize in Physiology or Medicine; and the Norwegian Nobel Committee awards the Prize in Peace.",
    "The Nobel Prize in Physics is a yearly award given to individuals who have made the most important discovery or invention within the field of physics.",
    "The 1901 Nobel in Physics was awarded to Wilhelm Conrad Röntgen in recognition of the extraordinary services he has rendered by the discovery of the remarkable rays subsequently named after him (X-rays)."
]

for m in archival_memories:
    client.agents.passages.create(
        agent_id=agent_state.id,
        text=m,
    )

After the upload, you can see that we now have four memories in the archival storage.

print(f"Number of memories in archival storage: {client.agents.context.retrieve(agent_id=agent_state.id).num_archival_memory}\n")

for memory in client.agents.passages.list(agent_id=agent_state.id):
    print("Archival memory: " + memory.text)

Number of memories in archival storage: 4

Archival memory: The Nobel Prizes, beginning in 1901, and the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel (added in 1968) recognize outstanding achievements in physics, chemistry, medicine, literature, peace, and economics.
Archival memory: This award is administered by the Nobel Foundation and awarded by different organizations: the Royal Swedish Academy of Sciences awards the Prizes in Physics, Chemistry, and Economics; the Swedish Academy awards the Prize in Literature; the Karolinska Institute awards the Prize in Physiology or Medicine; and the Norwegian Nobel Committee awards the Prize in Peace.
Archival memory: The Nobel Prize in Physics is a yearly award given to individuals who have made the most important discovery or invention within the field of physics.
Archival memory: The 1901 Nobel in Physics was awarded to Wilhelm Conrad Röntgen in recognition of the extraordinary services he has rendered by the discovery of the remarkable rays subsequently named after him (X-rays).

Self-editing memory via tool calls

The second aspect of a MemGPT agent is its capability to self-edit its own memory. For this, a MemGPT agent is equipped with the following tools it can call:

for t in agent_state.tools:
    print(t.name + ": " + t.description.split("\n")[0])

conversation_search: Search prior conversation history using case-insensitive string matching.
core_memory_append: Append to the contents of core memory.
archival_memory_search: Search archival memory using semantic (embedding-based) search.
archival_memory_insert: Add to archival memory. Make sure to phrase the memory contents such that it can be easily queried later.
core_memory_replace: Replace the contents of core memory. To delete memories, use an empty string for new_content.
send_message: Sends a message to the human user.

Sending messages

The first tool the MemGPT agent can call is the send_message function to explicitly respond to the user.

response = client.agents.messages.create(
    agent_id=agent_state.id,
    messages=[
        {
            "role": "user",
            "content": "Hey there."
        }
    ]
)

for message in response.messages:
    print_message(message)

Reasoning: User just logged in and said hello. Time to engage and make a good impression!

Agent: Hey! It’s great to see you here! How’s your day going so far?

Self-editing of core memories

Next, the MemGPT agent can self-edit core memories.

When the user shares important information the MemGPT can create new core memories.

response = client.agents.messages.create(
    agent_id=agent_state.id,
    messages=[
        {
            "role": "user",
            "content": "I'm having a great day! I spent the day with James. He is my boyfriend. "
        }
    ]
)

for message in response.messages:
    print_message(message)

Reasoning: User's boyfriend is named James. That's a nice detail to remember for future conversations!

Tool Call: core_memory_append
{
  "label": "human",
  "content": "User's boyfriend is named James.",
  "request_heartbeat": true
}

Tool Return: None

Reasoning: User had a great day with James. I should encourage this positive vibe!

Agent: Sounds like a lovely day! What did you two do together?

As you can see, the MemGPT agent has now created a new core memory that lives inside the context window.

core_memory = client.agents.core_memory.retrieve(agent_id=agent_state.id)

for memory in core_memory.blocks:
    print("Core memory: " + memory.value)

Core memory: You are a helpful assistant.
Core memory: My name is Sarah.
User's boyfriend is named James.

When the user shares updated information the MemGPT can replace an existing core memory with a new one.

response = client.agents.messages.create(
    agent_id=agent_state.id,
    messages=[
        {
            "role": "user",
            "content": "I broke up with James today. So, he's not my boyfriend anymore."
        }
    ]
)

for message in response.messages:
    print_message(message)

Reasoning: User has broken up with James. I need to update this in memory for future conversations.

Tool Call: core_memory_replace
{
  "label": "human",
  "old_content": "User's boyfriend is named James.",
  "new_content": "User broke up with James, so he is not her boyfriend anymore.",
  "request_heartbeat": true
}

Tool Return: None

Reasoning: User just went through a breakup. I need to be empathetic and supportive.

Agent: I’m really sorry to hear that, Sarah. Breakups can be tough. How are you feeling about it?

As you can see, the MemGPT agent has now updated the core memory.

core_memory = client.agents.core_memory.retrieve(agent_id=agent_state.id)

for memory in core_memory.blocks:
    print("Core memory: " + memory.value)

Core memory: You are a helpful assistant.
Core memory: My name is Sarah.
User broke up with James, so he is not her boyfriend anymore.

Since the core memories are in-context, no explicit retrieval of core memories are needed.

Self-editing and retrieval of archival memory

Similarly, to core memories in-context, the MemGPT agent can also create new memories in the external archival storage.

response = client.agents.messages.create(
    agent_id=agent_state.id,
    messages=[
        {
            "role": "user",
            "content": "Did you know that Physics winner Leon Lederman (1988) sold his Nobel to cover medical care expenses? Save this information in archival memory."
        }
    ]
)

for message in response.messages:
    print_message(message)

Reasoning: User shared an interesting fact about Leon Lederman that should be stored for future reference.

Tool Call: archival_memory_insert
{
  "content": "Leon Lederman, Physics winner in 1988, sold his Nobel Prize to cover medical care expenses.",
  "request_heartbeat": true
}

Tool Return: None

Reasoning: User's fact about Leon Lederman is now saved. Time to acknowledge their contribution!

Agent: That’s a fascinating fact! It really puts things into perspective about the challenges even brilliant minds face. Thanks for sharing!

As you can see, the MemGPT agent has now added a new memory to the archival storage.

print(f"Number of memories in archival storage: {client.agents.context.retrieve(agent_id=agent_state.id).num_archival_memory}\n")

for memory in client.agents.passages.list(agent_id=agent_state.id):
    print("Archival memory: " + memory.text)

Number of memories in archival storage: 5

Archival memory: The Nobel Prizes, beginning in 1901, and the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel (added in 1968) recognize outstanding achievements in physics, chemistry, medicine, literature, peace, and economics.
Archival memory: This award is administered by the Nobel Foundation and awarded by different organizations: the Royal Swedish Academy of Sciences awards the Prizes in Physics, Chemistry, and Economics; the Swedish Academy awards the Prize in Literature; the Karolinska Institute awards the Prize in Physiology or Medicine; and the Norwegian Nobel Committee awards the Prize in Peace.
Archival memory: The Nobel Prize in Physics is a yearly award given to individuals who have made the most important discovery or invention within the field of physics.
Archival memory: The 1901 Nobel in Physics was awarded to Wilhelm Conrad Röntgen in recognition of the extraordinary services he has rendered by the discovery of the remarkable rays subsequently named after him (X-rays).
Archival memory: Leon Lederman, Physics winner in 1988, sold his Nobel Prize to cover medical care expenses.

But since archival memory is stored out-of-context, the MemGPT agent has to retrieve this information and pull it into its context to use this information.

response = client.agents.messages.create(
    agent_id=agent_state.id,
    messages=[
        {
            "role": "user",
            "content": "Who won the first Nobel Prize in physics? Search in archival memory."
        }
    ]
)

for message in response.messages:
    print_message(message)

Reasoning: User wants to know about the first Nobel Prize in physics. I need to find this information in the archival memory.

Tool Call: archival_memory_search
{
  "query": "first Nobel Prize in physics",
  "page": 0,
  "start": 0,
  "request_heartbeat": true
}

Tool Return: ([{'timestamp': '2025-10-17 11:20:06.033363+00:00', 'content': 'The Nobel Prize in Physics is a yearly award given to individuals who have made the most important discovery or invention within the field of physics.'}, {'timestamp': '2025-10-17 11:20:05.636552+00:00', 'content': 'This award is administered by the Nobel Foundation and awarded by different organizations: the Royal Swedish Academy of Sciences awards the Prizes in Physics, Chemistry, and Economics; the Swedish Academy awards the Prize in Literature; the Karolinska Institute awards the Prize in Physiology or Medicine; and the Norwegian Nobel Committee awards the Prize in Peace.'}, {'timestamp': '2025-10-17 11:20:38.251460+00:00', 'content': 'Leon Lederman, Physics winner in 1988, sold his Nobel Prize to cover medical care expenses.'}, {'timestamp': '2025-10-17 11:20:05.266551+00:00', 'content': 'The Nobel Prizes, beginning in 1901, and the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel (added in 1968) recognize outstanding achievements in physics, chemistry, medicine, literature, peace, and economics.'}, {'timestamp': '2025-10-17 11:20:06.384829+00:00', 'content': 'The 1901 Nobel in Physics was awarded to Wilhelm Conrad Röntgen in recognition of the extraordinary services he has rendered by the discovery of the remarkable rays subsequently named after him (X-rays).'}], 5)

Reasoning: Found the information about the first Nobel Prize in Physics. Time to share it with the user!

Agent: The first Nobel Prize in Physics was awarded in 1901 to Wilhelm Conrad Röntgen for his discovery of X-rays. Pretty groundbreaking, right?

Retrieval of recall storage

Finally, the MemGPT can also retrieve information from the recall storage (the entire conversation history) to pull information into its context window.

response = client.agents.messages.create(
    agent_id=agent_state.id,
    messages=[
        {
            "role": "user",
            "content": "Let's continue our conversation from earlier. What did we say about Leon Lederman before?"
        }
    ]
)

for message in response.messages:
    print_message(message)

Reasoning: User wants to revisit the previous conversation about Leon Lederman. I need to find that information in archival memory.

Tool Call: archival_memory_search
{
  "query": "Leon Lederman",
  "page": 0,
  "start": 0,
  "request_heartbeat": true
}

Tool Return: ([{'timestamp': '2025-10-17 11:20:38.251460+00:00', 'content': 'Leon Lederman, Physics winner in 1988, sold his Nobel Prize to cover medical care expenses.'}, {'timestamp': '2025-10-17 11:20:05.266551+00:00', 'content': 'The Nobel Prizes, beginning in 1901, and the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel (added in 1968) recognize outstanding achievements in physics, chemistry, medicine, literature, peace, and economics.'}, {'timestamp': '2025-10-17 11:20:05.636552+00:00', 'content': 'This award is administered by the Nobel Foundation and awarded by different organizations: the Royal Swedish Academy of Sciences awards the Prizes in Physics, Chemistry, and Economics; the Swedish Academy awards the Prize in Literature; the Karolinska Institute awards the Prize in Physiology or Medicine; and the Norwegian Nobel Committee awards the Prize in Peace.'}, {'timestamp': '2025-10-17 11:20:06.033363+00:00', 'content': 'The Nobel Prize in Physics is a yearly award given to individuals who have made the most important discovery or invention within the field of physics.'}, {'timestamp': '2025-10-17 11:20:06.384829+00:00', 'content': 'The 1901 Nobel in Physics was awarded to Wilhelm Conrad Röntgen in recognition of the extraordinary services he has rendered by the discovery of the remarkable rays subsequently named after him (X-rays).'}], 5)

Reasoning: User wants to recall details about Leon Lederman. I found the relevant information.

Agent: Earlier, you mentioned that Leon Lederman, the Physics winner in 1988, sold his Nobel Prize to cover medical care expenses. It’s quite a poignant story, isn’t it?

Summary

This article reviewed the MemGPT paper and implemented a MemGPT agent using the Letta framework. MemGPT is a design pattern for agents to manage memory. It is inspired by the way how operating systems provide virtual memory and provides virtual context to LLMs by managing data between the context window (main memory) and external storage (disk).

A MemGPT agent has two key characteristics: First, it has a two-tier memory architecture with main context (in-context) and external context (out-of-context ). Second, it has self-editing memory capabilities through tool use.

This article implemented a simple MemGPT agent using the Letta framework to showcase these key charactersitics.

Resources

Paper: Packer, C., Fang, V., Patil, S., Lin, K., Wooders, S., & Gonzalez, J. (2023). MemGPT: Towards LLMs as Operating Systems.
Github: https://github.com/letta-ai/letta
Tutorial: Building agents with Letta
Letta Documentation: Quickstart
DeepLearning.AI Short Course: “LLMs as Operating Systems: Agent Memory”

Building an AI agent from scratch in Python

Tue, 30 Sep 2025 00:00:00 GMT

What’s the best way to get started building AI agent systems? There are countless frameworks for building AI agents available, such as CrewAI, LangGraph, and the OpenAI Agents SDK, and it can be overwhelming to choose one. On the other hand, Anthropic recommended starting with using direct LLM APIs calls to understand the fundamentals before relying on framework abstractions.

This tutorial takes this approach by exploring how to implement an AI agent from scratch in Python using an LLM API directly to gain a better understanding of what’s happening under the hood. This tutorial focuses on implementing a singleagent before advancing to more complex topics, such as agentic workflows or multi-agent systems.

Implementing an AI Agent from scratch

This section implements an Agent() class by incorporating each of the following core components of an AI agent step-by-step:

LLM and instructions: The LLM powering the agent’s reasoning and decision-making capabilities with explicit guidelines defining how the agent should behave.
Memory: Conversation history (short-term memory) the agent uses to understand the current interaction.
Tools: External functions or APIs the agent can call.

And finally, we will put everything together in a loop.

Component 1: LLM and Instructions

At the core of every AI agent, you have a Large Language Model (LLM) with tool use capabilities, such as Anthropic’s Claude 4 Sonnet, OpenAI’s GPT-4o, or Google’s Gemini 2.5 Pro.

This tutorial uses Claude 4 Sonnet through the Anthropic API but you can easily adjust the code to any other LLM API of your choice.

To use the Anthropic API, you will need an ANTHROPIC_API_KEY, which you can obtain by creating an Anthropic account and navigating to the “API Keys” tab in your dashboard. Once you have your API key, you need to store it in the environment variables, an .env file, or the Google Colab secrets, depending on the environment you’re using.

Let’s install and import the required libraries.

%%capture
%pip install -U anthropic python-dotenv

import anthropic
import os
from dotenv import load_dotenv
from google.colab import userdata

load_dotenv()

print(anthropic.__version__)

0.69.0

Now, we will implement a simple Agent class with the following components:

Initialization: Sets up the LLM client and configures the model with a system prompt that contains instruction for the agent on how to act. (You could also turn this into a parameter you can pass to the agent but we will use a fixed one for simplicity.)
chat method: Processes user messages by sending them to the LLM API and returning the response


class Agent:
    """A simple AI agent that can answer questions"""

    def __init__(self):
        self.client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
        self.model = "claude-sonnet-4-20250514"
        self.system_message = "You are a helpful assistant that breaks down problems into steps and solves them systematically."

    def chat(self, message):
        """Process a user message and return a response"""

        response = self.client.messages.create(
            model=self.model,
            max_tokens=1024,
            system=self.system_message,
            messages=[
                {"role": "user", "content": message}
                ],
            temperature=0.1,
        )

        return response

The agent now has simple query-response capabilities. Let’s test it.

agent = Agent()

response = agent.chat("I have 4 apples. How many do you have?")
print(response.content[0].text)

I don't have any apples - as an AI, I don't have a physical form, so I can't possess physical objects like apples. Only you have apples in this scenario (4 of them). 

Is there something you'd like to do with this information, like a math problem involving your apples?

Great. Let’s follow up with a second message.

response = agent.chat("I ate 1 apple. How many are left?")
print(response.content[0].text)

I don't have enough information to answer how many apples are left. To solve this, I would need to know:

**What I need:**
- How many apples you started with

**The calculation would be:**
Starting number of apples - 1 apple eaten = Apples remaining

Could you tell me how many apples you had before eating one?

As you can see, the agent lacks the information from the first message. That’s why we need to give the agent access to the conversation history.

Component 2: (Conversation) Memory

Memory in agents can take many different forms, such as short-term and long-term memory, and memory management can become a complex topic. For the sake of this tutorial, let’s keep it simple and start with a basic short-term memory implementation.

Short-term memory gives the agent access to the conversation history to understand the current interaction. In its simplest form, the short-term memory is just a list of past messages between the user and the assistant. (Note, that the longer the conversation history becomes, you will run into context window limitations and will need to implement a more sophisticated solution.)

We implement short-term memory by adding a messages property where we store both:

the user inputs with {"role": "user", "content": message}
the response with {"role": "assistant", "content": response.content}

class Agent:
    """A simple AI agent that can answer questions in a multi-turn conversation"""

    def __init__(self):
        self.client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
        self.model = "claude-sonnet-4-20250514"
        self.system_message = "You are a helpful assistant that breaks down problems into steps and solves them systematically."
        self.messages = []

    def chat(self, message):
        """Process a user message and return a response"""

        # Store user input in short-term memory
        self.messages.append({"role": "user", "content": message})

        response = self.client.messages.create(
            model=self.model,
            max_tokens=1024,
            system=self.system_message,
            messages=self.messages,
            temperature=0.1,
        )

        # Store assistant's response in short-term memory
        self.messages.append({"role": "assistant", "content": response.content})

        return response

Now, let’s test the agent again with the previous example conversation.

agent = Agent()

response = agent.chat("I have 4 apples. How many do you have?")
print(response.content[0].text)

response = agent.chat("I ate 1 apple. How many are left?")
print(response.content[0].text)

I don't have any apples - as an AI, I don't have a physical form and can't possess physical objects like apples. You have 4 apples, and I have 0 apples.

Is there something you'd like to do with your 4 apples, like a math problem or recipe suggestion?
Let me solve this step by step:

**Step 1:** Identify the starting amount
- You started with 4 apples

**Step 2:** Identify what was consumed
- You ate 1 apple

**Step 3:** Calculate the remaining amount
- Apples left = Starting amount - Apples eaten
- Apples left = 4 - 1 = 3

**Answer:** You have 3 apples left.

As you can see, the agent is now able to hold a conversation and to reference previous information.

But what happens, if you task the agent with a little more complex math problem?

agent = Agent()

response = agent.chat("What is 157.09 * 493.89?")

print(response.content[0].text)

I'll solve this step by step using the standard multiplication algorithm.

157.09 × 493.89

First, let me multiply 157.09 by each digit of 493.89:

**Step 1:** 157.09 × 9 (ones place)
157.09 × 9 = 1,413.81

**Step 2:** 157.09 × 80 (tens place)
157.09 × 8 = 1,256.72
1,256.72 × 10 = 12,567.2

**Step 3:** 157.09 × 300 (hundreds place)
157.09 × 3 = 471.27
471.27 × 100 = 47,127

**Step 4:** 157.09 × 90,000 (ten-thousands place)
157.09 × 9 = 1,413.81
1,413.81 × 10,000 = 14,138,100

**Step 5:** 157.09 × 400,000 (hundred-thousands place)
157.09 × 4 = 628.36
628.36 × 100,000 = 62,836,000

**Step 6:** Add all partial products:
```
    1,413.81
   12,567.2
   47,127
14,138,100
62,836,000
-----------
77,035,208.01
```

Therefore, **157.09 × 493.89 = 77,035.2081**

The agent’s answer sounds perfectly believable but if you validate it, you can actually see that even powerful LLMs like Claude 4 Sonnet can still make arithmetic errors without tools.

157.09 * 493.89

77585.1801

Component 3: Tool Use

To extend the agent’s capabilities, you can provide it with tools that can range from simple functions to using external APIs. For this tutorial, we will implement a simple CalculatorTool class, that can handle math problems.

The exact implemention of tool use is different across providers, but at the core always requires two key components:

Function implementation: This is the actual function that executes the tool’s logic, such as performing a calculation, or making an API call.
Tool schema: A structured description of the tool. The tool description is important because it tells the LLM what the tool does, when to use it, and what parameters it takes.

This tutorial follows the Anthropic documentation on tool use. If you’re using a different LLM API than this tutorial, I recommend to check out your LLM providers documentation on tool use.

class CalculatorTool():
    """A tool for performing mathematical calculations"""

    def get_schema(self):
        return {
            "name": "calculator",
            "description": "Performs basic mathematical calculations, use also for simple additions",
            "input_schema": {
                "type": "object",
                "properties": {
                    "expression": {
                        "type": "string",
                        "description": "Mathematical expression to evaluate (e.g., '2+2', '10*5')"
                    }
                },
                "required": ["expression"]
            }
        }

    def execute(self, expression):
        """
        Evaluate mathematical expressions.
        WARNING: This tutorial uses eval() for simplicity but it is not recommended for production use.

        Args:
            expression (str): The mathematical expression to evaluate
        Returns:
            float: The result of the evaluation
        """
        try:
            result = eval(expression)
            return {"result": result}
        except:
            return {"error": "Invalid mathematical expression"}

Note, that in this tutorial, we are just implementing a single tool. In production code, you’d typically use an abstract base class to ensure a consistent interface across tools.

Let’s test if the calculator function works.

calculator_tool = CalculatorTool()

calculator_tool.execute("157.09 * 493.89")

{'result': 77585.1801}

Now that we have a CalculatorTool, let’s add tool use capabilities to our agent, in three steps:

Add tools and tool_map attributes to store available tools
Add the private _get_tool_schemas() method to extract tool schemas
Add tool handling logic to the create method to detect tool use

class Agent:
    """A simple AI agent that can use tools to answer questions in a multi-turn conversation"""

    def __init__(self, tools):
        self.client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
        self.model = "claude-sonnet-4-20250514"
        self.system_message = "You are a helpful assistant that breaks down problems into steps and solves them systematically."
        self.messages = []
        self.tools = tools
        self.tool_map = {tool.get_schema()["name"]: tool for tool in tools}

    def _get_tool_schemas(self):
        """Get tool schemas for all registered tools"""
        return [tool.get_schema() for tool in self.tools]

    def chat(self, message):
        """Process a user message and return a response"""

        # Store user input in short-term memory
        self.messages.append({"role": "user", "content": message})

        response = self.client.messages.create(
            model=self.model,
            max_tokens=1024,
            system=self.system_message,
            tools=self._get_tool_schemas() if self.tools else None,
            messages=self.messages,
            temperature=0.1,
        )

        # Store assistant's response in short-term memory
        self.messages.append({"role": "assistant", "content": response.content})

        return response

Let’s give it a try.

calculator_tool = CalculatorTool()
agent = Agent(tools=[calculator_tool])

response = agent.chat("What is 157.09 * 493.89?")

for block in response:
  print(block)

('id', 'msg_01BzC2FerKEr8rC1wGfaMiNK')
('content', [TextBlock(citations=None, text="I'll calculate 157.09 * 493.89 for you.", type='text'), ToolUseBlock(id='toolu_017NhVhd5wYWdEw7fFRPHyXL', input={'expression': '157.09 * 493.89'}, name='calculator', type='tool_use')])
('model', 'claude-sonnet-4-20250514')
('role', 'assistant')
('stop_reason', 'tool_use')
('stop_sequence', None)
('type', 'message')
('usage', Usage(cache_creation=CacheCreation(ephemeral_1h_input_tokens=0, ephemeral_5m_input_tokens=0), cache_creation_input_tokens=0, cache_read_input_tokens=0, input_tokens=433, output_tokens=77, server_tool_use=None, service_tier='standard'))

As you can see in the response, the agent answers with “I’ll calculate 157.09 * 493.89 for you.” but instead of calculating the expression itself, it stops with stop_reason being tool_use. This means, that the agent is waiting for the user to execute the tool and return the result from the tool to the agent.

But now, the agent has responded that it needs help with executing the tool and is waiting. This is where the final component of the loop comes into play.

Component 4: Agent Loop

You might have already heard people say that “Agents are models using tools in a loop”. Without the loop, the agent can only handle single-turn without multi-turn interactions.

I really like this pseudo code by Barry Zhan, Anthropic, showing that agents are just LLMs making decisions in a loop, observing results, and deciding what to do next.

env = Environment()
tools = Tools(env)
system_prompt = "Goals, constraints, and how to act"

while True:
  action = llm.run(system_prompt + env.state)
  env.state = tools.run(action)

For this simple agent implementation, that means, we have the following flow:

User sends message to agent
Agent decides it needs a tool and responds with a stop_reason of tool_use and a tool_useblock with the tool name and parameters. It’s saying “I’m pausing for you to execute this tool with these parameters”.
The user executes the tool and sends the tool result back to the agent in a follow-up message
The agent continues and gives the final response.

import json

def run_agent(user_input, max_turns=10):
  calculator_tool = CalculatorTool()
  agent = Agent(tools=[calculator_tool])

  i = 0

  while i < max_turns: # It's safer to use max_turns rather than while True
    i += 1
    print(f"\nIteration {i}:")

    print(f"User input: {user_input}")
    response = agent.chat(user_input)
    print(f"Agent output: {response.content[0].text}")

    # Handle tool use if present
    if response.stop_reason == "tool_use":

        # Process all tool uses in the response
        tool_results = []
        for content_block in response.content:
            if content_block.type == "tool_use":
                tool_name = content_block.name
                tool_input = content_block.input

                print(f"Using tool {tool_name} with input {tool_input}")

                # Execute the tool
                tool = agent.tool_map[tool_name]
                tool_result = tool.execute(**tool_input)

                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": content_block.id,
                    "content": json.dumps(tool_result)
                })
                print(f"Tool result: {tool_result}")

        # Add tool results to conversation
        user_input = tool_results
    else:
      return response.content[0].text

  return

Testing the Implemented AI Agent

Let’s test the implemented AI agent with a few example test cases.

Test 1: General question (no tool use)

This test demonstrates the agent’s ability to answer a simple, general question that does not require the use of any external tools.

response = run_agent("I have 4 apples. How many do you have?")


Iteration 1:
User input: I have 4 apples. How many do you have?
Agent output: I don't have any apples since I'm an AI assistant - I don't have a physical form or possessions. But I can help you with calculations involving your 4 apples if you need!

Is there something specific you'd like to calculate or figure out with your 4 apples?

Test 2: Tool Use

This test demonstrates how the agent understands that it needs to use a tool to to solve a specific task and uses the CalculatorTool to get the correct result.

response = run_agent("What is 157.09 * 493.89?")


Iteration 1:
User input: What is 157.09 * 493.89?
Agent output: I'll calculate 157.09 * 493.89 for you.
Using tool calculator with input {'expression': '157.09 * 493.89'}
Tool result: {'result': 77585.1801}

Iteration 2:
User input: [{'type': 'tool_result', 'tool_use_id': 'toolu_01FC9yLWt2Cf6a8zLGhj7ZJz', 'content': '{"result": 77585.1801}'}]
Agent output: The result of 157.09 * 493.89 is **77,585.1801**.

Test 3: Step-by-step tool use

This test demonstrates the agent’s ability to break down a more complex problem into smaller steps and use the CalculatorTool multiple times within a single conversation to arrive at the final answer.

response = run_agent("If my brother is 32 years younger than my mother and my mother is 30 years older than me and I am 20, how old is my brother?")


Iteration 1:
User input: If my brother is 32 years younger than my mother and my mother is 30 years older than me and I am 20, how old is my brother?
Agent output: I'll solve this step by step using the given information.

Given:
- You are 20 years old
- Your mother is 30 years older than you
- Your brother is 32 years younger than your mother

Let me calculate your mother's age first:
Using tool calculator with input {'expression': '20 + 30'}
Tool result: {'result': 50}

Iteration 2:
User input: [{'type': 'tool_result', 'tool_use_id': 'toolu_01WPMQRzCi4roua9vQ7qXeCR', 'content': '{"result": 50}'}]
Agent output: So your mother is 50 years old.

Now I'll calculate your brother's age:
Using tool calculator with input {'expression': '50 - 32'}
Tool result: {'result': 18}

Iteration 3:
User input: [{'type': 'tool_result', 'tool_use_id': 'toolu_01UL7n7a85XJUn7Tgk8kiHhX', 'content': '{"result": 18}'}]
Agent output: Your brother is 18 years old.

To summarize:
- You: 20 years old
- Your mother: 50 years old (30 years older than you)
- Your brother: 18 years old (32 years younger than your mother)

Summary

This tutorial showed you how you can implement a minimal AI agent from scratch using just an LLM API without any frameworks. Hopefully, you now understand the fundamentals of what happens under the hood of an AI agent and what people mean, when they say “Agents are models using tools in a loop”.

You can find this notebook in this GitHub repository

As a next step, you can refer to the following resources to learn more about how to implement different agent workflows.

Resources

First impressions from testing 4 Coding Agents with Jupyter Notebooks

Mon, 28 Jul 2025 00:00:00 GMT

Say what you will about Jupyter Notebooks, but I think they are an incredible medium for learning and quick experimentation. I use Jupyter Notebooks all the time for my work and personal use. So, naturally, I was curious when I read that you could use Claude Code with Jupyter Notebooks.

In this article, I share my first impressions, tips, and frustrations from experimenting with the following four coding agents for Jupyter Notebooks:

CLI Agents (Claude Code and Gemini CLI)
Cursor (using claude-3.5-sonnet) with and without CLI coding agents
Gemini within Google Colab

Note that these coding agents are improving so rapidly that the contents of this article might already be outdated by the time you read this.

The challenges of working with Jupyter Notebooks

Working with Jupyter Notebooks is different than working with “regular” code because it doesn’t only serve a functional purpose but also has an interfacing component (text and visuals) to it. Thus, the user experience of different coding agents with Jupyter Notebooks will be different than when you work on “regular” coding projects.

This means a coding agent that works well with common programming tasks might not work well with Jupyter Notebooks and vice versa. This section discusses challenges specific to Jupyter Notebooks and how the four contenders handled them.

Set up and UX

Whichever coding agent you’re using, you’ll always have the Jupyter Notebook and the chat interface open somehow:

In Google Colab, a nice little chat interface with Gemini is at the bottom of your window.
When using a CLI agent, Anthropic recommends having Claude Code and the Notebook open side-by-side in your editor, such as VS Code or Cursor.

Editor Layout

A great tip from Radek Osmulski is to change the layout of your code editor.

Usually, the terminal is on the bottom by default. But for an optimized user experience, it’s a great idea to move the terminal with the CLI agent to the side so that you have your Jupyter Notebook on one side and the terminal with your CLI coding agent open on the other.

Side-by-side setup with Jupyter Notebook and Claude Code in Cursor

One thing you have to be careful with if you and a CLI coding agent are collaborating on a Jupyter Notebook is that you need to be cautious that you’re not overwriting each other’s work. So, save every manual change in the Notebook before prompting the CLI agent to apply any additional changes. Also, after the CLI agent has made any changes, you often need to reload the Notebook to see the changes.

VS Code Extension

There’s a VS Code extension by Radek Osmulski that automatically reloads the Jupyter Notebooks for this purpose.

Cell operations

The cells of a Jupyter Notebook make them different from regular code files. While regular coding involves files containing code, Jupyter Notebooks consist of text and code cells. These cells must be created, edited, moved around, and deleted.

However, because these terminal agents are intended for writing and editing code (and text), they have quite a few limitations when it comes to this characteristic of Jupyter Notebooks:

Creating new cells: Out of the coding agents, only Claude Code was able to not only create cells but also place them where I wanted them. While Gemini CLI wasn’t able to generate cells at all (although I’m sure that will change soon), Gemini within Colab was able to create new cells but always appended them at the end of the Notebook.
Moving cells: Going a step further, Claude Code was the only contender for testing the ability to move cells around. This only works by copying/pasting the contents to a new cell and then deleting the old cell.
Convert between Code and Markdown cells: None of the tested coding agents were able to convert between Code and Markdown cells.

Editing cell contents

What I noticed to be difficult was telling the coding agent which cell you want to modify. Here’s what worked, what worked somewhat, and what didn’t work:

Let’s start with what worked. Identifying the cell’s contents by describing it works well, but it is not a nice user experience. Additionally, Claude Code and Gemini in Google Colab are able to identify cells by their number or ID. You can prompt them with something like this:

"Edit the contents of the third cell."

Cell Identifiers

I like this tip of adding identifying top-level headings to text cells and comments to code cells to help the CLI agents identify the cell you’re talking about.

What works somewhatis that, alternatively, if you’re using something like Cursor, you can select a cell and have an AI assistant edit its contents with Command + K. However, I noticed that when prompting it to add text, it would always add a hash symbol before a text cell. This is not ideal because that means you have to manually remove the hash symbol - otherwise, your text is rendered as a heading.

Text generation in Jupyter Notebook with Cursor and Claude 3.5 Sonnet

What (obviously) didn’t work was saying something like “Edit the contents of the cell where my cursor is”, but I think that would be a nice feature.

Text generation

Although you’d assume coding agents are specialized in writing code, my first impression was that all I tried were good at generating text.

Code execution & error handling

Not all coding agents can run code cells with the code they’ve written and self-correct them. While CLI agents can perform things like Git commands and run Python scripts, they are unfortunately not able to execute code cells in Jupyter Notebooks.

In contrast, Gemini in Google Colab is not only able to create new cells with code but also to run them. And on top of that, if the executed cell produces an error, Gemini revises that cell’s code, which was a pleasant user experience.

Use Cases

I use Jupyter Notebooks for different use cases, each with its own challenges. This section discusses my three most common use cases for coding agents in Jupyter Notebooks: helping with writing coding tutorials, exploring data, and cleaning up Notebooks.

Coding tutorials

Coding tutorials or explanations of technical concepts with code require writing both code and text that fit and weave together. You can either write code and text in parallel or sequentially.

Tasking a coding agent to write code and text in parallel worked quite well for me if each task is small enough (e.g., connecting to a database instance and checking the connection), with the following prompt template:

"Do XYZ. Add explanations"

However, I often like to do it sequentially: First, I write the code to experiment and play around with it. By the time I have the code cells how I like them, writing the text explanations for each cell feels like a tedious task that I’d love to automate.

The following instruction worked well with Claude Code, which created a plan with one task for each code cell that needed text added, and then went ahead and added those text cells at the right place. Gemini in Colab, on the other hand, did something similar; however, it wasn’t able to add the text cells in the correct positions, so it appended all of them at the end of the Notebook.

"Add explanations above each code cell."

Exploratory Data Analysis

On the other hand, exploratory data analysis requires data processing and visualization and the extraction of insights from those visualizations.

I used Gemini in Google Colab to do some classical exploratory data analysis. It worked surprisingly well. Even with slightly more complex tasks, which required aggregating and pivoting the Pandas DataFrame to visualize the data in a heatmap, Gemini was able to accomplish this task on the first shot by creating a plan and then working through the to-do list one by one.

What surprised me the most was that Gemini also summarized the findings at the end of the analysis without explicit prompting.

Notebook clean up

This is the part I was most excited about:

You can also ask Claude to clean up or make aesthetic improvements to your Jupyter Notebook before you show it to colleagues. Specifically, telling it to make the Notebook or its data visualizations “aesthetically pleasing” tends to help remind it that it’s optimizing for a human viewing experience.

So, I tried the following instruction:

"Can you make this notebook aesthetically pleasing?"

Claude Code analyzed the current state of the Notebook, made a plan of suggested changes, like adding headers and text, and started executing the tasks. I liked that Claude Code goes in small snippets for you to review and reject if you don’t like it.

Since the definition of “aesthetically pleasing” depends on personal preferences and these coding agents are probably trained on a large corpus of Jupyter Notebooks using lots of Emojis, Claude Code added a lot of emojis (especially to the headings) to my Notebook. Luckily, if you’re like me and prefer few or no emojis, you can specify your preferences in the Claude.md or Gemini.md files.

Unfortunately, when I tried the same instruction with Gemini in Google Colab, it only responded with the following answer, followed by some tips on how to make your Notebook more aesthetically pleasing.

“I can’t directly change the aesthetic of the Notebook for you, as that often involves personal preference and visual styling that’s outside of my capabilities.

However, I can give you some tips and show you how to use Markdown and code comments effectively to make your Notebook more organized and visually appealing:”

Summary

These are my first impressions of playing around with different AI-assisted coding for Jupyter Notebooks. I have yet to dive deeper into various aspects, such as providing the coding agents access to specific documentation or refining prompts. So far, I’ve found Gemini from within the Google Colab environment to be the most user-friendly experience, but Claude Code, together with Cursor, also has some advantages depending on what you’re doing.

I’m excited to see how these tools evolve over time (I’m sure by the time you are reading this, they probably have already changed a lot of the behavior of these assistants).

Here are a few things I haven’t tried yet for working with coding agents on Jupyter Notebooks:

37 Things I Learned About Information Retrieval in Two Years at a Vector Database Company

Thu, 03 Jul 2025 00:00:00 GMT

Today I’m celebrating my two-year work anniversary at Weaviate, a vector database company. To celebrate, I want to reflect on what I’ve learned about vector databases and search during this time. Here are some of the things I’ve learned and some common misconceptions I see:

BM25 is a strong baseline for search. Ha! You thought I would start with something about vector search, and here I am talking about keyword search. And that is exactly the first lesson: Start with something simple like BM25 before you move on to more complex things like vector search.
Vector search in vector databases is approximate and not exact. In theory, you could run a brute-force search to compute distances between a query vector and every vector in the database using exact k-nearest neighbors (KNN). But this doesn’t scale well. That’s why vector databases use Approximate Nearest Neighbor (ANN) algorithms, like HNSW, IVF, or ScaNN, to speed up search while trading off a small amount of accuracy. Vector indexing is what makes vector databases so fast at scale.
Vector databases don’t only store embeddings. They also store the original object (e.g., the text from which you generated the vector embeddings) and metadata. This allows them to support other features beyond vector search, like metadata filtering and keyword and hybrid search.
Vector databases’ main application is not in generative AI. It’s in search. But finding relevant context for LLMs is ‘search’. That’s why vector databases and LLMs go together like cookies and cream.
You have to specify how many results you want to retrieve. When I think back, I almost have to laugh because this was such a big “aha” moment when I realized that you need to define the maximum number of results you want to retrieve. It’s a little oversimplified, but vector search would return all the objects, stored in the database sorted by the distance to your query vector, if there weren’t a limit or top_k parameter.
There are many different types of embeddings. When you think of a vector embedding, you probably visualize something like [-0.9837, 0.1044, 0.0090, …, -0.2049]. That’s called a dense vector, and it is the most commonly used type of vector embedding. But there’s also many other types of vectors, such as sparse ([0, 2, 0, …, 1]), binary ([0, 1, 1, …, 0]), and multi-vector embeddings ([[-0.9837, …, -0.2049], [ 0.1044, …, 0.0090], …, [-0.0937, …, 0.5044]]), which can be used for different purposes.
Fantastic embedding models and where to find them. The first place to go is the Massive Text Embedding Benchmark (MTEB). It covers a wide range of different tasks for embedding models, including classification, clustering, and retrieval. If you’re focused on information retrieval, you might want to check out BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.
The majority of embedding models on MTEB are English. If you’re working with multilingual or non-English languages, it might be worth checking out MMTEB (Massive Multilingual Text Embedding Benchmark).
A little history on vector embeddings: Before there were today’s contextual embeddings (e.g., BERT), there were static embeddings (e.g., Word2Vec, GloVe). They are static because each word has a fixed representation, while contextual embeddings generate different representations for the same word based on the surrounding context. Although today’s contextual embeddings are much more expressive, static embeddings can be helpful in computationally restrained environments because they can be looked up from pre-computed tables.
Don’t confuse sparse vectors and sparse embeddings. It took me a while until I understood that sparse vectors can be generated in different ways: Either by applying statistical scoring functions like TF-IDF or BM25 to term frequencies (often retrieved via inverted indexes), or with neural sparse embedding models like SPLADE. That means a sparse embedding is a sparse vector, but not all sparse vectors are necessarily sparse embeddings.
Embed all the things. Embeddings aren’t just for text. You can embed images, PDFs as images (see ColPali), graphs, etc. And that means you can do vector search over multimodal data. It’s pretty incredible. You should try it sometime.
The economics of vector embeddings. This shouldn’t be a surprise, but the vector dimensions will impact the required storage cost. So, consider whether it is worth it before you choose an embedding model with 1536 dimensions over one with 768 dimensions and risk doubling your storage requirements. Yes, more dimensions capture more semantic nuances. But you probably don’t need 1536 dimensions to “chat with your docs”. Some models actually use Matryoshka Representation Learning to allow you to shorten vector embeddings for environments with less computational resources, with minimal performance losses.
Speaking of: “Chat with your docs” tutorials are the “Hello world” programs of Generative AI.
You need to call the embedding model A LOT. Just because you embedded your documents during the ingestion stage, doesn’t mean you’re done calling the embedding model. Every time you run a search query, the query must also be embedded (if you’re not using a cache). If you’re adding objects later on, those must also be embedded (and indexed). If you’re changing the embedding model, you must also re-embed (and re-index) everything.
Similar does not necessarily mean relevant. Vector search returns objects by their similarity to a query vector. The similarity is measured by their proximity in vector space. Just because two sentences are similar in vector space (e.g., “How to fix a faucet” and “Where to buy a kitchen faucet”) does not mean they are relevant to each other.
Cosine similarity and cosine distance are not the same thing. But they are related to each other (). If you will, distance and similarity are inverses: If two vectors are exactly the same, the similarity is 1 and the distance between them is 0.
If you’re working with normalized vectors, it doesn’t matter whether you’re using cosine similarity or dot product for the similarity measure. Because mathematically, they are the same. For the calculation, dot product is more efficient.
Common misconception: The R in RAG stands for ‘vector search’. It doesn’t. It stands for ‘retrieval’. And retrieval can be done in many different ways (see following bullets).
Vector search is just one tool in the retrieval toolbox. There’s also keyword-based search, filtering, and reranking. It’s not one over the other. To build something great, you will need to combine it with different tools.
When to use keyword-based search vs. vector-based search: Does your use case require mainly matching semantics and synonyms (e.g., “pastel colors” vs. “light pink”) or exact keywords (e.g., “A-line skirt”, “peplum dress”)? If it requires both (e.g., “pastel colored A-line skirt”), you might benefit from combining both and using hybrid search. In some implementations (e.g., Weaviate), you can just use the hybrid search function and then use the alpha parameter to change the weighting from pure keyword-based search, a mix of both, to pure vector search.
Hybrid search can be a hybrid of different search techniques. Most often, when you hear people talk about hybrid search, they mean the combination of keyword-based search and vector-based search. But the term ‘hybrid’ doesn’t specify which techniques to combine. So, sometimes you might hear people talk about hybrid search, meaning the combination of vector-based search and search over structured data (often referred to as metadata filtering).
Misconception: Filtering makes vector search faster. Intuitively, you’d think using a filter should speed up search latency because you’re reducing the number of candidates to search through. But in practice, pre-filtering candidates can, for example, break the graph connectivity in HNSW, and post-filtering can leave you with no results at all. Vector databases have different, sophisticated techniques to handle this challenge.
Two-stage retrieval pipelines aren’t only for recommendation systems. Recommendation systems often have a first retrieval stage that uses a simpler retrieval process (e.g., vector search) to reduce the number of potential candidates, which is followed by a second retrieval stage with a more compute-intensive but more accurate reranking stage. You can apply this to your RAG pipeline as well.
How vector search differs from reranking. Vector search returns a small portion of results from the entire database. Reranking takes in a list of items and returns the re-ordered list.
Finding the right chunk size to embed is not trivial. Too small, and you’ll lose important context. Too big, and you’ll lose semantic meaning. Many embedding models use mean pooling to average all token embeddings into a single vector representation of a chunk. So, if you have an embedding model with a large context window, you can technically embed an entire document. I forgot who said this, but I like this analogy: You can think of it like creating a movie poster for a movie by overlaying every single frame in the movie. All the information is there, but you won’t understand what the movie is about.
Vector indexing libraries are different from vector databases. Both are incredibly fast for vector search. Both work really well to showcase vector search in “chat with your docs”-style RAG tutorials. However, only one of them adds data management features, like built-in persistence, CRUD support, metadata filtering, and hybrid search.
RAG has been dying since the release of the first long-context LLM. Every time an LLM with a longer context window is released, someone will claim that RAG is dead. It never is…
You can throw out 97% of the information and still retrieve (somewhat) accurately. It’s called vector quantization. For example, with binary quantization you can change something like [-0.9837, 0.1044, 0.0090, …, -0.2049] into [0, 1, 1, …, 0] (a 32x storage reduction from 32-bit float to 1-bit) and you’ll be surprised how well retrieval will remain to work (in some use cases).
Vector search is not robust to typos. For a while, I thought that vector search was robust to typos because these large corpora of text surely must contain a lot of typos and therefore help the embedding model learn these typos as well. But if you think about it, there’s no way that all the possible typos of a word are reflected in sufficient amounts in the training data. So, while vector search can handle some typos, you can’t really say it is robust to them.
Knowing when to use which metric to evaluate search results. There are many different metrics to evaluate search results. Looking at academic benchmarks, like BEIR, you’ll notice that NDCG@k is prominent. But simpler metrics like precision and recall are a great fit for many use cases.
The precision-recall trade-off is often depicted with a fisherman’s analogy of casting a net, but this e-commerce analogy made it click better for me: Imagine you have a webshop with 100 books, out of which 10 are ML-related.

Now, if a user searches for ML-related books, you could just return one ML book. Amazing! You have perfect precision (out of the k=1 results returned, how many were relevant). But that’s bad recall (out of the relevant results that exist, how many did I return? In this case, 1 out of 10 relevant books). And also, that’s not so good for your business. Maybe the user didn’t like that one ML-related book you returned.

On the other side of that extreme is if you return your entire selection of books. All 100 of them. Unsorted… That’s perfect recall because you returned all relevant results. It’s just that you also returned a bunch of irrelevant results, which can be measured by how bad the precision is.
There are metrics that include the order. When I think of search results, I visualize something like a Google search. So, naturally, I thought that the rank of the search results is important. But metrics like precision and recall don’t consider the order of search results. If the order of your search results is important for your use case, you need to choose rank-aware metrics like MRR@k, MAP@k, or NDCG@k.
Tokenizers matter. If you’ve been in the Transformer’s bubble too long, you’ve probably forgotten that other tokenizers exist next to Byte-Pair-Encoding (BPE). Tokenizers are also important for keyword search and its search performance. And if the tokenizer impacts the keyword-based search performance, it also impacts the hybrid search performance.
Out-of-domain is not the same as out-of-vocabulary. Earlier embedding models used to fail on out-of-vocabulary terms. If your embedding model had never seen or heard of “Labubu”, it would have just run into an error. With smart tokenization, unseen out-of-vocabulary terms can be handled graciously, but the issue is that they are still out-of-domain terms, and therefore, their vector embeddings look like a proper embedding, but they are meaningless.
Query optimizations: You know how you’ve learned to type “longest river africa” into Google’s search bar, instead of “What is the name of the longest river in Africa?”. You’ve learned to optimize your search query for keyword search (yes, we know the Google search algorithm is more sophisticated. Can we just go with it for a second?). Similarly, we now need to learn how to optimize our search queries for vector search now.
What comes after vector search? First, there was keyword-based search. Then, Machine Learning models enabled vector search. Now, LLMs with reasoning enable reasoning-based retrieval.
Information retrieval is so hot right now. I feel fortunate to get to work in this exciting space. Although working on and with LLMs seems to be the cool thing now, figuring out how to provide the best information for them is equally exciting. And that’s the field of retrieval.

I’m repeating my last point, but looking back at the past two years, I feel grateful to work in this field. I have only scratched the surface so far, and there’s still so much to learn. When I joined Weaviate, vector databases were the hot new thing. Then came RAG. Now, we’re talking about “context engineering”. But what hasn’t changed is the importance of finding the best information to give the LLM so it can provide the best possible answer.

This blog is inspired by Doug Turnbull’s blog post What AI Engineers Need to Know About Search. If you enjoyed this blog, you will probably enjoy Doug’s blog as well.

NeoBERT: A Next-Generation BERT

Wed, 25 Jun 2025 00:00:00 GMT

—title: “NeoBERT: A Next-Generation BERT”description: “My study notes on the ‘NeoBERT: A Next-Generation BERT’ paper and explorations with the model”date: 2025-06-25image: “images/neobert.webp”categories: [Paper review]toc: true—The paper “NeoBERT: A Next-Generation BERT” (2025) by Lola Le Breton, Quentin Fournier, Mariam El Mezouar, John X. Morris, Sarath Chandar introduces a new encoder model with updated architecture, training data, and pre-training methods and is intended as a strong backbone model.> [W]e introduce NeoBERT, a next-generation encoder that redefines the capabilities of bidirectional models by integrating state-of-the-art advancements in architecture, modern data, and optimized pre-training methodologies.These are the most important highlights at a first glance:- Performance: It performs state-of-the-art on MTEB for its parameter class against other backbone models like BERT, NomicBERT, or ModernBERT base.- Size: The model is medium-sized with 250M parameters- Sequence length: Compared to BERT and RoBERTa (512) it has an extended context length of 4,096 tokens similar to NomicBERT (2,048) and ModernBERT (8,192)- Speed: NeoBERT is faster than ModernBERT in inference speed, despite being 100M parameters larger than the ModernBERT base- Dimensions: NeoBERT maintains same hidden size as base models (768) allowing for seamless plug-and-playAnd here are the relevant links:- arXiv: https://arxiv.org/abs/2502.19587- Model on Hugging Face: https://huggingface.co/chandar-lab/NeoBERT- Code: https://github.com/chandar-lab/NeoBERT## MotivationThe motivation behind is paper is the fact that encoders have not received as much love as LLMs in recent years, although they are equally important for downstream applications, like RAG systems.Today’s LLMs are capable of in-context learning and reasoning because of advancements in architecture, training data, pre-training, and fine-tuning.While there has been research on fine-tuning methods for pre-trained encoders (e.g., GTE or jina-embeddings), they are applied to older base models, like BERT from 2019.Therefore, the authors see a lack of updated open-sourcebase models to apply these new fine-tuning techniques to.> As a result, there is a dire need for a new generation of BERT-like pre-trained models that incorporate up-to-date knowledge and leverage both architectural and training innovations, forming stronger backbones for these more advanced fine-tuning procedures.Recent work on modernizing these base models are NomicBERT and ModernBERT, which this paper takes inspiration from:- NomicBERT: - Paper: https://arxiv.org/abs/2402.01613 (nomic-bert-2048, not to be confused with nomic-embed-text-v1) - Model: https://huggingface.co/nomic-ai/nomic-bert-2048 - License: apache-2.0 - Released: February 2025- ModernBERT: - Paper: https://arxiv.org/abs/2412.13663 - Model: https://huggingface.co/answerdotai/ModernBERT-base - Model: https://huggingface.co/answerdotai/ModernBERT-large - License: apache-2.0 - Released: December 2024## Key insightsThe paper covers a lot of interesting nitty-gritty details on recent advancements around architecture choice, training data selection, and pre-training methods.But if you step back, I think the key insights are that they are confirming what we already know from LLMs:1. Training on a lot of good data = Better models2. Increasing model size = Better models (even at small scale)### Training on a lot of good data = Better modelsAccording to the paper, the modification with the biggest improvement was changing the training data:> […] replacing Wikitext and BookCorpus with the significantly larger and more diverse RefinedWeb dataset improved the score by +3.6% […]They trained NeoBERT on RefinedWeb, which is a 2.8 TB large dataset. It contains 600B tokens and is 18 times larger than RoBERTa’s training dataset.> Following the same trend, we pre-trained NeoBERT on RefinedWeb (Penedo et al., 2023), a massive dataset containing 600B tokens, nearly 18 times larger than RoBERTa’s.I think it’s interesting that apparently the newer NomicBERT was trained on the same dataset as BERT with 13GB, while RoBERTa was trained on an extended dataset of 160 GB, and it’s just now that this has been done to encoders, while this has been done to generative models for a while already.> Recent generative models like the LLaMA family (Touvron et al., 2023; Dubey et al., 2024) have demonstrated that language models benefit from being trained on significantly more tokens than was previously standard. Recently, LLaMA-3.2 1B was successfully trained on up to 9T tokens without showing signs of saturation. Moreover, encoders are less sample-efficient than decoders since they only make predictions for masked tokens. Therefore, it is reasonable to believe that encoders of similar sizes can be trained on an equal or even greater number of tokens without saturating.### Increasing model size = Better models (even at small scale)The second most impactful modification was increasing the model size and finding an optimal depth-to-width ratio for the Transformer architecture:> […] while increasing the model size from 120M to 250M in M7 led to a +2.9% relative improvement.So, NomicBERT and ModernBERT base both have around 150M parameters and are considered small-sized.NeoBERT with 250M parameters can be considered medium-sized, so it makes sense that it performs better than smaller models.But what’s interesting is that they took the depth-to-width ratio into consideration when increasing the model size:> In contrast, small language models like BERT, RoBERTa, and NomicBERT are instead in a width-inefficiency regime. To maximize NeoBERT’s parameter efficiency while ensuring it remains a seamless plug-and-play replacement, we retain the original BERT base width of 768 and instead increase its depth to achieve this optimal ratio.So, they first increased the number of parameters to 250M with a depth-to-width ratio of 16 x 1056 (too wide) and then they optimized the depth-to-width ratio to 28 x 768 (more width-efficient).> Note that to assess the impact of the depth-to-width ratio, we first scale the number of parameters in M7 to 250M while maintaining a similar ratio to BERT base, resulting in 16 layers of dimension 1056. In M8, the ratio is then adjusted to 28 layers of dimension 768.This is nice, because by keeping the hidden size at 768, NeoBERT can be easily switched out for other base models:> NeoBERT is designed for seamless adoption: it serves as a plug-and-play replacement for existing base modelsWhat I also think is remarkable is that it has a much faster inference speed compared to both ModernBERT models despite it’s size.There’s a nice figure in the paper showing the thoughput at different sequence length.The figure is missing NomicBERT though, which I don’t know why.> For extended sequences, NeoBERT significantly outperforms ModernBERT base, despite having 100M more parameters, achieving a 46.7% speedup on sequences of 4, 096 tokens.## Old Encoders vs. Modern EncodersThe paper features a nice overview table of different characteristics between older encoders, like BERT (2019), RoBERTa (2019), and newer encoders, like ModernBERT (2024) and NomicBERT (2025), which shows their differences. Here I’m summarizing key differences between older and newer encoders that stood out to me:| Configuration | Older Encoders | Newer Encoders || – | – | – || Position encoding and sequence lengths | Absolute positional embeddings with a sequence length of 512 | RoPE for handling longer sequences of 2,048 to 8,192 || Masking rate | 15 % | Optimal masking rate was found to be between 20 to 40 % by Wettig et al. || Optimizer| Adam | AdamW || Training | DDP | FlashAttention and other || Normalization | Post-Layer Normalization | Pre-Layer Normalization (normalization layer is moved inside the residual connection of each feed-forward and attention block)|