Building AI Memory Into the Enterprise Context Layer

AI systems get smarter when they can learn from corrections. When a user notices that an agent got something wrong and explains why, that correction should benefit everyone who asks a similar question later.

Consumer chatbots handle this by keeping your previous chats in context. That approach breaks down at enterprise scale. When hundreds or thousands of people query the same data across many sessions, no single context window can hold what the organization has learned. The deep insights and tribal knowledge live across users' sessions, not just within one person's history.

At Collate, we built a memory primitive that solves this by making memories a first-class part of our semantic knowledge graph — governed, shared, and retrievable by any agent or user with the right permissions.

This article covers three things:

What it takes to make enterprise data useful for AI
Why memory is essential at that scale
How Collate's memory architecture works

But before we jump in, it’s important to understand that adding enterprise memory to AI systems is part of a broader effort to make AI more useful in important ways.

Microsoft CEO Satya Nadella recently pointed to something crucial about the next generation of AI: local organizations should benefit from the learning that happens as they use these systems.

"This means the real opportunity is not in picking the best model but instead in building a learning loop on top of models where human capital and token capital compound. You can offload a task, or even a job, but you can never offload your learning. The future of the firm is the ability to compound that learning across people and AI."

Current AI architecture doesn't support local learning by default. Collate's memory does. It's a capability companies can put to work today. Let’s dive into why we built this new primitive and how it works.

Making Data Useful for AI: Context and Ontology

Collate rests on the premise that three things must be in place before AI can work well over enterprise data: Context, Ontology, and Memory.

Context — a map of metadata that describes your entire data landscape. This is the technical, business, and operational metadata: the structure of every table across every source, who owns each asset, where the data came from (lineage), which pipelines produced it, and signals about governance, quality, and sensitivity.

Ontology — a layer of meaning on top of that context. Two parts: mapping assets to standardized classes of data (Person, Customer, Product, Invoice) so an LLM can reason about "all people" or "all PII" without knowing every schema; and capturing agreed definitions for metrics and concepts in glossary entries.

Just pointing an LLM at raw metadata is not enough. Our recent benchmark showed that on text-to-SQL tasks, adding Collate's semantic layer based on context and ontology lifted execution accuracy from 10.8% to 76.5% — a 7.1x improvement.

With context and semantics in place, AI agents can manage the data landscape, power conversational interfaces for data teams and analysts, and support custom agents built with no-code tools or the SDK.

The next question is: How can we learn from experience? Memory is the third primitive which accomplishes that goal.

Why Enterprise Memory Matters

Anyone who has used AI seriously knows that LLM-powered agents need human feedback to reach the right answer.

Sometimes that feedback improves the request — the human clarifies what they meant. But when the subject is complex enterprise data, feedback also improves the AI's understanding of the asset itself. Two examples show what this looks like:

Cancelled orders. An analyst runs a revenue query against an orders table. The number is wrong because cancelled orders should have been subtracted using a separate cancelled-orders table. The memory captures this as a Q&A pair, named removing-cancelled-orders:

Q: Are cancelled orders included in this table?
A: Yes — to remove them, refer to the cancelled-orders table.

Table grain. An analyst doesn't know the granularity of prod.warehouse.orders. The memory, named orders-grain, records:

Q: What is the grain of the orders table?
A: One row per order_id; payments roll up to this grain.

Without memory, the next analyst who runs the same query gets the same wrong answer. Someone taught the system once, but the lesson didn't stick. Multiply that across an organization and the cost is significant.

Memory is what turns one person's learning into everyone's. This is what creates long-lasting organizational knowledge and differentiating intellectual property (IP). The asset you want to retain internally.

How Collate's Memory Works

Before the details, here's the framing we use.

Ontology answers "What could exist?" — the classes and relationships allowed in metadata.
Inside the ontology, the Glossary answers "What does this mean in our business?" — defining terms, metrics, and other concepts.
Context answers "What data is available and how is it stored?"
Memory answers "What have we learned about this particular thing?"

Five questions organize the rest of this section:

How and when are memories captured?
What format are they stored in?
How are they retrieved?
How are they governed?
What makes this architecture distinctive?

How memories are captured

Memories can enter Collate four ways.

Directly by a person. A data steward or authorized user writes one through the Collate UI.
Through AskCollate. When a user corrects an AI answer in conversation, AskCollate can propose the correction as a memory. It's stored as a draft, and a review task is opened for a steward.
Through custom agents in AI Studio. Agents can call client.memories().create(...) in the SDK when they detect a fact worth preserving. Depending on configuration, the memory may go through approval.
Through external integrations. Any program with an API key and the right permissions can POST a memory. Ingestion pipelines can extract them from source systems.

What the format looks like

OpenMetadata (OMD) — the open-source foundation of Collate — is schema-first. Every entity is defined by a JSON Schema, and Memory is no exception. Its schema lives at contextMemory.json.

Most fields are housekeeping (id, name, description, timestamps, versioning). The ones that matter for understanding how memory works are:

question — the canonical question this memory answers
answer — the retained guidance
memoryType — the kind of memory (note, runbook, decision, FAQ)
primaryEntity — the entity this memory is anchored to
relatedEntities — other entities this memory applies to

Modeling memories as question-and-answer pairs is deliberate. That shape is directly usable by LLMs. The text can hold anything: a definition, a warning, a runbook step, a decision.

How memories connect to entities

The important architectural choice is that memories are attached to specific entities in your metadata graph — not stored in one giant pool that must be scanned to be searched.

When a memory is created with primaryEntity pointing at the orders table, two things happen. The reference is stored on the memory's record for convenience, and a row is written to a relationship table:

<memory-id> | contextMemory | <table-id> | table | hasContextMemory

That row is the actual edge in the graph. Both directions are indexed, so:

Given a memory, find what it points to — one lookup.
Given a table (or dashboard, or pipeline), find every memory attached to it — one indexed query, no scans.

A memory can attach to almost anything in Collate:

Data assets — tables, databases, schemas, dashboards, charts, pipelines, stored procedures, topics, ML models, containers, search indexes, APIs, saved queries
People and groups — users, teams, bots, roles
Business context — domains, data products, glossary terms, tags
Measurements — metrics
Governance — policies, data contracts
Operational — incidents, test cases, workflows, conversation threads

Multiple memories can attach to one entity, and one memory can reference several entities.

How memories are retrieved

Collate exposes an MCP server at /mcp (that is https://<your-collate-host>/mcp). Any LLM or agent that speaks MCP can call its tools — no need to write SQL, SPARQL, or Elasticsearch queries. For memory, two tools do the work:

Structural lookup. "Give me every memory whose primaryEntity is prod.warehouse.orders" — or extend one hop to include memories on its columns, its upstream lineage, or its containing domain. This is a direct graph traversal against the indexed relationship table.

Hybrid semantic search. Pass a natural-language query with optional filters (asset FQN, memory type, domain). Vector similarity plus keyword ranking over the contextMemory index returns memories ranked by relevance. This catches lateral connections — a question about "order volume" can surface a memory attached to a metric definition rather than the orders table itself.

In practice, retrieval usually combines both: filter by attachment scope, then rank by semantic score.

Both go through role-based access control. If the caller can't see the underlying asset, the memory attached to it doesn't come back either. The same policy engine that guards tables and dashboards guards memories.

How memories are governed

Governance means ownership and approval, and memories inherit both from the base entity model.

Every memory has:

Owners — Users or Teams responsible for it. Edit and delete rights defer to them.
Reviewers — users who must approve proposed changes through the Tasks workflow.
Experts — designated subject-matter contacts (an owner is accountable; an expert is knowledgeable).
Certification — a stamp of Certified, Draft, or Deprecated.

Version numbers start at 0.1. Minor edits (like changing the answer text) bump the minor digit; backward-incompatible changes bump the major. Every change writes a diff to changeDescription and emits a change_event. The audit trail is automatic.

The typical lifecycle: a memory is created as Draft, a Task is opened for the owner, the owner reviews the diff, approves or rejects. On approval, certification flips to Certified. Later, when a memory is superseded, it flips to Deprecated and its version history preserves what it used to say.

What makes this architecture distinctive

Collate's memory isn't a demo feature. It runs at every level of the platform:

Memories span the entire data landscape — anything in metadata can carry one.
Every way of using Collate — UI, AskCollate, agents, SDK, external LLMs via MCP — can capture, store, and retrieve memories.
The amount of governance is tunable, from open capture to strict human-in-the-loop review.
Access is controlled just like any other entity.
The repository is designed to be managed and to scale, not to become a mess.

With this memory primitive, enterprises start to build the local learning loop that Nadella calls out as vital for AI success. With enterprise memory, the more you use your data, the smarter everyone gets about it.

Learn more about how Collate Context Center and memory help you maintain governed knowledge for your team and agents

How We Built Enterprise AI Memory into Collate