Knowledge¶

Knowledge is what an agent can know: an organisation's documents and data, stored so that a model can search them during a turn. Knowledge is a layer with several parts. Per use you choose how documents are processed, what they become searchable in and how an agent searches.

The parts, in the order they work¶

flowchart LR
    ext["External system<br/>website, SharePoint, Google Drive, S3, Zendesk"] -.->|"syncs"| col
    col["Collection<br/>where sources live"] --> ing["Ingest pipeline<br/>how a source becomes searchable"]
    ing --> idx["Index<br/>search by meaning and keyword"]
    ing --> kg["Graph<br/>search by entities and relations"]
    ing --> ts["Table set<br/>query tables by field"]
    idx --> tool["Search tool<br/>what the agent calls"]
    kg --> tool
    ts --> tool
    col --> tool
    tool --> agent["Agent"]

One collection can feed several indexes, graphs and table sets, each with its own pipeline. The documents are stored once; only the processing differs.

In a turn a model reaches knowledge through a tool. Usually that is a search tool: the model chooses the question, and which sources the tool reaches is yours to fix. When you offer the core__search_index tool, the model searches an index directly (MCP). Your application can also search an index itself through the API, outside a turn.

When to use this: an index, a graph or a table set¶

Start with an index. It works on any document without preparation. Add a table set when your documents hold tables -- spreadsheets, exports, price lists -- and questions are about amounts, counts or rows that meet a condition: an index finds the passage, a table set does the sum. Reading tables costs no model tokens. Add a graph when questions are about connections spread over many documents. A graph asks more of you: you write a schema, and reading facts out of text costs model tokens per chunk.

A table set is not a datastore. Core fills a table set from your documents and replaces a table when its document changes; your application or an agent writes the records of a datastore itself.

External sources¶

A collection can have a connected source: a website, a SharePoint site, a folder in Google Drive, an S3 bucket or Zendesk. Core syncs the documents into the collection. Each document becomes an ordinary source, so indexes, graphs, table sets and search tools work with it exactly as with a source you added yourself.

Room for more methods¶

The layer is laid out so that storage and search method stand apart. A pipeline is made of steps from a catalogue, and a search tool can combine several sources and search steps. Which steps and templates your installation offers is shown in the Templates segment of Knowledge and by GET /knowledge/pipeline-components and GET /knowledge/pipeline-templates. This guide describes what exists now: indexes, graphs, table sets and the search steps over them.

What is not knowledge: datastores and memory¶

Two parts resemble knowledge and stand apart from it, in the group Data and memory:

• A datastore holds structured records an agent reads and writes: a customer list, the settings of an integration. You look a record up by its key or filter the records on their fields, not with a search question (Datastores).
• Memory holds single facts that persist across calls, per person or shared: a customer's preference, an earlier decision (Memory).

The difference in one sentence: an agent searches knowledge, in documents the organisation supplies; a datastore and memory are data an agent also updates itself while it works.

Who reaches what¶

• The knowledge capability switches knowledge on for a key.
• The policy decides which collections, indexes, graphs and table sets a key reaches, by name or by label, and which sources a person finds inside an index. Inside a table set a person sees a table exactly when she may see the source it came from.
• Calling a search tool asks two things of the policy: a rule about Tools that permits the search tool, and rules that permit the index, graph, table set or collection behind it. Core asks both on every call. When one is missing, the tool does not exist for that key.

How it fits together¶

• An agent gets knowledge by including a search tool in its tools.
• A search tool is a tool like any other: it is called knowledge__<name>, fits in a toolset and falls under the rules about tools.
• Collections, pipelines, indexes and search tools can go into a publication, so a customer gets the same knowledge setup.
• Ingestion and search are metered as usage (Insight).

Read on¶

Tutorial: Give an agent knowledge from your own business information.