--- summary: "Vector + graph knowledge engine \u2014 collections of indexed documents,\ \ hybrid search, automatic graph extraction, NTFS-style per-resource ACLs." title: ScaiMatrix path: overview status: published --- # ScaiMatrix ScaiMatrix is the knowledge engine inside ScaiGrid. You group documents into collections, ingest them by upload or by crawling a site, and query them through vector search, graph traversal, or a hybrid of both. Every result is gated by an NTFS-style ACL so users never see chunks of documents they aren't allowed to read. It is built on top of ScaiGrid's inference, identity, and accounting layers. Embedding calls go through the same dispatch path as any other inference; permissions reuse ScaiKey users and groups; usage is metered against the tenant's budget. ## When to use it - You want semantic search over documents — PDFs, Markdown, HTML, plain text, source code. - You want a knowledge graph extracted from those documents — entities, relationships, multi-hop traversal. - You want recurring crawls of a website into a searchable corpus. - You need per-document permissions: a single collection that ten teams share but only some can see specific files. - You're building a RAG product (ScaiBot, ScaiPersona, your own application) and need the indexing layer underneath. If you only need raw embeddings, you can call `/v1/inference/embeddings` directly and persist them yourself — ScaiMatrix is for the case where you want storage, retrieval, and permissions managed for you. ## What you get - **Collections.** Named containers with their own embedding model, chunking strategy, and graph settings. Per-tenant slugged ids; forking is a first-class operation for embedding-model migrations. - **Document ingestion.** Upload, ingest-from-URL, ingest-from-ScaiDrive, or crawl a whole site. Async pipeline: extract -> chunk -> embed -> (optional) graph-extract. Each document has a live status you can poll. - **Hybrid search.** Vector, BM25 keyword, and combined modes; per-collection or global across everything you can read. Combined search blends a vector hit set with a graph traversal seeded from the same query. - **Knowledge graph.** Auto-extracted nodes and edges, manual CRUD, natural-language graph queries (Cypher generation + mutation guard), shortest-path search, BFS traversal, live SSE event stream, NDJSON export and bulk import. - **NTFS-style ACLs (v2).** Per-collection and per-document allow/deny ACEs, parent inheritance with per-resource break, transitive groups, owner concept, single-chokepoint search filtering. Mirrors ScaiDrive's permission shape one-for-one. - **Saved graph views.** Stash filter + tool + layout combinations as private or tenant-shared views. - **Crawl configs.** Schedule recurring crawls (hourly / daily / weekly / cron) or expose a webhook-triggered endpoint with HMAC verification. - **Re-chunk and re-extract.** In-place async jobs with progress counters for when chunking parameters or the graph extraction model change. - **Quotas.** Per-collection `storage_quota_bytes`, `graph_max_nodes`, `graph_max_edges` keep runaway extraction from filling the store. ## Two-minute mental model You manage four nouns and three verbs. - A **Collection** is a searchable namespace. It owns its embedding model, chunking strategy, graph settings, and a default-access policy. A tenant owns many collections. - A **Document** is a source inside a collection — it gets chunked, embedded, and optionally graph-extracted. Every document has its own ACL (v2). - A **Chunk** is the embedding-sized fragment that backs vector search. Chunks inherit their document's ACL — there's no separate chunk-level permission. - A **Node / Edge** is an extracted entity or relationship in the collection's knowledge graph. Each node carries the document ids that contributed to it. The verbs: - You **ingest** (upload / from-URL / from-ScaiDrive / crawl) -> the pipeline extracts text, chunks, embeds, and (optionally) extracts graph entries. - You **search** (vector / hybrid / graph / combined) -> ACL-gated results come back, never more than the caller is allowed to see. - You **grant** (ACEs on collections or documents) -> the right people see the right things, with NTFS-style deny rules for carve-outs. The admin UI surfaces all of this as Knowledge Dashboard, Collection Manager, Document Browser, Search Playground, Graph Explorer, Ingestion Monitor, and Crawl Manager. ## How it relates to other ScaiGrid modules - **ScaiBot** can use a ScaiMatrix collection as its knowledge base in `linked` mode — one collection powering many bots. - **ScaiPersona** uses ScaiMatrix for RAG retrieval behind a persona's responses. - **ScaiDrive** is the file store; ingestion can pull directly from a ScaiDrive file id, and the ACL model is intentionally mirrored so an operator who's configured one knows the other. - **ScaiInfer** runs the embedding model; ScaiMatrix calls it through the standard inference dispatch path so every embedding call is metered against your tenant budget. - **ScaiKey** owns user and group identity; ACEs name ScaiKey principals directly, and a nested-group mirror keeps transitive memberships in sync. ## When to call inference directly instead If you only need raw embeddings — say, you've already got your own vector store, or you're computing similarity in a pipeline that lives outside ScaiGrid — call `/v1/inference/embeddings` and persist the vectors yourself. ScaiMatrix is for the case where you want the storage, retrieval, permissions, and admin surface managed for you. The cost difference is marginal at small scale; the operational difference is large. ## What ScaiMatrix is not - **Not a primary file store.** Binaries live in ScaiDrive (or S3 under the hood); ScaiMatrix stores chunks, embeddings, and graph contents derived from them. - **Not a general-purpose graph database.** The graph surface is shaped for knowledge extracted from documents — labelled entities and labelled relationships with provenance. If you need a free-form property graph for non-document data, run Neo4j directly. - **Not a chat product.** Use ScaiBot or ScaiPersona on top of ScaiMatrix for that; ScaiMatrix is the indexing layer underneath. - **Not a structured-data warehouse.** Ingesting JSON / YAML works for retrieval but doesn't get structured query support — those are just text under the hood. ## Where to go next - [Quickstart](./quickstart) — collection, document, search, in five minutes. - [Architecture](./concepts/architecture) — how ingestion, storage, and search fit together. - [ACLs](./concepts/acls) — the v2 permission model in depth. - [Build a searchable knowledge base](./tutorials/build-a-knowledge-base) — end-to-end recipe. - [Crawl a docs site on a schedule](./tutorials/crawl-on-a-schedule) — recurring ingestion. - [API reference](./reference/api) — every endpoint. - [Permissions](./reference/permissions) — module permissions vs per-resource ACLs. ScaiMatrix's module ID inside ScaiGrid is `scaimatrix`; its API is mounted at `/v1/modules/scaimatrix/`.