Ingestion + fidelity

POST /v1/documents/ingest takes an existing .docx, .pptx, or .xlsx and returns a ScaiScribe spec representing it. You can then PATCH the spec, re-render, and produce a new binary. ScaiScribe is honest about what survives this round-trip: every ingest result carries a structured fidelity envelope describing what was lossy.

What the response looks like#

json
{
  "doc_id": "doc_01HXYZ...",
  "spec": {
    "type": "docx",
    "theme": "office",
    "body": [ ... ],
    "fidelity": {
      "score": 0.85,
      "warnings": [
        { "code": "INGEST_PANDOC_FOOTNOTE_DETECTED", "message": "...", "count": 3 },
        { "code": "INGEST_IMAGE_DROPPED", "message": "..." }
      ]
    }
  }
}

score ∈ [0, 1]. 1.0 = byte-perfect round-trip. Lower scores reflect detected loss — the lower it is, the more the document was structurally simplified to fit the spec model.
warnings — a list of structured records. Each carries a code (machine-readable), a message (operator-readable), and often a count or field pointing at what's affected.

You should surface this to end users when fidelity matters. ScaiScribe never silently drops content; anything that didn't translate becomes a warning, not a silent omission.

Format-specific paths#

DOCX#

Engine: mammoth (primary) + pandoc (audit, when available).

Mammoth converts the DOCX to semantic HTML. The walker maps the HTML to spec elements:

Headings → heading (with level)
Paragraphs → paragraph (markdown)
Lists (<ol>/<ul>) → list
Tables → table
Images → image (or warning if asset extraction isn't configured)

When pandoc is installed on the worker host, it runs in parallel and walks the resulting AST for things mammoth doesn't surface cleanly:

Warning code	Trigger
`INGEST_PANDOC_FOOTNOTE_DETECTED`	Footnotes — surfaced as inline text by mammoth; flagged with count
`INGEST_PANDOC_MATH_DETECTED`	Math equations — degrade to plain text in mammoth path
`INGEST_PANDOC_CITATION_DETECTED`	Citation references
`INGEST_PANDOC_DEFINITION_LIST_DETECTED`	Definition lists — render as alternating paragraphs
`INGEST_PANDOC_LINE_BLOCK_DETECTED`	Line blocks (poetry, addresses) — line breaks lost

The body is mammoth's; pandoc is purely an honest audit. Without pandoc, ingest proceeds mammoth-only and these warnings just don't show.

PPTX#

Engine: JSZip + manual OOXML walk (no general-purpose library at the quality of mammoth for PPTX).

Extracted:

Slide structure (count, order, layout — matched against the theme's master registry by walking each slide's _rels → slideLayoutN.xml)
Text frames (titles, body, bullet lists)
Tables and basic shapes
Images (with positions)
Speaker notes

Not extracted (each surfaces as a fidelity warning):

SmartArt (preserved as raw_ooxml escape)
Complex shape compositions
Animations / transitions (dropped)
Embedded video/audio (retained as asset references)

Per-slide fidelity scores are also surfaced — useful when one slide is the problem and the rest are clean.

XLSX#

Engine: exceljs read mode.

High-fidelity for: cell values, formulas, basic formatting, named ranges, defined tables.

Lossy for (each surfaces a warning):

Macros — preserved verbatim in raw_ooxml; ScaiScribe doesn't execute or modify VBA
Pivot tables — preserved as raw_ooxml
Advanced conditional formatting beyond the theme repertoire
External data connections — dropped

Documents ScaiScribe rendered carry an embedded JSON breadcrumb at scaiscribe/breadcrumbs.json inside the zip. When the ingester sees this, it skips the structural reconstruction entirely and uses the breadcrumb verbatim — giving you full round-trip equivalence with fidelity_score = 1.0.

This means:

ScaiScribe → render → ingest → ScaiScribe = lossless. Use it whenever you have a document the system itself produced.
Externally-authored docs → ingest = best-effort with honest fidelity reporting. The mammoth/pandoc/exceljs paths above kick in.

The breadcrumb is a small zip part (typically <10 KB) registered under application/json content type. Word, LibreOffice, and Excel all tolerate the extra part without complaint.

For ScaiScribe-authored DOCX, there's also a slow-path recovery for code_block, callout, and page_break elements via mammoth's styleMap — even if the JSON breadcrumb is stripped, the OOXML style names (Code Block, Callout Info, etc.) still encode enough to reconstruct these element types. Chart and TOC don't have slow-path recovery; they need the breadcrumb.

Common warning codes#

Code	What happened	What to do
`INGEST_IMAGE_DROPPED`	An image was encountered but asset extraction isn't wired up yet.	The text around it survives. Re-add via `scribe_add_image` if needed.
`INGEST_CHART_REDUCED_TO_IMAGE`	A PPTX chart couldn't be parsed structurally; surfaces as an image element.	Spec the chart from scratch via `scribe_add_chart` if you need editable.
`PPTX_INGEST_LAYOUT_UNMATCHED`	A slide's layout couldn't be matched to a theme master.	Slide defaults to `"content"`; set the right layout via `scribe_set_slide_layout`.
`UNMAPPED_STYLE`	A custom Word style wasn't in the theme.	The text survives; styling is approximate. Add the style to the theme or accept the loss.
`INGEST_PANDOC_FOOTNOTE_DETECTED` (and other `INGEST_PANDOC_*`)	Pandoc spotted structure mammoth dropped.	Informational. The visible text survives in the body.

The full list lives in Troubleshooting.

Decision tree: should I trust this round-trip?#

scdoc

Is the document from ScaiScribe?
├── Yes — has the breadcrumb. Round-trip is lossless. Done.
└── No  — external document.
    ├── fidelity_score >= 0.9 and no critical warnings?
    │   → Edit and re-render with confidence. Re-check the warning list on output.
    ├── fidelity_score 0.6–0.9?
    │   → Inspect warnings; structural fidelity may not survive non-trivial edits.
    └── fidelity_score < 0.6?
        → Treat as a content extract, not a round-trip. Consider authoring from scratch.

Where to next#

Tutorials → Ingest and edit — concrete worked example.
Charts — chart-specific round-trip behaviour.
Troubleshooting — full warning code catalogue.