---
audience: developers
summary: Reading binary documents into the spec model, with honest reporting about
  what survived.
title: Ingestion + fidelity
path: concepts/ingestion
status: published
---


# Ingestion + fidelity

`POST /v1/documents/ingest` takes an existing `.docx`, `.pptx`, or `.xlsx` and returns a ScaiScribe spec representing it. You can then PATCH the spec, re-render, and produce a new binary. ScaiScribe is honest about what survives this round-trip: every ingest result carries a structured `fidelity` envelope describing what was lossy.

## What the response looks like

```json
{
  "doc_id": "doc_01HXYZ...",
  "spec": {
    "type": "docx",
    "theme": "office",
    "body": [ ... ],
    "fidelity": {
      "score": 0.85,
      "warnings": [
        { "code": "INGEST_PANDOC_FOOTNOTE_DETECTED", "message": "...", "count": 3 },
        { "code": "INGEST_IMAGE_DROPPED", "message": "..." }
      ]
    }
  }
}
```

- **`score`** ∈ [0, 1]. 1.0 = byte-perfect round-trip. Lower scores reflect detected loss — the lower it is, the more the document was structurally simplified to fit the spec model.
- **`warnings`** — a list of structured records. Each carries a `code` (machine-readable), a `message` (operator-readable), and often a `count` or `field` pointing at what's affected.

You should surface this to end users when fidelity matters. ScaiScribe never silently drops content; anything that didn't translate becomes a warning, not a silent omission.

## Format-specific paths

### DOCX

Engine: **mammoth** (primary) + **pandoc** (audit, when available).

Mammoth converts the DOCX to semantic HTML. The walker maps the HTML to spec elements:
- Headings → `heading` (with level)
- Paragraphs → `paragraph` (markdown)
- Lists (`<ol>`/`<ul>`) → `list`
- Tables → `table`
- Images → `image` (or warning if asset extraction isn't configured)

When pandoc is installed on the worker host, it runs in parallel and walks the resulting AST for things mammoth doesn't surface cleanly:

| Warning code | Trigger |
|---|---|
| `INGEST_PANDOC_FOOTNOTE_DETECTED` | Footnotes — surfaced as inline text by mammoth; flagged with count |
| `INGEST_PANDOC_MATH_DETECTED` | Math equations — degrade to plain text in mammoth path |
| `INGEST_PANDOC_CITATION_DETECTED` | Citation references |
| `INGEST_PANDOC_DEFINITION_LIST_DETECTED` | Definition lists — render as alternating paragraphs |
| `INGEST_PANDOC_LINE_BLOCK_DETECTED` | Line blocks (poetry, addresses) — line breaks lost |

The body is mammoth's; pandoc is purely an honest audit. Without pandoc, ingest proceeds mammoth-only and these warnings just don't show.

### PPTX

Engine: **JSZip + manual OOXML walk** (no general-purpose library at the quality of mammoth for PPTX).

Extracted:
- Slide structure (count, order, **layout** — matched against the theme's master registry by walking each slide's `_rels` → `slideLayoutN.xml`)
- Text frames (titles, body, bullet lists)
- Tables and basic shapes
- Images (with positions)
- Speaker notes

Not extracted (each surfaces as a fidelity warning):
- SmartArt (preserved as `raw_ooxml` escape)
- Complex shape compositions
- Animations / transitions (dropped)
- Embedded video/audio (retained as asset references)

Per-slide fidelity scores are also surfaced — useful when one slide is the problem and the rest are clean.

### XLSX

Engine: **exceljs** read mode.

High-fidelity for: cell values, formulas, basic formatting, named ranges, defined tables.

Lossy for (each surfaces a warning):
- Macros — preserved verbatim in `raw_ooxml`; ScaiScribe doesn't execute or modify VBA
- Pivot tables — preserved as `raw_ooxml`
- Advanced conditional formatting beyond the theme repertoire
- External data connections — dropped

## The §4.7 OOXML breadcrumb

Documents ScaiScribe rendered carry an embedded JSON breadcrumb at `scaiscribe/breadcrumbs.json` inside the zip. When the ingester sees this, it skips the structural reconstruction entirely and uses the breadcrumb verbatim — giving you **full round-trip equivalence** with `fidelity_score = 1.0`.

This means:

- ScaiScribe → render → ingest → ScaiScribe = lossless. Use it whenever you have a document the system itself produced.
- Externally-authored docs → ingest = best-effort with honest fidelity reporting. The mammoth/pandoc/exceljs paths above kick in.

The breadcrumb is a small zip part (typically <10 KB) registered under `application/json` content type. Word, LibreOffice, and Excel all tolerate the extra part without complaint.

For ScaiScribe-authored DOCX, there's also a slow-path recovery for `code_block`, `callout`, and `page_break` elements via mammoth's styleMap — even if the JSON breadcrumb is stripped, the OOXML style names (`Code Block`, `Callout Info`, etc.) still encode enough to reconstruct these element types. Chart and TOC don't have slow-path recovery; they need the breadcrumb.

## Common warning codes

| Code | What happened | What to do |
|---|---|---|
| `INGEST_IMAGE_DROPPED` | An image was encountered but asset extraction isn't wired up yet. | The text around it survives. Re-add via `scribe_add_image` if needed. |
| `INGEST_CHART_REDUCED_TO_IMAGE` | A PPTX chart couldn't be parsed structurally; surfaces as an image element. | Spec the chart from scratch via `scribe_add_chart` if you need editable. |
| `PPTX_INGEST_LAYOUT_UNMATCHED` | A slide's layout couldn't be matched to a theme master. | Slide defaults to `"content"`; set the right layout via `scribe_set_slide_layout`. |
| `UNMAPPED_STYLE` | A custom Word style wasn't in the theme. | The text survives; styling is approximate. Add the style to the theme or accept the loss. |
| `INGEST_PANDOC_FOOTNOTE_DETECTED` (and other `INGEST_PANDOC_*`) | Pandoc spotted structure mammoth dropped. | Informational. The visible text survives in the body. |

The full list lives in [Troubleshooting](/docs/scaiscribe/troubleshooting/getting-started).

## Decision tree: should I trust this round-trip?

```
Is the document from ScaiScribe?
├── Yes — has the breadcrumb. Round-trip is lossless. Done.
└── No  — external document.
    ├── fidelity_score >= 0.9 and no critical warnings?
    │   → Edit and re-render with confidence. Re-check the warning list on output.
    ├── fidelity_score 0.6–0.9?
    │   → Inspect warnings; structural fidelity may not survive non-trivial edits.
    └── fidelity_score < 0.6?
        → Treat as a content extract, not a round-trip. Consider authoring from scratch.
```

## Where to next

- [Tutorials → Ingest and edit](/docs/scaiscribe/tutorials/ingest-and-edit) — concrete worked example.
- [Charts](/docs/scaiscribe/concepts/charts) — chart-specific round-trip behaviour.
- [Troubleshooting](/docs/scaiscribe/troubleshooting/getting-started) — full warning code catalogue.