Turning messy docs into structured Markdown using Azure Document Intelligence + Pydantic + Jinja2

How many times have you received a document that's messy, unstructured, or just plain hard to work with? Now imagine you need a whole stack of them as source material for an AI app. That was my task: Word documents, some laid out as decision trees, some as plain narratives. Structure tells a model what matters and how the pieces connect, so the goal was to turn those documents into a clean, structured Markdown format. To get there, I built a pipeline where Azure Document Intelligence extracts the content, Pydantic enforces the structure, and Jinja2 renders the Markdown. I also used Claude from Anthropic for every step that needed an LLM, but you can swap in any LLM you prefer.

The pipeline

flowchart TD
    A[.docx file] --> B["1. Extract text, tables and layout from the document<br/>using Azure Document Intelligence."]
    B --> C["2. Classify the document shape<br/>(decision tree or plain narrative)"]
    C --> E["3. Map the extracted text onto a Pydantic model,<br/>prompted with the few-shot examples<br/>for that shape."]
    E --> F["4. Validate the Pydantic model<br/>deterministically to catch structural issues."]
    F -->|validation flags something| G["5. Give the LLM one chance<br/>to repair it."]
    F -->|no issues| H["6. Render the Pydantic model<br/>to Markdown using Jinja2."]
    G --> R{"Re-validate: fewer warnings<br/>and no content lost?"}
    R -->|yes: keep the repaired version| H
    R -->|no: keep the original<br/>and its warnings| H
    H --> I["Structured Markdown (final.md)<br/>+ warnings.txt for human review"]

The pipeline runs in six stages. The Azure Document Intelligence prebuilt-layout model extracts text, tables, and layout from the source document. A classifier then determines whether the document is a decision tree or a plain narrative, and that label decides which few-shot examples go into the mapping prompt. The LLM maps the extracted text onto a Pydantic model, which is then validated deterministically to catch structural issues. If it flags a problem, the LLM gets one chance to repair it. If the repair doesn't help, the original version is kept for a human to review; otherwise, the repaired version moves on. Finally, the Pydantic model is rendered to Markdown using Jinja2.

The idea behind the design is that no stage trusts the one before it. Mapping rebuilds the structure instead of assuming extraction kept it, validation checks the mapping without asking an LLM, and a repair is only kept if the same checks, run again, show fewer warnings and no lost content.

For this pipeline, a document can be one of two shapes:

• a decision tree or
• a plain narrative

A decision tree is a branching structure, where each step leads to one or more outcomes.

"who is calling? → is the identity confirmed? → go to step 4"

A plain narrative is linear text, read top to bottom, with no branching.

"This document has information for..."

The goal is simple: turn either shape into a structured Markdown file that can be easily read and processed.

Before we start: the sample documents for the demo

The pipeline needs documents to run on, so the first thing to build was a small set of samples. They are written as generic "customer support call scripts", but the domain is arbitrary. The same pattern applies just as well to onboarding SOPs, IT runbooks, compliance procedures — anywhere structured and unstructured documentation coexist.

For this demo, I created 3 synthetic tree documents and 3 synthetic narrative documents, designed to represent the two document shapes the pipeline is built to handle:

• tree documents follow a decision-tree structure
• narrative documents are plain, linear text, read top to bottom.

Those six are then split two ways, and the split matters. Four of them stay in synthetic_docs/ and become the few-shot examples the model is shown at mapping time. They don't reach the model as Word files. The script src/create_few_shot_examples/create_raw_examples.py first runs each one through the same Azure extraction the pipeline uses and saves the result to the few_shot_examples/ folder. There, I paired each output with its gold standard. The remaining two, one of each shape, are held out in sample_inputs/ as the documents the demo actually processes. Nothing the pipeline runs on has ever been shown to the model as an answer.

Feel free to swap in your own documents immediately — this step is just there so you can run the demo out of the box, without needing to supply anything yourself.

Step 1: Extracting data with Azure Document Intelligence

Azure Document Intelligence reads a document and returns its text along with the layout it detects: headings, paragraphs, tables, and the boxes of a drawing. To run the demo, you'll need an Azure Document Intelligence resource, plus its endpoint and key.

The code snippets in this article are trimmed to the essentials: imports and logging are left out, and docstrings are shortened. The full code is in this repository.

endpoint = os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"]
key = os.environ["AZURE_DOCUMENT_INTELLIGENCE_KEY"]

Note: I built this whole demo on the free tier (F0), which gives you 500 pages a month at no cost. The F0 only reads the first two pages of each document, so if your file is longer, the rest is simply dropped. The sample documents are short, so it did not matter here, but longer files would require a paid tier. For details, see the Azure Document Intelligence pricing page.

In this demo, I am going to use two documents, one in a decision tree shape and one in a plain narrative shape. You can find these documents in the sample_inputs/ folder.

In order to extract the text from these documents, I will use the prebuilt-layout model of Azure Document Intelligence. It handles a wide range of layouts, and it can return its output as Markdown. That matters, because the layout it detects is encoded right in the text: section titles come back as # or ## headings instead of plain lines, and tables come back as pipe tables instead of a loose run of cell values. The next steps can then work with that structure instead of rebuilding it from scratch. If you want to go deeper on what the model can do, the Azure Document Intelligence documentation is a good next stop.

def extract_text(source_path: str | Path) -> str:
    """
    Call Azure AI Document Intelligence (prebuilt-layout) on a real file.
    """

    client = DocumentIntelligenceClient(
        endpoint=endpoint, credential=AzureKeyCredential(key)
    )

    with open(source_path, "rb") as source_file:
        poller = client.begin_analyze_document(
            "prebuilt-layout",
            body=source_file,
            # output the extracted text in Markdown format
            output_content_format=DocumentContentFormat.MARKDOWN,
        )
    result = poller.result()
    return result.content

Step 2: Classifying the document shape

The next step is the classification of the extracted text: is the document a decision tree or a plain narrative? The label determines which set of few-shot examples gets loaded into the LLM's prompt for the mapping step, and showing the model examples of the right shape is what helps it understand the document's structure and content.

Note: A Vision Language Model (VLM) could do this classification instead, since it sees what text extraction loses, like arrow direction and spatial layout. But a VLM costs more and needs an extra step to render the document as an image, so this demo classifies from the extracted text alone. If you'd rather use a VLM, feel free to swap in your own classifier here.

def classify(raw_text: str) -> str:
    """
    Classify a document as "tree" or "narrative".
    """
    client = get_client()  # Anthropic client
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=16,  # Keep it short, we only need a label.
        thinking={"type": "disabled"},
        system=CLASSIFY_PROMPT,  # Tells the model how to classify.
        messages=[{"role": "user", "content": raw_text}],
    )

    # _first_text extracts the first text block from the response, which should contain the label.
    label = _first_text(response).strip().strip(".").lower()
    print(f"Label is {label!r}")
    if label not in VALID_LABELS:
        raise ValueError(f"Unexpected classification label {label!r}.")
    return label

Few-shot examples live in the few_shot_examples/ folder, organized one subfolder per category. Within each category, examples are paired by filename:

• a <name>_raw.txt file holding the example document,
• matched with a <name>_expected.json file holding what that document should map to.

Drop a new <name>_raw.txt / <name>_expected.json pair into a category folder, and the pipeline picks it up automatically. Just make sure every raw file has its matching expected file.

You might be wondering whether classifying first is really necessary, rather than just showing the model every few-shot example regardless of shape. Honestly, with only one or two examples per category, it isn't, because cost and latency won't be an issue either way. It's really up to you and what your project needs. That said, I'd still lean toward including it, since it means you won't have to rewrite the pipeline later if things scale up.

Step 3: Mapping onto a Pydantic model

Defining the data model

Pydantic is a data validation library for Python: you define your data as typed models, and it makes sure whatever comes in actually matches that shape. That's exactly what we need here, a way to catch inconsistencies or errors in the LLM's output early, before they ripple further down the pipeline.

Rather than two separate schemas, one for trees and one for narratives, I used a single CallScript model: the fields both shapes share, plus steps and outcomes for trees and main_text for narratives. A single validator then enforces the rule that ties it together: exactly one of steps or main_text is filled in, never both and never neither.

Why use Pydantic at all, when the LLM only needs a JSON input_schema? Function calling features in most LLMs already let you define a schema and get structured output, so why not just write that schema by hand? Because here the schema comes from Pydantic: the input_schema handed to the LLM is generated with CallScript.model_json_schema(). One class is both the schema the LLM sees and the validator that checks its answer, so the two can never drift apart. On top of that, Pydantic can enforce rules a JSON schema can't express, like the tree-or-narrative rule, which runs as a Python validator once the response arrives.

Every script has a few shared fields, plus a few that depend on its shape:

class CallScript(BaseModel):
    """
    A CallScript describes either a decision-tree call flow or a
    plain-narrative procedure, never both.
    """

    # common fields
    title: str = Field(..., description="The name of the procedure.")
    category: str | None = Field(
        default=None,
        description="High-level grouping, e.g. 'Billing', 'Technical Support'.",
    )
    attention_points: list[AttentionPoint] = Field(
        default_factory=list,
        description=(
            "Warnings or caveats the agent must not miss, preserved in the "
            "order and nesting they appeared in the source document."
        ),
    )

    # Decision-tree shape
    steps: list[DecisionStep] = Field(default_factory=list)
    outcomes: list[Outcome] = Field(default_factory=list)

    # Plain-narrative shape
    main_text: str | None = Field(
        default=None,
        description="Free-form procedure text, used for non-tree documents.",
    )

    @model_validator(mode="after")
    def check_single_shape(self) -> "CallScript":
        """
        Enforce that a document is a tree OR a narrative, never both and never neither.
        """
        is_tree = bool(self.steps)
        is_narrative = bool(self.main_text)

        if is_tree and is_narrative:
            raise ValueError(
                "A CallScript cannot have both `steps` and `main_text` set — "
                "pick one shape."
            )
        if not is_tree and not is_narrative:
            raise ValueError(
                "A CallScript must have either `steps` or `main_text` set."
            )
        return self

Mapping the extracted text onto the model

With the text extracted and the shape classified, the next step is mapping that text onto the CallScript model. The LLM has no built-in knowledge of the model or its fields, so the request has to spell everything out. It includes:

• Instructions (the system prompt) describing the model's structure and how to map the text onto it
• Few-shot examples for the classified shape, passed as earlier conversation turns, so the model can see how similar documents were mapped before
• The extracted text of the document itself

The demo uses Claude, but any LLM that supports structured output will work. The output is then validated and turned into a CallScript object.

All the prompts used in the pipeline live in src/prompts/ folder. Keep in mind that they're built for this demo, not as a general-purpose template. The mapping prompt starts with the basics: never invent information and never drop it. Then it covers the trickier edge cases:

• Pick exactly one shape, tree or narrative. Never both, never neither.
• Make every branch point to a step or outcome that actually exists.
• Treat outcomes as endpoints. They never route anywhere else.
• Keep every sentence of narrative content, without summarizing.

def map_to_schema(raw_text: str, category: str) -> CallScript:
    """
    Map raw text into a validated CallScript.
    """
    few_shot = load_few_shot_examples(category)

    messages = []
    for example_raw, example_expected in few_shot:
        # Each example becomes a user turn (its extracted text) and an
        # assistant turn (its hand-written CallScript), as a worked answer.
        messages.append({"role": "user", "content": example_raw})
        messages.append({"role": "assistant", "content": example_expected})
    # The document to map goes last: the user turn the model must answer.
    messages.append({"role": "user", "content": raw_text})

    # Forced tool use: Claude must return a CallScript, which is then validated.
    # An incomplete reply raises an error.
    return request_call_script(
        system=AGENT_PROMPT,
        messages=messages,
        tool_description="Record the CallScript structure extracted from the source text.",
    )

Note: Instead of forced tool use, you could use Claude's native structured outputs, which guarantee the response matches your schema.

Step 4: Validating the Pydantic model

LLMs can hallucinate or misread the source text, so trusting the mapping step's output blindly is risky. Instead, I added a deterministic validation step that checks the CallScript object before it reaches the rendered Markdown.

Each check is its own function, and which ones apply depends on the document's shape:

Note: Three of these checks compare wording, and all three use the same rule: two pieces of text match when at least 80% of the distinct words in one are found in the other. What differs is the direction:

• Narrative coverage and Attention-point coverage start from the source. For each source sentence or attention-point line, they ask: did it make it into the output?
• Ungrounded nodes starts from the output. For each step and outcome, it asks: does it come from a line in the source?

80% is loose enough to let the model reflow a paragraph or fix a typo, and strict enough to catch a summarized or dropped sentence.

The last check, Shape conflicts, is a safety net: the Pydantic model already rules them out, but a CallScript built with model_construct() skips validation, so it can still reach this step if someone builds one by hand.

def sanity_check(script: CallScript, raw_text: str) -> list[str]:
    """
    Run all structural checks and return a list of warning strings.
    """
    warnings: list[str] = []

    # Narrative-only checks
    if script.main_text:
        warnings.extend(_check_narrative_coverage(script, raw_text))
        warnings.extend(_check_narrative_size(script, raw_text))

    # Tree checks (no-ops on narratives, which have no steps or outcomes)
    warnings.extend(_check_dangling_references(script))
    warnings.extend(_check_unreachable_nodes_and_outcomes(script))
    warnings.extend(_check_grounding(script, raw_text))

    # Shape-agnostic checks
    warnings.extend(_check_attention_coverage(script, raw_text))

    # Shape-agnostic: this is a safety net for the case where a
    # CallScript bypasses the Pydantic validator
    warnings.extend(_check_shape_conflict(script))

    return warnings

The first two tree checks only look at the graph, so they can't tell a real box from one the model made up. If a branch points at an invented outcome, the reference resolves and the outcome is reachable, so the graph looks perfect. The _check_grounding goes back to the source: it compares every step and outcome against the extracted text, line by line. An invented box won't match any line, so it gets flagged.

Narratives have the same blind spot. _check_narrative_coverage walks the source and asks whether each sentence made it into main_text, so a sentence that exists only in the output is never examined, and its extra words can even help other sentences reach the threshold. The _check_narrative_size closes that gap with a simple count: main_text restates the source body word for word, so its size should never be larger than the body's.

Attention points have a blind spot too. Narrative coverage stops at the "Attention points" heading, and grounding only looks at steps and outcomes, so a dropped caveat would pass every check, even though caveats are exactly what the agent must not miss. The attention-point coverage check applies the same 80% threshold to every line under that heading, matching each one against a single attention point, nested ones included.

Finally, none of these checks stop the pipeline. Instead of raising an error, each one adds its warnings to a single list, and that list decides what happens next: if it's empty, the script goes straight to rendering; if not, the warnings go to the repair step, so the model knows what to fix.

Note: A script can pass every check and still be wrong. The validation step is not a guarantee of correctness, only a way to catch common structural issues that are easy to check for deterministically. It is still possible for the model to misinterpret the source text or make other mistakes that the checks don't catch. The goal is to catch the low-hanging fruit, so human review can focus on what the checks can't see.

Step 5: Repairing the Pydantic model

If validation flags anything, the LLM gets exactly one attempt to repair it: it's shown the warnings, along with the current object and the source text, and asked to fix them. I capped it at one try on purpose. Without a limit, the model could keep "fixing" the same problem in a loop, burning time and tokens without ever getting it right.

Here's the part worth calling out, because it's easy to get wrong: checking that the warning count went down isn't enough. Say the model gets a "dangling reference" warning. The easiest way to make it disappear is to simply delete the branch that causes it. The warning is gone, but the script has lost content, so it's actually worse than before.

That's why a repair is only accepted if it passes two checks:

1. It reduces the number of warnings.
2. It doesn't lose any content. The CallScript model's content_size property adds up every step, outcome, and branch in a tree, every word in a narrative, and every attention point in both, nested ones included. The pipeline compares it before and after the repair. If the number drops, something was deleted, and the repair is rejected.

@property
def content_size(self) -> int:
    """
    Rough size metric used to detect a repair that deletes content.
    """
    branch_count = sum(len(step.branches) for step in self.steps)
    narrative_words = len(self.main_text.split()) if self.main_text else 0
    attention_count = len(flatten_attention_points(self.attention_points))
    return (
        len(self.steps)
        + len(self.outcomes)
        + branch_count
        + narrative_words
        + attention_count
    )

Note: At the end of the repair step, the remaining warnings are stored in a warnings.txt file in the outputs/ folder alongside the final Markdown output. The file only covers what's left after repair: if a repair was accepted, the warnings it fixed aren't recorded anywhere. For a full history of what was flagged and fixed, you'd need to add your own logging around attempt_repair's return values.

Putting it together, the repair step sends the source text, the mapped CallScript and its warnings back to the LLM, validates the reply into a new CallScript, and runs the checks on it again. If it passes both checks above, it replaces the original; otherwise, the original is kept. Either way, the pipeline moves on to rendering, and any warnings that remain end up in warnings.txt.

def attempt_repair(
    script: CallScript, warnings: list[str], raw_text: str
) -> tuple[CallScript, list[str], bool]:
    """
    Attempt exactly one repair call and apply the acceptance gate.
    """
    if not warnings:
        return script, warnings, False
    # Repair the script and re-validate it.
    candidate = llm_repair(script, warnings, raw_text)
    candidate_warnings = sanity_check(candidate, raw_text)
    # check that the repair reduced the number of warnings and didn't lose content
    warnings_improved = len(candidate_warnings) < len(warnings)
    size_preserved = candidate.content_size >= script.content_size

    if warnings_improved and size_preserved:
        return candidate, candidate_warnings, True

    return script, warnings, False

Note: Why does the repair step need the original text? Some warnings can be fixed from the object alone: if a branch points to a step or outcome that doesn't exist, the model can see that and repair it on the spot. But if the model dropped a paragraph of narrative text, nothing in the object shows what went missing. Only the source does, which is why the repair prompt includes it.

You'll see a repair at work on a real document in the decision tree example below.

Step 6: Rendering the Pydantic model to Markdown with Jinja2

The last step turns the CallScript model into Markdown with Jinja2, a Python templating engine that lets you define exactly how each field of the model is written out — layout, formatting, structure, all of it. The same template handles both decision trees and plain narratives, so there's no need to maintain two versions. Rendered this way, the output ends up structured underneath and easy to read on top.

The template for this demo outputs the title, category and attention points unconditionally. Even when a section is empty, it still renders with a placeholder like "None." rather than being skipped. That's deliberate: an empty section is a recorded fact ("this document has no attention points"), while an omitted one would be indistinguishable from something the mapping step simply missed.

From there, the template has two shape-specific sections:

• when script.steps is set, it outputs the call flow and outcomes (the tree shape);
• when script.main_text is set, it outputs the narrative text.

The template checks the two conditions separately rather than as an if/else. The schema already guarantees exactly one is set, so this only matters if a CallScript skips validation, for example one built with model_construct(): the template still renders whatever it's given instead of breaking.

You can find the template in templates/call_script.md.j2.

def render_markdown(script: CallScript) -> str:
    """Render a CallScript into final Markdown using the Jinja2 template."""
    jinja_env = Environment(
        loader=FileSystemLoader(TEMPLATES_DIR),
        trim_blocks=True,
        lstrip_blocks=True,
    )
    template = jinja_env.get_template("call_script.md.j2")
    rendered = template.render(script=script)
    rendered = re.sub(r"\n{3,}", "\n\n", rendered)
    return rendered.strip() + "\n"

Outputs

In the outputs/ folder, each document gets its own subfolder, named after the document's path under sample_inputs/ without the extension. Inside that subfolder, you'll find:

• final.md — the rendered Markdown output
• raw.txt — the raw text extracted from the source document using Azure Document Intelligence
• warnings.txt — the warnings that survived repair, or No warnings - all checks passed. if the script passed every check.

If a document fails, its folder holds only an error.txt with the reason.

Narrative example

The narrative case is the simpler of the two, because the source is already linear text. The pipeline still extracts, maps, validates, and renders it, but the mapping stage doesn't have to reconstruct a graph from labels alone.

What it does instead is decompose the document. sample_inputs/complaints_handling_overview.docx arrives from the extractor as one undifferentiated block of text, and every line of it has to be routed to the field it belongs to:

Complaints Handling Overview                       →  title
Category: Complaints Process                       →  category  ("Complaints Process")

This document explains how customer complaints…    →  main_text
A complaint is any expression of dissatisfaction…      (the body, in full)
Every complaint is recorded on the day…
…

Attention points                                   →  attention_points
Log any expression of dissatisfaction…                 (as a nested list)
Record the following on the day…:
The customer's own wording, not a paraphrase.
…

Here's what the rendered Markdown looks like, from outputs/complaints_handling_overview/final.md:

# Complaints Handling Overview

**Category:** Complaints Process

## Attention points

- Log any expression of dissatisfaction, whether or not the customer calls it a complaint.
…

## Main Text

This document explains how customer complaints are recorded, handled, and closed. It describes the principles consultants work to rather than a fixed sequence of steps.

…

Every line of the source lands in exactly one place.
Decomposing a document is exactly the operation where content goes quietly missing — a line that belongs to no field is a line that gets dropped — so this is the property worth checking, and it's checkable rather than a matter of impression:

• all 17 sentences of the body appear in main_text at 100% word overlap, so the text was carried across intact rather than paraphrased. The 80% threshold is there to let the model reflow a paragraph without that counting as data loss, but on this document it never had to.
• all 10 attention-point lines are carried through verbatim, wording untouched;
• the title and category are captured as their own fields, so they can be indexed and grouped without parsing prose.

To verify this yourself, compare raw.txt with final.md in outputs/complaints_handling_overview/: every line of the source appears in the output, in the section it belongs to. To check extraction too, open the original sample_inputs/complaints_handling_overview.docx next to final.md.

Decision tree example

This is the case worth looking at closely, because the pipeline has to recover something that extraction destroyed.

The source is sample_inputs/password_reset_procedure.docx: a flowchart drawn in Word, with decision diamonds, terminal boxes, and labelled arrows connecting them. Here's what Azure Document Intelligence returns for it, annotated with what the mapping stage turned each line into:

# Password Reset Procedure                                     → title
Category: Reset Process                                        → category: "Reset Process"
This document describes the decision process...                → (boilerplate, not stored)
Customer requests password reset                               → (the flowchart's start box, not stored)

## Identity verified via standard two-way authenticator?       → STEP_1 (question)
## No                                                          → STEP_1 branch "no"
## Yes                                                         → STEP_1 branch "yes"
## Reject the request                                          → OUTCOME_1
## Suspicious activity detected on the account?                → STEP_2 (question)
## No                                                          → STEP_2 branch "no"
## Yes                                                         → STEP_2 branch "yes"
## Inform the Security Team                                    → OUTCOME_2
## Password reset link sent                                    → OUTCOME_3

Every label survived extraction. The arrows did not.
All nine ## lines of the flowchart are flat — same heading level, reading order only, no indication of which Yes/No belongs to which question, or where either one leads. The topology, which is the entire content of a decision tree, is gone. Rebuilding it is the actual job handed to the mapping stage: match each Yes/No back to its question, and each terminal line to the branch that leads to it, using the few-shot examples and the wording as the only signals.

Note: Two lines are intentionally left out of the output. The first is the intro sentence, a short overview that every sample tree starts with. The agent doesn't need it to follow the flow, so it isn't stored. The second is the flowchart's entry point, Customer requests password reset. It's the trigger that tells the agent when to use the flow, and a tree CallScript has no field for it. If you'd rather keep either one, add a field to the CallScript model and update the mapping prompt to fill it.

Here's what the rendered Markdown looks like, from outputs/password_reset_procedure/final.md:

# Password Reset Procedure

**Category:** Reset Process

## Attention points

- Never process a reset without completing identity verification first.
- Temporary passwords must only be sent through a verified channel, such as the registered email address or phone number.
- If suspicious activity is detected, place a security hold and refer to fraud prevention before resetting.

## Call flow

**STEP_1.** Identity verified via standard two-way authenticator?
- If **no** → `OUTCOME_1`
- If **yes** → `STEP_2`

**STEP_2.** Suspicious activity detected on the account?
- If **yes** → `OUTCOME_2`
- If **no** → `OUTCOME_3`

## Outcomes

- **OUTCOME_1** (Reject the request): Reject the request
- **OUTCOME_2** (Inform the Security Team): Inform the Security Team
- **OUTCOME_3** (Password reset link sent): Password reset link sent

Every branch lands on the outcome it should: an unverified identity is rejected, a verified one moves to the suspicious-activity check, which routes to either the Security Team or the reset link — exactly the arrows drawn in the original flowchart, reconstructed from labels alone. Validation passed every check: no dangling references, no unreachable steps or outcomes, no ungrounded nodes, and no missing attention points.

Still, a model rebuilding a flowchart from labels alone can point a branch at a step that was never defined, and that's exactly what _check_dangling_references catches.

As with the narrative, you can verify this yourself: compare raw.txt with final.md in outputs/password_reset_procedure/, and every question and outcome box appears in the output. raw.txt has no arrows, though, so to check that each Yes/No became a branch that leads to the right place, open the flowchart in sample_inputs/password_reset_procedure.docx next to final.md.

When the mapping gets it wrong

The mapping above came out clean, so validation had nothing to flag and the repair step never ran. To see both at work, I took that same mapping of sample_inputs/password_reset_procedure.docx and introduced two mistakes into it:

1. A missing outcome. The "Inform the Security Team" outcome was deleted, while the "yes" branch that should lead to it stayed in place, pointing at an outcome that no longer existed.
2. An invented outcome. An extra "Escalate to a supervisor" outcome that appears nowhere in the source, and that no branch leads to.

STEP_2: Suspicious activity detected on the account?
  yes -> OUTCOME_2      # no OUTCOME_2 defined
  no  -> OUTCOME_3
Outcomes:
  OUTCOME_1: Reject the request
  OUTCOME_3: Password reset link sent
  OUTCOME_4: Escalate to a supervisor

Pydantic accepts this object without complaint, since every field has the right type. The structural checks from Step 4 don't. They report one warning for the first mistake and two for the second:

- Dangling reference: step 'STEP_2' branch 'yes' points to undefined id 'OUTCOME_2'.
- Unreachable outcome: 'OUTCOME_4' (Escalate to a supervisor) is defined but no branch routes to it.
- Ungrounded node: 'OUTCOME_4' ('Escalate to a supervisor') does not match any line in the source (best word overlap 50%).

The two mistakes need different fixes, and the repair step is allowed to make only one of them.

The missing outcome can be fixed by adding content, and the extracted text (the same text saved as raw.txt), which the repair step receives along with the warnings, still has exactly what's needed: the line "Inform the Security Team". The repair puts OUTCOME_2 back with that wording, and the dangling branch resolves again.

The invented outcome could only be fixed by deleting it, and that is the one thing the repair step never does: the repair prompt forbids it, and the content_size check would reject it anyway. So the repair leaves OUTCOME_4 untouched: it doesn't remove it, rename it to some line from the source, or connect it to a branch just to make the warning go away.

STEP_2: Suspicious activity detected on the account?
  yes -> OUTCOME_2
  no  -> OUTCOME_3
Outcomes:
  OUTCOME_1: Reject the request
  OUTCOME_2: Inform the Security Team
  OUTCOME_3: Password reset link sent
  OUTCOME_4: Escalate to a supervisor

The warning count drops from 3 to 2 and nothing is lost, so the repair is accepted. The two warnings that remain end up in warnings.txt:

- Unreachable outcome: 'OUTCOME_4' (Escalate to a supervisor) is defined but no branch routes to it.
- Ungrounded node: 'OUTCOME_4' ('Escalate to a supervisor') does not match any line in the source (best word overlap 50%).

Note: This run is saved in outputs/password_reset_repair/.

That's the intended split. What can be restored from the source gets restored automatically, and what would need content removed is left for a human to decide.

What if the source document is wrong?

The pipeline can only be as good as the documents it reads. The sample documents for this demo are clean and correct: in sample_inputs/password_reset_procedure.docx, for example, every box can be reached, every scenario has its own branch, and every branch ends where it should. Real documents aren't always like that. If a source tree has a box that nothing leads to, or a scenario with no branch for it, that's a mistake in the source, and the pipeline won't fix it. It shouldn't: fixing it would mean guessing what the author meant, which is exactly the kind of invention the pipeline is built to avoid. Its rule is never invent, never drop, so a broken source produces a broken output. The checks compare the output against the source, not the source against reality.

Note: Sometimes a warning points at the source rather than the mapping. A box in a decision tree that nothing leads to, for example, can come through as an unreachable outcome. When that happens, the fix belongs in the original document, not in the pipeline.

Wrapping up

So, back to the question I opened with: what do you do with a document that's messy, unstructured, or just plain hard to work with? This pipeline is an answer, not the only one, but it works well enough to be worth sharing.

A messy Word document goes in. Azure Document Intelligence extracts the text and layout, a classifier decides whether it's a decision tree or a narrative, an LLM maps it onto a Pydantic model, deterministic checks flag what may have gone wrong, one gated repair attempt fixes what it can, Jinja2 renders the result as structured Markdown, and anything still flagged is left for a human to review.

The part that matters is that no stage trusts the one before it.

The goal was never a pipeline that never gets it wrong — it's one that tells you when it might have, and is honest about the few cases where it can't.

Worth being upfront about the rough edges too: this only works as well as extraction does, and extraction has real limits. Nothing downstream can recover what never comes back from it.

• Some shapes come back empty. Word can store a shape's text in more than one internal format, and if the label isn't in the one extraction reads, the box arrives with no text at all. The step or outcome it held is simply missing, and the mapping stage has no way of knowing it was ever there. For example, if the Inform the Security Team box in the password-reset flowchart came back empty, the tree would have no such outcome.
• Connector arrows never survive. Arrows are lines without text, so extraction drops them and returns only the labels, in reading order. That's the whole reason the mapping stage has to rebuild the routing, and why a confusing layout can lead to a wrong route that every check accepts. In the password-reset flowchart, for example, nothing in the text says Reject the request belongs to No rather than Yes; if the model had picked Yes, nothing would flag it.
• ASCII flowcharts are a risk. A flowchart typed with characters like +---+, | and --> instead of drawn with shapes comes back as plain lines of text, with runs of spaces collapsed. Those spaces were what kept side-by-side boxes in separate columns and lined each connector up under its box. Without them, two labels on the same line read as one, and the connectors no longer point at anything. The validator won't catch a wrong split: grounding checks that a label's words appear in a source line, and they do.

The first two showed up while building this; the third is why I'd convert any ASCII flowchart into real shapes before running it through. Knowing where a pipeline breaks is as useful as knowing where it works, and it's the first thing to check before pointing it at your own documents.

If you're dealing with your own pile of decision trees, SOPs, runbooks, or narrative procedures, the same pattern should carry over. The six stages stay the same; what you adapt is the schema, the prompts and the few-shot examples for your documents. You can find the full code is in this repository.