Video and Audio as Knowledge Sources: Content Understanding into Microsoft Foundry IQ

Your largest corpus is the one nobody indexed. Here's how to turn recordings into grounding data an agent can cite.

The corpus you already have and never indexed

Every organization is sitting on years of recorded meetings, support calls, training sessions, conference talks, and screen recordings. Almost none of it is retrievable. When someone asks "what did we decide about the vendor migration," the answer exists — in a 47-minute recording nobody will ever scrub through.

The reflex fix is to bolt a transcription service onto your RAG pipeline: run Whisper, dump the text into a blob container, index it. It works, sort of, and then you discover what you lost. The transcript has no speakers, no timestamps you can link back to, no slide content, no distinction between the presenter reading a bullet and someone in the room disagreeing with it. Your agent can now quote the meeting but can't tell you when it happened or who said it.

Azure Content Understanding is Microsoft's answer to that gap: it ingests documents, audio, images, and video and extracts the most critical information to power well-grounded generative and agentic solutions, combining Document Intelligence's traditional AI with LLM-based content reasoning. And as of Build 2026 it is integrated with Foundry IQ standard mode for built-in content extraction inside Microsoft's retrieval and agent workflows.

That integration is the subject of this article. Specifically, the part everyone gets wrong: there are two ways to get media into a knowledge base, they are not equivalent, and the one people assume exists is the one you should verify before you plan a sprint around it.

This is a companion to my earlier piece on connecting a Foundry IQ knowledge base to LangGraph over MCP. That one covered retrieval. This one covers what you feed it.

Architecture

Figure 1 — Two paths into the same knowledge base. The native path is a flag on the knowledge source; the explicit path runs analyzers yourself and lands the output as text.

The two paths, stated plainly

Path A — native extraction inside the ingestion pipeline. Setting the contentExtractionMode property to standard on file-based indexed knowledge sources (Azure Blob, SharePoint, OneLake) enables Content Understanding functionality within the ingestion pipeline. One property. No orchestration code. This shipped in the Foundry IQ 2026-05-01-preview release, which focused on richer Content Understanding extraction and image serving for multimodal agentic retrieval.

Path B — explicit analysis, then index the output. You run Content Understanding yourself, write the resulting Markdown and fields to blob storage, and point a normal knowledge source at that. More moving parts, complete control.

Here is the honest caveat, and I would rather you hear it from me than discover it in week three: Microsoft's Foundry IQ extraction announcements emphasize document understanding — layout, tables, figures, and document-embedded images — rather than audio and video ingestion. Content Understanding standard mode itself is documented for documents, images, audio, and video. Whether your blob knowledge source will accept an .mp4 today, in your region, at your API version, is a question you should answer with a five-minute test rather than an assumption.

So: test Path A first, build on Path B if it doesn't cover your media types. Path B is what this tutorial walks through in detail, because it works regardless, and because understanding it makes Path A trivial to adopt when it covers you.

Prerequisites

• A Microsoft Foundry resource, and a Content Understanding resource.
• An LLM deployment for analyzers. Analyzers are powered by LLM and embedding models you deploy in Foundry, and GPT-5.2 improves custom field extraction enough to avoid prompt-engineering gymnastics on mixed layouts, domain-specific language, and multilingual content. Analyzers built on GPT-4.1 continue to run unchanged.
• An Azure AI Search service for the knowledge base.

pip install 'markitdown[az-content-understanding]' azure-search-documents azure-identity

Pin your API version before writing a line of code

Content Understanding's GA API is 2025-11-01. The preview versions 2024-12-01-preview and 2025-05-01-preview were slated for retirement on July 15, 2026 — a date that has now passed, so if you inherited a codebase targeting either, it is already broken or about to be.

Note the asymmetry with Foundry IQ, which is on 2026-05-01-preview for the content-extraction features. You will be running a GA content service against a preview retrieval service. Plan your support expectations accordingly.

Step 1 — Choose the analyzer, and know what each modality costs you

Content Understanding ships prebuilt analyzers per modality, and MarkItDown auto-selects among them: documents route to prebuilt-documentSearch, video to prebuilt-videoSearch, and audio to prebuilt-audioSearch.

They are not interchangeable, and the differences determine what your agent can cite:

The row that matters most is natural citation anchor. A document chunk cites a page; an audio chunk cites a timestamp. If you flatten audio into plain text before indexing, you throw away the only anchor that makes a recording navigable — and no amount of clever chunking downstream will recover it.

Step 2 — The fastest path from a recording to indexable text

MarkItDown with the Content Understanding backend is the shortest route, and it is genuinely a few lines. Zero configuration auto-selects the analyzer per file type:

from markitdown import MarkItDown

md = MarkItDown(cu_endpoint="<content_understanding_endpoint>")

doc = md.convert("report.pdf")     # → prebuilt-documentSearch
video = md.convert("meeting.mp4")  # → prebuilt-videoSearch
audio = md.convert("call.wav")     # → prebuilt-audioSearch

print(video.markdown)

The output is Markdown with headings, tables, and figure descriptions inline — exactly the shape downstream chunkers and embedding models prefer. That last clause is the whole argument for this approach: you are not inventing a format, you are producing the format the rest of the stack already wants.

With a custom analyzer, the output carries extracted fields as YAML front matter above the body:

md = MarkItDown(
    cu_endpoint="<content_understanding_endpoint>",
    cu_analyzer_id="my-meeting-analyzer",
)
result = md.convert("standup-2026-08-14.mp4")

---
contentType: video
fields:
  MeetingTitle: Vendor migration review
  Decision: Proceed with phased cutover
  Owner: A. Rivera
---
<!-- 00:04:12 -->
...

That front matter is not decoration. It is your metadata filter, your citation payload, and the difference between "the agent found a relevant meeting" and "the agent told me we decided to proceed, and here is the timestamp."

Step 3 — Design a field schema worth extracting

Prebuilt analyzers give you good transcripts. Custom analyzers give you answers to questions you ask repeatedly — and those are what make retrieval feel like it understands your business rather than your file formats.

Two constraints to plan around:

Custom analyzers are built in Content Understanding Studio, not the Foundry portal. The Foundry portal surfaces prebuilt analyzers and a playground, with a deep link into CU Studio that preserves your project context, but custom analyzer creation lives in Studio. Expect to move between two tools.

Standard mode is per-file. Standard mode handles single files with straightforward field extraction — documents, images, audio, or video without cross-file analysis or complex reasoning. Pro mode is for multi-step reasoning and cross-file analysis. Since Foundry IQ's integration is with standard mode, per-file extraction is your ingestion contract. Questions like "which meetings contradicted the Q2 plan" are retrieval-time work for the agentic retrieval engine, not ingestion-time work for the analyzer.

A schema that earns its keep on meeting recordings:

Resist the urge to extract a summary field. The retrieval engine synthesizes at query time against the actual question; a pre-baked summary just adds a lossy paraphrase your agent might cite instead of the source.

One operational note from Microsoft that applies whenever you change models: run side-by-side against your existing eval set before flipping production traffic, since confidence scores, latency, and output accuracy can all shift with a new model.

Step 4 — Land the output where a knowledge source can reach it

Write the Markdown to blob storage, one file per recording, with the front matter intact:

import pathlib
from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient
from markitdown import MarkItDown

md = MarkItDown(cu_endpoint=CU_ENDPOINT, cu_analyzer_id="my-meeting-analyzer")
blobs = BlobServiceClient(
    account_url="https://stfoundryiqprod.blob.core.windows.net",
    credential=DefaultAzureCredential(),
).get_container_client("meeting-transcripts")


def ingest(path: str) -> str:
    """Analyze one recording and land the result as indexable Markdown."""
    result = md.convert(path)
    name = pathlib.Path(path).stem + ".md"
    blobs.upload_blob(name, result.markdown.encode("utf-8"), overwrite=True)
    return name

Keep the source recording and the derived Markdown in separate containers. You want the knowledge source pointed at text only, and you want the original media addressable for playback when a citation resolves. Mixing them means either indexing binaries you can't use or losing the link back to the thing a user actually wants to watch.

Step 5 — Create the knowledge source and the knowledge base

Now it is an ordinary Foundry IQ ingestion. Foundry IQ automates document chunking, vector embedding generation, and metadata extraction for indexed knowledge sources, and schedules recurring indexer runs for incremental refresh.

from azure.identity import DefaultAzureCredential
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    AzureBlobKnowledgeSource,
    AzureBlobKnowledgeSourceParameters,
    KnowledgeBase,
    KnowledgeSourceReference,
)

client = SearchIndexClient(endpoint=SEARCH_ENDPOINT,
                           credential=DefaultAzureCredential())

source = AzureBlobKnowledgeSource(
    name="meeting-recordings-ks",
    description=(
        "Transcribed and structured meeting recordings: decisions, owners, "
        "due dates and unresolved questions, with timestamps. "
        "Use for questions about what was decided, by whom, and when."
    ),
    azure_blob_parameters=AzureBlobKnowledgeSourceParameters(
        connection_string=BLOB_CONNECTION,
        container_name="meeting-transcripts",
        # Path A: enable Content Understanding inside the ingestion pipeline.
        # Test this against your actual media types before relying on it.
        content_extraction_mode="standard",
    ),
)
client.create_or_update_knowledge_source(knowledge_source=source)

client.create_or_update_knowledge_base(knowledge_base=KnowledgeBase(
    name="meetings-kb",
    knowledge_sources=[KnowledgeSourceReference(name="meeting-recordings-ks")],
    retrieval_instructions=(
        "Prefer the most recent meeting when decisions conflict. "
        "Always surface the timestamp and speaker with any quoted decision."
    ),
))

Two things to be deliberate about.

The description is load-bearing. The agentic retrieval engine uses it to plan queries and select sources, so write it the way you would brief a new colleague: what is in here, and what kinds of question it answers. "Meeting transcripts" is a wasted field.

The content_extraction_mode="standard" flag is your Path A test. Point it at a container holding one .mp4 and one .md, run the indexer, and look at what got indexed. If the media file produced chunks, you can skip Steps 2 through 4 for that file type. If it didn't, your pipeline is already correct and you have lost five minutes.

What happens at query time

Figure 2 — Ingestion runs once per recording; retrieval runs per question. The timestamp survives both, which is the point.

Nothing about retrieval changes because the source was a video. That is the entire payoff: the agentic retrieval engine plans queries, selects sources, runs parallel searches, and aggregates results, returning extractive data with citations so agents can reason over raw content and trace answers to source documents. Your recordings are now just documents that happen to have timestamps.

The 2026-05-01-preview release also added image serving — surfacing document-embedded images during agentic retrieval. For a slide-heavy deck or a demo recording, that is the difference between "the presenter showed an architecture diagram" and actually returning the frame. Combined with figure extraction from Office files, where each figure is retrievable by ID via GET /contentunderstanding/analyzerResults/{operationId}/files/figures/{figureId}, you can build citations that resolve to a picture rather than a paraphrase of one.

Making citations survive the pipeline

This is where most media-RAG implementations quietly fail. The transcript is indexed, retrieval works, the answer is correct — and the citation says "meeting-2026-08-14.md," which is useless to someone who wants the 12 seconds where the decision was made.

Three rules:

1. Never strip the front matter during chunking. If your chunker treats YAML as noise, the extracted fields never reach the index and your metadata filters silently match nothing.
2. Keep timestamp markers inside the chunk text, not only in metadata. Extractive retrieval returns content; if the timestamp lives only in a sidecar field, the model has nothing to quote.
3. Store a deterministic link back to the media. A blob URL plus a timestamp fragment costs you one field at ingestion and turns every citation into a playable link.

The test: ask your agent a question, take the citation it returns, and try to get to the exact moment in the recording. If you can't do it in one click, the pipeline isn't finished, however good the answer text looks.

Troubleshooting

Where to go next

A few things landed or were slated for July 2026 that change the calculus here, and are worth confirming against current docs before you design around them: a synchronous API for Read and Layout, an agentic understanding mode for complex documents, data zone and global zone processing for residency, improved custom analyzer training from your own examples, and labeled training data no longer being stored in CU so training inputs stay in your own storage. That last one matters most if compliance review is what's blocking you.

The natural follow-on build is a routing layer: send short call recordings straight through prebuilt-audioSearch, send slide-heavy sessions through prebuilt-videoSearch so you keep the visual channel, and reserve custom analyzers for the recording types you query weekly. Extraction quality and cost both track how well that routing matches your corpus.

References

1. What is Foundry IQ? — knowledge sources, indexing automation, agentic retrieval and ACL enforcement.
2. Content Understanding standard and pro modes — per-file vs. cross-file analysis, and the preview API retirement notice.
3. Content Understanding models and deployments
4. Foundry vs. Content Understanding Studio — which tool does what.
5. Supported document formats and input file limits
6. Build a RAG solution with Content Understanding
7. Content Understanding in LangChain
8. What's new in Azure Content Understanding at Build 2026 — the Foundry IQ standard-mode integration, MarkItDown backend, GPT-5.2 analyzers, and the July roadmap.
9. Improved data processing features in Foundry IQ: richer content extraction and data enrichment — contentExtractionMode, SharePoint indexing expansion, and image serving.
10. Foundry IQ: improve recall by up to 54% with knowledge bases
11. Build 2026 Session BRK242 — "Turn your agents into action" — agentic understanding mode and the Foundry IQ integration, demoed end to end.
12. MarkItDown on GitHub — the CU-backed converter used throughout this article.
13. Microsoft Agent Framework — for registering CU as an agent tool instead of a pipeline stage.