Markdown looks simple because it is easy to type. That simplicity can be deceptive when you build an automated publisher.
Quick solution:
def extract_hero_and_caption(ast_nodes):
hero_index = find_hero_image_node(ast_nodes)
if hero_index is not None and is_valid_caption_block(ast_nodes[hero_index + 1]):
return ast_nodes[hero_index + 2:]
return ast_nodes[hero_index + 1:]
~~~{% endraw %}
[CommonMark 0.31.2](https://spec.commonmark.org/0.31.2/) defines link and image syntax; caption handling is an application-level convention that the transform must preserve explicitly.
A regular expression removes a hero image. Another inserts an internal link. A third strips a caption. The pipeline passes [unit tests](https://raylabs.app/articles/how-to-build-robust-viewmodel-unit-tests/), but the rendered article suddenly contains an orphan caption or an unnatural sentence where one word has been replaced by an entire destination title.
The problem is not Markdown itself. The problem is treating structured editorial text as an unstructured string.
Two transformations deserve particular care:
1. extracting an image and its visible caption;
2. inserting contextual links without rewriting the author's voice.
## Standard Markdown does not give you a universal caption node
CommonMark defines links and images, but there is no universal native image-plus-visible-caption construct.
Publishers therefore use conventions such as:{% raw %}
~~~markdown
*The pipeline keeps content and deployment state separate.*
~~~
or raw HTML:
~~~html
`<figure>`
`<img src="hero.jpg" alt="Architecture overview">`
`<figcaption>`The pipeline keeps content and deployment state separate.`</figcaption>`
`</figure>`
~~~
The first form is just an image block followed by a paragraph. Your pipeline may decide that the paragraph is a caption, but that interpretation is application-specific.
That means caption extraction needs an explicit rule.
## The greedy whitespace bug
Consider this input:
~~~markdown
*The publication path has three independent gates.*
The article begins here with a normal paragraph.
~~~
A regex designed to remove the image may accidentally consume too much or too little whitespace.
If it removes only the image token, the caption remains at the top of the article as ordinary prose.
If it greedily absorbs multiple following paragraphs, the first real paragraph disappears too.
The dangerous pattern is matching the image, then matching arbitrary whitespace and text.
The safer rule is structural:
1. locate the image block;
2. identify exactly the next block;
3. classify that block as a caption only if it matches your caption convention;
4. remove those exact blocks;
5. preserve all other whitespace and prose.
## Prefer an AST when transforms become important
For one tightly controlled format, a carefully tested parser can be enough.
As soon as your pipeline performs several Markdown transformations, an abstract syntax tree becomes easier to reason about than stacked regular expressions.
The conceptual flow becomes:
~~~text
Markdown
-> parse blocks/nodes
-> identify hero image node
-> inspect immediate next paragraph
-> classify caption
-> transform nodes
-> render Markdown or HTML
~~~
Now the next paragraph is an actual node, not a substring discovered by a greedy expression.
An AST also helps distinguish links inside paragraphs, headings, code fences, image alt text, and existing link nodes.
That matters for internal-link insertion.
## Contextual links should preserve an existing phrase
Imagine the article contains:
~~~text
A reliable serverless publishing workflow needs explicit retry boundaries.
~~~
You want to link [serverless publishing workflow](https://raylabs.app/articles/designing-a-reliable-serverless-ai-publishing-workflow/) to a relevant internal article.
The good result is:
~~~markdown
A reliable [serverless publishing workflow](https://raylabs.app/articles/designing-a-reliable-serverless-ai-publishing-workflow/) needs explicit retry boundaries.
~~~
A bad automation finds only the word workflow and replaces it with the destination page title:
~~~markdown
A reliable serverless publishing [Designing a Reliable Serverless AI Publishing Workflow](https://raylabs.app/articles/designing-a-reliable-serverless-ai-publishing-workflow/) needs explicit retry boundaries.
~~~
The URL may be correct, but the sentence no longer sounds authored.
The rule should be simple: link an existing natural phrase, and do not rewrite prose to manufacture an anchor.
## Score candidate phrases instead of replacing tokens
A practical link inserter can work from phrase candidates.
For each destination:
1. collect phrases already present in the article;
2. reject phrases inside headings, code, existing links, and captions;
3. prefer multiword phrases with clear semantic overlap;
4. choose the first contextually useful occurrence;
5. wrap exactly that phrase;
6. do not alter surrounding words.
A basic score can favor three-to-six-word phrases, exact topic terms, and body paragraphs over navigation-like text.
The scoring system does not need artificial intelligence. It needs predictable editorial constraints.
## Never link inside code, image syntax, or an existing link
String replacement is especially risky because the same phrase may appear in code, image alt text, headings, or a link that already exists.
A syntax-aware transform can exclude those node types.
A regex stack tends to solve this by adding exceptions until maintenance becomes harder than parsing Markdown correctly.
## Link insertion should be idempotent
Run your publishing transform twice.
The second run should not:
- add a second copy of the same link;
- nest Markdown links;
- change the anchor phrase;
- reinsert a removed caption;
- move the hero;
- change unrelated whitespace.
Idempotency is one of the strongest tests for editorial automation because production jobs are retried.
If the same draft can be transformed twice with a different output, recovery becomes dangerous.
This is part of the same design problem discussed in [Automating Medium Draft Workflows](https://raylabs.app/articles/automating-medium-draft-workflows/): retries should continue the same artifact rather than mutate it unpredictably.
## Separate editorial transformation from privacy validation
A Markdown transformer may also need to reject private repository URLs or internal hostnames.
Keep that as a separate validator.
Caption extraction answers which blocks belong to the hero presentation. Internal-link insertion answers which existing phrase can safely become an anchor. Privacy validation answers whether any forbidden URL or identifier remains in the final document.
RayLabs treats private-link detection as its own boundary in [Preventing Private Repository Links in Public Articles](https://raylabs.app/articles/preventing-private-repository-links-in-public-articles/).
## Verify the rendered result, not only the source
A Markdown file can look correct and still render incorrectly under the target parser.
Your final test should render the transformed document through the same stack used in production and inspect the result.
For caption handling, confirm that exactly one hero exists, the caption appears in the intended location, and the caption does not appear again as the first article paragraph.
For contextual links, verify that anchor text matches the intended phrase, the destination is the expected existing URL, no nested anchors exist, and code and headings remain unchanged.
## Execution Checklist
- [ ] Confirm caption extraction isolates hero image and caption blocks without consuming subsequent paragraphs.
- [ ] Verify contextual link insertion wraps exact natural phrases without altering surrounding text.
- [ ] Execute the publishing transform twice to guarantee idempotency.
- [ ] Render the final output through the production parser to validate HTML structure.
