Short answer: generate each property-tour video once, publish it only after validation, and let a separate delivery path serve immutable renditions; this keeps rendering failures away from viewers and makes storage, cache, and retry costs attributable.
The page says tour_playback_gap: an edtech course launch has 42 published lessons, but only 39 promo clips are playable. The on-call does not need a generic "video failed" alarm. They need three identifiers immediately: the course release, the property-tour asset, and the current lifecycle state. If the asset is still rendering, inspect the generation queue. If it is published but playback misses, inspect the delivery manifest and cache path. Those are different incidents, with different owners and different rollback choices.
I've been paged by missed jobs and duplicate deliveries. The recurring lesson is plain: don't let a single success counter stand in for the whole pipeline. A renderer can report success while publication never happens; a retry can publish twice while every individual request looks healthy. The boundary between generation and delivery is therefore an operational contract, not a box on an architecture diagram.
How should property-tour video generation and delivery boundaries be selected?
Choose the boundary at the first durable, validated rendition. Before that point, work is mutable: prompts change, source photos arrive late, a render can be retried, and a codec or container choice can be rejected. After that point, the object should be treated as immutable and addressable by a content version. Generation owns everything required to reach that state. Delivery owns moving that exact state to viewers.
For an edtech company generating short promo videos for property-tour training courses, this split also keeps the two demand shapes from interfering. A course launch may create a burst of render work from a small editorial team. Playback comes from many learners, may spike after a campaign email, and is latency-sensitive. Sharing a synchronous request path couples a slow creative task to an impatient viewer.
Don't do it.
The contract should be small enough to test. A generation worker accepts a job key, the prompt revision, ordered media inputs, and an output profile. It produces a candidate rendition plus metadata. A validator confirms that the candidate meets the chosen media profile and that required frames or audio are present. Publication then records a versioned object key and flips one logical asset version to ready. Delivery reads only ready versions; it never reaches into a renderer's scratch directory.
There is a catch. This architecture is not suitable when every view must synthesize a unique, private tour from live inputs. In that case, there may be no reusable published rendition, so a session-scoped generation path with strict admission control is the more honest design. Likewise, stick with a simpler synchronous pipeline for a low-volume internal preview tool when operating a queue, validator, publication ledger, and cache would cost more engineering time than the delays they prevent. The boundary earns its keep only when retries, reuse, or independent scaling matter.
Work backward from the page
Start with the alert payload, because it forces vague architecture into observable states. course_id, asset_id, asset_version, and stage should be dimensions in logs and traces, while metrics should use bounded labels such as stage and result. Asset identifiers belong in exemplars or structured logs rather than unbounded metric labels. The page should link to a runbook view that answers one question first: did generation fail to produce a publishable object, or did delivery fail to expose an already published object?
A useful state model is accepted -> rendering -> validating -> published. Failure is recorded beside a stage and attempt, not substituted for the last good state. Delivery is allowed to resolve only published. That single rule prevents a half-written object from becoming a viewer problem. Publication should behave like a compare-and-set on (asset_id, asset_version): repeated completion messages can confirm the same result, but they cannot create a second logical release.
Make retries boring.
The generation job key can be derived from stable inputs: course, property, prompt revision, ordered source revisions, and output profile. A worker that receives the same key checks the publication ledger before spending render capacity. If the same version is already published, it acknowledges the duplicate. If an earlier attempt stopped before publication, it may resume or render to a fresh temporary key, but only the validated final object is made visible. This is the idempotency reflex that turns an at-least-once queue from a source of duplicate clips into a routine transport detail.
For the first alert, page on user-visible risk rather than every retry. A single failed attempt that succeeds on retry is useful diagnostic data, not necessarily an on-call event. A published lesson whose current promo asset has no resolvable rendition is different. Route that signal according to the failed boundary: generation ownership when the release deadline is approaching and required assets remain unpublished; delivery ownership when published assets cannot be resolved or playback starts fail. I'm not sure one static time threshold fits both a two-minute clip and a template with many source images. Measure render-duration distributions per output profile, then set deadlines from observed behavior and the course release objective.
Instrument the transition, not just the worker
Worker counters show activity. Transition records show correctness. Emit an event whenever an asset changes stage, and include the prior state, next state, attempt, stable job key, object version, and elapsed time. Then derive three views: work accepted but not published by its deadline, duplicate completion events suppressed by the publication guard, and published versions that delivery cannot resolve. The second view matters even if no learner notices; it is an early warning that retry pressure or acknowledgment timing has changed.
The following Go sketch keeps the important invariant close to the transition. It is deliberately a generic interface, because the storage engine and queue are deployment choices. The ledger must implement the conditional publish atomically.
package pipeline
import (
"context"
"errors"
"time"
)
type Candidate struct {
AssetID string
Version string
ObjectKey string
JobKey string
}
type Ledger interface {
PublishIfAbsent(ctx context.Context, c Candidate) (published bool, err error)
}
type Events interface {
Transition(assetID, version, from, to, jobKey string, elapsed time.Duration)
}
func Publish(ctx context.Context, ledger Ledger, events Events, c Candidate, started time.Time) error {
if c.AssetID == "" || c.Version == "" || c.ObjectKey == "" || c.JobKey == "" {
return errors.New("candidate metadata is incomplete")
}
published, err := ledger.PublishIfAbsent(ctx, c)
if err != nil {
return err
}
if published {
events.Transition(c.AssetID, c.Version, "validating", "published", c.JobKey, time.Since(started))
}
return nil
}
This function does not make delivery depend on the generation process. It commits a reference to a validated object. A delivery resolver can then map the logical asset and version to that object, while the page can distinguish "not published" from "published but not resolved." Keep those messages specific in internal tooling. An ambiguous 404 from a dashboard hides whether the ledger has no version, the manifest points to the wrong key, or the viewer requested an old release.
Test the invariant under concurrency, not only on the happy path. Submit the same job key twice, delay the first acknowledgment, and allow both workers to finish. Exactly one call may establish the logical publication, while both completions can be acknowledged. Next, publish version B while viewers still request version A and verify that both resolve during the defined retention window. Finally, inject a validation rejection and confirm that no delivery reference becomes visible. These tests are more valuable than a large set of mocks that never race.
Storage and cache costs follow lifecycle choices
Storage and cache cost should influence the boundary without becoming the entire design. The expensive mistake is often uncontrolled multiplicity: temporary renders that never expire, identical jobs that create separate final objects, or tiny prompt changes that retain every rendition forever. Record object class and lifecycle purpose in the ledger: scratch, candidate, published, or retained rollback. Scratch data gets a short, explicit expiry. Published data follows the course retention policy. A previous version stays only as long as rollback and active viewer sessions require.
Cache behavior should follow immutability. Give each published rendition a versioned key so a long cache lifetime cannot serve new bytes under an old identity. A small manifest or resolver chooses the current version. Updating that pointer is cheap; rewriting a large object in place is operationally ambiguous and risks mixed playback across cache layers. Media format selection belongs before publication as well. Browser and device support varies by container, codec, audio format, and delivery context, so confirm the target compatibility matrix against the MDN media formats guide rather than assuming one output plays everywhere.
Track cost as units engineers can act on: candidate bytes created, published bytes retained, cache-origin bytes, and renders suppressed by idempotency. Avoid a single blended "media cost" chart. It may reveal a bill change, but it cannot tell the team whether the remedy is shorter scratch retention, fewer output profiles, better cache reuse, or less duplicate generation. Your mileage may vary on the right retention window; release rollback policy and measured access patterns should decide it.
The same clarity helps capacity planning. Render concurrency should be governed by queue age and release deadlines. Delivery capacity should be governed by viewer demand and playback signals. Scaling both from one CPU graph recreates the coupling the architecture was meant to remove.
Tune the early warning without training people to ignore it
The earlier signal is not "the worker logged an error." It is a required asset consuming too much of its publication budget, or a growing gap between accepted and published work for an upcoming course release. Instrument that gap before tuning a page. Begin with a dashboard and ticket-level notification, observe normal queue-age and render-duration distributions, and page only when the remaining release margin makes human action useful.
False positives have a real cost. Page too early and normal long renders wake someone up; the team learns to discount the alert, and the next genuine missed publication waits. Page too late and the launch dashboard becomes the detector. The threshold should therefore combine the asset deadline, current stage, observed duration for its output profile, and whether retry capacity remains. It should not be one universal number copied across every property-tour template.
Close the runbook with an action for each state. For queued work, inspect age and admission limits. For rendering work, inspect attempt history without launching an untracked duplicate. For validating work, preserve the rejection reason and source revision. For published work with playback trouble, verify resolver state, object reachability, and format compatibility. If an alert has no distinct action, demote it until the signal is precise enough to deserve interruption.
That is the boundary test I trust: a responder can identify the failed stage from the page, retry without duplicating a logical asset, and explain which stored bytes and cache traffic the recovery will create.
References
- MDN, "Media Formats Guide": https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats
