Short answer: validate an image before any expensive transformation, then validate the transformed artifact again before it reaches OCR or the community feed. The first gate protects the worker and the bandwidth budget; the second catches what decoding, resizing, or metadata handling changed.
I learned to think about this as a delivery problem, not a single moderation API call. A UGC feed can receive the same upload twice, receive a retry after the user has deleted it, or receive a perfectly valid JPEG whose dimensions make a decoder allocate far more memory than expected. In an OCR product, a safety miss is also a data-leak risk: text extracted from a private document can be indexed before the image is cleared for publication.
The incident pattern: a clean upload becomes unsafe later
The production shape is familiar. A Node.js edge service accepts a multipart upload, stores an immutable original, and emits image.received. A queue worker makes a thumbnail and an OCR-sized derivative. Another worker scans the derivative and publishes a feed record. The queue is at-least-once, so every consumer must tolerate duplicates.
The failure I worry about is the gap between those steps. A file can pass a MIME check while its magic bytes identify another format. An image can contain an ICC profile, an animation frame, or an EXIF comment that a downstream library interprets differently. A thumbnailer may strip one field but preserve another. If the only safety check runs after upload, the raw object can still be fetched by a preview endpoint or copied into a backup before the check completes. If it runs only before transformation, the derivative becomes an unexamined new input. During a postmortem, I draw this as a timeline: upload at 10:02:11, quarantine at 10:02:12, thumbnail at 10:02:14, OCR enqueue at 10:02:15. The missing event is not mysterious; it is the absent decision between the last two timestamps. That gap is where a retry, a second decoder, or an overlooked metadata field can turn a passing upload into a published problem.
That gives us an invariant: no object gets a public URL, an OCR job, or a feed row until the specific bytes used by that action have a passing decision. “Validated” is not a permanent property of an object name; it is a decision attached to a content digest and a processing stage.
That is the whole rule.
How should community image safety validation protect an OCR feed?
Use a small state machine with explicit ownership of each transition. For example: received -> quarantined -> decoded -> transformed -> scanned -> ocr_allowed -> published. Store the SHA-256 digest, detected format, width and height, animation status, scan policy version, and decision timestamp beside each artifact. Keep the original private even after a derivative is approved; retention and deletion are separate controls.
The pre-transform gate should be cheap and deterministic. Check the byte limit before buffering, inspect magic bytes rather than trusting a client header, reject unsupported containers, and cap pixel count and frame count. Decode in a sandbox with CPU and memory limits. A scan verdict can be “allow,” “reject,” or “review”; treating review as allow is how a queue backlog turns into a public incident.
The post-transform gate receives the derivative by digest. It repeats format and dimension checks, scans pixels and metadata, and verifies that the derivative is derived from the approved original. A perceptual hash can connect near-duplicates, but it must not replace the content decision. OCR starts only from ocr_allowed, never from “thumbnail exists.”
Here is the sort of admission function I keep close to the queue consumer. It is deliberately boring: idempotency and bounded work matter more than cleverness.
package pipeline
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"io"
"time"
)
type Artifact struct {
Digest string
Stage string
Bytes int64
Pixels int64
Policy string
Decision string
DecisionTime time.Time
}
type Scanner interface {
Check(ctx context.Context, digest string) (string, error)
}
type Store interface {
Find(ctx context.Context, digest, stage string) (*Artifact, error)
Put(ctx context.Context, artifact Artifact) error
}
func AdmitDerivative(ctx context.Context, r io.Reader, pixels, maxBytes, maxPixels int64, policy string, scan Scanner, store Store) (Artifact, error) {
h := sha256.New()
limited := io.LimitReader(r, maxBytes+1)
n, err := io.Copy(h, limited)
if err != nil {
return Artifact{}, err
}
if n > maxBytes {
return Artifact{}, errors.New("artifact exceeds byte limit")
}
digest := hex.EncodeToString(h.Sum(nil))
if old, err := store.Find(ctx, digest, "transformed"); err == nil && old != nil {
return *old, nil // duplicate delivery: return the recorded decision
}
// Decode and inspect dimensions in the isolated transformer before this call.
// The worker supplies the measured pixel count as part of its artifact record.
artifact := Artifact{Digest: digest, Stage: "transformed", Bytes: n, Pixels: pixels, Policy: policy}
if artifact.Pixels > maxPixels {
return Artifact{}, errors.New("artifact exceeds pixel limit")
}
decision, err := scan.Check(ctx, digest)
if err != nil {
return Artifact{}, err
}
artifact.Decision = decision
artifact.DecisionTime = time.Now().UTC()
if decision != "allow" {
return artifact, store.Put(ctx, artifact)
}
artifact.Stage = "ocr_allowed"
return artifact, store.Put(ctx, artifact)
}
The pixel count in this excerpt comes from the isolated decoder, rather than a value supplied by an HTTP client; production code should reject zero or unknown measurements. The important contract is that a retry sees the same digest and decision instead of invoking OCR twice.
Before upload or after transformation: choose by threat and cost
“Before” and “after” are not competing switches. They protect different boundaries.
| Gate | Catches early | Costs or misses | Best use |
|---|---|---|---|
| Before decode/transform | spoofed types, oversized payloads, hostile containers | cannot see derivative-only changes | quarantine admission and resource control |
| After transform | decoder surprises, retained metadata, derivative policy violations | spends CPU and storage first | the exact bytes sent to OCR or publication |
| Both | failures across the full lifecycle | two verdicts to operate and audit | public UGC with downstream extraction |
Bandwidth changes the balance. If mobile uploads are expensive, reject obviously invalid content at the edge and return a small, actionable error. Do not download an object twice just to satisfy two services; pass a digest and let the scanner read from private storage. If the scanner is slow, keep the feed state pending and expose no fallback URL. A missing thumbnail is less damaging than an unreviewed one.
There are cases where the full two-gate path is not suitable. A private, short-lived internal attachment with no transformation and no OCR may use one quarantine scan, provided its access policy is equally strict. Conversely, a public image editor that accepts user-supplied filters should add a policy check after every filter chain. Your mileage may vary because the threat model, not a universal checklist, sets the boundary.
Operating the pipeline when queues misbehave
The runbook starts with evidence. Alert on the age of the oldest quarantined item, the count of review decisions, scan latency, decoder memory, and the ratio of OCR jobs to ocr_allowed artifacts. Track decisions by policy version so a policy change can be replayed without pretending that an old verdict was new.
When a consumer times out, retry with a bounded backoff and the same idempotency key: digest + stage + policy. Never create a second public object as a retry side effect. A dead-letter item should retain the digest and reason code, not a copy of the user's image in logs. During an incident, pause publication and OCR independently; keeping quarantine ingestion alive lets operators recover without accepting new public data.
Test the transitions, not just the happy-path upload. Feed truncated files, mismatched headers, huge dimensions, animated images, metadata with control characters, and the same digest concurrently. Assert that exactly one OCR task is created after two successful deliveries. Run a property test that no path reaches published unless a post-transform allow decision exists for the same digest.
I am not sure any static test corpus will represent your community's future abuse patterns. That uncertainty belongs in the design: version the scanner policy, sample decisions for human review, and make deletion propagate to originals, derivatives, queue payloads, and OCR text. “We scanned it once” is not an audit trail.
A decision rule that survives a postmortem
For a B2B SaaS community feed that extracts text from photos, place the first safety gate at quarantine admission and the second immediately before OCR or publication. Keep artifacts private until the latter passes, bind every decision to a digest and policy version, and make queue consumers idempotent.
Choose a lighter path only when there is no transformation, no public exposure, and no sensitive extraction. Stick with a single quarantine scan for that narrow case; otherwise the saved CPU is not worth the blind spot. The operational win is not a particular scanner. It is knowing which bytes were examined, which bytes were served, and why those two sets are the same.
