Asynchronous Form Schema Discovery: Idempotent Retries and Secure Temporary Files

Use a durable job record as the unit of work for form schema discovery: validate the document at admission, submit one asynchronous extraction job under a correlation ID, poll it with bounded retries, and treat the temporary files on disk as scratch space that nothing downstream is allowed to trust. In a healthtech service that watermarks discharge packets before they go to an outside clinic, the constraint that decides the design shows up before the fidelity-versus-render-cost trade ever does. It is whether you can say, six months later, exactly which bytes left the network and under which policy.

Everything below follows from that one requirement.

The failure that pages you: a duplicate packet leaves the building

Two shapes of page dominate cron and queue infrastructure: a job that never ran, and a delivery that ran twice. Field extraction feeding a watermarking step can produce both in the same night, because it sits exactly where request handlers, queues and a slow external job meet.

Walk the second one through. A worker leases a message for an admitted 14-page referral packet, submits the extraction, and dies before it writes the returned job identifier anywhere durable. The lease expires. A replacement worker reads the same message, finds no job identifier on the correlation record, and submits again — so there are now two extractions, two renders, and, if the delivery step is equally naive, two watermarked packets addressed to the same clinic intake mailbox. Nothing alerts, because every individual call succeeded. The mistake was never a failed request; it was letting a retry mean "start over" instead of "continue".

Standard queues are at-least-once. That is not a defect to work around, it is the contract, and consumer idempotency is the price of admission. Two habits cover most of it: write the job identifier to the correlation record before the first poll, and put a client-supplied idempotency key on the submit so a replayed request resolves to the same job rather than creating a second one. HTTP 429 gets the same treatment — a retried POST that carries an idempotency key is safe to retry; one that doesn't isn't.

What the manifest must hold before patient data goes out

Validation belongs at admission, before a document occupies a worker slot: MIME type, page count, byte size, checked against policy, with a rejection recorded against the correlation ID and a stable reason returned to the caller. Oversized or malformed uploads that reach the worker pool become a latency amplifier under load, and the amplification is worst exactly when traffic is highest.

Then there is the temporary-file question, which in health data handling is really a retention question. Give each correlation ID its own private working directory created with mode 0700 — Node's fs.promises.mkdtemp is fine for this — never join a client-supplied filename into a path, and keep inputs and outputs in separate prefixes so an "output" bucket policy can be audited on its own. Delete the scratch directory when the job reaches any terminal state, success or failure. What survives is the manifest.

The manifest outlives the files. That is the point.

Mine carries the correlation ID, the upstream job identifier, the validated MIME type and page count and size, the input digest, the policy version, the output digest, the render decision, and the timestamps for each transition. The render decision matters more than it looks: this is where the fidelity-versus-cost choice gets recorded. A cheap overlay pass and a full re-render place watermarks differently on a form with tight field geometry, and when someone eventually reports that a signature block was covered, you want to answer from a manifest rather than from a screenshot argument in a chat thread. Store the manifest where retention is 30 days or longer, per your own policy, and store it apart from the artifacts it describes.

How do I implement form schema discovery with asynchronous jobs, retries, and validation?

Submit the extraction as an explicit job with POST /v1/pdf/form/extract, persist the returned identifier, then poll GET /v1/pdf/job/get/{job_id} with exponential backoff, jitter, and a hard deadline taken from your sharing SLO rather than from a number someone typed once. Honour Retry-After when it arrives. No tight loops, and no worker sleeping while holding a lease it could have released.

Here is the shape in Go — a submit-and-poll probe I keep next to the service so the production contract can be exercised without the service running.

package main

import (
    "bytes"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

type submitResp struct {
    JobID string `json:"job_id"`
}

type jobResp struct {
    Status string `json:"status"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    base := strings.TrimSuffix(os.Getenv("INFRAI_BASE_URL"), "/")
    body := os.Getenv("EXTRACT_REQUEST")  // JSON built from the capability's published request schema
    corr := os.Getenv("CORRELATION_ID")   // one per admitted document, durable, reused on every retry
    if key == "" || base == "" || body == "" || corr == "" {
        fmt.Fprintln(os.Stderr, "set INFRAI_API_KEY, INFRAI_BASE_URL, EXTRACT_REQUEST, CORRELATION_ID")
        os.Exit(2)
    }
    // Terminal state names stay in config, not compiled in: they come from the
    // capability's response schema, which the discovery surface publishes.
    terminal := strings.Split(os.Getenv("TERMINAL_STATES"), ",")

    client := &http.Client{Timeout: 20 * time.Second}
    deadline := time.Now().Add(10 * time.Minute)

    jobID, err := submit(client, base, key, corr, body)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }

    status, err := poll(client, base, key, jobID, terminal, deadline)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Printf("correlation=%s job=%s status=%s\n", corr, jobID, status)
}

func submit(c *http.Client, base, key, corr, body string) (string, error) {
    req, err := http.NewRequest(http.MethodPost, base+"/v1/pdf/form/extract", bytes.NewBufferString(body))
    if err != nil {
        return "", err
    }
    req.Header.Set("Authorization", "Bearer "+key)
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", corr) // a replayed submit resolves to the same job
    raw, err := do(c, req)
    if err != nil {
        return "", err
    }
    var out submitResp
    if err := json.Unmarshal(raw, &out); err != nil {
        return "", err
    }
    if out.JobID == "" {
        return "", errors.New("no job id on the submit response")
    }
    return out.JobID, nil
}

func poll(c *http.Client, base, key, jobID string, terminal []string, deadline time.Time) (string, error) {
    path := strings.ReplaceAll("/v1/pdf/job/get/{job_id}", "{job_id}", url.PathEscape(jobID))
    delay := time.Second
    for time.Now().Before(deadline) {
        req, err := http.NewRequest(http.MethodGet, base+path, nil)
        if err != nil {
            return "", err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        raw, err := do(c, req)
        if err != nil {
            return "", err
        }
        var out jobResp
        if err := json.Unmarshal(raw, &out); err != nil {
            return "", err
        }
        for _, t := range terminal {
            if out.Status != "" && out.Status == strings.TrimSpace(t) {
                return out.Status, nil
            }
        }
        time.Sleep(delay)
        if delay < 30*time.Second {
            delay *= 2
        }
    }
    return "", errors.New("polling deadline reached before a terminal state")
}

func do(c *http.Client, req *http.Request) ([]byte, error) {
    delay := time.Second
    for attempt := 0; attempt < 5; attempt++ {
        if attempt > 0 && req.GetBody != nil {
            b, err := req.GetBody()
            if err != nil {
                return nil, err
            }
            req.Body = b
        }
        res, err := c.Do(req)
        if err != nil {
            return nil, err
        }
        raw, err := io.ReadAll(res.Body)
        res.Body.Close()
        if err != nil {
            return nil, err
        }
        if res.StatusCode >= 200 && res.StatusCode < 300 {
            return raw, nil
        }
        if res.StatusCode != http.StatusTooManyRequests {
            return nil, fmt.Errorf("%s %s: status %d: %s", req.Method, req.URL.Path, res.StatusCode, raw)
        }
        if s, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil && s > 0 {
            delay = time.Duration(s) * time.Second
        }
        time.Sleep(delay)
        if delay < 30*time.Second {
            delay *= 2
        }
    }
    return nil, errors.New("rate-limit retry budget exhausted")
}

Two details in there earn their keep. The idempotency key is the correlation ID, so a retry after a lost lease lands on the same job instead of creating a sibling. And terminal state names are configuration rather than constants, because the field names and enumerations belong to the capability schema, not to my poller.

I'm not sure what backoff ceiling is right for your workload — 30 seconds is a starting point, not a recommendation. Arrival rate, page-count distribution and the upstream completion-time distribution decide it, and a load test with production-shaped documents is the only thing that resolves the question honestly.

Vendor shortlist for the extraction step, and where each one hurts

Four candidates, deliberately different in deployment model, because that is the axis a compliance review will actually ask about.

Infrai fits that last row for this workflow — one REST API in front of the document work, with a self-describing discovery surface anyone can read without a key, so you can swap the vendor behind a capability without editing the service code that calls it. Its per-call response metadata — cost, latency, vendor, request identifier — drops straight into the manifest, which is convenient when the audit question is "who processed this document, and when". Stick with a self-hosted engine when your compliance program says protected health information never crosses a boundary you don't operate, or when a trial shows a specific engine handles your form geometry materially better. That is a real limit, and it is the first thing to settle, not the last.

Whatever you shortlist, feed every candidate the same corpus of real documents, compare extracted field identity and geometry against an approved truth set, and run the concurrency test from the region your workers actually live in. Your mileage will vary with scanned pages, embedded fonts and forms built by three different vendors over ten years.

Rollout, verification, and the rollback you will want at 3 a.m.

Ship it in shadow first: run discovery and the manifest write for real traffic, skip the watermark and the delivery, and diff the manifests against what the old path produced. Then let a small share of documents through end to end. The metric that tells you whether it worked is not average latency; it is the count of correlation IDs with more than one upstream job identifier attached, which should be flat zero.

Before external sharing is enabled, five checks: one durable job identifier per correlation record, a terminal status recorded, output stored apart from input, digests and the render decision present in the manifest, and the scratch directory gone. An output with no manifest is not ready to share. A manifest pointing at a missing artifact is worse, because it looks fine on a dashboard.

Rollback should be a flag, not a deploy. Flip new admissions back to the previous path, let in-flight jobs finish against their existing correlation records rather than cancelling them mid-render, and keep every manifest either way — the ones written during a rollback are the ones you will read first when you work out what happened.

Nothing here is clever. That is deliberate.

References

• Gotenberg documentation — https://gotenberg.dev/docs/getting-started/introduction
• pdf-lib documentation — https://pdf-lib.js.org/
• Apryse SDK documentation — https://docs.apryse.com/
• PSPDFKit / Nutrient developer documentation — https://www.nutrient.io/guides/
• RFC 9110, HTTP Semantics: Retry-After — https://www.rfc-editor.org/rfc/rfc9110#name-retry-after
• MDN, Blob — https://developer.mozilla.org/en-US/docs/Web/API/Blob
• Node.js documentation, fs.promises.mkdtemp — https://nodejs.org/api/fs.html#fspromisesmkdtempprefix-options
• OWASP File Upload Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html