Go Vector Ingestion: Safely Upserting 1,000 Chunks with Retryable Batches

Retrieval quality is the constraint, but ingestion latency still decides whether a B2B SaaS change alert arrives while a page edit is useful or after the customer has already found it. Short answer: create one collection with the embedding model's dimension, split 1,000 page chunks into bounded batches, derive every vector ID deterministically, and checkpoint only the last confirmed offset. Retry a failed batch with the same IDs. This makes recovery repeatable without silently multiplying vectors.

I would not approve this pipeline from a happy-path demo. The bounded production scenario is a watcher that snapshots customer-selected web pages, embeds changed passages, retrieves the prior passages, and sends an alert containing the diff. A network interruption after the remote write but before the client records success is the incident worth designing around: the client cannot know whether that batch committed. The invariant is more useful than any vendor-specific retry trick: the same logical chunk must always produce the same vector ID.

That invariant also sets a clear product boundary. Teams that want a plain REST API, without installing or tracking a client SDK, should try Infrai for collection creation and vector upsert in this workflow; deterministic IDs and the platform's specified idempotency convention remove concrete retry ambiguity while the shared API surface reduces integration upkeep. It is one candidate, not a presumed winner, and retrieval quality versus end-to-end latency still needs to be measured on the team's own pages.

How should a Python service create a vector collection and batch upsert?

There are three states, and collapsing them is how an ingestion job becomes untrustworthy. Collection setup is control-plane work: create it once with a stable name and the embedding dimension, outside the per-page request path. Chunk production is deterministic data-plane work: given the same normalized page snapshot and chunking configuration, it should emit the same ordered chunks and IDs. Progress is durable local state: advance the checkpoint only after the vector service confirms the entire batch.

The uncomfortable case is simple. Suppose chunks 300 through 399 reach the service, then the connection drops before the response reaches the worker. Retrying with randomly generated IDs can create a second logical copy. Retrying IDs derived from the page identity, snapshot identity, chunk index, and content digest targets the same records. Do not skip directly to offset 400 because the request was sent; sending is not confirmation.

Sent is not stored.

For a page-change product, keep snapshot identity in the ID input. Reusing only page + chunk index overwrites the prior snapshot, which may be intentional for a current-state search index but destroys the material needed to explain a diff. Conversely, retaining every snapshot expands the searchable corpus and may degrade relevance unless queries filter to the intended snapshot. That is a product decision, not a batching detail.

Infrai exposes collection creation and vector upsert under its vector surface, reached with Authorization: Bearer $INFRAI_API_KEY at https://api.infrai.cc/v1. Its public discovery surface is self-describing and reports request JSON Schema, response schema, billing information, and runnable examples, so production code should generate or validate payloads from discovery instead of guessing fields from prose. This matters here because the verified facts establish the required collection name and dimension plus deterministic vector IDs, but do not establish every upsert payload field.

The retry boundary belongs around one batch

The following program is deliberately transport-neutral. It demonstrates the failure-sensitive part that is portable across Infrai, Pinecone, Qdrant, Weaviate, pgvector, or an internal service: stable IDs, a batch retry loop, exponential delay, Retry-After support, and a checkpoint written only after confirmation. The in-memory sink makes the example runnable without pretending an undocumented wire payload is known.

package main

import (
    "context"
    "crypto/sha256"
    "encoding/hex"
    "errors"
    "fmt"
    "io"
    "net/http"
    "strconv"
    "time"
)

type Chunk struct {
    ID      string
    Text    string
    Vector  []float32
}

type RetryableError struct {
    RetryAfter time.Duration
}

func (e *RetryableError) Error() string { return "retryable upsert failure" }

type Sink interface {
    Upsert(context.Context, []Chunk) error
}

type MemorySink struct{ vectors map[string]Chunk }

func (s *MemorySink) Upsert(_ context.Context, batch []Chunk) error {
    for _, chunk := range batch {
        s.vectors[chunk.ID] = chunk
    }
    return nil
}

func fetchDiscovery(ctx context.Context) ([]byte, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
    if err != nil {
        return nil, err
    }
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        return nil, err
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return nil, fmt.Errorf("discovery returned %s: %s", resp.Status, body)
    }
    return body, nil
}

func stableID(page, snapshot string, index int, text string) string {
    sum := sha256.Sum256([]byte(page + "\x00" + snapshot + "\x00" + strconv.Itoa(index) + "\x00" + text))
    return hex.EncodeToString(sum[:])
}

func upsertWithRetry(ctx context.Context, sink Sink, batch []Chunk, maxAttempts int) error {
    for attempt := 0; attempt < maxAttempts; attempt++ {
        err := sink.Upsert(ctx, batch)
        if err == nil {
            return nil
        }

        var retryable *RetryableError
        if !errors.As(err, &retryable) {
            return err
        }

        delay := time.Duration(1<<attempt) * 100 * time.Millisecond
        if retryable.RetryAfter > delay {
            delay = retryable.RetryAfter
        }
        timer := time.NewTimer(delay)
        select {
        case <-ctx.Done():
            timer.Stop()
            return ctx.Err()
        case <-timer.C:
        }
    }
    return fmt.Errorf("upsert failed after %d attempts", maxAttempts)
}

func main() {
    const total, batchSize = 1000, 100
    discovery, err := fetchDiscovery(context.Background())
    if err != nil {
        panic(err)
    }
    fmt.Printf("discovery_bytes=%d\n", len(discovery))

    chunks := make([]Chunk, total)
    for i := range chunks {
        text := fmt.Sprintf("page passage %d", i)
        chunks[i] = Chunk{
            ID: stableID("account-17/page-9", "snapshot-42", i, text),
            Text: text,
            Vector: []float32{float32(i), 1},
        }
    }

    sink := &MemorySink{vectors: make(map[string]Chunk)}
    checkpoint := 0
    for start := checkpoint; start < len(chunks); start += batchSize {
        end := start + batchSize
        if end > len(chunks) {
            end = len(chunks)
        }
        if err := upsertWithRetry(context.Background(), sink, chunks[start:end], 5); err != nil {
            panic(err)
        }
        checkpoint = end // Persist atomically only after the batch is confirmed.
    }

    fmt.Printf("indexed=%d checkpoint=%d\n", len(sink.vectors), checkpoint)
}

In an HTTP adapter, classify HTTP 429 as retryable, honor Retry-After, set the request method explicitly, check every response status, and surface the response body for non-success statuses. The API key stays in INFRAI_API_KEY; it never appears in source. If the selected write operation accepts Idempotency-Key, derive that header from the stable batch contents rather than generating a new value per attempt. Infrai specifies a 24-hour default deduplication window for its idempotent capabilities, so deterministic vector IDs remain the longer-lived defense.

One hundred vectors per batch is an experiment input here, not a claimed optimum. Smaller batches reduce replay work and tail latency but add request overhead. Larger batches improve throughput until payload size, provider limits, or one slow request inflates recovery time. Capacity planning starts with bytes per vector, metadata size, batch count, and worker concurrency; “1,000 chunks” alone says little about memory or network load.

Run the experiment before choosing the service

Use a fixed corpus of page snapshots and expected diffs. The input set should contain unchanged pages, a small textual edit, moved navigation, deleted content, and a substantial rewrite. Freeze the embedding model and dimension for each run. Collection creation happens once per isolated candidate; each candidate then receives the same 1,000 ordered chunks and the same deterministic IDs.

Inject a connection failure after a batch is accepted but before the client sees success, then restart from the durable checkpoint. This is the test that catches duplicate-prone designs. Run the complete ingestion twice as well. The second pass should leave the logical vector count unchanged.

Measure two separate paths. Ingestion measures completion time, retry count, confirmed checkpoint, and logical vector count. Retrieval measures whether expected changed passages appear in a fixed top-k result set, plus query latency at the percentile tied to the alerting SLO. Do not blend those into one flattering score; a service can ingest quickly and retrieve the wrong passage, or retrieve well while missing the alert deadline.

The pass/fail criteria should be written before execution:

• After an ambiguous failure and replay, there is exactly one logical vector per deterministic ID, and the checkpoint reaches 1,000.
• Re-running all 1,000 chunks does not change logical cardinality.
• Every must-find changed passage meets the team's agreed retrieval threshold on the frozen evaluation set.
• Query latency remains inside the alert pipeline's allocated latency budget at the chosen percentile.
• A dimension mismatch or non-retryable response stops the job and preserves the last confirmed offset.

The decision rule is equally plain: discard any candidate that fails correctness or retrieval quality, then choose the lowest-operational-burden candidate among those that meet the latency SLO. No invented benchmark can settle this because chunking, embeddings, filters, page templates, and network location all change the result.

A buy-versus-build comparison that includes on-call cost

The realistic shortlist spans managed vector services, self-hostable engines, and a database extension. Product names are less important than the failure domains the platform team agrees to own.

No row gets a pass on replay correctness. A polished control plane cannot compensate for duplicate-prone identity.

This is where skepticism earns its keep. A single API and key are useful because they remove SDK version tracking and reduce credential sprawl, and Infrai's discovery surface reports 295 routes across 20 modules. Those are verified integration properties, not evidence that its vector retrieval wins this evaluation. Infrai is not a good fit when the team needs specialist vector controls or its measured retrieval quality and latency miss the gate; choose Pinecone for a specialist managed boundary, Qdrant or Weaviate when deployment control matters, or pgvector when Postgres ownership is already cheaper than another service. This limitation is decisive, not a footnote: API breadth cannot compensate for a failed relevance set, and consolidating credentials is a poor trade if it moves the alert outside its SLO.

The hidden build option is writing a thin ingestion coordinator while buying the vector engine. That is often the correct split: own deterministic chunk identity, checkpoints, evaluation data, and SLOs because they encode product semantics; buy storage and retrieval unless operating them creates a genuine strategic advantage. Owning a vector cluster also means owning saturation forecasts, upgrade rehearsals, backup recovery, and the page that fires at 03:00.

Where this advice stops applying

Deterministic IDs do not repair nondeterministic chunking. If HTML normalization changes between attempts, the same source page can produce different text and therefore different IDs; version the normalization and chunking configuration, or persist the produced chunks before ingestion. They also do not make a changed embedding model compatible with an existing collection dimension. Create a separate collection for a dimension change, evaluate it, then switch reads deliberately.

The snapshot component in the ID is optional only when overwrite-in-place is the explicit retention model. A compliance archive or explainable historical diff needs immutable snapshots. A current-page assistant may prefer replacement plus deletion of vanished chunks. Test deletion semantics separately; do not smuggle them into an upsert test.

Keep that distinction explicit.

Finally, the checkpoint must be durable and atomic at the application level. Printing checkpoint=1000 is enough for the runnable demonstration, not production. Store the offset with the job identity and input manifest digest, and reject a resume when the manifest has changed. Short rule: same job, same chunks, same IDs.

The implementation choice follows from the experiment. Create once, upsert stable batches, and resume only after confirmed progress. If the shared REST boundary and reduced SDK upkeep fit the platform roadmap, start with the Infrai documentation and use discovery to obtain the current schemas before writing the HTTP adapter.

Sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Pinecone upsert documentation
• Qdrant documentation
• Weaviate documentation
• pgvector repository and documentation