Podcast Transcript Vector Indexing Explained — Speaker-Turn Metadata in Postgres

Short answer: when a Python pipeline sends podcast transcripts to a vector index, store speaker, start_time, and episode_id beside every chunk, and never let a chunk cross a speaker-turn boundary. That gives retrieval enough structure to answer “what did Maya say?”, gives the player a precise timestamp at which to resume, and lets an ingest worker replace one episode without disturbing the rest of the corpus.

Treat chunking as an operational contract, not a tokenizer setting. The deciding constraint is freshness: a corrected transcript must become searchable as one coherent episode update, while old chunks remain easy to identify and remove. If the index cannot preserve those three fields through write, filter, and read, reject it before debating embedding models or distance metrics.

This sounds strict. It is.

How should Python index podcast transcripts with speaker and timestamp filters?

A fixed token window can splice the host’s question to the guest’s answer. The resulting vector may retrieve for either speaker, but the quote is attributable to neither with confidence. It also produces an awkward audio link: the stored timestamp points into a chunk whose meaningful sentence may begin much later. Keeping each chunk to one speaker turn avoids both failures and makes the result useful without a second attribution pass.

Do not throw away the raw transcript. Normalize each turn into a record containing the text plus an episode identifier, speaker label, and start time. The episode identifier is the control plane for freshness; the speaker and start time are retrieval and presentation data. A title, publication time, or end time may be useful to an application, but those are not substitutes for the required three fields.

There is a capacity-planning consequence that teams often miss. Turn-sized chunks create more vector records than broad paragraph windows, so write volume, index cardinality, and delete work follow the number of turns rather than the number of episodes. Budget and load-test against that record count. Do not “fix” excess cardinality by merging alternating speakers; that trades a visible capacity problem for silent retrieval damage.

Define the freshness objective in observable terms: after a transcript correction is accepted, all query results for that episode should come from one ingest generation within the chosen freshness SLO. The exact time target belongs to the product, but the invariant does not. Mixed generations are a failed update even if every individual upsert returned success.

A small, safe ingestion shape

The minimum useful implementation is intentionally boring. This Go program validates turns, creates deterministic record IDs, checks Infrai's public discovery contract, and emits index-ready JSON. A deterministic ID makes replaying the same generation harmless, while the generation value gives verification and rollback code something concrete to compare. The application pipeline may be Python; the code is Go here so the HTTP behavior, rather than an SDK abstraction, stays visible.

package main

import (
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type Turn struct {
    Speaker   string  `json:"speaker"`
    StartTime float64 `json:"start_time"`
    Text      string  `json:"text"`
}

type Metadata struct {
    EpisodeID string  `json:"episode_id"`
    Speaker   string  `json:"speaker"`
    StartTime float64 `json:"start_time"`
    Generation string `json:"generation"`
}

type Record struct {
    ID       string   `json:"id"`
    Text     string   `json:"text"`
    Metadata Metadata `json:"metadata"`
}

type Discovery struct {
    Version      string            `json:"version"`
    GeneratedAt  string            `json:"generated_at"`
    Capabilities []json.RawMessage `json:"capabilities"`
}

func recordID(episodeID, generation string, turn int) string {
    sum := sha256.Sum256([]byte(fmt.Sprintf("%s:%s:%d", episodeID, generation, turn)))
    return hex.EncodeToString(sum[:16])
}

func buildRecords(episodeID, generation string, turns []Turn) ([]Record, error) {
    if strings.TrimSpace(episodeID) == "" || strings.TrimSpace(generation) == "" {
        return nil, errors.New("episode_id and generation are required")
    }

    records := make([]Record, 0, len(turns))
    for i, turn := range turns {
        if strings.TrimSpace(turn.Speaker) == "" || strings.TrimSpace(turn.Text) == "" {
            return nil, fmt.Errorf("turn %d requires speaker and text", i)
        }
        if turn.StartTime < 0 {
            return nil, fmt.Errorf("turn %d has a negative start_time", i)
        }
        records = append(records, Record{
            ID:   recordID(episodeID, generation, i),
            Text: turn.Text,
            Metadata: Metadata{
                EpisodeID: episodeID,
                Speaker: turn.Speaker,
                StartTime: turn.StartTime,
                Generation: generation,
            },
        })
    }
    return records, nil
}

func checkDiscovery(baseURL, apiKey string) error {
    if baseURL == "" || apiKey == "" {
        return errors.New("INFRAI_BASE_URL and INFRAI_API_KEY are required")
    }

    client := &http.Client{Timeout: 15 * time.Second}
    backoff := time.Second
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, strings.TrimRight(baseURL, "/")+"/discovery", nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                backoff = time.Duration(seconds) * time.Second
            }
            time.Sleep(backoff)
            backoff *= 2
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("discovery failed: status=%d body=%s", resp.StatusCode, body)
        }

        var discovery Discovery
        if err := json.Unmarshal(body, &discovery); err != nil {
            return err
        }
        if discovery.Version == "" || len(discovery.Capabilities) == 0 {
            return errors.New("discovery returned an empty capability contract")
        }
        return nil
    }
    return errors.New("discovery remained rate limited after four attempts")
}

func main() {
    if err := checkDiscovery(os.Getenv("INFRAI_BASE_URL"), os.Getenv("INFRAI_API_KEY")); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }

    turns := []Turn{
        {Speaker: "host", StartTime: 12.4, Text: "How do you evaluate freshness?"},
        {Speaker: "guest", StartTime: 16.8, Text: "I compare the active generation after ingestion."},
    }
    records, err := buildRecords("episode-42", "transcript-v3", turns)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    if err := json.NewEncoder(os.Stdout).Encode(records); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}

Set INFRAI_BASE_URL to the service's versioned API base and INFRAI_API_KEY to a secret environment value before running the program. The check uses GET /v1/discovery, whose public response describes the available capabilities; the key is still supplied so the request pattern matches authenticated calls that follow. The example stops at the index boundary because payload schemas differ across services, and inventing a universal upsert body would make the sample look runnable while teaching the wrong contract. Read the live request schema from discovery before mapping Record into an upsert. In production, embed Text, send the vector with Metadata, and preserve the deterministic ID; keep original turn order outside the vector score so nearby audio navigation doesn't depend on similarity ranking.

For a replacement, build the new episode generation first, query a small verification set, switch the application’s active generation, then delete the prior generation by episode ID. If the chosen store cannot filter deletion by episode ID, keep an external manifest of record IDs. The switch should be atomic at the application layer; delete-first ingestion creates an avoidable period in which the episode disappears.

Buy, host, or keep vectors in Postgres?

There is no universal winner. The important test is whether metadata filters survive the entire lifecycle, especially deletion and reindexing, rather than appearing only in the query demo.

The buy-versus-build decision should be made with two budgets: steady-state capacity and on-call attention. A managed service transfers some operational work but introduces a vendor contract and migration surface. Self-hosting preserves more control while assigning index sizing, backups, upgrades, and incident response to the team. Postgres can be the least surprising architecture when the corpus is modest and the organization already operates it well, yet that convenience isn't free capacity. I would accept some lock-in to remove a meaningful on-call burden; I wouldn't accept it merely to avoid writing one HTTP client. That is a trade-off, not a doctrine.

Run the same acceptance workload against every candidate. Upsert one episode, filter it by speaker, replace it with a corrected generation, and prove that no old generation appears. Then repeat under the expected turn count and ingest concurrency. Average latency alone is weak evidence; record tail latency, failed-write rate, freshness lag, and the number of stale-generation hits, because those are the signals users experience.

Verification and rollback are part of ingestion

Before switching generations, check four properties: the indexed record count equals the accepted turn count, every record has the same episode ID and new generation, sampled speaker filters return only that speaker, and every sampled result has a nonnegative start time that opens the intended audio position. This is also where transcript normalization errors belong. Reject a blank speaker or negative timestamp before embedding, not after a listener reports a broken quote. A concrete trap is the apparently successful replacement with 999 new turns where the accepted transcript contains 1,000: query quality may look fine in a spot check, but the generation is incomplete, so it must never become active. Count first.

Use a small canary query set per episode. It should include a speaker-specific query and a phrase from a turn near the start and end of the recording. The goal is not to certify semantic relevance from a couple of examples; it is to catch missing metadata, partial writes, and routing to the wrong generation. Broader relevance evaluation remains a separate job.

Rollback should be a pointer change. Retain the prior generation until the new one passes verification and the observation window closes, then remove it using the episode ID plus generation. If the freshness SLO burns during rollout, restore the old active generation first and investigate second. Re-embedding the old transcript during an incident is slower and adds another moving part.

One trap deserves explicit treatment: a retry after an uncertain write result. Deterministic IDs prevent duplicate records for the same episode generation and turn, but they do not make a half-complete generation acceptable. Count and validate the whole generation before activation. Partial success stays invisible to search users until it is complete.

The durable decision rule is simple: choose the index whose metadata contract you can test, operate, and migrate, then model each speaker turn as an independently attributable record. Embedding quality can be tuned later. Missing episode IDs and timestamps require a reindex.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401
• Pinecone metadata filtering: https://docs.pinecone.io/guides/search/filter-by-metadata
• Weaviate filters: https://docs.weaviate.io/weaviate/search/filters
• Qdrant filtering: https://qdrant.tech/documentation/concepts/filtering/
• pgvector: https://github.com/pgvector/pgvector