I listen to a lot of podcasts. The problem was never finding episodes, it was using them afterwards: a 90-minute conversation contains maybe four ideas worth keeping, and by the next week I can remember the idea but not the episode, the timestamp, or the guest's name.
Notes apps did not solve this. I would write "the bit about pricing floors" and six months later have no way back to the source.
This post is about the architecture that finally worked: treat the podcast library as a queryable store, and expose it to an LLM through the Model Context Protocol so the agent can search it the way it searches a codebase.
Why transcripts alone are not enough
The obvious move is transcription. That gives you text, but raw transcripts are hostile to retrieval:
- 90 minutes of speech is roughly 13,000 words, most of it filler.
- Speaker turns are unlabeled unless you diarize.
- A question asked at minute 12 is answered at minute 40, so the useful span does not sit in one chunk.
- Names and numbers are transcribed inconsistently, which breaks keyword search.
So the pipeline needs a structure layer before retrieval: summary, key points, and a topic map, all anchored to timestamps. Then retrieval can operate on structure and fall back to transcript spans.
What MCP changes
Without MCP, every client integration is bespoke: one plugin for ChatGPT, one for Cursor, one for a CLI. With MCP, the server exposes tools once and any MCP-capable client can call them.
The tool surface I ended up with is small on purpose:
| Tool | Input | Returns |
|---|---|---|
search_episodes |
query, optional date range | episode IDs with matched spans and timestamps |
get_episode |
episode ID | summary, key points, chapter map |
get_transcript |
episode ID, optional time window | verbatim text for that window |
list_recent |
limit | latest imported episodes |
The important constraint is that search_episodes returns spans with timestamps, not whole documents. An agent that receives 13,000 words will summarize the middle and lose the point. An agent that receives six 200-word spans with timestamps can answer precisely and cite.
Connecting it
If you want to try the shape of this without building it, Podwise runs a hosted MCP server at https://mcp.podwise.ai/mcp that exposes a library you have imported there: Apple Podcasts, Spotify, YouTube, RSS and private feeds, with summaries, transcripts, mind maps and search across everything. One OAuth sign-in from Claude, ChatGPT or Cursor and the tools appear. There is a CLI as well (brew install hardhackerlabs/podwise-tap/podwise) if you prefer scripted imports.
The client side is unremarkable: point the client at the server, complete OAuth, and the four tools above show up in the tool list.
What actually works in practice
Three query patterns earned their place:
- "Where did someone argue X?" — span search across the library, returns timestamps, no hallucination risk because the answer is quoted.
- "Summarize everything this guest said about Y across three appearances." — cross-episode aggregation, which is where a plain transcript search tool falls over.
- "What did I listen to last month that mentioned Z?" — recency filter plus search, useful for writing.
What does not work well yet: questions that require understanding audio the transcript missed (heavy accents, crosstalk), and anything that needs the visual context of a video podcast.
The design takeaway
If you are building anything that ingests long-form audio, the ordering matters: structure first, retrieval second, agent third. Agents are good at reasoning over six precise spans and bad at reasoning over a dumped transcript. Most of the engineering effort belongs in the structure layer, not the prompt.
