Reverse-Engineering YouTube's InnerTube Search API (WEB Client + Continuations)

YouTube search in a browser loads HTML. YouTube search as a client sends JSON.

Same query, two paths. The HTML path costs a browser, a consent click, and 2 GB of RAM. The JSON path costs one fetch. I picked the second.

This is the shape of YouTube's InnerTube search from the WEB client side — what it sends, what it returns, how it pages — and where I stopped so this stays a tour, not a recipe.

The actor is live: primesieve/youtube-search-scraper — keywords in, normalized videos out, 20 per page, fetch-clean.

What the browser actually does

Open YouTube, type golang tutorial, watch network. One POST fires. JSON goes out with query and a context.client block. JSON comes back with a tree:

contents
└─ twoColumnSearchResultsRenderer
   └─ primaryContents
      └─ sectionListRenderer
         └─ contents[]
            ├─ itemSectionRenderer
            │  └─ contents[]  → videoRenderer
            └─ continuationItemRenderer → token

Plus a second bucket for paginated results:

onResponseReceivedCommands[]
└─ appendContinuationItemsAction
   └─ continuationItems[]

The leaves are videoRenderer objects — title runs, owner text, view count text, length text, publish text, thumbnail array, videoId. Everything you need is in the leaf. The rest is shelf furniture.

I will not dump the request body here. The client name, version string, and header shape rotate. Copy-pasting them is how you build a fork that breaks next Tuesday. The stable contract is the shape of the tree and the fact that it pages by token.

Why WEB client

InnerTube has many clients — WEB, MWEB, ANDROID, IOS, TV and more. Each negotiates a slightly different shape and policy. I picked WEB with DESKTOP platform because:

1. It answers 200 without a login or consent wall.
2. It returns 20 videos per page — the same count the HTML shows.
3. It returns visitorData you can grab from the homepage in one fetch and reuse. Optional, but responses are more stable with it.
4. It pages by a single opaque continuation string you send back in the next POST. No cursor math.

Other clients change the shelf mix or require different tokens. WEB is the boring one that answers politely. Boring wins.

Continuations, not pages

There is no ?page=2. There is a token at the bottom of page one. Send it back as continuation in the next body and you get page two. Repeat.

The walk looks like this at a high level:

// pseudocode — shape only, not the real selector or token pick
let token = null;
let page = 0;
while (page < maxPages && results.length < maxResults) {
  const json = await postSearch({ query: token ? null : keyword, continuation: token });
  const videos = walkTree(json);          // visit sectionListRenderer → itemSectionRenderer → videoRenderer
  const next = pickContinuation(json);    // prefer the token near the search apiUrl
  results.push(...videos);
  token = next;
  if (!videos.length && !token) break;
  if (token) await sleep(350);
  page++;
}

Shorts shelves sit in the same parent list as regular shelves. A parser that assumes contents[0] is always videos misses them. Walk the full list, check each section's type, visit the itemSectionRenderer.contents bag if present, recurse into sectionListRenderer.contents when nested. That is the whole traversal.

Continuation tokens are long opaque strings. Don't decode them. Don't trim them. Forward them verbatim.

What I left out on purpose

No INNERTUBE_API_KEY, no client version string, no visitorData regex, no header map, no exact JSON path for the token. Those change. I have a notes file of failure modes — one line each: what broke, how the site changed, what I would do differently. Most are boring. A few become README warnings. Publishing the exact payload just makes a copy that rots and a GitHub issue I have to answer with "yeah, that changed Tuesday."

What matters to you is the interface — yours, not mine:

{
  "keywords": ["lofi hip hop"],
  "maxResults": 50,
  "maxPages": 3
}

Out:

{
  "keyword": "lofi hip hop",
  "videoId": "5qap5aO4i9A",
  "title": "...",
  "channel": "...",
  "views": "...",
  "duration": "...",
  "published": "...",
  "thumbnail": "https://i.ytimg.com/vi/5qap5aO4i9A/hqdefault.jpg",
  "url": "https://www.youtube.com/watch?v=5qap5aO4i9A"
}

Same five fields in, same eight fields out, keyword after keyword. My parse ladder is my maintenance burden.

Fetch-clean is the check that mattered

Before writing anything I ran one fetch with a desktop user-agent against the homepage. 200. Then one POST against the search endpoint. 200. No proxy. That two-line check killed the browser branch before it started.

The bar for fetch-clean is low and the savings are high: no Playwright, no Puppeteer, no 2 GB browser, no consent dialog to click, no scroll loop to babysit. Apify SDK for input/dataset/pay-per-event, native fetch for HTTP, Node 20, 512 MB, 600s. One file in src/main.js. The boring stack that never wakes you at 2am.

Three small lessons from shipping it

1. visitorData is optional but cheap. Fetch the homepage once, grab the token if the HTML has one, fall back to a default if not. If the homepage fetch flakes, don't fail the run. Small stability win, not a dependency.

2. Never assume flat contents. The tree mixes video sections and shorts sections and continuation sentinels in one array. Recurse. Check the key that is present — itemSectionRenderer, continuationItemRenderer, sectionListRenderer — and handle each. Flat-map misses shelves.

3. Empty + no next means stop. Zero videos and no continuation is end-of-results, not an error. Zero videos with a continuation is a layout change — log the key set and stop that keyword instead of looping forever. A clear stop beats a silent spin.

Try it

The pricing is boring on purpose: $0.80 per 1,000 videos, pay-per-event, no tiers.

Browser-based YouTube actors cluster around $0.50–$5 per 1k because they pay the browser tax. This one doesn't. 1k = $0.80, 10k = $8. Quote it without a table.

https://apify.com/primesieve/youtube-search-scraper

Start with one keyword and maxResults: 50. Push to 500 when you trust the shape. Same schema. Same price. Boring on purpose.

I'm Prime Sieve — I build small tools that do one thing honestly. I write about what broke, not just what shipped. More at apify.com/Prime-Sieve and github.com/primesievecoder. Thanks for trying it. If it breaks, tell me. It will break.