Reading Google Play and App Store reviews straight from their JSON, no browser

I spent an afternoon last year fighting a headless Chrome that scraped Play Store reviews. CAPTCHA, then a proxy bill, then a week later the DOM shifted and the whole thing returned empty arrays. Threw it out.

Turns out I never needed the browser. Both Google Play and the Apple App Store serve reviews as plain JSON you can hit with an HTTP request. No login, no proxies, no Playwright. This is the request shapes, the pagination caps that aren't in any docs, and the one place Google's format bit me.

All Node.js, all on got-scraping — a drop-in got replacement that copies a real browser's TLS and header fingerprint. That fingerprint earns its keep. The identical request from stock axios or fetch would sometimes come back 403 while got-scraping walked right through, because Play is fingerprinting the TLS handshake, not reading your User-Agent.

Apple first, because Apple made it easy

Apple publishes reviews as an RSS feed in JSON. One endpoint:

https://itunes.apple.com/{country}/rss/customerreviews/page={1-10}/id={appId}/sortby={mostrecent|mosthelpful}/json

• country — a storefront code (us, gb, de...). Every storefront keeps its own reviews.
• appId — the numeric id from the store URL: apps.apple.com/us/app/whatsapp-messenger/id310633997.
• page — 1 to 10, and 10 is the wall. Fifty reviews a page, so ~500 per storefront. Coming back to that.

import { gotScraping } from 'got-scraping';

async function fetchAppleReviews(appId, { country = 'us', maxReviews = 200 } = {}) {
    const out = [];
    const pages = Math.min(10, Math.ceil(maxReviews / 50));
    for (let page = 1; page <= pages; page++) {
        const url = `https://itunes.apple.com/${country}/rss/customerreviews/page=${page}/id=${appId}/sortby=mostrecent/json`;
        const res = await gotScraping({ url, responseType: 'json' });
        const entries = res.body?.feed?.entry ?? [];
        // The first entry is sometimes app metadata, not a review — guard on im:rating.
        for (const e of entries) {
            if (!e['im:rating']) continue;
            out.push({
                id: e.id.label,
                author: e.author.name.label,
                rating: Number(e['im:rating'].label),
                title: e.title.label,
                text: e.content.label,
                version: e['im:version'].label,
                date: e.updated.label,
            });
        }
        if (!entries.length) break;
    }
    return out.slice(0, maxReviews);
}

That 500-review cap is hard. No continuation token gets you past it — I looked. What does work: reviews are scoped per storefront, so pull us, then gb, au, ca, de, and the rest of the English-language storefronts, deduping on review id. There are about ten of them; at ~500 each that's roughly 5,000 recent reviews, which is usually plenty.

App metadata — title, average rating, total count, current version — lives at a second, simpler endpoint:

https://itunes.apple.com/lookup?id={appId}&country=us

Google Play, and the batchexecute rabbit hole

Play has no clean REST endpoint. The store front-end talks to an internal RPC called batchexecute, and the payload is ugly and documented nowhere. The payoff for climbing through it: Play paginates with no ceiling. Apple caps you at 500 a storefront; Play just keeps going.

The endpoint:

POST https://play.google.com/_/PlayStoreUi/data/batchexecute?hl=en&gl=us
Content-Type: application/x-www-form-urlencoded;charset=UTF-8

The body is a URL-encoded f.req parameter wrapping the RPC id UsvDTd and its arguments:

function buildBody(pkg, { count = 100, token = null, sort = 2 } = {}) {
    const tok = token ? `\\"${token}\\"` : 'null';
    const inner = `[null,null,[2,${sort},[${count},null,${tok}],null,[]],[\\"${pkg}\\",7]]`;
    const freq = `[[["UsvDTd","${inner}",null,"generic"]]]`;
    return 'f.req=' + encodeURIComponent(freq);
}

sort is 2 for newest, 1 for relevance, 3 for rating. pkg is the package name (com.whatsapp). token is the continuation cursor the previous response handed back.

The response is where it gets weird. It opens with an anti-JSON-hijacking guard, the literal )]}', and then a nested envelope where the real data sits as a JSON string inside the outer JSON. You parse twice:

function parse(raw) {
    const envelope = JSON.parse(raw.slice(raw.indexOf('[')));   // strip )]}'
    const inner = envelope?.[0]?.[2];                            // a JSON *string*
    if (!inner) return { reviews: [], nextToken: null };
    const data = JSON.parse(inner);
    const reviews = (data[0] ?? []).map((r) => ({
        id: r[0],
        author: r[1][0],
        rating: r[2],
        text: r[4],
        date: new Date(r[5][0] * 1000).toISOString(),
        thumbsUp: r[6],
        reply: r[7]?.[1] ?? null,     // developer reply text
        appVersion: r[10],
    }));
    const nextToken = data[1]?.[1] ?? null;
    return { reviews, nextToken };
}

Then loop, feeding nextToken back until you've got enough or it comes back null:

async function fetchPlayReviews(pkg, { maxReviews = 200 } = {}) {
    const out = [];
    let token = null;
    while (out.length < maxReviews) {
        const res = await gotScraping({
            url: 'https://play.google.com/_/PlayStoreUi/data/batchexecute?hl=en&gl=us',
            method: 'POST',
            body: buildBody(pkg, { count: 150, token }),
            headers: { 'content-type': 'application/x-www-form-urlencoded;charset=UTF-8' },
        });
        const { reviews, nextToken } = parse(res.body);
        if (!reviews.length) break;
        out.push(...reviews);
        if (!nextToken) break;
        token = nextToken;
    }
    return out.slice(0, maxReviews);
}

A few things I only learned by running this against real apps:

• Play reviews carry no title. Just a rating and a body. Apple gives you both. If you're merging the two stores into one schema, title has to be nullable or you'll drop half your Play data on a strict validator.
• Developer replies hide in slot [7] on Play — text at [7][1], timestamp at [7][2][0]. Apple's public feed doesn't surface replies at all.
• The loop fires requests back to back with no delay. Proxyless, from my laptop and from a datacenter, I haven't been rate-limited doing this — but it's the assumption most likely to break at tens of thousands of reviews, and I'd put a throttle in front of it before trusting it at that scale.

And here's where it got me. I first pulled appVersion from slot [8], eyeballed one response, saw a version string, shipped it. Some apps came back with a country code there instead. The version is [10]. The index parsing is brittle by design — Google can reshuffle slots whenever they like, and nothing tells you. So I pinned a test against a known app with a review I can eyeball and assert the fields on it. When Google moves something, that test screams before my users notice.

Is this actually worth skipping the browser?

For me, yes — and the number that convinced me: 250 Play reviews land in about 0.6 seconds this way. You're reading the exact API the store's own frontend reads, so a store redesign doesn't touch you. My Playwright version was 10 to 20 times slower, wanted a proxy budget the moment I scaled it, and died on the next UI refresh. I don't miss it.

If you'd rather not babysit the slot indices

I bundled both stores into one actor on Apify — one schema, handles the pagination and the batchexecute envelope, runs proxyless: App Reviews Scraper. It's mostly there so I stop re-fixing the [10]-versus-[8] kind of thing every quarter. But the code above is the whole trick, and rolling your own is very doable.

The batchexecute envelope eats afternoons if you go in blind — if you get stuck on it, leave a comment and I'll dig in.