Goodreads scraper: the book page has no API, only a GraphQL cache hiding in a script tag. Here's how we read it.
Quick answer
Goodreads retired its public API in December 2020. Book pages still carry the data — but it lives inside a <script id="__NEXT_DATA__"> blob as a normalized Apollo GraphQL cache, not as HTML you can select() your way through. Entities sit in a flat dict keyed "Typename:opaqueId", and every relationship between them is a {"__ref": "<key>"} pointer you have to resolve yourself. Miss that, and you'll either get nothing or — worse — the wrong field, because some fields exist twice under different keys.
Why is there no <div class="rating"> to select?
Goodreads runs on Next.js. The server renders the full Apollo cache into __NEXT_DATA__ and lets the client hydrate from it — so the page's actual content is one JSON tree, not markup:
NEXT_DATA_RE = re.compile(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', re.DOTALL)
Once parsed, the payload you want is at props.pageProps.apolloState — a dict where every key looks like "Book:kca://book/amzn1.gr.book.v1.YaoKZD8xVx72w5T1ZgR1YQ". A live capture of The Hunger Games looks like this:
{
"Book:kca://book/...": {
"__typename": "Book",
"legacyId": 2767052,
"title": "The Hunger Games",
"work": { "__ref": "Work:kca://work/..." },
"primaryContributorEdge": {
"node": { "__ref": "Contributor:kca://author/..." }
}
}
}
work and primaryContributorEdge.node aren't the author's name and the work's rating stats — they're pointers. You resolve a __ref by looking its key straight up in the same cache dict:
def resolve_ref(cache, ref):
if not isinstance(ref, dict):
return None
key = ref.get("__ref")
return cache.get(key) if key else None
That's the whole trick, and it's also where a naive scraper silently breaks: treat book_entity["work"] as a value and you get a {"__ref": ...} dict in your dataset instead of a publish date.
Why does Book.legacyId need str() before you can use it?
Small, easy to miss: legacyId — the number that appears in the URL (/book/show/2767052-the-hunger-games) — arrives as a JSON int, not a string, on both Book and Work entities. Our fixture, captured live against book_id=2767052, confirms it. If your output schema declares book_id as a string (ours does — Pydantic won't silently coerce a stray type mismatch across a whole pipeline), you cast explicitly at the boundary:
book_id = book_entity.get("legacyId")
# ...
return {"book_id": str(book_id), ...}
Small bug, but it's exactly the kind that a spot-check on one book won't catch and a batch run of 500 will.
Why does the same book carry two different description keys?
This is the one that actually cost debugging time. Apollo's InMemoryCache suffixes a field's cache key with its JSON-stringified GraphQL arguments whenever that field takes arguments. A real Goodreads Book entity carries both:
"description": "<p>Raw HTML...</p>"
"description({\"stripped\":true})": "Plain text, no tags..."
Grab the bare description key and you get raw HTML entities in your dataset. Grab the wrong one inconsistently and different books in the same run come back with different formatting. The fix is a small helper that checks the arg-suffixed form first:
def get_field_with_args(entity, field_prefix):
suffix = f"{field_prefix}("
for key, value in entity.items():
if key.startswith(suffix):
return value
return entity.get(field_prefix)
Reviews hit the same pattern at a different scope: they live under ROOT_QUERY's getReviews(...)-prefixed key (also argument-suffixed, also resolved the same way), and each Review entity's author field is named creator — not author, which is what you'd naturally guess and get a silent None for.
What happens to a book with a missing required field?
Not every entity resolves cleanly — a stub entity, a partially loaded cache, a book with no listed author. Rather than propagate a broken row, the parser checks the identity-critical fields up front and skips the book entirely when any of them (book_id, title, title_complete, url) is missing:
if book_id is None or not title or not title_complete or not url:
return None
Everything softer than that — series, genres, awards, edition details — degrades to None or an empty list instead of raising. A book with no series isn't a scraper bug; a book with no title is.
How does search actually work without the retired API?
Two paths, tried in order. Goodreads' own auto_complete JSON endpoint is primary — it returns a plain top-level JSON list, each entry already carrying an explicit, pre-1-indexed rank field (we verified this live; a scraper that assumes array position is the rank will silently agree with the endpoint 99% of the time and be wrong exactly when Goodreads reorders results server-side). If auto_complete fails, we fall back to parsing the /search/index HTML results page directly.
FAQ
Do I need to log in or use an API key?
No — none of the five routes this Actor hits (book page, author page, list/shelf page, auto_complete, /search/index) require authentication.
Can I pull reviews as well as book metadata?
Yes — turn on includeReviews and Goodreads returns up to maxReviewsPerBook reviews embedded on the same page fetch, with ratings, like/comment counts, and spoiler flags.
Does search return the exact rank Goodreads itself shows?
Yes, when auto_complete succeeds — we use its own rank field rather than deriving one from list position.
What if a page just doesn't have __NEXT_DATA__ at all?
The parser returns None and the row is skipped with a warning — a malformed or unexpected page never becomes a bad dataset row.
Packaged and ready to run: Goodreads Book & Reviews Scraper — book metadata, ratings, series, genres, ISBNs and reader reviews from book, author, or list/shelf URLs, or free-text/ISBN search, as clean JSON, CSV, or Excel.
We do the dirty work so your dataset stays clean. 😈
