And the equivalent implementation using the official Python SDK:



```python title="scrape_amazon_basic.py" {5-9}

client = alterlab.Client(api_key="YOUR_API_KEY")

response = client.scrape(
    url="https://www.amazon.com/dp/B08F7PTF54",
    render_js=True,
    wait_for_selector="#corePrice_feature_div"
)

# The fully rendered HTML is now available for parsing
html_content = response.text
print(f"Successfully retrieved {len(html_content)} bytes of HTML.")

# Helper function with fallbacks for resilient extraction
def get_text(selectors):
    for selector in selectors:
        element = soup.select_one(selector)
        if element and element.text.strip():
            return element.text.strip()
    return None

# Title selectors
title = get_text(['#productTitle', '.product-title-word-break'])

# Price selectors (Amazon splits dollars and cents in the DOM)
price_whole = get_text(['.a-price-whole'])
price_fraction = get_text(['.a-price-fraction'])
price = f"{price_whole}{price_fraction}" if price_whole else None

# Rating selectors
rating = get_text(['#acrPopover', 'span[data-hook="rating-out-of-text"]'])

# Review count selectors
reviews = get_text(['#acrCustomerReviewText', 'span[data-hook="total-review-count"]'])

return {
    "title": title,
    "price": price,
    "rating": rating.split(' ')[0] if rating else None,
    "reviews": reviews.split(' ')[0] if reviews else None
}

### Understanding the DOM Structure

When inspecting Amazon's DOM, you will notice heavy use of utility classes (often starting with `a-`). 
*   **Title:** Usually consistently found under `id="productTitle"`.
*   **Price:** Often split into multiple `<span>` elements (e.g., `<span class="a-price-whole">19</span><span class="a-price-fraction">99</span>`). You must concatenate these during parsing.
*   **Variations:** If a product has multiple sizes or colors, the default price shown in the HTML might change based on the default selection.

## Best practices

When building automated data collection systems, reliability and compliance must be your top priorities. A poorly designed scraper will fail frequently and place unnecessary load on the target servers.

<div data-infographic="steps">
  <div data-step data-number="1" data-title="Review Rules" data-description="Always check robots.txt and adhere to stated crawling policies."></div>
  <div data-step data-number="2" data-title="Limit Rates" data-description="Implement concurrency limits and exponential backoff to respect server load."></div>
  <div data-step data-number="3" data-title="Extract Publicly" data-description="Ensure you are only targeting publicly available, non-authenticated data."></div>
</div>

### Respect Rate Limits and Concurrency
Do not flood the target servers with thousands of concurrent requests. Implement intelligent rate limiting in your scraping pipeline. If you receive an HTTP 429 (Too Many Requests) or a 503 (Service Unavailable) status code, your scraper should automatically trigger an exponential backoff routine, pausing execution and retrying after a progressively longer delay.

### Adhere to robots.txt
Always inspect `https://www.amazon.com/robots.txt` before initiating a scrape. This file dictates which paths the site administrators prefer bots to avoid. While search engine crawlers and data pipelines rely on public data, respecting these guidelines ensures you are operating a well-behaved bot.

### Handle Missing Data Gracefully
Because e-commerce DOMs are highly volatile, your parsing logic must not crash if a field is missing. As shown in the code example above, always use helper functions that accept a list of fallback selectors and return `None` (or a default value) rather than throwing a `NullReferenceException`. 

## Scaling up

Scraping a single product page is straightforward. Scraping 100,000 product pages daily requires a distributed architecture.

When scaling your Python scraping operations, you need to transition from synchronous scripts to asynchronous task queues. A standard modern stack for this involves:
1.  **Job Queue:** Celery or AWS SQS to hold the URLs that need to be scraped.
2.  **Workers:** Python workers running `asyncio` or multithreading to pull URLs from the queue and send requests to the AlterLab API.
3.  **Storage:** Amazon S3 or a PostgreSQL database to store the parsed JSON blobs.

By offloading the heavy lifting of browser rendering and network management to AlterLab, your worker nodes remain lightweight. They only need enough CPU and memory to dispatch HTTP POST requests and parse the returned strings via BeautifulSoup.

Managing proxy pools, headless browser clusters, and handling dynamic anti-bot protections in-house requires dedicated DevOps resources. Utilizing a managed API ensures predictable costs and higher success rates. For detailed information on volume tiers, review the [AlterLab pricing](/pricing) page.

## Key takeaways

*   **Public Data Only:** Focus exclusively on publicly available product information and always review the target site's Terms of Service and `robots.txt` before deploying a crawler.
*   **Rendering is Mandatory:** Modern e-commerce sites rely heavily on JavaScript. Using raw HTTP clients like `requests` will result in missing pricing and variation data.
*   **Resilient Parsing:** A/B testing changes DOM structures frequently. Implement fallback CSS selectors in your BeautifulSoup logic to prevent pipeline failures.
*   **Managed APIs Reduce Overhead:** Offloading network and headless browser management to tools like AlterLab allows your engineering team to focus on data parsing rather than proxy maintenance.

## Related guides

Expanding your e-commerce data coverage? Check out our technical guides for other major platforms:
*   [How to Scrape Walmart](/blog/how-to-scrape-walmart-com)
*   [How to Scrape eBay](/blog/how-to-scrape-ebay-com)
*   [How to Scrape Etsy](/blog/how-to-scrape-etsy-com)