How to Scrape Websites with Claude Code (2026 Guide)

Claude Code can edit files, run shell commands, and write tests — but it cannot fetch live web pages on its own. Connect it to CrawlForge MCP and it gains 20 scraping tools that run straight from your terminal.

npm install -g crawlforge-mcp-server
npx crawlforge-setup  # paste your API key
# Now in Claude Code:
# > Fetch https://news.ycombinator.com and list the top 5 stories

This guide shows you how to scrape websites with Claude Code using CrawlForge MCP, from installation to stealth-mode bypass. Every code block below is runnable.

Table of Contents

• The Problem: Claude Code Cannot Fetch URLs
• Prerequisites
• Step 1: Install CrawlForge MCP
• Step 2: Get Your API Key
• Step 3: Register the MCP Server with Claude Code
• Step 4: Verify the Connection
• Step 5: Your First Scrape
• Full Working Example: Scrape a Pricing Page
• Advanced: Scrape JavaScript-Rendered Sites
• Tool Quick Reference
• Troubleshooting
• FAQ

The Problem: Claude Code Cannot Fetch URLs

By default, Claude Code has no network access. Ask it to "read this blog post" and it will tell you it cannot open URLs. The built-in WebFetch helper exists in Claude Desktop but is limited, rate-capped, and frequently blocked by Cloudflare, Akamai, and other edge protections.

CrawlForge MCP solves this by exposing 20 scraping tools — fetch_url, extract_content, scrape_structured, stealth_mode, deep_research, and more — as Model Context Protocol tools that Claude Code can call like any other function. For more background on the protocol itself, see the complete guide to MCP web scraping.

+------------+     MCP       +-------------+     HTTPS     +-----------+
| Claude Code| <-----------> | CrawlForge  | <-----------> |  Target   |
| (terminal) |  JSON-RPC     |   Server    |               |   site    |
+------------+   over stdio  +-------------+               +-----------+
                                    |
                                Credits +
                                Stealth +
                                Actions

Prerequisites

• Node.js 18+ — check with node --version
• Claude Code — install with npm install -g @anthropic-ai/claude-code
• A CrawlForge account — free at crawlforge.dev/signup (1,000 credits included, no credit card)

Step 1: Install CrawlForge MCP

npm install -g crawlforge-mcp-server

Verify the install:

crawlforge-mcp-server --version
# crawlforge-mcp-server 3.0.16

Step 2: Get Your API Key

1. Go to crawlforge.dev/signup and create an account.
2. Open the dashboard at crawlforge.dev/dashboard/api-keys.
3. Copy the key — it starts with cf_live_.

Step 3: Register the MCP Server with Claude Code

The fastest path is the setup wizard:

npx crawlforge-setup

It writes the correct entry to ~/.config/claude-code/mcp.json (Linux/macOS) or %APPDATA%\claude-code\mcp.json (Windows).

{
  "mcpServers": {
    "crawlforge": {
      "command": "crawlforge-mcp-server",
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_your_key_here"
      }
    }
  }
}

Step 4: Verify the Connection

Open Claude Code and run:

/mcp

You should see crawlforge listed as connected with 20 tools available. If not, jump to Troubleshooting.

Step 5: Your First Scrape

Paste this prompt into Claude Code:

Fetch https://news.ycombinator.com using CrawlForge and give me the
top 5 story titles with their URLs as a JSON array.

Claude Code will call fetch_url (1 credit), parse the HTML, and return something like:

[
  { "title": "Show HN: My side project", "url": "https://example.com/post/1" },
  { "title": "Why X is changing Y", "url": "https://example.com/post/2" }
]

That is it. You are scraping.

Full Working Example: Scrape a Pricing Page

Here is a realistic task: extract pricing tiers from a SaaS site. Paste this prompt:

Use scrape_structured to extract pricing from https://crawlforge.dev/pricing.
Return an array of { plan, price, credits, features[] }.

// What Claude Code sends to CrawlForge via MCP
const response = await fetch('https://crawlforge.dev/api/v1/tools/scrape_structured', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.CRAWLFORGE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://crawlforge.dev/pricing',
    selectors: {
      plan: '.pricing-card h3',
      price: '.pricing-card .price',
      credits: '.pricing-card .credits',
      features: '.pricing-card ul li',
    },
  }),
});

const data = await response.json();
console.log(data.plan); // ['Free', 'Hobby', 'Professional', 'Business']

Advanced: Scrape JavaScript-Rendered Sites

Some sites render pricing or product data through client-side React. fetch_url returns the pre-hydration HTML shell and misses the data. Switch to scrape_with_actions (5 credits):

// Prompt Claude Code with the goal, and it generates this call
const payload = {
  url: 'https://app.example.com/dashboard',
  actions: [
    { type: 'wait', selector: '.data-grid', timeout: 5000 },
    { type: 'click', selector: 'button.load-more' },
    { type: 'wait', timeout: 1500 },
  ],
  formats: ['json', 'markdown'],
};

const result = await fetch('https://crawlforge.dev/api/v1/tools/scrape_with_actions', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.CRAWLFORGE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(payload),
}).then(r => r.json());

For Cloudflare and Akamai-protected sites, use stealth_mode (also 5 credits). The fingerprint-rotation tradeoffs are covered in the stealth mode deep dive.

Tool Quick Reference

Full list in the 20-tools overview.

Troubleshooting

which crawlforge-mcp-server

npm install -g crawlforge-mcp-server

export CRAWLFORGE_API_KEY="cf_live_..."

FAQ

Next Steps

• Read the CrawlForge quick start for five copy-paste examples
• Browse the getting started docs for full API reference
• Compare MCP clients in Claude Desktop vs Claude Code
• Evaluate alternatives at Firecrawl alternative

Start Free — 1,000 Credits Included