I built an open-source browser agent that turns any website into structured JSON

I built an open-source browser agent that turns any website into structured JSON

Most web scraping tools still think the internet is made of clean HTML.

It is not.

The modern web is made of:

• JavaScript-heavy pages
• login-ish flows
• dropdowns
• old government forms
• pricing pages with no API
• websites that change their DOM every week
• pages where the useful data is behind clicks, tabs, filters, and forms

So I built Browsewright:

Give an LLM a URL and a goal. It drives a real Chrome browser, completes the task, and returns structured data instead of raw HTML.

GitHub: https://github.com/krishnashakula/browsewright

pip install browsewright

Then run:

bw "https://stripe.com" "what does this company do and who is it for"

Instead of writing selectors, XPath, browser scripts, retry logic, and extraction code, you give it intent.

Browsewright figures out the path.

The idea

Playwright automates a browser you script.

Browsewright is the browser that scripts itself.

With traditional browser automation, you write code like:

page.goto(url)
page.click("#pricing")
page.wait_for_selector(".plans")
page.locator(".price-card").all()

That works until the website changes.

Browsewright flips the model.

You say:

bw "https://example.com/pricing" "extract all plans, prices, limits, and billing options as JSON"

The LLM reads the page, decides what matters, drives the browser only when needed, and returns structured output.

Why I built it

I kept seeing the same problem in AI agents, lead enrichment systems, market research tools, and n8n-style automations:

The LLM is smart enough to understand the task.

But it cannot reliably operate the web.

Most systems stop at one of these layers:

1. Fetch raw HTML
2. Convert the page to Markdown
3. Ask the LLM to summarize it
4. Hope the answer is useful

That is fine for simple pages.

But real business workflows need action:

• Fill this form
• Find pricing
• Search this registry
• Compare competitors
• Track stock availability
• Extract lead data
• Monitor brand mentions
• Submit a query
• Return clean JSON

That is not just scraping.

That is browser work.

What Browsewright does

Browsewright is an open-source Python tool that lets an LLM operate a real browser.

It can:

• read a page
• decide what to click
• fill forms
• submit forms
• extract structured JSON
• use cheaper shortcuts before launching Chrome
• run locally with your own API key
• expose itself as an MCP server for AI assistants

Install:

pip install browsewright

Use it from the CLI:

bw "https://news.ycombinator.com" "what is the top story right now"

Use JSON mode:

bw "https://example.com" "find the pricing plans" --json

Debug visually:

bw "https://example.com" "debug this page" --no-headless --verbose

Use it in Python:

import asyncio
from browsewright import search

res = asyncio.run(
    search(
        "https://stripe.com",
        "what does this company do and who is it for?"
    )
)

print(res.answer)
print(res.stage)
print(res.tokens_total)
print(res.elapsed_s)

It does not just read the web. It acts on the web.

Most AI scraping tools can summarize a page.

Browsewright can operate a page.

Example:

bw-tasks form \
  "https://registers.maryland.gov/RowNetWeb/Estates/frmEstateSearch2.aspx" \
  --profile examples/sample_profile.json

That is not a clean JSON API.

That is not a modern developer-friendly endpoint.

It is an old form-based website.

Browsewright can inspect the form, understand the fields, map a profile to the inputs, choose valid dropdown options, submit the form, and extract results.

No selectors.

No XPath.

No custom script for that one website.

Just intent.

The cheap path comes first

One design goal was simple:

Do not launch a browser unless you actually need a browser.

A lot of sites can be answered from cheaper paths:

• public APIs
• RSS feeds
• JSON endpoints
• public archives
• static content

So Browsewright tries a pre-flight pipeline first.

Only if that fails does it spin up Chrome.

That makes it useful for workflows where cost matters.

In my benchmark, Browsewright extracted data from 50 real websites for about 4.7 cents total.

Reproduce it:

python examples/batch_test.py

Built-in workflows

Browsewright ships with task commands for common business automation use cases.

1. Competitor watch

bw-tasks watch "https://competitor.com/pricing"

Use this to monitor pricing pages, feature pages, changelogs, positioning, and landing pages.

2. Lead enrichment

bw-tasks enrich "https://company.com"

Returns CRM-style fields and a personalized first line for outreach.

Example output:

{
  "company_name": "ExampleCo",
  "industry": "AI SaaS",
  "icp_fit_score_1_to_10": 8,
  "personalized_cold_email_first_line": "I noticed your team is expanding into enterprise AI workflows..."
}

3. Agentic form fill

bw-tasks form "https://example.com/form" --profile profile.json

Useful for authorized internal workflows, testing, research forms, and repetitive manual data-entry tasks.

4. Price and stock tracking

bw-tasks track "https://store.com/product"

Track product availability, price movement, and page changes.

5. Brand monitoring

bw-tasks brand "Browsewright" "https://site1.com" "https://site2.com"

Collect mentions, sentiment, and summaries across selected URLs.

Works with MCP

Browsewright can also run as an MCP tool.

Install with MCP support:

pip install "browsewright[mcp]"

Add it to your MCP client:

{
  "mcpServers": {
    "browsewright": {
      "command": "bw-mcp"
    }
  }
}

Now your AI assistant gets a tool like:

read_page(url, goal)

That means an AI assistant can research the web with a real browser instead of only guessing from static text.

Why structured JSON matters

A summary is nice.

JSON is useful.

For automation, you usually need data that can go into another system:

• a CRM
• a spreadsheet
• a database
• Slack
• n8n
• Make
• Zapier
• a vector database
• an internal dashboard
• another AI agent

Browsewright has a core structured extraction primitive:

import asyncio
from browsewright import extract_structured

schema = {
    "headline": "string",
    "open_roles": [
        {
            "title": "string",
            "team": "string",
            "location": "string"
        }
    ]
}

data = asyncio.run(
    extract_structured(
        "https://example.com/careers",
        schema,
        instruction="Extract the page headline and every open job posting."
    )
)

print(data["open_roles"])

This is the part I care about most:

Not “AI browsed a website.”

More like:

AI turned a messy website into a reliable automation input.

Where this is useful

Browsewright is useful when you need web data but the site does not give you a clean API.

Examples:

AI agents

Give your agent the ability to inspect pages, fill forms, and return structured output.

n8n workflows

Use Browsewright as the browser/research layer, then pass JSON into the rest of your workflow.

Sales automation

Enrich leads from company websites, pricing pages, career pages, and recent announcements.

Market research

Extract competitor positioning, product pages, integrations, pricing, and messaging.

Recruiting

Extract job listings from company career pages into clean structured data.

Monitoring

Track changes on important URLs and alert when something changes.

Internal operations

Automate repetitive browser tasks on systems where you are authorized to operate.

A simple n8n-style flow

Here is the kind of workflow Browsewright is designed for:

Schedule trigger
   ↓
Run Browsewright on competitor pricing page
   ↓
Extract structured JSON
   ↓
Compare with previous snapshot
   ↓
If changed, send Slack alert
   ↓
Save result to database

The browser is only one piece.

The output is what makes the automation useful.

Important note on responsible use

Browsewright is designed for authorized research, your own properties, internal automation, and websites where automated access is allowed.

Use polite mode.

Respect robots.txt.

Respect rate limits.

Respect terms of service.

Do not use browser automation to abuse websites, bypass rules, collect private data, or violate laws.

The best use case is not “scrape everything.”

The best use case is:

Replace repetitive manual browser work with reliable, transparent, authorized automation.

How it compares conceptually

Browsewright is not trying to replace every scraper.

It is for the cases where plain scraping is too brittle and full manual browser scripting is too much work.

Try it

Install:

pip install browsewright

Run:

bw "https://example.com" "find the pricing and return the plans as JSON"

Clone:

git clone https://github.com/krishnashakula/browsewright
cd browsewright
pip install -e .

Use MCP:

pip install "browsewright[mcp]"

GitHub:

https://github.com/krishnashakula/browsewright

What I want feedback on

I am looking for feedback from developers building:

• AI agents
• scraping systems
• research agents
• n8n workflows
• sales automation
• market intelligence tools
• browser automation tools
• MCP servers

Especially:

1. What websites should I test next?
2. What built-in tasks should Browsewright support?
3. What MCP workflows would make this more useful?
4. What would make you trust browser-agent output in production?
5. What should the API look like for teams using this at scale?

If this saves you from writing one more fragile scraper, please star the repo.

GitHub: https://github.com/krishnashakula/browsewright

The web was built for humans.

Browsewright lets agents use it like humans — and return data like machines.