Keenable: Agent-First Search API Architecture and the 100B-Page Index Trade-Off

Agents don't search like humans. They issue hundreds of queries per session, need structured extraction over snippet relevance, and care more about p95 latency than the perfect top result. Keenable built a search API around those constraints with a 100B+ page proprietary index, SQL-like query interface, and continuous benchmarking against agent-like workloads.

The founders (Amazon AGI web grounding, Yandex search lead) are betting that wrapping existing search APIs won't cut it when agents become the primary consumers of web data. The architecture reveals what changes when you optimize for machine callers instead of human eyeballs.

Why Agent Search Needs Different Plumbing

Human search optimizes for the first three results and tolerates 500ms variance. Agent search runs in tight loops where every query blocks downstream tool calls. The contract shifts:

• Query volume: Agents issue 10-100x more queries per task than humans per session
• Latency budget: p95 matters because agents serialize tool calls; tail latency compounds across multi-step workflows
• Result consumption: Agents parse structured data, not blue links; relevance scoring for human click-through doesn't align with extraction success
• Query patterns: Agents use precise filters (date ranges, domain constraints, schema hints) that humans rarely specify

Traditional search APIs built for human traffic handle agent workloads poorly. Rate limits assume sporadic queries. Pricing tiers penalize high-volume programmatic access. Relevance models optimize for engagement metrics that don't exist in agent contexts.

The 100B-Page Index Decision

Keenable maintains its own crawl and index instead of wrapping Google, Bing, or Brave. This is expensive but unlocks control over:

Crawl strategy: Agents need fresh data on niche domains that human-centric crawlers deprioritize. A proprietary crawl can target high-churn sources (job boards, pricing pages, event listings) and re-crawl on agent-driven schedules rather than PageRank-weighted intervals.

Index schema: Human search indexes optimize for snippet extraction and keyword matching. Agent search needs structured fields (publish date, author, price, location) extracted at index time, not query time. The index becomes a queryable database, not a ranked document store.

Latency control: Owning the index means co-locating query processing with storage. Third-party API wrappers add network hops and rate-limit unpredictability. Keenable reports <250ms p95 in US East because the entire stack runs in the same region.

Cost structure: At scale, API wrapper economics break. If an agent workflow issues 1,000 queries per user session and you're paying $5 per 1K queries to an upstream provider, you're spending $5 per session before any other infrastructure costs. A proprietary index shifts that to fixed crawl and serving costs that amortize across all queries.

The trade-off: you're now responsible for crawl politeness, duplicate detection, spam filtering, and index freshness. You need petabyte-scale storage and distributed query processing. This only makes sense if agent query volume justifies the fixed cost.

SQL-Like Query Interface

Keenable exposes a SQL-like syntax for web queries. Instead of keyword strings, agents construct queries with explicit filters and projections:

SELECT title, price, publish_date 
FROM web 
WHERE domain IN ('example.com', 'another.com') 
  AND publish_date > '2026-08-01' 
  AND content CONTAINS 'API pricing'
ORDER BY publish_date DESC 
LIMIT 10

This changes the orchestration contract. Traditional search APIs return ranked lists; the agent parses snippets and hopes the LLM extracts the right fields. SQL-like queries return structured records where the search engine handles extraction.

Orchestration impact: Agents can issue parallel queries with different filters instead of iterating through paginated results. A pricing comparison agent might run:

queries = [
    "SELECT price FROM web WHERE domain='competitor1.com' AND content CONTAINS 'enterprise plan'",
    "SELECT price FROM web WHERE domain='competitor2.com' AND content CONTAINS 'enterprise plan'",
    "SELECT price FROM web WHERE domain='competitor3.com' AND content CONTAINS 'enterprise plan'"
]
results = await asyncio.gather(*[search_api.query(q) for q in queries])

This parallelizes what would otherwise be sequential LLM calls to parse unstructured snippets.

Failure modes: SQL-like queries fail hard when the schema doesn't match the page structure. If price isn't extracted correctly at index time, the query returns empty results instead of degrading gracefully to keyword matches. You need robust extraction pipelines and fallback strategies.

Benchmarking Agent-Like Queries

Keenable publishes a live benchmark (NEEDLE) that compares search APIs on "agent-like queries" refreshed daily. The methodology matters because it defines what "agent-like" means:

• Query generation: Queries are synthetic but modeled on real agent workflows (fact-checking, price comparison, event discovery)
• Freshness: New queries daily to prevent overfitting; results can't be cached
• Judging: An LLM judge evaluates result quality by comparing against a pooled "ultimate" ranking (best results from all providers combined)
• Metrics: Mean share of ultimate performance (how close each provider gets to the oracle ranking) and cost per 1K queries

The benchmark reveals that no provider dominates on all axes. Keenable trades absolute quality for cost and latency. Perplexity has higher quality but 3x latency and 5x cost.

Benchmark limitations: The LLM judge introduces bias toward providers whose result format the judge parses easily. "Agent-like queries" are still synthetic; real agent workloads have domain-specific patterns the benchmark doesn't capture. Freshness requirements prevent caching, which penalizes providers optimized for repeated queries.

Latency Optimization and State Management

Agent workflows serialize tool calls. If a search query takes 600ms at p95 and the agent issues 10 queries per task, you've added 6 seconds of latency before any LLM inference. Keenable's <250ms p95 target reflects this compounding effect.

Architectural choices for low latency:

• Regional co-location: Index shards, query processors, and API endpoints in the same availability zone
• Pre-computed extractions: Structured fields extracted at index time, not query time
• Query planning: SQL-like queries compile to optimized index scans; the planner avoids full-text search when filters can narrow the candidate set
• Connection pooling: Agents reuse HTTP/2 connections across queries instead of establishing new TLS handshakes

State management: Agents often issue related queries (initial search, refinement, pagination). Keenable doesn't expose session state or query context APIs. Each query is stateless. This simplifies the API but pushes context management to the orchestration layer. The agent must track which queries it's already issued and merge results client-side.

Observability gap: The API doesn't expose query plans or cache hit rates. You can't debug why a query was slow or understand which filters triggered a full index scan. This matters when you're optimizing agent workflows and need to know if a query structure is pathological.

Deployment Shape and Security Boundaries

Keenable is API-only. No self-hosted option, no on-premise deployment. This is a deliberate trade-off:

Advantages:

• No infrastructure management for customers
• Keenable controls the entire stack for latency optimization
• Updates and index refreshes happen transparently

Constraints:

• Data leaves your VPC; you're trusting Keenable with query logs
• No air-gapped deployments for regulated industries
• Rate limits and quotas are opaque; you can't scale beyond what the API allows

Security boundaries: The API uses standard bearer token auth. Query logs are retained for debugging and benchmark generation. If your agent queries contain sensitive context (customer names, internal project codes), those leak to Keenable's logs. You need to sanitize queries or accept the data residency risk.

Rate limiting: 100K free requests per month, then $1 per 1K requests at 100+ RPS. The rate limit is per API key, not per agent or workflow. If you're running multiple agents, you need to implement client-side throttling or request a higher tier.

When the Economics Break

A proprietary index only makes sense at scale. Keenable's bet is that agent query volume will justify the fixed cost of crawling and indexing 100B+ pages. The math:

• Break-even point: If you're issuing <10M queries per month, wrapping an existing API is cheaper. At 10M queries, you're paying $10K/month to Keenable or $20-50K to competitors. The fixed cost of running your own index (crawl, storage, compute) is likely higher unless you're at 100M+ queries per month.

• Crawl freshness: Agents need fresh data. If your use case requires hourly re-crawls of specific domains, a general-purpose index won't cut it. You either build your own crawler or pay Keenable to prioritize your domains (custom pricing).

• Query diversity: If your agents issue the same queries repeatedly, caching dominates and latency matters less. Keenable's architecture optimizes for diverse, non-cacheable queries. If your workload is repetitive, you're paying for infrastructure you don't need.

Likely Failure Modes

Index coverage gaps: 100B pages sounds large but the web is bigger. If your agent needs data from paywalled sites, dynamic JavaScript-rendered pages, or APIs disguised as web content, Keenable's crawler won't reach it. You need fallback strategies (direct API calls, browser automation).

Extraction errors: SQL-like queries depend on accurate field extraction at index time. If the price extractor misparses "$1,000" as "$1", your agent gets wrong data with no error signal. You need validation logic in the orchestration layer.

Rate limit surprises: The 100 RPS tier is per API key. If you're running 10 agents in parallel and each issues 20 queries per second, you hit the limit. The API returns 429 errors but doesn't expose queue depth or retry-after hints. Your orchestration needs exponential backoff and circuit breakers.

Query language brittleness: SQL-like syntax is powerful but fragile. A typo in a field name returns empty results instead of degrading to keyword search. Agents need robust error handling and fallback to simpler query modes.

Technical Verdict

Use Keenable when:

• Your agents issue >1M diverse queries per month and latency compounds across multi-step workflows
• You need structured extraction (prices, dates, locations) and can't afford LLM parsing overhead
• You're building agent infrastructure where $1 per 1K queries is cheaper than your current search API costs
• You can tolerate API-only deployment and data residency outside your VPC

Avoid Keenable when:

• Your query volume is <100K per month (use the free tier but don't build critical paths on it)
• You need air-gapped deployment or strict data residency controls
• Your agents query paywalled or JavaScript-heavy sites that crawlers can't reach
• You need query plan visibility and cache control for latency debugging

The architecture is a bet that agent search is a distinct workload category. If agents become the primary consumers of web data, optimizing for their access patterns (high volume, structured extraction, latency sensitivity) makes sense. If agents remain a niche use case, the fixed cost of a proprietary index won't amortize and API wrappers will stay cheaper.

Source Links

• Keenable
• NEEDLE Benchmark
• Show HN Discussion