Semantic Scholar vs Valyu: Which Research API Should Your Agent Call?

If you are building a research agent, the first real decision you make is what comes back from the retrieval call. A paper record, or the paragraph inside the paper that answers the question. Those are different products, and picking the wrong one shows up three weeks later as a RAG pipeline you did not plan to build.

Valyu is a search and DeepResearch API that retrieves full text with structured citations across academic, clinical, scientific, patents & regulatory sources. Semantic Scholar is an academic search engine and metadata API from the Allen Institute for AI.

Quick answer

Pick Valyu when the unit of work is a passage of evidence: search inside papers at query time, pull a clinical trial protocol and an FDA label alongside the literature, get a cited answer or a full multi-step report without operating your own retrieval stack.

Pick Semantic Scholar when the unit of work is a record: find papers by topic, walk a citation graph, rank by influence, pull author profiles, get recommendations. It is free, the corpus is enormous, and nothing else gives you that citation graph as cleanly.

The short version in code:

# Valyu returns the text that answers the question, with the citation attached
{"title": "...", "content": "In the phase 3 cohort (n=847), median PFS was...",
 "citation": {"doi": "10.1056/...", "authors": [...], "fragment": "#:~:text=median%20PFS"}}

# Semantic Scholar returns records you then have to go and read
{"paperId": "649def34...", "title": "...", "abstract": "...", "citationCount": 1893}

What can Semantic Scholar index?

The Semantic Scholar Academic Graph covers 214 million papers, 2.49 billion citations, and 79 million authors. Coverage spans every field, assembled from publisher feeds, preprint servers, and web crawling.

The API is organised as three services:

Full text is not a query-time product here. It lives in S2ORC: 8 million-plus full-text papers, alongside 81 million paper nodes and 73 million abstracts, distributed as a bulk download. There is also a snippet search endpoint over open-access papers, which returns short extracts rather than the retrieval-depth passages a RAG pipeline usually wants.

So: metadata and abstracts at query time, full text as a corpus you host yourself.

What can Valyu index?

Valyu full-text-indexes roughly 4 million open-access papers, plus licensed journal content:

PubMed's complete 37 million-record abstract corpus is available as an opt-in via include_abstracts. More on that flag below, because it is the single most commonly misreported detail about this API.

Beyond the literature, the index covers ClinicalTrials.gov (500K+ trials), FDA drug labels from DailyMed (150K+), SEC filings (3M+), USPTO patents (8M+) and EPO patents (6M+) with full text and figures, plus genomics and chemistry sources. One query can span several of those at once, which is the part that matters if your agent needs to go from a mechanism in a paper to the trial testing it to the label of the approved drug.

Round 1: Finding papers

Both do discovery. Here is Semantic Scholar, rewritten from the docs example as something you would actually put in an agent rather than an interactive prompt loop:

import os
import requests

S2_BASE = "https://api.semanticscholar.org"
HEADERS = {"X-API-KEY": os.environ["S2_API_KEY"]}


def search_papers(query: str, limit: int = 10):
    r = requests.get(
        f"{S2_BASE}/graph/v1/paper/search",
        headers=HEADERS,
        params={
            "query": query,
            "limit": limit,
            "fields": "title,abstract,year,citationCount,externalIds,url",
        },
        timeout=30,
    )
    r.raise_for_status()
    return r.json().get("data", [])


def recommendations(paper_id: str, limit: int = 10):
    r = requests.get(
        f"{S2_BASE}/recommendations/v1/papers/forpaper/{paper_id}",
        headers=HEADERS,
        params={"fields": "title,year,citationCount,url", "limit": limit},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()["recommendedPapers"]


for p in search_papers("chimeric antigen receptor T cell exhaustion"):
    print(f"{p['citationCount']:>6}  {p['year']}  {p['title']}")

The fields parameter is the thing to learn first. Ask for nothing and you get almost nothing back, and every extra field is a join on their side, so keep the list tight.

For anything above a few thousand results, use bulk search with its continuation token instead of paging the relevance endpoint:

import json
import requests

url = "https://api.semanticscholar.org/graph/v1/paper/search/bulk"
params = {"query": "(cold -temperature) | flu", "fields": "title,year", "year": "2023-"}

retrieved = 0
with open("papers.jsonl", "a") as f:
    while True:
        r = requests.get(url, params=params, timeout=60).json()
        for paper in r.get("data", []):
            print(json.dumps(paper), file=f)
        retrieved += len(r.get("data", []))
        if "token" not in r:
            break
        params["token"] = r["token"]

print(f"Retrieved {retrieved} papers")

Note the query syntax on the bulk endpoint: | is OR, a leading - negates, and parentheses group. It is not the same syntax as the relevance search endpoint, which trips people up.

Valyu does discovery too, over a smaller but full-text index:

import os
from valyu import Valyu

# Reads VALYU_API_KEY from the environment when no key is passed
valyu = Valyu(api_key=os.environ["VALYU_API_KEY"])

response = valyu.search(
    "Phase 3 melanoma immunotherapy trials",
    search_type="proprietary",
    included_sources=["valyu/valyu-pubmed", "valyu/valyu-clinical-trials"],
    max_num_results=15,
    response_length="large",  # full methodology and results, not just abstracts
)

for result in response.results:
    print(result.title, result.url)
    # content is str | list | dict; structured sources return objects
    print(result.content)

There are TypeScript and Rust SDKs if Python is not your stack, and the REST endpoints are there if you would rather not add a dependency at all.

Round 2: Getting the actual full text

Semantic Scholar. To search inside papers you download S2ORC, chunk it, embed it, store it, and query your own index. That is a real pipeline: object storage, an embedding job, a vector database, and a refresh strategy when the corpus updates. Perfectly reasonable if you want control over chunking and embeddings, and genuinely the right call for some teams. It is just not a thing you get from an API call.

Valyu. Full text is the default return value:

response = valyu.search(
    "mechanisms of acquired resistance to KRAS G12C inhibitors",
    search_type="proprietary",
    included_sources=["valyu/valyu-pubmed"],
    max_num_results=10,
    response_length="large",
)

for r in response.results:
    print(r.title)
    print(r.content[:500])          # relevant full-text chunks, not the abstract
    print(r.citation.doi, r.citation.fragment)  # fragment deep-links the passage

The fragment field is worth calling out separately. It is a text-fragment deep link to the exact cited passage, so a reviewer can click a citation in your agent's output and land on the sentence rather than the paper. If anyone is ever going to audit what your agent claimed, that field is the difference between "reviewable" and "take my word for it".

The include_abstracts Gotcha

• Default: PubMed search returns papers for which full text is available, giving you the abstract plus the most relevant chunks of full text.
• include_abstracts=true: expands the search to PubMed's complete 37 million-record abstract corpus, where papers without full text return their abstract only.

Full-text-first is the default. The complete abstract corpus is the opt-in. If you have read the opposite somewhere, that is the error.

# Deep evidence, narrower net (default)
valyu.search(query, included_sources=["valyu/valyu-pubmed"])

# Wide net, shallower evidence for the papers that lack full text
valyu.search(query, included_sources=["valyu/valyu-pubmed"], include_abstracts=True)

Use the default for evidence synthesis. Flip the flag for coverage sweeps and systematic-review-style screening, where missing a paper is worse than only having its abstract.

Round 3: Synthesis

The Semantic Scholar API has no answer-generation endpoint. Ai2 does ship Ai2 Scholar QA separately: an open-source cited-synthesis system over 11 million-plus full-text papers and 100 million-plus abstracts, available as a Docker app, an async API, or a Python package. You bring your own Semantic Scholar, Anthropic, and OpenAI keys, and you host it. It is a good piece of software. It is also infrastructure you now operate.

Valyu ships synthesis as managed API surface. The Answer API returns a cited answer in one call. DeepResearch runs autonomous multi-step investigation: planning, searching, extraction, fact verification, and report writing.

import os
from valyu import Valyu

valyu = Valyu(api_key=os.environ["VALYU_API_KEY"])

task = valyu.deepresearch.create(
    query=(
        "What is the current evidence that GLP-1 receptor agonists reduce "
        "major adverse cardiovascular events in patients without diabetes? "
        "Cover trial design, effect sizes, and where the evidence conflicts."
    ),
    mode="standard",
    search={
        "search_type": "proprietary",
        "included_sources": ["academic"],  # arXiv, PubMed, bioRxiv/medRxiv, ChemRxiv
        "start_date": "2021-01-01",
    },
    research_strategy=(
        "Prioritise randomised controlled trials and systematic reviews over "
        "observational studies. Separate primary endpoints from secondary and "
        "post-hoc analyses. Flag any conflicting or null results explicitly."
    ),
    report_format=(
        "Structured review with: evidence summary table (trial, n, population, "
        "endpoint, effect size, CI), narrative synthesis, conflicting findings, "
        "and evidence gaps."
    ),
    output_formats=["markdown", "pdf"],
)

result = valyu.deepresearch.wait(task.deepresearch_id)

if result.status == "completed":
    print(result.output)
    print("cost:", result.cost)
    for s in result.sources:
        print(f"{s.title} | {s.doi or s.url}{s.fragment or ''}")

research_strategy and report_format are the two parameters that do the most work. They are where you encode the methodology a domain expert would apply, and they are the difference between a report you can hand to someone and a wall of summarised abstracts.

Point it at a single dataset when you know exactly where the evidence lives, and ask for a spreadsheet instead of prose:

task = valyu.deepresearch.create(
    query=(
        "Summarise reported mechanisms of acquired resistance to KRAS G12C "
        "inhibitors in non-small-cell lung cancer, with supporting evidence "
        "for each mechanism."
    ),
    mode="fast",
    search={
        "search_type": "proprietary",
        "included_sources": ["valyu/valyu-pubmed"],
        "start_date": "2022-01-01",
    },
    deliverables=["xlsx"],           # mechanism-by-evidence table
    tools={"code_execution": True},  # required for xlsx/pptx/docx
)

DeepResearch also supports webhooks and human-in-the-loop checkpoints, which is what you want when a run takes minutes rather than milliseconds and you do not want to hold a request open.

Rate limits and cost, the part that bites in production

Valyu is usage-based, priced per thousand results by source type, with max_price as a per-query spend cap. Sources priced above the cap are excluded and the response returns a 206 partial-success warning rather than a surprise on the invoice. Free credits are available to start, $10 without a card and $20 with a work email.

Semantic Scholar is free, which is good. The limits are the catch, and they are widely misread:

• Unauthenticated: 1,000 requests per second shared collectively across every unauthenticated user on the planet, with additional throttling under heavy use. That number looks generous and is not yours.
• With an API key: a dedicated 1 request per second across all endpoints. Higher rates are available following review.

One request per second is a hard ceiling on any fan-out design. Batch endpoints and bulk search exist precisely because of it, and you should build around them from day one rather than discovering the limit in production.

Feature comparison

Which workflows fit each

Semantic Scholar fits work where the graph is the point:

• Bibliometrics and influence ranking
• Forward and backward citation chasing
• Building a candidate set for a systematic review
• Author disambiguation and profile lookups
• Recommendation and "more like this" features
• Any offline corpus work where a bulk download beats an API

Valyu fits work where the evidence is the point:

• Humans and Agents that need to quote and cite, not just link
• Cross-domain runs spanning literature, trials, labels, and filings
• RAG without owning a RAG stack
• Recurring literature reviews and competitive or regulatory monitoring
• Any output a human will audit, thanks to passage-level provenance

What the same query returns from each

Identical input, different object types, which is the clearest way to see the design split.

A topic search for "melanoma immunotherapy" on Semantic Scholar returns paper records: titles, abstracts, citation counts, author links, external IDs. The same query on Valyu returns full-text passages with citations attached to each one.

Ask "which papers cite this one" and Semantic Scholar answers directly through /paper/{id}/citations. Valyu has no equivalent, because it is not a graph.

Ask "what does the evidence say about X" and Valyu answers with passages, or with a cited answer if you call the Answer API. Semantic Scholar returns candidate records for you to go and read.

This is a difference in design goal, not in quality. One is built for graph discovery. The other is built for evidence retrieval.

Integrations

Semantic Scholar is a REST API returning JSON, with community client libraries in most languages. That is the whole integration story, and for many teams it is enough.

Valyu ships a hosted MCP server, a CLI, a Claude Code plugin, agent skills, and framework integrations for the Vercel AI SDK, LangChain, LlamaIndex, AWS Bedrock AgentCore, and n8n, plus tool definitions for Anthropic, OpenAI, and Google. If you are wiring retrieval into an existing agent framework, that is usually a config change rather than a client to write.

For a sense of scale in production: RevisionDojo uses Valyu to deliver academic research to more than 450,000 students, integrating the JavaScript SDK and the Valyu AI SDK across search, citation discovery, and structured literature reviews for IB Extended Essays.

FAQ

Does Semantic Scholar give me full text?

Not at query time. Full text ships as the S2ORC bulk dataset, 8 million-plus full-text papers alongside 81 million paper nodes and 73 million abstracts, which you download and index yourself. There is also a snippet search endpoint over open-access papers that returns short extracts.

Does Semantic Scholar offer answer generation?

Not in the API. Ai2 separately ships Ai2 Scholar QA, an open-source cited-synthesis system over 11 million-plus full-text papers and 100 million-plus abstracts, available as a Docker app, async API, or Python package. It requires your own Semantic Scholar, Anthropic, and OpenAI keys, and you host it.

Does Valyu return abstracts for every PubMed record by default?

No, and this is commonly stated backwards. PubMed search defaults to papers with full text available, returning the abstract plus relevant full-text chunks. Set include_abstracts=true to expand to the complete 37 million-record abstract corpus, where papers without full text return their abstract.

How many academic papers does Valyu index?

Roughly 4 million full-text open-access papers: PubMed 2.5M, arXiv 1M, bioRxiv 350K, medRxiv 80K, ChemRxiv 8K, plus licensed journal content. PubMed's complete 37 million-record abstract corpus is available via include_abstracts.

What is the difference between Search, Answer, DeepResearch, and Workflows?

Search returns structured results. Contents extracts clean content from URLs. Answer adds cited answer generation on top of search. DeepResearch runs autonomous multi-step investigation with file deliverables. Workflows are templated, versioned DeepResearch runs for repeatable work.

Does Valyu cover clinical trials outside the United States?

Coverage currently centres on US trials and FDA-approved drugs, with 24 to 48 hour update delays on trial data.

Can I use both in the same pipeline?

Yes, and it is usually the right answer. Semantic Scholar identifies which works matter through the citation graph, and Valyu retrieves the full-text passages that support specific claims. See the code above.

Do preprints count as peer-reviewed papers?

No. Preprints from arXiv, bioRxiv, medRxiv, and ChemRxiv are not peer-reviewed, and both services index them alongside peer-reviewed articles. Filter on source or publication type when evidence quality matters.