I Split One Fact Across Two Files to Test If RAG Retrieval Actually Works

Building a Knowledge Base from Scratch, EP02

The promise from last time

Last episode I ran three experiments on the "paste a file into chat" habit: half a document and the model improvises, the wrong edition and it waves you through, a new window and it remembers nothing. I promised that next time we'd build a real knowledge base. This is that episode.

The setup: a fictional small store ("Cozy Home," home goods and small appliances) with three Markdown support FAQs: shipping policy, returns & warranty, payments & invoices. Each file is 1-2KB. Plus one deliberate trap.

The trap

I split the cash-on-delivery information across two files. Which regions support COD lives in the shipping policy. The 3 RMB collection fee lives in the payments doc. This mirrors how real companies operate: ops writes shipping rules, finance writes payment rules, and nobody answers for the other document's completeness.

If retrieval works the way RAG claims, asking "does COD cost extra?" should surface both files at once. If it's just fancy keyword matching, it won't.

Building the library

Honest disclosure first: there is no CLI command for creating the knowledge base. The bl CLI's knowledge group currently has exactly one command, retrieve. Ingesting documents happens in the web console: standard edition, file connector, drag in three files, wait for the status to flip to active. Ten minutes, zero code. The store owner could have done it.

The console hands you a knowledge base ID (mine was zj0knmrbye). Every retrieval command points at it.

The retrieval

bl knowledge retrieve --index-id zj0knmrbye --query "新疆可以货到付款吗"

The query is Chinese because the documents are Chinese. The CLI doesn't care. What comes back is JSON:

{
  "data": {
    "total": 3,
    "nodes": [
      {
        "score": 0.8299149870872498,
        "metadata": {
          "doc_name": "01-发货与物流政策",
          "content": "……货到付款仅支持华东、华南主要城市。偏远地区（新疆、西藏等）不支持货到付款，请选择在线支付。货到付款订单需额外支付 3 元代收服务费。……"
        }
      },
      {
        "score": 0.48945897817611694,
        "metadata": { "doc_name": "03-支付与发票说明" }
      }
    ]
  }
}

score is the relevance ranking. doc_name tells you which file the answer came from. content is the original passage, verbatim, not a processed conclusion.

Five queries, three findings

The humidifier query had no keyword overlap with the docs at all. It hit at 0.75, returning 「电器类商品（如加湿器、小夜灯）提供 1 年保修，非人为损坏免费维修或换新」, which translates to "electrical products (humidifiers, night lights) carry a 1-year warranty, free repair or replacement for non-artificial damage." The query says "broken"; the passage says "non-artificial damage." That's semantic retrieval, not string matching.

The coupon query returned exactly one result out of a three-document library: 「新人券：首单满 59 减 10，仅限新注册会员，不可与其他优惠叠加」, "the newcomer coupon cannot be combined with other offers." "Stack with" matched "combined." The other two docs weren't dragged in to pad the answer. If you're wiring this into a downstream system, don't assume the nodes array has a fixed length.

The COD fee query was the payoff. Shipping policy at 0.82, payments doc at 0.77. Shipping returned 「货到付款仅支持华东、华南主要城市（上海、杭州、南京、广州、深圳、厦门等）」, "COD is supported in major east and south China cities only." Payments returned 「需额外支付 3 元代收服务费」, "an extra 3 RMB collection fee applies." The retrieval assembled both without being told where anything lived. That's the entire value proposition of RAG in one response: you don't need to know which file the answer lives in.

Three small observations for builders

Small documents don't get chunked. Each KB-scale FAQ came back as one whole slice. Bigger documents will get sliced, and where the cuts land determines retrieval quality. That's the next episode's topic.

No model selection. The command has no --model flag. The knowledge base ships with its retrieval model built in.

The output is JSON by default. No extra flags. jq one-liner gets you the top hit's content. Wiring retrieval into an existing system costs close to nothing.

Honest limits

The library was tiny (three files), so chunking never came up, which is the best case. Retrieval returns raw passages, not phrased answers; don't paste them straight into customer replies. And score is a within-query ranking signal, not a percentage: in my runs, 0.54 was the only correct answer to one question while 0.49 was a bystander in another. Never filter by absolute score.

Try it

npm install -g bailian-cli
bl auth login --api-key sk-xxxxx

Write three Markdown files from your own domain, build the library in the console (ten minutes), copy the index ID, and run one retrieval. Then ask a question whose answer spans two of your files. Watch what comes back.

All five retrievals ran for real on the Bailian CLI against a Model Studio knowledge base. Commands and full JSON responses are kept in the project repo. CLI install: Bailian CLI docs. API key: get one free.