Why Resource Loading Matters in Distributed Query Systems
In distributed systems like Milvus, efficient and reliable resource management is critical. Analyzer files—key resources used during query processing—must be fetched, validated, and kept consistent across multiple components. Failures or stale data in this process can cause query errors, inconsistent results, or even system instability. This article explores a recent Milvus improvement that overhauled how analyzer file resources are prepared, moving from legacy RPCs and local caches prone to transient failures, to a robust streaming client with centralized metadata management.
The Problem with Legacy Resource Loading
Previously, Milvus fetched analyzer file resources using older RPC mechanisms and local caching strategies. These legacy approaches suffered from several issues:
- No batch fetching: Resources were requested individually, increasing overhead and latency.
- Lack of centralized metadata: Without a single source of truth, metadata could become inconsistent or stale.
- Poor concurrency coordination: Multiple components might redundantly download the same resource without coordination.
- Inadequate retry and expiration handling: Transient IO errors often caused failures, and caches lacked time-to-live (TTL) policies, leading to stale data.
These limitations led to validation failures in the query coordinator and runners, inconsistent resource states, and unreliable query execution.
The New Streaming Client and AnalyzerResourceManager
The core of the fix is introducing a streaming client that supports a new batch FetchFileResources RPC. This API can fetch multiple resource metadata and content items by their IDs or names in a single request, dramatically improving efficiency.
Centralized Metadata Management
The AnalyzerResourceManager acts as the central authority for managing resource metadata. It:
- Fetches metadata in batches via the streaming client.
- Coordinates concurrent download requests to avoid duplicate work.
- Implements shared cancellation so that if one component cancels, others depending on the same resource are notified.
Robust Download and Caching
Downloads are retried locally on transient failures, reducing the chance of query errors due to momentary network glitches. Files are cached with TTL-based expiration, ensuring that stale data is refreshed proactively.
Validation Integration
Validation logic is routed through the streaming client, which verifies resource integrity against the fetched metadata. This ensures that corrupted or incomplete resources are detected early, with precise status errors returned to the caller.
Runner Lifecycle Improvements
Runners—components that execute queries—now pin exact resource bindings and include schema IDs in their identity. This guarantees that the runner's resource environment is stable and consistent throughout its lifecycle.
Additionally, if a runner is unavailable, it can be prepared on demand using retry logic after releasing its admission scope. This ensures that resources are always available during write-ahead log (WAL) replay and query execution.
Lessons Learned
This change highlights several important principles for distributed resource management:
- Batching and streaming reduce overhead: Fetching multiple resources in a single streaming RPC is more efficient than many individual requests.
- Centralized metadata prevents inconsistency: A single source of truth for resource metadata avoids stale or conflicting information.
- Coordinated concurrency avoids redundant work: Managing concurrent downloads and cancellations reduces wasted network and computation resources.
- Explicit lifecycle management improves stability: Pinning resources to runner lifecycles and including schema IDs ensures consistent execution environments.
- Robust retry and caching policies reduce transient failures: Handling IO errors locally and using TTL caches minimizes query disruptions.
Together, these improvements make Milvus's distributed query execution more reliable and performant.
This article is based on the PR "fix: prepare analyzer file resources through streaming client" merged into the milvus-io/milvus repository. Read the full breakdown on nossletter.tech.
Originally published on Nossletter.
