Why a Core Storage Library Matters for Primary Key Indexing
When working with large-scale vector databases like Milvus, efficient and consistent primary key indexing is crucial. Primary keys serve as the backbone for data retrieval, and mishandling their storage can lead to fragmented logic, duplicated code, and performance bottlenecks. Before this enhancement, Milvus lacked a unified core storage library for primary key indexes, resulting in scattered implementations and maintenance challenges.
This article dives into how introducing a dedicated core storage library for primary key indexes addresses these issues, streamlines storage lifecycle management, and sets the stage for incremental improvements without disrupting existing workflows.
The Problem: Fragmented Storage Logic for Primary Keys
In Milvus’s architecture, primary key indexes are essential for locating vector data efficiently. However, prior to this enhancement, there was no shared, core storage library dedicated to managing these indexes. Instead, primary key handling was embedded in various nodes or test-only memory engines.
This fragmentation led to several issues:
- Duplicated Storage Logic: Different components implemented their own versions of primary key storage and retrieval, increasing maintenance overhead.
- Inconsistent Behavior: Without a unified approach, lifecycle management of storage layers, such as LSM-tree generations, was handled inconsistently.
- Limited Efficiency: Embedding logic in nodes or memory engines limited the ability to optimize write paths and manage storage transitions smoothly.
The root cause was clear: the absence of a dedicated, shared library meant no single source of truth or standardized mechanism for primary key storage.
The Solution: Introducing internal/pkindex Core Library
To tackle these challenges, the PR introduces a new core storage library named internal/pkindex. This library encapsulates all primary key index storage logic, providing a consistent, efficient, and reusable foundation.
Key Components of the Library
- Order-Preserving Encoding of Primary Keys
The library implements an encoding scheme that preserves the order of primary keys. This is critical for range queries and efficient lookups, ensuring that the storage engine can maintain sorted keys without additional overhead.
- SSTable Read/Write via Pebble
The library leverages Pebble (version 1.1.5), a performant key-value store inspired by LevelDB, to manage SSTables (Sorted String Tables). By integrating Pebble, the library handles persistent storage on local files, enabling robust and efficient disk-backed storage.
- Engine Managing Write Paths and Lifecycle Transitions
One of the standout features is an engine within the library that manages the write path and lifecycle transitions of LSM (Log-Structured Merge) tree generations. Crucially, it does so without introducing write pauses, enabling smooth compaction and generation transitions.
- Error Categorization and Backward Compatibility
The library categorizes errors to facilitate better error handling and debugging. It also maintains backward-compatible error message formatting, ensuring existing components that rely on previous formats continue to function without disruption.
How It Works in Practice
Imagine a policy that needs to write primary key data as part of indexing. Instead of each node or engine implementing its own storage logic, it now calls into internal/pkindex. The library encodes the keys preserving their order, writes them to SSTables using Pebble, and manages the underlying lifecycle transitions automatically. This separation allows for incremental adoption; components can start using the library without needing to overhaul their entire storage logic.
Lessons Learned: The Power of Separation and Incremental Adoption
This enhancement underscores the importance of isolating core storage logic into a dedicated library. By doing so, Milvus achieves:
- Consistency: Uniform handling of primary key storage and lifecycle management across components.
- Maintainability: Reduced code duplication and clearer separation of concerns.
- Performance: Efficient write paths and seamless lifecycle transitions without write pauses.
- Flexibility: The ability to incrementally adopt the new library across the system without breaking existing behavior.
In complex distributed systems, especially those dealing with storage and indexing, having a well-defined core library not only simplifies development but also lays a strong foundation for future improvements.
This article is based on the recent pull request merged into the Milvus repository. Read the full breakdown on nossletter.tech.
Originally published on Nossletter.
