Valkey 9.0 ‘unlocks’ atomic slot migration & straddles real-time workloads
Valkey is an open source key-value database that resides under the Linux Foundation. It is built for high performance, horizontal scalability and observability.
As nicely defined by AWS, a key-value database is a type of non-relational database (also known as NoSQL database) that uses a simple key-value method to store data.
“It stores data as a collection of key-value pairs in which a key serves as a unique identifier. Both keys and values can be anything, ranging from simple objects to complex compound objects. Key-value databases (or key-value stores) are highly partitionable and allow horizontal scaling at a level that other types of databases cannot achieve,” explains AWS.
Valkey today (valkey.io) supports a variety of data structures and workloads, from caching and messaging to primary in-memory storage, all under a permissive BSD license. The latest version offers horizontal scaling enhancements that reduce downtime, memory usage and the time needed to scale, making large database deployments faster and more resilient.
Atomic slot migration
This version introduces expiration dates for hash fields, atomic slot migration and multiple databases in cluster mode, which fortify Valkey’s use at scale.
The team explain atomic slot migration by saying that, “Atomic slot migrations fundamentally changes how Valkey migrates data from node-to-node inside the cluster. Prior to Valkey 9.0, data was migrated in the cluster key-by-key. This approach works for most situations, but corner cases can lead to degraded performance, operational headaches, and, at worst, blocked node migrations and lost data.”
A Valkey 9.0 cluster can support over 1 billion requests per second, delivering on new capabilities for scalability, reliability, and efficiency in distributed data environments, all while reducing cost and overhead for engineering teams.
Valkey 9.0 continues the project’s mission to provide a fully open, community-driven in-memory data store that evolves with modern application needs. When compared with version 8.1, users of Valkey 9.0 experience up to 40% more throughput in their applications of the project.
This version represents what the team promise is “significant progress” in horizontal scalability and compute efficiency – areas that have traditionally challenged distributed systems.
“One of our goals is to make Valkey as resilient as possible, so that our users can depend on it for serving their application traffic.” said Madelyn Olson, maintainer of Valkey and principal software engineer at AWS. “Valkey 9.0 is a major step forward in that direction with larger and more stable clusters that can more easily scale to meet user demands.”
Valkey reshards data during horizontal scaling operations using slot migration. In the past, this process led to errors, processing overhead, and data loss for operators and clients during large key movements. Atomic slot migration, included in Valkey 9.0, is a major breakthrough in reliability and performance, ensuring horizontal scaling operations are seamless and error-free.
Forks & streams
Valkey’s atomic slot migration solves this by introducing a snapshot-based migration process that is triggered by a single command.
A “child process” first forks and streams data incrementally from the source to the target node, allowing both nodes to remain active throughout migration. Once all data is migrated, Valkey performs an atomic handoff and clients are redirected instantly to the target node, all without downtime or errors previously encountered in traditional slot migration.
Valkey 9.0’s numbered databases are highly scalable and can be spread over a breadth of clusters, all with zero overhead when unused.
