BACKGROUND

Active-active storage systems enable data storage processors (also referred to herein as “storage nodes”) in high-availability (HA) clusters to have exclusive or shared input/output (IO) access to storage objects such as volumes, logical units (LUs), file systems, or other storage resources. Active-active storage systems typically employ locks to control and synchronize such storage access by peer nodes in the HA clusters. Each lock is associated with a logical block address (LBA) of a storage object. Before writing/reading data of a write/read IO operation to/from an LBA of a storage object, a storage node in an HA cluster requests a lock for accessing the LBA of the storage object. Such a lock is generally defined as a lock object, which contains a waiting queue and a lock state for the lock. The storage node sends a remote procedure call (RPC) message to a remote peer node in the HA cluster to synchronize the lock state with the remote peer node, thereby guaranteeing coherency of data in the HA cluster. Based on a type of the requested lock (e.g., exclusive or shared), either the lock is granted immediately, or the lock request is placed in the waiting queue until the lock becomes available.

SUMMARY

Synchronizing lock states between storage nodes in high-availability (HA) clusters using RPC messaging can be problematic due to increased latency that can result from RPC messaging. For example, at least four waiting periods can be associated with RPC messaging, namely, (i) a first waiting period, in which a remote peer node waits for a next cycle of a polling process to recognize an RPC message from a storage node, (ii) a second waiting period, in which the remote peer node waits for a request contained in the RPC message to be serviced, (iii) a third waiting period, in which the storage node waits for a next cycle of a polling process to recognize an RPC reply from the remote peer node, and (iv) a fourth waiting period, in which the storage node waits for the RPC reply to be serviced. However, such waiting periods associated with RPC messaging can contribute to increasing not only an amount of time required to acquire a lock, but also an overall IO latency in an HA cluster.

Techniques are disclosed herein for providing an RPC-less locking mechanism based on RDMA CAW for a storage cluster with an active-active architecture. In the disclosed techniques, locks can be associated with logical block addresses (LBAs) of pages of storage objects such as volumes, logical units (LUs), file systems, and so on. Each lock can be defined as a lock object, which can contain at least a waiting queue and a reference to an atomic lock state (ALS) object. Each ALS object can include at least three fields, namely, (i) a target object reference field for a logical block address (LBA) of a page of a storage object, (ii) an ownership field that specifies one or more storage nodes as owners of the lock, and (iii) a lock type field that specifies a type of the lock (e.g., exclusive or shared). A plurality of ALS objects can be maintained in an ALS object array on memory visible to at least a storage node and a remote peer node in a high-availability (HA) cluster. In the disclosed techniques, the storage node and the remote peer node can be configured to communicate over a remote direct memory access (RDMA) channel. Further, each allocated ALS object in the ALS object array can be maintained on a region of memory registered for remote direct memory access over the RDMA channel.

The disclosed techniques can include, in response to receipt at the storage node of a request for a lock (a “lock request”) to access an LBA of a storage object, synchronizing, between the storage node and the remote peer node, a state of the lock (a “lock state”) for the LBA. The synchronizing of the lock state can include locating or allocating an ALS object having a target object reference field containing the LBA of the storage object, updating an ownership field of the ALS object to include the storage node, and setting a lock type field of the ALS object to a type of lock specified in the lock request. In the disclosed techniques, if the storage node locates the allocated ALS object in an ALS object array maintained on visible memory of the remote peer node, then the storage node can update and/or set corresponding fields of the ALS object by performing atomic compare-and-write (CAW) operations supported by the RDMA channel between the storage node and the remote peer node. By synchronizing, between a storage node and a remote peer node in an HA cluster, a state of a lock for an LBA of a storage object by performing atomic CAW operations over an RDMA channel, the use of RPC messaging between the storage node and the remote peer node can be reduced. As a result, an amount of time required to acquire the lock, as well as an overall IO latency in the HA cluster, can also be reduced.

In certain embodiments, a method of locking an entity of a storage object in a storage cluster with an active-active architecture includes maintaining a lock state object on a memory of a first storage node. The lock state object includes a first field for a lock state of an entity of a storage object, and the memory is registered for remote direct memory access (RDMA). The method further includes, in response to receipt at a second storage node of a request for a lock to access the entity of the storage object, accessing, by the second storage node, the lock state object by performing a first RDMA operation between the first storage node and the second storage node, and updating, by the second storage node, the first field of the lock state object by performing a second RDMA operation between the first storage node and the second storage node, thereby synchronizing the lock state between the first storage node and the second storage node.

In certain arrangements, the performing of the first RDMA operation between the first storage node and the second storage node includes performing a remote compare-and-write (CAW) operation against a mutex value associated with a lock state object bucket including the lock state object.

In certain arrangements, the performing of the second RDMA operation between the first storage node and the second storage node includes setting the first field of the lock state object to a value indicating that at least the second storage node is an owner of the lock.

In certain arrangements, the accessing of the lock state object includes allocating the lock state object on the memory of the first storage node by performing a third RDMA operation between the first storage node and the second storage node.

In certain arrangements, the lock state object includes a second field, and wherein the performing of the third RDMA operation between the first storage node and the second storage node includes setting the second field of the lock state object to a value of the entity of the storage object.

In certain arrangements, the lock state object includes a third field, and the accessing of the lock state object includes performing a fourth RDMA operation between the first storage node and the second storage node. The fourth RDMA operation includes setting the third field of the lock state object to a value indicating a type of lock specified by the request.

In certain arrangements, the lock state object includes a second field, and the accessing of the lock state object includes locating the lock state object on the memory of the first storage node by performing a third RDMA operation between the first storage node and the second storage node.

In certain arrangements, the performing of the third RDMA operation between the first storage node and the second storage node includes inspecting the first field of the lock state object to determine that at least the second storage node is the owner of the lock.

In certain arrangements, the inspecting of the first field of the lock state object includes determining that the second storage node is the sole owner of the lock.

In certain arrangements, the method further includes releasing, by the second storage node, the lock by performing a fourth RDMA operation between the first storage node and the second storage node.

In certain arrangements, the lock state object includes a second field having a value of the entity of the storage object, and the performing of the fourth RDMA operation between the first storage node and the second storage node includes replacing the value of the entity in the second field of the lock state object with a predetermined unspecified entity value.

In certain arrangements, each of the first storage node and the second storage node is the owner of the lock, the releasing of the lock includes performing a fifth RDMA operation between the first storage node and the second storage node, and the performing of the fifth RDMA operation includes updating the first field of the lock state object to a value indicating that the first storage node is the sole owner of the lock.

In certain arrangements, the lock includes a waiting queue and a reference to the lock state object, the waiting queue is on the memory of the first storage node, and the waiting queue includes a process thread waiting for the lock. The method further includes registering the lock state object with a poller on the first storage node, and periodically polling, by the poller, the lock state object to detect the updating of the first field of the lock state object to the value indicating that the first storage node is the sole owner of the lock.

In certain arrangements, the method further includes, having detected the updating of the first field of the lock state object, providing a notification of the updating to the process thread in the waiting queue.

In certain embodiments, a system for locking an entity of a storage object in a storage cluster with an active-active architecture includes a memory and processing circuitry configured to execute program instructions out of the memory to maintain a lock state object on a memory of a first storage node. The lock state object includes a first field for a lock state of an entity of a storage object, and the memory is registered for remote direct memory access (RDMA). The processing circuitry is further configured to execute the program instructions out of the memory, in response to receipt at a second storage node of a request for a lock to access the entity of the storage object, to access, by the second storage node, the lock state object by performing a first RDMA operation between the first storage node and the second storage node, and to update, by the second storage node, the first field of the lock state object by performing a second RDMA operation between the first storage node and the second storage node, thereby synchronizing the lock state between the first storage node and the second storage node.

In certain arrangements, the processing circuitry is further configured to execute the program instructions out of the memory to perform the first RDMA operation between the first storage node and the second storage node, including performing a remote compare-and-write (CAW) operation against a mutex value associated with a lock state object bucket including the lock state object.

In certain arrangements, the processing circuitry is further configured to execute the program instructions out of the memory to perform the second RDMA operation between the first storage node and the second storage node, including setting the first field of the lock state object to a value indicating that at least the second storage node is an owner of the lock.

In certain embodiments, a computer program product includes a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method of locking an entity of a storage object in a storage cluster with an active-active architecture. The method includes maintaining a lock state object on a memory of a first storage node. The lock state object includes a first field for a lock state of an entity of a storage object, and the memory is registered for remote direct memory access (RDMA). The method further includes, in response to receipt at a second storage node of a request for a lock to access the entity of the storage object, accessing, by the second storage node, the lock state object by performing a first RDMA operation between the first storage node and the second storage node, and updating, by the second storage node, the first field of the lock state object by performing a second RDMA operation between the first storage node and the second storage node, thereby synchronizing the lock state between the first storage node and the second storage node.

Other features, functions, and aspects of the present disclosure will be evident from the Detailed Description that follows.

