Organizations use storage virtualization systems to manage data objects stored on multiple network attached storage (“NAS”) systems. A storage virtualization system can migrate data objects from one NAS system (the “source,” “source server” or “src”) to another NAS system (the “destination,” “destination server” or “dst”). A storage virtualization system can also synchronize data objects between two NAS systems. In order to perform these operations, a storage virtualization system will typically be placed as an intermediary between the NAS system that stores the data objects and the client computer that accesses or requests access to the stored data objects. The client computer, storage virtualization system and the associated NAS system may be connected over a network. The storage virtualization system may begin performing its migration or synchronization operations once a second NAS system is added to the network. Storage virtualization provides a mechanism for real-time or near real-time data object management without requiring a shutdown or disconnection of NAS resources, or any intervention from users. As such, storage virtualization provides an effective and efficient way to manage data objects stored on multiple NAS systems without causing downtime or reducing productivity.
In order to efficiently perform storage virtualization operations, the storage virtualization system must be able to identify and locate the data objects stored on each of its associated NAS systems. For example, the storage virtualization system must monitor all of the data files, data directories, hard links, soft links and devices on its associated NAS systems. The storage virtualization system may therefore internally store a data structure containing information or “metadata” about the data objects stored on its associated NAS systems. The data structure may be a mapping index, a database, a File Identifier (“FID”) table, or directory tree that identifies each data object and its location on the associated NAS systems. The data structure may also include relevant path address, file attributes, timestamps or other such information for those data objects. Metadata in the data structure may originate from the associated NAS system that stores the data object, or the storage virtualization system that manages the data object. Using such a data structure enables the storage virtualization system to have an updated “snapshot” of the data objects it is managing. Changes made to those data objects, such as renaming or editing data object content, may be reflected in the data structure.
Due to hardware and performance constraints, a storage virtualization system may only have a finite amount of internal physical memory in which to store the data structure. For example, a storage virtualization system with 16 gigabytes of internal memory may only be able to store information on 50-60 million data objects. As a result, the storage virtualization system may only be able to manage 50-60 million data objects. NAS system manufacturers are continuously increasing the memory and data object storage capacity of their NAS systems. As the number of data objects that can be stored on these NAS systems grows, a storage virtualization system associated with these larger NAS systems will need to keep up with the increased storage demands.
What is therefore needed is a way to increase the storage capacity of present and future storage virtualization systems so that they can manage larger numbers of data objects. The data objects themselves may be stored on a single NAS system or on multiple NAS systems associated with a storage virtualization system. What is further needed is a way to optimize the storage virtualization system's data object management capabilities without affecting performance or resulting in data loss.