Due to the ever increasing demand for data storage in today's digital world, distributed storage systems have received a lot of attention. In distributed storage systems, data as a whole or different portions of data are stored over several physically independent computer storage servers (also known as storage nodes). The total storage capacity of the distributed storage system is designed to be much greater than that of a single storage node.
There is an exponentially increasing amount of data produced and stored worldwide in distributed storage systems. A large fraction of that data is duplicated, for instance, as a result of unintentional copies of all or part of the data being stored multiple times, therefore leading to wasted storage resources. In particular, company/enterprise data, consisting mostly of structured formats such as documents and spreadsheets, is often replete with unnecessary duplicate data sometimes estimated over ˜70%. Therefore, an enterprise-wide storage system can make significant space savings if the data can be de-duplicated and stored only once. However, in many distributed database systems, deduplication is difficult to implement as the data models supported by, for example, relational database management systems (RDBMS), are not designed with deduplication in mind.
File-system level deduplication products are alternatives that aim to work with raw data on disks underneath any higher level storage abstractions such as databases, thereby leaving the upper or higher layers to operate as normal (e.g., unaware of the deduplication process underneath it). However, it has been shown that file-system level deduplication does not lead to optimal storage resource savings (e.g., for databases, especially) as there are too many small changes (e.g., file headers, timestamps, etc.) that make blocks of data subtly different and thereby defeat deduplication efforts. File-system level deduplication can also introduce additional data loss risks, as the database on top can make incorrect assumptions about which critical data could be lost due to a physical disk error.
In view of the above, there is a need for efficient systems and methods for data storage that can overcome these challenges and store the data in a distributed storage system securely, reliably and with minimal duplication.