Reliable storage of digital data in a memory system is effected by using error correcting codes, erasure codes, redundant data storage, or the like. One method of storing data in a reliable way is to use a RAID (Reliable Array of Inexpensive Disks) architecture. Except for RAID 0 (which does not provide redundancy), the various RAID types (e.g. 1, 3, 4, 5, 6, etc.) store an extent of data by writing chunks of the data to several independent memories and redundancy data to other independent memories, with the objective that, should one or more of the memories encounter a failure, the data can be recovered from the remaining stored data.
Originally the concept developed using disk drives (mechanical rotating magnetic media platters). A stripe of data comprising a plurality of strips of data may be written, the strips being chunks of data of the extent of data, such that each of the strips of the stripe is written to a separate disk, and a parity strip is computed over the chunks of data and stored to another disk. In this single parity case, the loss of a disk storing a chunk of the extent may be overcome by computing the exclusive-OR of the remaining chunks and the parity data to produce the lost data. This is a simple example of single-parity RAID, but a person of skill in the art would understand the extension to multiple parity, distributed parity, and other similar techniques which may provide protection against the loss of multiple disks.
In many of the RAID mechanizations, considerable effort is expended to reduce the number of disk operations to be performed, as the number of operations per second and the bandwidth of mechanical disk devices is limited primarily by rotational and seek latencies. In particular, the time to assemble the data from all of the strips of a stripe is a significant limitation on the processing speed of a large data base system. Computer technology has far outstripped the ability of the rotating disk to store or retrieve data in a timely manner.
The cost of non-volatile solid state memory has been reduced, primarily due to the increase production of NAND flash devices for use in consumer electronics such as cell phones, tablets, audio players and the like. Memory systems using NAND flash have been produced that emulate the protocols of rotating disks, and such SCSI-compatible devices, for example, are now found in laptops and have been used in arrays to replace mechanical disks in larger storage systems. Flash memory, however, has its own limitations, which may be characterized as being related to timing and wear out.
Data is stored in a flash memory circuit in pages, with a plurality of pages forming a block, and a plurality of blocks being found in a memory circuit (“a chip”). A characteristic of NAND flash memory is that while data may be written in pages in sequential order, the memory cannot be overwritten without being first erased, and the erase operation must be performed on integral blocks. The page read time, page write time and block erase time are not the same time duration, with the write time typically being a multiple of the read time and the block erase time typically being a multiple of the write time. Although these times vary with the specific NAND flash type and manufacturer, an example would be read time (100 us), write time (500 us) and block erase time (5 ms). Moreover, a read operation cannot be performed on a chip when a write or block erase operation is in progress. We do not herein discuss the issue of wear out, or the details of the management of the memory so as to manage the pages of memory where the data has been modified, except to say that this important aspect of a flash memory system is addressed by a which may also include such operations as “garbage collection” and “wear leveling.”
If the data of a RAID stripe was written as strips to, for example, separate memory chips, then RAID may be used in a flash memory system. However, if the individual memory chips are operated as independent storage devices, each chip may be in a different state when a request for data from a RAID stripe is received. That is, the chip may be performing, for example, a write or an erase operation and not be available to read data until the write or erase operation has completed. This significant limitation on the performance of a RAIDed flash memory may be overcome by appropriately scheduling the operations of the memory circuits or processing the data received from the memory circuits, such as has been described in U.S. Pat. No. 8,200,887 entitled “Memory Management System and Method, issued Jun. 12, 2012, which is commonly owned, and which is incorporated herein by reference.
For convenience, the technique is termed “erase hiding.” This is not intended to exclude an aspect of the technique that may be called “write hiding”. As will be seen, the erase hiding and write hiding are both performed in a similar manner, and may often be combined, where the choice of performing a write operation or an erase operation may depend on the nature of the pending operations for any memory chip. In a broader sense, the term “erase hiding” is understood to mean that the operations of the memory system are scheduled or controlled such that there are periods of time where operations that could interfere with low-latency reading of data from the memory system are either inhibited or permitted so as to permit low-latency read operations to be routinely performed. Where such operations are not pending, the particular memory module may also perform read or write operations, some of which may be background operations. Typical operations that may result in increases in latency are erase of flash memory and writing of data. The term “chip” is used to signify a memory circuit having a plurality of blocks of storage locations, or pages, and may include a plurality of memory circuits in a memory package, a plurality of memory packages, and even larger scale assemblies that are operated in a coordinated and consistent manner.
Large flash memory arrays in a single chassis are being produced using these concepts, with scale sizes of 10-40 Terabytes (TB). Even larger scale memory arrays may be assembled from these building blocks. Along with the need for larger and larger memory arrays, with low-latency performance, the reliability requirements that led to RAID are even more stringent today, and the costs of lost data, or the need to perform restoral from a backup are significant to a business.
Today, one may distribute the strips of the data of a RAID stripe over a plurality of modules of memory in chassis, and widely distribute the data over the physical memory space. However, there is a need to protect against the unlikely event of the loss of a full chassis. So, the strips of the data of a RAID stripe may be spread over a plurality of individual chassis, and the chassis may not be located in a same rack or adjacent rack, or facility. Yet, the low latency performance of a memory system using erase hiding is still important.