1. Field of Invention
This invention relates to multiprocessing systems and more particularly to means for recovering from failures in part of the system.
2. Description of the Prior Art
With the advent of Very Large-Scale Integrated circuit (VLSI) technology there has arisen a corresponding need for new fault-handling methods and apparatus, uniquely adapted to the requirements of the new technology.
In data processing systems, fault-handling cycles have three phases: a fault occurs, the fault is detected, and then a recovery procedure is activated. As decribed in the above-identified Budde, et al. application Ser. No. 296,025, now U.S. Pat. No. 4,438,494 the detection phase may include detection, diagnosis, logging, and reporting of the fault. During this phase, system resources detect the presence and extent of the failure and pass this information on to recovery mechanisms. After fault detection, the recovery mechanisms are employed to mask the effects of the fault from the rest of the system and possibly isolate and repair the faulty subsystem.
A system is made up of a hierarchy of levels of fault-handling cycles. Faults can occur at many different levels in the system. Each level has its own characteristics. Faults not handled at one level will propagate up to the higher levels in the system. Higher levels have more complex environments which make recovery a more complex and slow task. Failure modes increase in complexity, the interaction between subsystems grows, and the original source of the failure becomes more ambiguous.
It is an object of the present invention to provide a recovery mechanism in which recovery from faults is accomplished at the lowest possible level.
By performing detection and recovery from hardware failures at a low level in the system, a more general and complete solution is possible to the problems of handling system failures. This approach divides the responsibilities of fault tolerance, allowing faster and simpler solutions to fault recovery. The mechanisms for detection and recovery from software errors need only address the set of faults that can be generated at those levels. By controlling and reducing the amount of errors that are reflected up to the next level, parallel and independent development may proceed on different levels (hardware, system software, applications). The designers at one level can assume that lower levels will always provide consistent and correct operation.
It is desirable that all of the fault-handling mechanisms, both detection and recovery mechanism, be orthogonal, so that expansion of bus bandwidth, logical resources, detection capabilities, or redundancy may be done without any side effects on the rest of the system.
It is therefore a further object of the present invention to provide a recovery mechanism that enables a system to have a very flexible and modular basis for growth and adaptation to the application environment, such that system capabilities may be added or removed without any impact on the application software.