The present invention relates generally to data processing systems and more particularly to automatic detection and recovery for problems arising on a channel that interconnects queue managers.
The MQSeries software from International Business Machines Corporation is a type of message oriented middleware. In particular, MQSeries is messaging software that allows application programs to exchange information across different software platforms. The MQSeries software resolves problems of interconnectivity. Hence, an application that runs on a first type of platform may communicate with another application that runs on a different type of platform. The differences in platforms may be due, for example, to differences in operating systems.
The MQSeries software utilizes queues to facilitate the exchange of information between applications. As shown in FIG. 1, a first application 10 may send a message 12 to a queue 14. Subsequently, second application 16 may receive the message 12 from the queue 14. The transfer of the message 12 may be performed asynchronously such that the application 10 need not wait on the application 16 and application 16 need not wait on the application 10.
With the MQSeries software, a message is a string of bytes that has meaning to the applications that use the message. Each message includes application data and a message descriptor. The message descriptor identifies the message and contains control information such as type information and the priority assigned to the message by the sending application. The application data may take many forms. The content of the application data is controlled by the application that constructs the message.
A queue is a data structure that stores messages. Each queue exists independently of the applications that use the queue. As shown in FIG. 2, queues are managed by queue managers. The queue manager insures that messages are put on the proper queue and routed to the appropriate destinations. Each queue manager is manifest as a separate object of the queue manager or object type. As shown in FIG. 2, the application 10 sends a message 12 to the queue manager 18, which stores the message 12 on the queue 14. The queue manager 18 retrieves the message 12 from the queue 14 and forwards the message to the application 16. As will be described in more detail below, the application programs are written using a common program interface, known as message queue interface (MQI). The MQI allows application programs that are running on different computers and/or different operating systems at different locations to communicate with queue managers. The applications communicate with the queue manager by issuing MQI calls.
Communication between applications need not be in one direction alone. As shown in FIG. 3, applications 10 and 16 may both receive messages and transmit messages. In the example depicted in FIG. 3, message 12 is sent from application 10 through queue 14 onto application 16. Conversely, message 20 is sent through queue 22 from application 16 to application 10.
The MQSeries software exploits the concept of a channel. A channel is a logical communication link between two queue managers. FIG. 4 depicts an instance wherein two channels 38 and 40 interconnect queue managers 32 and 36. Channels are available in two varieties: message channels and MQI channels. FIG. 4 depicts some examples of message channels. Message channels are unidirectional and serve to transfer messages from one queue manager to another. Message channels connect the queue managers via message channel agents (MCA""s). In the example depicted in FIG. 4, queue manager 32 manages queue 30 and queue manager 36 manages queue 34. Message channel 38 links queue manager 32 and 36 so that messages may be sent over the message channel from queue manager 32 to queue manager 36. Message channel 40 carries messages from queue manager 36 to queue manager 32.
An MQI channel connects a client to a queue manager on a server machine. A client is a section of code that is installed on a machine to accept MQI calls from applications and pass them on to an MQI server machine. An MQI server is a queue manager that provides queuing services to one or more clients. In the example shown in FIG. 5, client 40 is interconnected by MQI channel 48 with queue manager 44. The queue manager 44 manages access to queue 42. The MQI channels, unlike message channels, are bi-directional so that both calls and responses may be sent over a single channel.
As mentioned above, message channels interconnect MCA""s. An MCA is a program that controls the sending and receiving of messages. An MCA is positioned at each end of a channel. One MCA takes messages from a transmission queue (defined below) and puts the messages on the communication link. The other MCA receives 35 messages and delivers the messages to a remote queue manager. The MCA that initiates a communication is called a caller MCA, whereas the MCA that responds to a communication is a responder MCA. As shown in FIG. 6, MCA""s 62 and 64 are at respective ends of channel 60. Channel 60 interconnect queue manager 50 and queue manager 52.
Queue manager 50 includes a transmission queue 54. A transmission queue is a special type of local queue that is used to store messages temporarily before the messages are transmitted by an MCA to a remote queue manager. In the example shown in FIG. 6, MCA 62 transmits messages from the transmission queue 54 to the remote queue manager 52.
The queue manager 50 includes an initiation queue 56. The initiation queue 56 holds messages that are used to trigger a channel initiator 66 to start the MCA 62. Hence, before a message is sent, the message is first stored in the transmission queue 54. If the message satisfies triggering criteria for the queue, a separate message is sent to the initiation queue 56, causing the triggering of the channel initiator 66. The channel initiator 66 acts as a trigger monitor for sending MCA""s (such as sender MCA 62). The channel initiator 66 causes the sender MCA 62 to grab the message from the transmission queue 54 and send the message over the channel 60 to the remote queue manager 52.
The remote queue manager 52 includes a channel listener 68. A channel listener 68 is a program that monitors the channel to determine when to start the responder MCA 64. The responder MCA 64 is started in response to a start-up request from sender MCA 62. The channel listener 68 detects incoming network requests and starts the associated channel.
One difficulty with systems developed using the MQSeries software is that problems may arise relating to a channel. For example, a channel may fail. The MQSeries provides no built-in mechanism for automatically detecting such a failure and responding by attempting to restart the channel. Similarly, channel initiators and listeners may fail. Once again, there is no built-in mechanism within the MQSeries software for automatically restarting or reviving channel initiators or listeners.
The MQSeries software allows messages to be sent in predefined sequences over channels. Sequence numbers may be attached as tags to messages so that messages are sent and received in the same order. Unfortunately, there is no mechanism provided within the MQSeries software for automatically detecting and recovering from sequence errors when the messages get out of sequence.
The present invention addresses the limitations of conventional systems by providing a mechanism for automatically detecting and recovering from problems that arise relating to a channel in a distributed computing system. Specifically, in one embodiment of the present invention, failure of the channel is automatically detected. Upon detection of failure of the channel, the embodiment of the present invention attempts to automatically recover by reviving or restarting the channel. If such efforts fail, notifications in the form of reports, pages or emails may be sent to appropriate parties to advise the parties of the problem. Similarly, automatic detection and recovery is provided for problems relating to channel listeners and initiators.
The present invention may also provide for detection and recovery of sequence errors. The present invention may provide a mechanism for identifying when messages are out of sequence, resulting in a sequence error. Recovery mechanisms may be used to attempt to resolve the sequence error.
In accordance with one aspect of the present invention, a method is practiced in a distributed system that has a first queue for holding messages and a second queue for holding messages. A first queue manager is provided for managing the first queue, and a second queue manager is provided for managing the second queue. A channel interconnects the first queue manager with the second queue manager. An initiator is provided for initiating the sending of the message of the channel and a listener is provided for initiating the receiving of a message over the channel. A determination is made whether the listener or the initiator is inactive. Where it is determined that the listener or the initiator is inactive, corrective action is taken to revive the inactive listener or initiator.
In accordance with another aspect of the present invention, a determination is made whether the channel has stopped. If it is determined that the channel has stopped, corrective action is automatically taken to start the channel without prompting by user. Notifications may be generated to inform the appropriate parties of the problem.
In accordance with a further aspect of the present invention, a sequence of messages to be sent over a channel are provided. Each message is tagged with an associated sequence value that identifies the position of the message within the sequence. An out of sequence condition is detected at one of the queue managers, indicating a sequence error for at least one of the messages in the sequence. A reset of the sequence value to a known sequence value is initiated such that the next message to be sent in the sequence is tagged with the known sequence value.
In accordance with an additional aspect of the present invention, a computer system is provided in a distributed system. The computer system includes a channel monitor for monitoring activity with respect to the channel to detect problems. The computer system includes a recovery component for automatically recovering from problems in response to the detection from the channel monitor.