TECHNICAL FIELD

The present disclosure relates generally to distributed networks, and more specifically to a system and method for detecting root cause of an exception error in a task flow in a distributed network.

BACKGROUND

System stability testing and diagnosis in a distributed network of computing devices is a challenging problem. Computing devices in a distributed network can fail due to many possible causes. It is challenging to detect a cause of a failure in the distributed network. Currently, in case of a failure, the computing devices in the distributed network undergo manual inspection by operators. This process is time-consuming and error-prone. Current technology is not configured to provide a reliable and efficient solution to detect the root cause of an exception error reported by a device in a distributed network.

SUMMARY

Current technology is not configured to provide a reliable and efficient solution to detect the root cause of an exception error reported by a device in a distributed network. This disclosure contemplates systems and methods for detecting the root cause of an exception error in a task flow in a distributed network.

For example, assume that a user wants to access their account from a web application. To enable the user to access their account, a series of steps may be performed by a series of servers in the distributed network. The distributed network may include a first server that may be configured to verify user login credentials to enable the user to enter their user portal on the web application, a second server that may be configured to retrieve user information to be displayed on the user's portal, a third server that may be configured to store the user information, and other servers. The other servers may perform other steps to enable the user to access their account and view their information. In the distributed network, a plurality of servers may communicate with each other and exchange messages to enable the user to access their account, and view their information. In some cases, a server may not receive a message as expected. For example, assume that a first server may send a message to a second server to retrieve particular user information, e.g., a residential address, a phone number, a document, etc. Also, assume that the second server does not respond to the received message. In this example, the first server reports an exception error because the first server does not receive the requested user information as expected from the second server. The disclosed system is configured to detect the root cause of such exception errors. Accordingly, the disclosed system provides a practical application of detecting the root cause of exception errors in a distributed network.

In one embodiment, the system for detecting a root cause of an exception error in a distributed network comprises a processor and a memory. The processor receives communication data that comprises a plurality of messages being transmitted between a plurality of servers in a distributed network, where the plurality of servers comprises a first server, a second server, and a third server. The processor receives communication acknowledgement data that indicates whether each message from among the plurality of messages is being received by a respective server from among the plurality of servers. The communication acknowledgement data comprises a first acknowledgement data that indicates whether the second server received a first message from the first server, and a second acknowledgement data that indicates whether the second server received a second message from the third server. The processor determines whether each message from among the plurality of messages is received by a respective server from among the plurality of servers based at least in part upon the communication acknowledgement data. The processor determines, from the first acknowledgement data, that the second server has received the first message from the first server. The processor determines, from the second acknowledgement data, that the second server has reported an exception error that indicates that the second server did not receive the second message from the third server, and that the third server did not send the second message to the second server. The processor determines that the third server is associated with the root cause of the exception error in response to determining that the third server did not send the second message to the second server. The memory is operably coupled with the processor. The memory is operable to store the communication data and the communication acknowledgement data.

The disclosed system provides several practical applications and technical advantages, which include: 1) technology that detects a server that reports an exception error by implementing an exception monitoring module in each server from among a plurality of servers in a distributed network, and implementing an exception listener module in a central server that is configured to receive communications exchanged between the plurality of servers; 2) technology that identifies servers that are in direct communication with the server that reported the exception error, e.g., by implementing a multi-layer detection module; 3) technology that determines the health of communications among the servers, i.e., determines whether each message transmitted between each pair of servers is successfully received or failed, e.g., based on acknowledgement messages that indicate whether each message is received or not; 4) technology that generates a communication matrix whose elements indicate successful and failed messages between the servers, where the successful messages are indicated by logic number “1” and the failed messages are indicated by logic number “0”; 5) technology that, using the communication matrix, identifies the earliest failed message based on determining the sequence associated with the messages and the acknowledgement messages exchanged between the servers; 6) and technology that determines that the root cause of the exception error is associated with the server that is associated with the earliest failed message.

As such, the disclosed system may improve the current data processing, distributed system monitoring, and fault detection technologies by detecting a root cause of an exception error in a task flow in a distributed system. Accordingly, the disclosed system may be integrated into a practical application of improving processing and memory storage capacity utilization for detecting the root cause of the exception error that would otherwise be spent using the current technology. For example, by detecting the root cause of the exception error, the disclosed system may report the root cause to an operator, and the root cause that has led to the exception error can be addressed and perhaps resolved.

The disclosed system may further be integrated into an additional practical application of improving the underlying operations of computer systems in the distributed network (e.g., servers, databases, etc.) that are tasked to carry out steps to grant a user request, such as to access an account and retrieve particular information from the account. For example, using the current technology where the root cause of the exception error has remained undetected, the computer systems in the distributed network continue to communicate with other computer systems which lead to more failed messages reported by the computer systems. This leads the processors and memories of the computer systems to have to process multiple failed messages, which leads to reducing the efficiency of utilizing the processing resources and storage capacity of these computer systems. In contrast, upon detecting the earliest failed message, the disclosed system processes the earliest failed message and detects a root cause of the earliest failed message. Upon detecting the earliest failed message, the disclosed system may pause the undergoing task flow, thus, preventing other servers to report failed messages as a consequence of the earliest failed message. Accordingly, the disclosed system improves the processing resources and storage capacity utilization in the computer systems of the distributed network more efficiently.

The disclosed system may further be integrated into an additional practical application of improving network communication between the computer systems of the distributed network. For example, by detecting the root cause of the exception error, and reporting the root cause of the exception error, the root cause can be addressed earlier compared to the current technology, and thus, the network communication bandwidth among the computer systems is not wasted.

Furthermore, current technology is also not configured to provide reliable and efficient system corrections after a root cause of an exception error is determined. This disclosure contemplates systems and methods for automatically identifying a corrective system change for resolving a detected exception error, optionally testing the corrective system change in an improved test environment, and automatically implementing the corrective system change.

Embodiments of this disclosure may be integrated into the practical application of a solution monitoring system which generates an intelligent solution registry that can automatically resolve exception errors with various detected root causes. The solution monitoring system monitors changes that are implemented for various detected root causes and systems. The solution monitoring system intelligently extracts and stores change descriptions in a reusable manner, such that similar exception errors that are detected in the future can be resolved using the same or similar solutions. The solution monitoring system facilitates more rapid and efficient responses to exception errors than was previously possible. For example, responses may be implemented with decreased consumption of computing resources (e.g., memory, processing, and/or networking resources). For instance, a solution or response may be implemented without wasting networking resources to communicate between the different systems used to evaluate, plan, and implement a solution using previous technology. Information obtained by the solution monitoring system may reduce or remove bottlenecks in responding to exception errors, such that there is little or no downtime between when an exception error is detected and a corrective action is taken. The solution monitoring system may also or alternatively prevent unnecessary repeated testing of the same or similar exception errors (e.g., with the same or similar root causes) by efficiently linking, or associating, previously observed exception errors to solutions that are known to be effective.

In certain embodiments, this disclosure is integrated into the practical application of a solution implementation system that automatically implements corrective changes in response to a detected exception error. In certain embodiments, the solution implementation system may leverage the specially configured information determined by the solution monitoring system. When an exception error is detected, the solution implementation system determines system(s) impacted by the root cause of the exception error. A corrective action is determined that is tailored to both the root cause and the impacted system(s). The determined corrective action is then automatically implemented. The solution implementation system provides more reliable, rapid, and efficient resolution of exception errors than was previously possible. For example, corrective actions may be implemented with decreased consumption of computing resources (e.g., memory, processing, and/or networking resources). For instance, rather than relying on human intervention or a one-size-fits-all response, a tailored solution may be implemented without wasting computational resources. The solution implementation system may provide near real-time corrections to exception errors, such that there is little or no downtime between when an exception error is detected and the corrective action is implemented. The solution implementation system may also or alternatively prevent unnecessary repeated testing of the same or similar exception errors (e.g., with the same or similar root causes) by automatically implementing a predetermined corrective action for a given combination of root cause and system configuration.

In certain embodiments, this disclosure is integrated into the practical application of an intelligent test environment for evaluating root causes and/or possible corrective actions for an exception error. A system implementing the test environment detects that an exception error is associated with functions implemented by tightly coupled systems and, in response, converts code for testing the tightly coupled systems to containerized code for separate systems. In this way, the containerized functionality can be separately tested to determine a root cause and/or test potential corrective actions. In certain embodiments, the test environment may include only the systems involved in the communication flow for an exception, such that operation of the test environment consumes fewer computational resources than was previously possible. In some embodiments, the intelligent test environment includes only the minimum components needed for testing, such that tests can be run at near real-time, for example, to further improve operations of the solution monitoring and/or solution implementation systems described in this disclosure. The intelligent test environment of this disclosure may also free up other system resources, such that more tests can be run in parallel to more rapidly and reliably determine root causes and/or test corrective actions.

Certain embodiments of this disclosure may include some, all, or none of these advantages. These advantages and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.

In an embodiment, an exception-resolution monitoring system includes a memory and processor communicatively coupled to the memory. The processor determines an exception error is detected corresponding to a failed transmission of a message between communicatively coupled servers of a distributed network. Changes are monitored that are tested and implemented to correct the detected exception errors. For each monitored change, a root cause is determined of the exception error corrected by the change. Properties are determined of the distributed network at the time of the exception error. One or more servers of the distributed network are determined that are impacted by the monitored change. A file location is determined of the one or more servers that are impacted by the monitored change that is altered by the monitored change. A value provided in the determined file location is determined to execute the monitored change. Based at least in part on the determined file location and the determined value, an executable corrective action is determined. The executable action is stored in the memory and linked to the root cause of the exception error and the properties of the distributed network at the time of the exception error.

In another embodiment, a solution implementation system includes a memory configured to store a solution registry with a corrective action for each of a plurality of predefined root causes of exception errors. A processor is communicatively coupled to the memory. The processor receives communication data transmitted through a distributed network comprising a plurality of servers. An exception error is detected corresponding to a failed transmission of the communication data between communicatively coupled servers of the distributed network. A first root cause of the exception error is determined. The first root cause is associated with a first server of the plurality of servers of the distributed network. The first server failed to relay the communication data to another server in the distributed network. Properties of the distributed network at the time of the detected exception error are determined. Using the solution registry, a first corrective action is determined for the first root cause that is associated with the determined properties of the distributed network at the time of the detected exception error. The first corrective action is automatically implemented at the first server.

In yet another embodiment, a system includes a distributed network includes a plurality of servers. Operations of a first server and a second server of the distributed network are tightly coupled. An exception resolution system includes a processor that determines that an exception error is detected in the distributed network. The exception error corresponds to a failed transmission of communication data between communicatively coupled servers of the distributed network. A data transfer route is determined associated with failed transmission of the communication data. The data transfer route includes the servers of the distributed network that fail to transmit or receive the communication data as a result of the exception error. The tightly coupled first server and second server are determined to be included in the shortest data transfer route. A subset of components of the tightly coupled first server and second server is determined that are involved in operations associated with the exception error. Based on the data transfer route and the subset of components, a containerized test environment is generated with replica components corresponding to (1) the servers included in the data transfer route other than the first and second servers and (2) the subset of the components of the tightly coupled first and second servers that are involved in operations associated with the exception error. A change is provided to a property of a first replica component of the containerized test environment. The property of the first replica component corresponds to a data transfer configuration property of the first server. Based on results provided by the containerized test environment, it is determined that the provided change resolves the exception error.

