COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains material to which a claim for copyright is made. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records but reserves all other copyright rights whatsoever.

TECHNICAL FIELD

This disclosure relates generally to the fault tolerance of computer systems. In particular, this disclosure relates to the implementation of disaster recovery in a computing infrastructure. Specifically, embodiments relate to systems and methods for disaster recovery that provide efficient and controlled failover in a distributed multi-services computing environment. Even more specifically, embodiments relate to systems and methods for disaster recovery that provide controlled one-click, hot-standby disaster recovery for distributed multi service computing environments.

BACKGROUND

In today's modern world it is now typical for an enterprise (e.g., a business entity, corporation, institution, organization, government agency, a non-profit organization, or the like) to utilize a highly complex distributed computing environment (e.g., a collection of computing devices such as servers, desktops, laptops or the like all connected via a computing network). Such a computer environment is used by enterprises so that their personnel can access and utilize certain software applications and data. The continuous availability of the computer systems of these enterprise environments is vital to the operation of these enterprises.

Thus, these enterprises take tremendous steps to ensure availability of these systems and data and to minimize what is referred to as “down-time” for these systems. These down-times can be caused by facility disaster, natural disasters, hardware failures, software application failures, purposeful attacks from virus or simply scheduled (e.g., periodic) maintenance of one or more of the computer systems. From the perspective of the enterprise, however, the cause of the down-time is less germane than its effects. When users of an enterprise cannot access that enterprise's systems, those users may not be able to perform their duties in an adequate capacity, reducing or interrupting the functioning of the enterprise, and in turn possibly reducing the enterprise's productivity or profitability.

To address these down-time issues many enterprises have implemented some form of disaster recovery (DR). Disaster recovery involves a set of policies, computing infrastructure, tools or procedures to enable the recovery or continuation of vital technology infrastructure and systems following a natural or human-induced disaster or other cause of failure or downtime for a primary (also referred to as a production) computing environment. Disaster recovery configurations are used in some cases to provide protection against loss of data due to failures, not only in the computer systems themselves but in the surrounding environment (e.g. loss of electrical power, acts of nature, fire, etc.).

Many of these DR configurations involve maintaining a dormant replica of servers of the primary computing environment at a secondary site or environment that is usually geographically separated from the primary environment by hundreds of miles or more (e.g., to protect against natural disasters, power grid failures, etc.). The state of the servers at the secondary site are synchronized with the state of the corresponding server at the primary site in order to reduce disruption when service is switched over to the secondary site upon loss of the server at the primary site. Accordingly, as each server within the primary computing environment at the primary site may have a number of applications (also referred to herein as services) executing thereon, a corresponding server within the DR computing environment at the secondary site will thus have those same applications (e.g., in the same state) executing on that server.

In modern computing distributed enterprise computing environments then, with hundreds or thousands (or more) distributed servers with associated services, it becomes highly impractical to mirror the state of a primary enterprise computing environment at a secondary site. Consequently, the deployment of DR solutions with respect to these multiple services deployed on multiple servers usually requires either a tedious manual process or a third-party orchestration tool to failover to the secondary DR site.

What is desired therefore, are improved systems and methods for integrated solutions for disaster recovery that provide for the simple configuration of primary or secondary DR environments, and substantially automated failover and recovery in association with these sites.

SUMMARY

As mentioned, to address down-time issues many enterprises have implemented some form of disaster recovery. Many of these DR configurations involve maintaining a dormant replica of servers of the primary computing environment at a secondary site or environment that is usually geographically separated from the primary environment. The state of the servers at the secondary site are synchronized with the state of the corresponding server at the primary site in order to reduce disruption when service is switched over to the secondary site upon loss of the server at the primary site. Accordingly, as each server within the primary computing environment at the primary site may have a number of applications (also referred to herein as services) executing thereon, a corresponding server within the DR computing environment at the secondary site will thus have those same applications (e.g., in the same state) executing on that server.

This requirement is impractical, as it requires an almost one to one correspondence between the configuration of the primary site and the DR site. Specifically, modern computing distributed enterprise computing environments then may have hundreds or thousands (or more) distributed servers with associated services. These services may be deployed on specific servers within the primary computing environment. To mirror this configuration then, a corresponding DR site may be required to have the same number of servers with the same service deployed on those servers in a manner that reflects or mirrors the deployment of those services within the primary computing environment. Moreover, when a failover is necessitated, this failover may be required to performed in a multi-step process, either manually or using previously configured third-party orchestration tool. Improved systems and methods for disaster recovery that provide for the simple configuration of primary or DR environments, and substantially automate failover and recovery in association with these sites.

To address these needs and problems, among others, embodiments as disclosed are directed to systems and methods for a service based disaster recovery system for a computing environment that includes a primary environment (also referred to herein interchangeably as a site) having primary servers with primary services, and a DR environment having DR servers with DR services. At a high level, embodiments may include the ability to configure and deploy the DR environment (e.g., through a configuration interface), including providing the ability to configure a DR service for one or more deployed primary services in the primary environment. An environment management database holds DR configuration data including the status of each service deployed in the computing environment. An interface may be used to initiate or terminate a failover for the entire computing environment or for a single server or service. Code or other logic built into the services may identify an associated current state for the service and perform activation (e.g., wake up) or enter a standby mode (e.g., sleep) depending on the identified state.

Specifically, in certain embodiments, a service based disaster recovery system may include an environment management database storing a DR configuration, comprising a representation of the servers of each computing environment, a designation whether each server is part of the primary environment or the DR environment, a representation of the services of each computing environment, an identification of the server on which each of the services is deployed, and mappings between corresponding services in the primary and DR computing environments. Thus, the DR configuration may include DR configuration data on primary servers of the primary computing environment, the primary services of the primary computing environment, and the primary servers on which those primary services are deployed (e.g., are executing). The DR configuration may also include DR configuration data on servers of the DR computing environment, the DR services of the DR computing environment, the DR servers on which those DR services are deployed, and a mapping between each DR service and a corresponding primary service.

The environment management database may provide an interface through which requests for this DR configuration data may be requested or retrieved. For example, this interface may receive a request for DR configuration data for a particular service (e.g., as referenced by the service identifier (ID) for that service) and may return associated DR configuration data for that service (e.g., associated with the service ID for that service in the DR configuration data stored in the environment management database), including for example, the server on which the service is deployed, the status or other data associated with that server, the identification of any corresponding service and the status of the corresponding service or the server on which that corresponding service is installed.

This DR configuration may, for example, be established by a user or administrator associated with an enterprise when the enterprise is establishing the DR computing environment. To aid in configuring the DR computing environment, the DR system of embodiments may provide a DR interface such as Graphical User Interface (GUI) or the like that interacts with the environment management database and may be used to by a user to designate a DR configuration for a computing environment, including defining primary servers and primary services of a primary computing environment and DR servers and DR services of a DR computing environment, designating a primary server of the primary computing environment on which each primary service is deployed, designating a DR server of the DR computing environment on which each DR service is deployed, and designating a correspondence between primary services in the primary computing environment and DR services in the DR computing environment. In this manner, a primary service on a primary server and a DR service on a DR server may be defined as a DR service pair through the DR configuration data maintained at the environment management database.

The service based disaster recovery system may also include an environment manager that may be utilized by each of the services (e.g., primary or DR services) deployed in each of the primary or DR computing environments. This environment manager may be code or a library incorporated into, or accessed by, a service. The environment manager may also be deployed remotely from the service, and can be implemented, for example, as its own service (such as a web service or the like) that may be accessed by a service through an interface such as a web services interface or Application Programming Interface (API) or the like). The environment manager may provide an interface through which a service can check to see if any action is required by that service (e.g., if that service should be asleep or awake).

In normal operation then, the primary services deployed in the primary environment are awake (or active). In other words, they may be actively servicing requests to allow users (e.g., human or machine) to access the services or functionality that they provide. Conversely, the DR services deployed in the DR computing environment are asleep or inactive. Thus, the DR services may not be actively servicing requests or may not be allowing users to access the services or functionality that they provide. However, even when asleep or inactive, these DR services may still be up and running in a standby mode awaiting activation (e.g., to be given a “wake up” action, as will be discussed).

Accordingly at regular intervals (e.g., using a heartbeat message or the like) each service (e.g., both primary services and DR services) may report its status to the environment management database, which may record the status of that service in association with the representation of that service. The environment management database can then update the DR configuration data with a status for the service and the last time a message was received through the service. In some embodiments, this heartbeat message may be sent through the environment manager. In these types of embodiments, the environment manager may also offer an interface for such heartbeat message such that a service may report a status through that interface and the environment manager, in turn, may report that status to the environment management database.

Additionally, at regular intervals (which may be a different interval or the same interval as the heartbeat messages), each service may send a request or otherwise access the interface of the environment manager to determine if the service should take an action. More specifically, the service may access the interface of the environment manager with a service ID for the service to request an action check for the service. The environment manager can then return an action to take to the requesting service, where this action may be a wake action or a sleep action. If the action is a wake action, and the service is not already awake, the service may wake up (e.g., by beginning to actively service requests to allow users to access the services or functionality that the service provides). If the service is already awake the service may wait for the expiration of the checking interval before checking again with the environment manager. If the action is a sleep action, and the service is not currently asleep, the service may go to sleep (e.g., by stopping the servicing of requests or stopping the access of users to the services or functionality that the service provides). If the service is already asleep the service may wait for the expiration of the checking interval before checking again with the environment manager.

When the environment manager receives a request from a service, therefore, it will make a determination of the action to return to the requesting service. To make such a determination, embodiments of an environment manager may request DR configuration data associated with the service ID for the service from the environment management database. In response to this request, the environment management database may return the DR configuration data associated with the service, including DR configuration data on the server on which the service is deployed (e.g., including, for example, if that server is down). Moreover, if the requesting service is part of a DR service pair, the environment management database may return DR configuration data on any corresponding service (e.g., if the service is a primary service, the corresponding service may be a corresponding DR service while if the service is a DR service the corresponding service may be a primary service) or server on which a corresponding service is deployed.

When the environment manager receives the DR configuration data associated with the requesting service from the environment management database, it can be determined if a corresponding service exists (or has been configured) from the DR configuration data associated with the service returned from the environment management database. If a corresponding service does not exist the environment manager may return a wake action to the requesting service. If however, a corresponding service exists it can then be determined if the corresponding service is installed. This installation may mean that not only does the corresponding service exist but that, additionally, the service has been configured with a particular server and is executing on that server. If a corresponding service is not installed, the environment manager may return a wake action to the requesting service.

If a corresponding service is installed, the environment manager can then determine if the corresponding service is awake. If a corresponding service is awake the environment manager may return a sleep action to the requesting service. If the corresponding service is not awake, the environment manager can then determine if the primary server of primary service of the DR pair (e.g., a primary service and corresponding DR service) including the requesting service is down. If the primary server is down and the requesting service is the primary service of the DR service pair the environment manager may return a sleep action to the requesting service, while if the requesting service is not the primary service the environment manager may return a wake action to the requesting service. Similarly, if the primary server is not down and the requesting service is the primary service of the DR service pair the environment manager may return a wake action to the requesting service, while if the requesting service is not the primary service the environment manager may return a sleep action to the requesting service.

When the requesting service receives the action returned by the environment manager the requesting service can then take an appropriate action based on the type of the returned action. If the returned action is the same as the current state of the service the service may take no further action until the expiration of the checking interval at which point the service may once again requesting an action from the environment manager. However, if the requesting service is asleep and the action is a wake action the service may wake up (e.g., by beginning to actively service requests to allow users to access the services or functionality that the service provides) and if the returned action is a sleep action and the service is not currently asleep, the service may go to sleep (e.g., by stopping the servicing of requests or spotting the access of users to the services or functionality that the service provides). To ensure that both services of the DR pair are not simultaneously awake, in some embodiments before a service wakes up it may check the status of the corresponding service of the DR pair as returned in the DR configuration data to ensure the corresponding service is idle or has not been active (e.g., as determined from its heartbeat message) for a certain threshold amount of time.

Embodiments may thus provide for single-server or even single service fail over. In particular, embodiments may provide an interface by which a user (e.g., an administrator associated with an enterprise or the like) may initiate a fail over by designating a server (e.g., a primary server in the primary environment) or even a particular service (e.g., a primary service in the primary environment) as down. Such a designation may be made, for example, using the DR configuration interface as previously discussed or through a separate interface associated with the environment management database. This interface may interact with the environment management database to indicate that the server or service indicated as down by the user through the interface is indicated as such in the DR configuration data associated with that server or service. Thus, when the respective services of any DR service pairs associated with the server that the user marked as down next requests an action status (e.g., through the environment manager), the DR configuration data returned to the environment manager from the environment management database for that service will indicate the status of the server marked as down by the user. Each of the respective services of the DR service pairs associated with that service can then take the appropriate (e.g., wake or sleep) actions (e.g., without any further user involvement). Consequently, when a primary server is indicated as down by a user, the corresponding DR service for each primary service deployed on that server will wake up and become active without any further involvement from the user, while each primary service deployed on the indicated primary server will sleep (if needed) without any further user involvement.

Similarly, the user may interact with the same interface to designate a server (or service) as up (e.g., when it is desired to restore the server). The interface may interact with the environment management database to indicate that the server designated as up is indicated as such in the DR configuration data associated with that server. Thus, when the respective services of any DR service pairs associated with the server the user marked as up next requests an action status (e.g., through the environment manager) the DR configuration data returned to the environment manager from the environment management database for that service will indicate the status of the server marked as up by the user. Each of the respective services of the DR service pairs associated with that service can then take the appropriate actions (e.g., without any further user involvement). Thus, when a primary server is indicated as up by a user, the corresponding DR service for each primary service deployed on that primary server will go to sleep without any further involvement from the user, while each primary service deployed on the indicated primary server will wake up without any further user involvement.

As can be seen then, embodiments as disclosed provide a number of advantage over previous failover tools and methodologies. For example, embodiments may allow simple (e.g., “one-click”) on demand failover or recovery for services and, moreover, may provide this functionality without the need for a third-party orchestration tool. Such functionality may also be accesses by users through a single tool or interface that allows both configuration of the DR environment and the ability to institute failover or recovery from the same interface or tool. Accordingly, embodiments may save deployment time and remove the complexity of deploying a DR solution in computing environment while simultaneously shortening the time required to failover a computing system during a disaster (e.g., from hours to minutes), thus preventing data loss and extending system uptime.

Additionally, embodiments may allow quick and seamless implementations of such failover and recovery, as DR services in the DR environment are never fully inactive and may be up and running in a standby mode waiting to activate based on associated DR configuration data. This type of functionality can be referred to as a “hot-standby” mode of operation. Embodiments are thus server or service centric, meaning that individual servers or services may be taken down or restored and each service may individually access the DR configuration data of the environment management database associated with that service to make an individual determination about the status of that service and take a corresponding action with respect to that service (e.g., wake up or go to sleep). In this manner, embodiments may free the DR computing environment from the constraints of having to mirror the primary computing environment, as there no longer have to be an identical number (or type) of servers in the DR environment, and services no longer have to be deployed on identical servers. More generally, then, the servers in the DR computing environment do not have to be equivalent to (e.g., either in processing power or in number or otherwise) to those in the primary environment. As each service may be make individual determinations it is relatively easy to integrate new services into such a computing environment and provide disaster recovery with respect to these newly integrated services.

In one embodiment, an environment management database stores DR configuration data on a primary computing environment comprising a primary server and a primary service deployed on the primary server and a disaster recovery (DR) environment comprising a DR server and a DR service deployed on the DR server. The DR service corresponds to the primary service and the DR configuration data includes DR configuration data on the primary server, the primary service, the DR server and the DR service. Additionally, the DR configuration data includes a designation of the DR server as a DR server, a status of the primary server, and a mapping between the primary service and the DR service.

A first environment manager associated with the DR service can receive a first request for a first action from the DR service and request DR configuration data for the DR service from the environment management database. The first environment manager receives DR configuration data for the DR service from the environment management database, including the designation of the DR server as a DR server and the status of the primary server. The environment management database returns such DR configuration data to the first environment manager based on the mapping between the DR service and the primary service. Based on the DR configuration data, the first environment manager can determine if the status of the primary server is down, returning a wake action as the first action to the DR service when the status of the primary server is down and returning a sleep action as the first action to the DR service when the status of the primary server is not down. The DR service is adapted to wake up when the first action is the wake action and sleep when the first action is a sleep action.

Similarly, a second environment manager associated with the primary service can receive a second request for a second action from the primary service and request DR configuration data for the primary service from the environment management database. The second environment manager receives DR configuration data for the primary service from the environment management database, including the designation of the DR server as a DR server and the status of the primary server. The environment management database returns such DR configuration data to the second environment manager based on the mapping between the DR service and the primary service. Based on the DR configuration data, the second environment manager can determine if the status of the primary server is down, returning the sleep action as the second action to the primary service when the status of the primary server is down and returning the wake action as the second action to the primary service when the status of the primary server is not down. The primary service is adapted to wake up when the second action is the wake action and sleep when the second action is a sleep action.

In some embodiments, the first environment manager is adapted to determine if the primary service is awake and return the sleep action as the first action to the DR service when the primary service is awake and the second environment manager is adapted to determine if the DR service is awake and return the sleep action as the second action to the primary service when the DR service is awake.

In particular embodiments, the first environment manager and second environment manager are adapted to determine the first action and the second action based on a determination that the DR service is deployed on the DR server from the DR configuration data.

In another embodiment, the first environment manager is incorporated into the DR service and the second environment manager is incorporated into the primary service.

In other embodiments, the request for DR configuration data for the DR service includes a first service identifier for the DR service and the request for DR configuration data for the primary service includes a second service identifier for the primary service.

In a specific embodiment, the first request for the first action is sent by the DR service at a checking interval and the second request for the second action is sent by the primary service at the checking interval.

These, and other, aspects of the disclosure will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following description, while indicating various embodiments of the disclosure and numerous specific details thereof, is given by way of illustration and not of limitation. Many substitutions, modifications, additions and/or rearrangements may be made within the scope of the disclosure without departing from the spirit thereof, and the disclosure includes all such substitutions, modifications, additions and/or rearrangements.

