Each and every one of these documents, as well as all documents cited therein, is hereby incorporated by reference as if fully recited herein.

This Application claims priority of each and every one of these documents, as well as to all documents incorporated therein, to the fullest extent possible.

BACKGROUND

Field of the Disclosure

This Application can relate to cross silo time stitching, and other matters.

For example, this Application can include information relating to cross silo time stitching in a distributed network monitoring environment.

Other and further possibilities are described herein.

Related Art

One problem that has arisen, particularly in the field of network monitoring, is that when network devices provide status data, such as in a distributed network monitoring environment, there can be many different types of network devices. For example, there are communication networks, network routers, computing devices, virtual machines, virtual desktops, virtual desktop implementations, data storage elements, applications, users, and other types of network devices, in a distributed network monitoring environment. Each type of network device can have a different set of status data information, which can be formatted in a different type of status data message packet. This can pose the problem that a network monitoring device that is attempting to reconcile status data information about those different types of network devices, often is involved in a great deal of work when attempting to make comparisons. For example, it can be difficult for a network monitoring device to determine which of several network devices has reported its status first, or which is involved in an alert circumstance that is higher priority. Accordingly, status data information about different types of network devices can be difficult to compare.

Moreover, when a network monitoring device attempts to report the status of the distributed network monitoring environment, it can be difficult for the network monitoring device to determine the nature of connections or other associations between different types of network devices. For example, it might be difficult for a network monitoring device to determine which to report first a connection between a computing device and a data storage element, or a connection between a network router and an virtual application.

Similarly, but not identically, status data information can also apply to connections between “endpoints” (network devices or users) and other network elements. For example, an “endpoint” can refer to a 5-tuple «sender IP address, sender port, destination IP address, destination port, application ID». Alternatively, an “endpoint” can refer to a 7-tuple «sender IP address, sender port, sender interface, destination IP address, destination port, destination interface, application ID». Each of these sets of identification can serve to identify one “endpoint” from another. This can have the effect that each pair of endpoints can serve to identify a pathway in the network a route traveled by significant amounts of traffic in the network, or a connection of some importance in the network.

It can be difficult for a network monitoring device to determine which to report first connections between the same type of endpoints, connections between different types of endpoints, and/or otherwise. Even if the network monitoring device can make that determination, it cannot be sure that its determination will remain accurate. For example, the priority of different endpoints can change with time, with which other endpoints those endpoints are connected to, and with the nature of other connections in the distributed network monitoring environment.

This can present problems for monitoring devices in the distributed network monitoring environment.First, when a monitoring device receives information indicating status data for different amounts of time for distinct network devices, the monitoring device might have difficulty reconciling, within the meaning of the distributed network monitoring environment, the nature of the problem (if any) identified by the status data provided by a first device, as compared with the status data provided by a second device.Second, when a monitoring device receives information indicating differing types of status data for distinct network devices, that monitoring device might have to determine, by itself and without much further information, a meaningful and reconcilable set of knowledge about the status of the distributed network monitoring environment. For example, status data provided by the first device (in a first format) and status data provided by a second device (in a second format) might be substantially irreconcilable, and might not Indicate anything of Interest to the monitoring device, without the latter including significant domain knowledge.Third, when a monitoring device requests status data from A particular network device, either the monitoring device or the particular network device, or both, might be unable to handle the request within a sufficiently short time duration that the status data can still be included with other status data in a meaningful network monitoring report. This can have the effect that certain classes of status data, such as those that ae slow to collect at the generating device (which might be the most important status data of all), or those that are slow to be interpreted at the monitoring device (which might themselves instead be the most important status data of all), might be omitted from consideration when the monitoring device attempts to diagnose the network.Fourth, when a monitoring device requests status data from a particular network device (or endpoint), either (A) the monitoring device or (B) the particular network device, or both, might be unable to handle their other tasks within a relatively reasonable time duration. This can have the effect that the request by the monitoring device for status data from the network device, or the response from the network device with data for the monitoring device, might degrade the ability of one or the other device, or both, to operate within the distributed network monitoring environment.

One possibility is sometimes referred to as “virtual infrastructure operations management,” The possibility can provide that virtual machines implemented at the network device are each outfitted with their own local monitoring elements. Those local monitoring elements might be disposed to measure resource utilization metrics, to report (post mortem, that is, after the fact) any errors discovered about performance of the network device or its virtual machines, or to perform capacity management. While this possibility might have the capability of performing these functions at the network device, with the effect that: the monitoring device is not burdened with those functions, the possibility can be subject to several drawbacks.One drawback is that the local monitoring elements cannot conveniently or easily be disposed to obtain information from more than one “silo,” that is, information with respect to a function performed by the network device. Thus, the push data and the pull data cannot conveniently ox easily be coordinated to provide a unitary in-order time record. Moreover, status data in differing formats or embodying differing concepts would have to be collated by each individual network device's local monitoring elements, with the strong probability that difference or errors would creep into the implementation of those local monitoring elements.Another drawback is that the network device's local monitoring element cannot conveniently or easily be disposed to be coordinated with status data with respect to any other network device. For example, a virtual machine operating on a server might be able to provide status data its own operation, but it would not: be able to coordinate its own status data with another network device, such as a data storage element. Moreover, a local monitoring element for a virtual machine operating on a server would not be able to conveniently or easily manipulate status data in another format, such as status data from another network device.

Another possibility might be to install a reporting element, such as a software program including instructions capable of being interpreted by the network device, or another computing device accessible to the network device, to collect status data and send that information to one or more monitoring devices, in a manner convenient to those monitoring devices. While this possibility might have the capability of ameliorating difficulties the monitoring devices might have in processing status data they receive from network devices, the possibility can be subject to several drawbacks.One drawback is that the reporting element might be incompatible with some other element of the network device, whether hardware, software, or otherwise. For example, if the reporting element relies on a particular aspect of the network device's operating system, or of a guest operating system in a virtual machine in the network device, there is always a chance that any upgrades or other changes in one or more of those operating systems will cause the reporting element to perform improperly, or vice versa.

Another drawback is that, for these and other reasons, historically, operators of network devices have been substantially hostile to such reporting elements.

Some Drawbacks of the Known Art

Each of these issues, either alone or in combination with others, at some times, or in some conditions, can difficulty in aspects of effective and efficient collation of status data from more than one network device, more than one type of network device, or more than one format or type of status data, or otherwise, particularly in a distributed network monitoring environment.

A system includes apparatus, such as a network monitoring device, that can ameliorate at least some of the drawbacks noted above.

In one possible implementation (in a push circumstance for status data), the network monitoring device receives message packets from a network device that can include status data information, such as in the case of network traffic status data) a number of message packets and a number of octets processed by the network device in a recent time duration (sometimes referred to herein as a “dock tick”). Similarly, but not identically, in another possible implementation (in a pull circumstance for state data), the network monitoring device receives message packets from a network device that can include status data information, such as (in the case of processor coupled to the network and available for use by request) a degree to which the processor is busy, has high or low latency in responding to requests, and a degree to which the processor is slowed by handling requests from other process requests than any contemplated by new uses.For example, the network device cart send message packets each second, which each include status data information for the previous clock tick. In such cases, the network device can send each message packet as a wrapper, the wrapper including sets of status data information, each set of status data information being associated with an earlier dock tick. As network traffic is sometimes delayed, it is not necessarily so that the status data message packets arrive in order. Accordingly, the network monitoring device maintains a buffer of approximately one to two minutes (that is, approximately 60-120 seconds or, equivalently, approximately 60-120 Clock ticks). The inventors have found that with a buffer of one minute, approximately 95% of alt status data message packets are retrieved by the network monitoring device before the buffer is recycled, and with a buffer of two minutes, approximately 99% of all status data message packets are retrieved by the network monitoring device before the buffer is recycled. An even larger buffer would be likely to maintain an even greater probability that status data message packets would arrive before the buffer had to be recycled. Whether this is worth it is up to the operator of the network monitoring device and its users.In one embodiment, the monitoring device can assign each such wrapper in a status data message packet, with a beginning time stamp and an ending time stamp. This can have the effect that the monitoring device can determine the time duration of the period for which the network device is reporting, usually a predetermined exact number of clock ticks (such as exactly one clock tick, but it is possible to be more or less); and whether the status data message packet is appropriately associated with the most recent clock tick, or whether the status data message packet was delayed in transit, and belongs to an earlier clock tick. In either case, the system assigns the status data in the message packet to the proper clock tick, maintains that information in the buffer, and when the buffer is recycled to that point, emits one or more messages to users to present live (or recordable) status data, thereto.In one embodiment, the monitoring device assumes that actual status data indicates activity that was processed by the network device it a substantially uniform distribution. For example, if the status data message packet indicates that the network device processed 600 items in the past 10 clock ticks, the monitoring device assumes, unless told otherwise, that there were 60 data items for each such clock tick, if a status data message packet crosses a clock tick boundary, the network monitoring device can divide the message packet into more than one such message packet, assigning data items to each portion of the original message packet in response to where it crossed the clock tick boundary. This is described in other and further detail herein.As described herein, although the monitoring device assumes there is a uniform distribution of activity that generated the status data, in the context of the invention, there is no particular requirement that this is so. For example, the monitoring device can use a CDF or PDF representing a model of the amount of status data that might arrive, and can divide the activity according to that CDF or PDF. For example, if it is known from domain knowledge, or from past observance of the network, that network traffic is substantially bursty, the monitoring device can use a CDF or PDF associated with bursty traffic to generate a sequence of artificial “virtual packets,” one for each clock tick.Moreover, although the monitoring device can divide the status data into a number of distinct “virtual packets,” each representing the status data for activity for precisely one clock tick, in the context of the invention, there is no particular requirement for that, either. For example, the monitoring device can generate a single marker (or alternative type of “virtual packer”) that can represent a set of activity represented by the received status data. For example, as described above, if the network device processed 600 items in the past 10 clock ticks, the monitoring device can make an entry recording the (600:10) ratio (beginning at, say, 11:59:30), and can reduce the entry as tire passes through the status report. In such examples, the (600:10) ratio beginning at 11:59:30 can be decremented at 11:59:30 to become a (540:9) ratio beginning at 11:59:31, a (480:8) ratio beginning at 11:59:32, and similarly.

In different circumstances (that is, in a pull circumstance for status data), the monitoring device can obtain status data message packets from a network device by communicating with the network device in a similar manner as a client-server relationship. In such cases, the monitoring device would be similar to the client, thus making requests for status data from the network device, and the network device would be similar to the server, making responses including that status data information. However, in many cases, such as with vmWare devices, the network device is unwilling to provide status data message packets as often as each clock tick, so the network device accumulates status data for longer, such as about 20 seconds for data storage access information maintained by virtual machines. Even this value can vary, as resource usage at the virtual machine can cause the virtual machine to provide status data message packet less frequently or with less status data, such as possibly as little as only five seconds for data storage access information.In such cases, the network device can provide a set of average usage values for the reported time duration as the status data (sometimes referred to herein as “cooked” status data), or can provide a set of register values at the start and end of the reported time duration as the status data (sometimes referred to herein as “raw” status data). In the latter such case, the monitoring device determines the format of the status data information in the message packet, that is, whether raw or cooked, computes the cooked status data on behalf of the network device, and maintains the cooked status data in the buffer at the appropriate one or more clock ticks. Similar to, but not identical to, as noted above, the monitoring device can generate one or more “virtual packets,” delayed in time to reflect the delay in reporting, and possibly also delayed in time to reflect flight time across the network, and place those cooked virtual packets in their proper position on the monitoring device's time line.Similar to the process described above, the monitoring device allocates the status data, from the network device, among the clock ticks, assuming that activity follows a substantially uniform distribution. As noted above, the activity need not follow a substantially uniform distribution and in the context of the invention, there is no such particular requirement. As also noted above, the status data need not be allocated in multiple “virtual packets” across multiple clock ticks, and in the context of the invention, there is no such particular requirement.

Moreover, the monitoring device manages its communication with the network device, so as to manage how much status data it can retrieve, how much load it is placing on its “server,” the network device, and how much load it is placing on itself. When the monitoring device places excess load on the network device, the latter has the possibility of throttling back the amount of status data it provides, or the number of message packets it provides, or the fidelity of the status data to actual measurements, or even whether it is willing to communicate with the monitoring device at all.

Other and further details ae included herein.

This Application

After reading this application, those skilled in the art would recognize that techniques shown in this application are applicable to more than just the specific embodiments shown herein. For example, the applicability of the techniques shown herein can broadly encompass a wide variety of network monitoring techniques. These can include “push” techniques, in which the network device pushes the status data out to the network monitoring device, “pull” techniques, in which the network monitoring device explicitly requests status data information from the network device, “polling” techniques in which the network monitoring device looks to each network device in a round-robin or similar fashion to determine if any status data information is available, “shared memory” techniques, in which the network monitoring device and the network device can each include one or more portions of memory in which status data information can be maintained, and otherwise.

Moreover, after reading this application, those skilled in the art would recognize that techniques shown in this application are applicable, or can be made applicable with relatively small effort that does not include undue experiment or further invention, to circumstances in which the status data information is fuzzy, probabilistic, unclear, unknown, or otherwise. For example, while this Application is primarily directed to status data information that can be explicitly stated and maintained in non-volatile (or volatile) storage, or in memory or mass storage, in the context of the invention, there is no particular requirement for any such limitation. In such cases, the status data can include information that is only meaningful when examined over a period of time, or when combined with other information, or when interpreted by a user—or by another computing device, a machine learning system, an Artificial Intelligence system, one or more human beings (possibly with expert knowledge).

Moreover, after reading this application, those skilled in the art would recognize that techniques shown in this application are applicable, or can be made applicable with relatively small effort that does not include undue experiment or further invention, to circumstances in which the status data information is maintained in a data structure other than a buffer, such as when the status data information is maintained due to circumstances other than network delay. For example, the status data can be maintained in a data structure that includes one or more hashing techniques, one or more hierarchical techniques (such as a tree structure, directed graph, or lattice), one or more holographic techniques (such as a content-addressable memory, a Kohonen network, a biochemical computing device, or otherwise), or some other technique.

Moreover, after reading this application, those skilled in the art would recognize that techniques shown in this application are applicable, to manty other circumstances not explicitly described, such as status data that is distinguished by its application to activity with respect to location in an area or region (such as a particular set of network devices or endpoints in one or more selected places), or in another state-space (such as a particular set of network devices or endpoints using one or more virtual machines, virtual machine applications, real or virtual machine communication ports, or otherwise.

Possible Applicability

After reading this Application, those skilled in the art would recognize that techniques shown herein are applicable to more than just the specific embodiments shown herein, are within the scope and spirit of the invention, and would not require undue experiment or further invention.

Some particular implementations could include one or more of the following:Use of cross silo data collection in other types of network environments.Use of cross silo data collection from other types of devices, such as in an “Internet of Things” environmentUse of cross silo data collection in circumstances in which network operators, or other users, have determined a relative priority between distinct types of status data. For example, if a network operator determines that response latency from VM (virtual machine) instances is more important than response latency from VSD's (virtual storage devices), the monitoring element can exert whatever control it is capable of exerting on the network in response to that determination. For example, in this context “more important” can mean that, say, VM instance response latency is three times as important as VSD response latency, or can mean that lesser VM instance response latency is always better than lesser VSD response latency, or can mean that they have one comparison metric up to a selected threshold, after which a different comparison, metric is used.

Other and further techniques, also shown or suggested by this Application, are also applicable to more than just the specific embodiments described herein.

