FIELD OF THE DISCLOSURE

The present application generally relates to management of data indexing in a cloud-based log aggregation solution.

BACKGROUND OF THE DISCLOSURE

Log aggregation systems, such as the cloud-based log aggregation systems Splunk, provided by Splunk Inc. of San Francisco, Calif., and Sumo, provided by Sumo Logic of Redwood City, Calif., allow for aggregation and data analysis of computer-generated log messages (e.g. audit trails, event logs, audit records, etc.). However, these services are typically all-or-nothing; users must send all the log data that may potentially be needed for later analysis. Since it is very difficult to anticipate what data will ever be needed, especially in the context of an emergency outage, enterprises typically send all of the available logging data, including logs generated by user devices, network appliances, servers, Internet-of-Things (IoT) devices, or any other type and form of device. The amount of data may be quite large, resulting in significant expense, as the cloud-based log aggregation systems typically charge fees for processing and indexing the data based on size. Most of the expense of these systems comes from the huge amount of compute and storage requirements for indexing all the data that is ingested. Additionally, large amounts of bandwidth are needed, as well as significant computing resources on the client side in order to collect and provide this data to the cloud-based log aggregation systems. Discarding data may reduce costs, but may result in loss of data required for troubleshooting of systems and software. Similarly, retaining logging data at the client devices for later aggregation in the event of an emergency may result in loss of data in some events, and may incur significant delays when troubleshooting is required due to transmission and processing time.

BRIEF SUMMARY OF THE DISCLOSURE

The system and methods discussed herein provide for classification of data at the source into sets of critical and non-critical data (sometimes referred to as priority and non-priority data, high priority and low priority data, indexed data and archived data, data to be indexed and data not to be indexed or to be archived, or similar terms). Critical data may be indexed and processed normally, while non-critical data may be provided to and stored by the cloud-based log aggregation system without indexing, at significantly lower cost in terms of processing and storage. In the event that non-critical data is required for troubleshooting or analysis, the non-critical data may be indexed dynamically on request. Because the non-critical data is stored at the cloud-based log aggregation system, it may be quickly indexed and added to the critical data, without additional consumption of bandwidth or delays due to transmission. Dynamic selection and classification of critical and non-critical data may thus allow an enterprise to balance costs and availability of indexed data.

In one aspect, the present disclosure is directed to a method for cloud-based log management. The method includes retrieving, by a log manager of a computing device, a plurality of logs. The method also includes selecting, by the log manager, a first subset of the plurality of logs. The method also includes classifying, by the log manager, the first subset of the plurality of logs as critical, high priority, or to be indexed. The method also includes transmitting, by the log manager to a log aggregator for indexing, the first subset of the plurality of logs, responsive to the classification of the first subset of the plurality of logs as critical, high priority, or to be indexed. The method also includes transmitting, by the log manager to a storage device of the log aggregator, a remaining subset of the plurality of logs.

In some implementations, the method includes retrieving the plurality of logs by retrieving, by the log manager from a plurality of additional computing devices, the plurality of logs. In some implementations, the method includes appending, by the log manager, an indexing control identifier to each log of the first subset of the plurality of logs. In a further implementation, the method includes appending the indexing control identifier to metadata of each log of the first subset of the plurality of logs.

In some implementations, the log aggregator stores the remaining subset of the plurality of logs without indexing responsive to the remaining subset of the plurality of logs not being classified as critical, high priority, or to be indexed. In some implementations, the method includes selecting the first subset of the plurality of logs by selecting a log of the plurality of logs, by a machine learning system of the computing device; and adding the log to the first subset, by the machine learning system, responsive to the log having characteristics matching characteristics of logs previously requested for review. In a further implementation, the characteristics comprise routines that generated the logs previously requested for review, computing devices corresponding to the logs previously requested for review, generation times of logs previously requested for review, or types of logs previously requested for review.

In some implementations, the method includes selecting the first subset of the plurality of logs by, for each log of the first subset of the plurality of logs: identifying, by the log manager, a routine that generated the log, the routine comprising an indexing command; and adding the log to the first subset, by the log manager, responsive to the identified routine comprising the indexing command.

In some implementations, the method includes transmitting, by the log manager to the log aggregator, a request to index at least one log of the remaining subset of the plurality of logs, the log aggregator indexing the at least one log of the remaining subset of the plurality of logs responsive to receipt of the request.

In another aspect, the present disclosure is directed to a system for cloud-based log management. The system includes a computing device comprising a network interface and a processor executing a log manager. The log manager is configured to: retrieve a plurality of logs; select a first subset of the plurality of logs; and classify the first subset of the plurality of logs as critical, high priority, or to be indexed. The log manager is also configured to: transmit, to a log aggregator for indexing, the first subset of the plurality of logs, responsive to the classification of the first subset of the plurality of logs as critical, high priority, or to be indexed; and transmit, to a storage device of the log aggregator, a remaining subset of the plurality of logs.

In some implementations, the log manager is further configured to retrieve, from a plurality of additional computing devices, the plurality of logs. In some implementations, the log manager is further configured to append an indexing control identifier to each log of the first subset of the plurality of logs. In a further implementation, the log manager is further configured to append the indexing control identifier to metadata of each log of the first subset of the plurality of logs.

In some implementations, the log aggregator stores the remaining subset of the plurality of logs without indexing responsive to the remaining subset of the plurality of logs not being classified as critical, high priority, or to be indexed. In some implementations, the log manager further comprises a machine learning system configured to: select a log of the plurality of logs; and add the log to the first subset, responsive to the log having characteristics matching characteristics of logs previously requested for review. In a further implementation, the characteristics comprise routines that generated the logs previously requested for review, computing devices corresponding to the logs previously requested for review, generation times of logs previously requested for review, or types of logs previously requested for review.

In some implementations, the log manager is further configured to, for each log of the first subset of the plurality of logs: identify a routine that generated the log, the routine comprising an indexing command; and add the log to the first subset, by the log manager, responsive to the identified routine comprising the indexing command.

In some implementations, the log manager is further configured to transmit, to the log aggregator, a request to index at least one log of the remaining subset of the plurality of logs, the log aggregator indexing the at least one log of the remaining subset of the plurality of logs responsive to receipt of the request.

In still another aspect, the present disclosure is directed to a tangible computer-readable storage medium comprising instructions that, when executed by the processor of a computing device, cause the processor to: retrieve a plurality of logs; select a first subset of the plurality of logs; classify the first subset of the plurality of logs as critical, high priority, or to be indexed; transmit, to a log aggregator for indexing, the first subset of the plurality of logs, responsive to the classification of the first subset of the plurality of logs as critical, high priority, or to be indexed; and transmit, to a storage device of the log aggregator, a remaining subset of the plurality of logs.

In some implementations, the computer readable medium further comprises instructions to append an indexing control identifier to each log of the first subset of the plurality of logs.

The details of various embodiments are set forth in the accompanying drawings and the description below.

