BACKGROUND

This specification relates generally to processing computer security data. A cybersecurity intelligence platform can receive computer security data logs, or simply, data logs, that record information about events that have occurred during one or more processes executed by computing devices on a client network. The platforms can also analyze the data logs or provide data derived from the data logs as part of determining whether there is anomalous activity on the client network. For example, an event can include a request for or receipt of information sent between computing devices.

Computer security data logs can be formatted in many different ways. For example, various cybersecurity software applications may format logs differently, while even the same company may produce different software products that generate data logs having different formats. Information can be extracted from a data log using a parser and data logs that are formatted differently generally need to be parsed using different parsers.

SUMMARY

This specification describes technologies for identifying parsers that are likely to successfully extract desired information from a given data log. For example, these technologies can include a machine learning model that is trained to determine, from a set of data log types, a probability distribution over the set of data log types that likely correspond to the given data log. A data log is input to the machine learning model, and a probability distribution over the data log types is generated as output. Using a mapping from data log types to parsers corresponding to the log types, a data analysis system can select the parser having the highest likelihood of successfully extracting information from the data log. The data analysis system can then execute the selected parser on the data log.

In general, one innovative aspect of the subject matter described in this specification can be implemented in methods that include the actions of: obtaining an unlabeled computer security data log and processing the unlabeled computer security data log using a machine learning model to generate a probability distribution that includes a respective probability for each of a plurality of possible log types. Each of the plurality of possible log types is associated with a corresponding parser that parses logs of the possible log type to extract structured computer security data. The actions further include selecting the possible log type having the highest probability and parsing the unlabeled computer security data log using the parser corresponding to the selected possible log type.

Other implementations of this aspect include one or more non-transitory computer readable media storing instructions that when executed by one or more computers cause the one or more computers to perform the actions of the methods. Implementations of the methods can include one or more of the following features.

In some implementations, the actions further includes maintaining a mapping from each of a plurality of log types to a parser corresponding to the log type. One or more of the plurality of log types can include a plurality of log subtypes.

In some implementations, the actions further include determining that the parser corresponding to the selected possible log type did not successfully parse the unlabeled computer security data log and in response, parsing the unlabeled computer security data log using the parser corresponding to the possible log type having the second highest probability.

In other implementations, the actions further include determining that the parser corresponding to the possible log type having the second highest probability successfully parsed the unlabeled computer security data log. The actions also include generating training data, the training data including the unlabeled computer security data log and a label that identifies the possible log type having the second highest probability. The actions further include training the machine learning model using the training data to predict the unlabeled computer security data log is of the possible log type having the second highest probability.

Another innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of: generating training data, the training data including a plurality of training computer security data logs and, for each training computer security data log, a label that identifies a log type of the training computer security data log. The actions further include training a machine learning model to predict log types of unlabeled computer security data logs using the training data.

In some implementations, the actions include receiving labeled training data. For each training computer security data log of the labeled training data, the actions further include the operations of: determining a first log type for the training computer security data log using a corresponding label for the training computer security data log. The operations also include determining, using a mapping from each of a plurality of log types to a parser corresponding to the log type, a parser that corresponds to the first log type. The operations further include parsing the training computer security data log using the parser that corresponds to the first log type and determining that the parser successfully parsed the training computer security data log. The operations also include, in response to determining that the parser successfully parsed the training computer security data log, adding the training computer security data log and the first log type to the training data.

In some implementations, the actions further include receiving unlabeled training data. The actions can further include, for each training computer security data log of the unlabeled training data, performing the operations of parsing the training computer security data log of the unlabeled training data using parsers selected from a plurality of parsers until a particular parser successfully parses the training computer security data log of the unlabeled training data. Each parser of the plurality of parsers corresponds to a different log type. The operations can further include determining a particular log type corresponding to the particular parser and adding the training computer security data log and the particular log type to the training data.

In some implementations, the training data further includes a label that identifies a log subtype of the training computer security data log. In other implementations, a parser that successfully parses a computer security data log labeled with a first type and a first subtype also successfully parses a computer security data log labeled with the first type and a second subtype. In yet other implementations, a first log type includes a plurality of log subtypes.

A centralized server may aggregate data logs received from many other computing devices running various cybersecurity applications. The centralized server may be unable to know which application generated a particular data log or when the application was last updated. Therefore, it is not possible for the centralized server to ascertain the log type of any particular data log, without a label that identifies the log type.

Conventional systems that parse information from data logs may maintain hundreds or thousands of parsers and use a brute force method of information extraction that includes executing many different parsers on a data log without knowing which, if any, parser will successfully extract desired information. Using such a brute force approach can be computationally expensive when considering the number of parsers that a system may maintain.

Among other advantages, embodiments feature a data analysis system including a data log identification machine learning model that is able to output a probability distribution over possible data log types, allowing the data analysis system to make an informed selection of a parser for an unlabeled data log. Therefore, the data analysis system can use less computation time and resources when determining a parser to use, as compared to conventional systems that employ a brute force method of parser selection. In addition, because the machine learning model can generate a probability distribution over possible data log types, if a first highest probability data log type is incorrect, the data analysis system can use the parser corresponding to the data log type having the next highest probability. When the data analysis system successfully determines the data log type, the system can use this information to improve future predictions made by the machine learning model.

Other advantages will be evident from the description, drawings, and claims.

