FIELD

The present disclosure relates to tracking sensitive data in big data environments.

BACKGROUND

Large data sets may exist in various sizes and organizational structures. With big data comprising data sets as large as ever, the volume of data collected incident to the increased popularity of online and electronic transactions continues to grow. For example, billions of records (also referred to as rows) and hundreds of thousands of columns worth of data may populate a single table. The large volume of data may be collected in a raw, unstructured, and undescriptive format in some instances. However, traditional relational databases may not be capable of sufficiently handling the size of the tables that big data creates.

As a result, the massive amounts of data in big data sets may be stored in numerous different types of data storage. Sensitive data may be copied and stored in various locations across the different types of data storage for various use cases. Tracking the sensitive data may be difficult as users may copy and distribute data. Typically, raw data is mapped or derived into output data. The mapped values may change as the raw source data changes. The change in raw data may result in sensitive data, such as personally identifying information (PII), appearing unexpectedly in output columns or files. Varying source data contents may thus impede tracking sensitive data in a big data environment.

SUMMARY

A system, method, and computer readable medium (collectively, the “system”) is disclosed for scanning a big data management system to identify sensitive data. The system may retrieve a pattern from a pattern database with the pattern identifying a type of sensitive data. The system may also retrieve data identified by a variable from a big data management system. The system may then match the data to the pattern to detect the type of sensitive data in the data. An output may be generated in response to the data matching the pattern.

In various embodiments, a variable access permission may be retrieved for the variable from a permissions repository, a sensitive data permission may be retrieved for the type of sensitive data from the permissions repository, and the variable access permission may be compared to the sensitive data permission to detect a discrepancy. The system may schedule retrieving the pattern from the pattern database to execute automatically at a predetermined time. The output may comprise a log, an output file, an email, and/or a message that flags at least one of the data, the variable access permission, or the sensitive data permission for review. A new pattern may be added to the pattern database to identify a new type of sensitive data. The pattern may comprise at least one of a regular expression or a string. The system may assign a priority of execution to the comparing the data to the pattern to run the comparing the data to the pattern as a background process.

The forgoing features and elements may be combined in various combinations without exclusivity, unless expressly indicated herein otherwise. These features and elements as well as the operation of the disclosed embodiments will become more apparent in light of the following description and accompanying drawings.

