<commit_before>#!/usr/bin/env python
# -*- coding: utf-8 -*-
#        1         2         3         4         5         6         7         8         9        9
# 3456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
"""
ReguLog - Unpacks and analyzes log archives with regular expressions

ReguLog can be used via a GUI (start script with no argument) or on the command line to:

 - Search for log files according to regexp patterns into directories and archive files. The zip,
    tar and gzipped-tar archive formats are supported. Data is searched recursively into directories
    and archive files down to the deepest level. At the end of the scan process, an overview of the
    identified log files is shown with their earliest and latest modification times.

 - Extract the identified log files into a destination directory, and apply several schemes for
    file concatenation (log4j style) and directory re-ordering.

 - Search for regexp patterns in log files, in archive or directories, and extract fields (using the
    Python syntax e.g. to extract a 2-digit day field "(?P<day>[0-3][0-9])"). The found occurrences
    can be displayed on match or on change, and may include references to fields (e.g. "{day}").
    Additional pre-defined fields are available such as the timestamp, extracted with a regexp too.
    Python code can as well be entered and executed on search start, event match and search end.

 - Export the data of the found events in XML/CSV files for post-processing.

 - Store a working pattern into a new or existing patterns XML file, then re-use patterns for
    other searches.

Known issues:
 - A tar file packed in a zip file cannot be read ("seek" error)
 - During search, too many displayed events can lead to HMI freeze (QT issue) or crash. As a
   workaround set the verbosity to "Quiet".
 - "Kill" button does not work (issue in bfScriptInterface and QT, not specific to regulog)
"""

#Revisions:
# 0.1     : First version with tar file support
# 0.2     : Use of ScriptInterface for GUI
# 0.3-d1  : Changed name to regulog
# 0.3-d2  : Associated options to commands
# 0.3-d3  : Implement unpack, make dist with new ScriptInterface
# 0.3-d4  : Added support for MIDF for input files and dirs, renaming of options and commands
# 0.3-d5  : Re-factored patterns, add support for input XML file with log event definition
# 0.3-d6  : Re-factored log archive opening
# 0.3-d7  : Added LogSource class, implemented simple version of unpack
# 0.3-d8  : Consolidated unpack for all kinds of sources and HMI re-ordering
# 0.3-d9  : Renamed "unpack" to "extract", completed "keep source dirs", changed default values
# 0.3-d10 : Update with new bfScriptInterface supporting multiple files/dirs selection
# 0.3-d11 : Re-factored extract procedure to support file re-ordering and concatenation
# 0.3-d12 : Completed extract procedure with options - Issue PR1: very long extraction time
# 0.3-d13 : Split extract procedure, rewrote procedures to process files in archive order
# 0.3     : Removed default test values for release, added list of known issues
# 0.4.1   : Set last modification time to value found on original file
# 0.4.2   : Corrected hanging with reducedirs if all files in source have different names
# 0.4.3   : Some re-factoring for pathfilter, change path regex to filename regex in event
# 0.4.4   : Fixed today's timestamp on joined files
# 0.5.1   : First version of log analysis with patterns
# 0.5.2   : Implementation of event save, display on match, display on change, timestamp
# 0.5.3   : Consolidation of ElementTree patch, fix of CDATA removal while saving, re-factoring
# 0.5.4   : Re-factoring of many elements, chronological order, newline in HMI
# 0.5.5   : Support of multiline patterns, improvement of timestamp pattern and logic
# 0.5.6   : Case sensitive option, hide timestamp option
# 0.5.7   : Fixed timestamp in ipsec.log, implemented global source
# 0.5.8   : Lookup functions, re-factoring search function, change of syntax to {field}-style
# 0.5.9   : XML/CSV export
# 0.6.0   : Python execution
# 0.6.1   : HMI changes following new ScriptInterface, widgets location, re-wording
# 0.6.2   : Python execution consolidation, displayIfChanged introduced
# 0.6.3   : Introduced checkPathFilter, named groups in path filter now create dirs
# 0.6.4   : Reducedir separately in each output dir, avoid duplicate files in tar
# 0.6.5   : Moved code to new EventSet class with lookup, corrections in scanPath
# 0.6.6   : Added Python functions, fully pseudo-path in _source_path, fixed multiline issue
# 0.6.7   : Further Python functions (e.g. get_fields), __str__ for Event, _user_fields

__version__ = "0.6.7-aib"

# TODO support cascaded event types (Parent, ParentFile), with includes of patterns in other files
# TODO prevent changing XML structure and comments when saving event type
# TODO improve events search performance<commit_msg>update as target-unspecific version (no tag in version name)<commit_after>#!/usr/bin/env python
# -*- coding: utf-8 -*-
#        1         2         3         4         5         6         7         8         9        9
# 3456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
"""
ReguLog - Unpacks and analyzes log archives with regular expressions

ReguLog can be used via a GUI (start script with no argument) or on the command line to:

 - Search for log files according to regexp patterns into directories and archive files. The zip,
    tar and gzipped-tar archive formats are supported. Data is searched recursively into directories
    and archive files down to the deepest level. At the end of the scan process, an overview of the
    identified log files is shown with their earliest and latest modification times.

 - Extract the identified log files into a destination directory, and apply several schemes for
    file concatenation (log4j style) and directory re-ordering.

 - Search for regexp patterns in log files, in archive or directories, and extract fields (using the
    Python syntax e.g. to extract a 2-digit day field "(?P<day>[0-3][0-9])"). The found occurrences
    can be displayed on match or on change, and may include references to fields (e.g. "{day}").
    Additional pre-defined fields are available such as the timestamp, extracted with a regexp too.
    Python code can as well be entered and executed on search start, event match and search end.

 - Export the data of the found events in XML/CSV files for post-processing.

 - Store a working pattern into a new or existing patterns XML file, then re-use patterns for
    other searches.

Known issues:
 - A tar file packed in a zip file cannot be read ("seek" error)
 - During search, too many displayed events can lead to HMI freeze (QT issue) or crash. As a
   workaround set the verbosity to "Quiet".
 - "Kill" button does not work (issue in bfScriptInterface and QT, not specific to regulog)
"""

#Revisions:
# 0.1     : First version with tar file support
# 0.2     : Use of ScriptInterface for GUI
# 0.3-d1  : Changed name to regulog
# 0.3-d2  : Associated options to commands
# 0.3-d3  : Implement unpack, make dist with new ScriptInterface
# 0.3-d4  : Added support for MIDF for input files and dirs, renaming of options and commands
# 0.3-d5  : Re-factored patterns, add support for input XML file with log event definition
# 0.3-d6  : Re-factored log archive opening
# 0.3-d7  : Added LogSource class, implemented simple version of unpack
# 0.3-d8  : Consolidated unpack for all kinds of sources and HMI re-ordering
# 0.3-d9  : Renamed "unpack" to "extract", completed "keep source dirs", changed default values
# 0.3-d10 : Update with new bfScriptInterface supporting multiple files/dirs selection
# 0.3-d11 : Re-factored extract procedure to support file re-ordering and concatenation
# 0.3-d12 : Completed extract procedure with options - Issue PR1: very long extraction time
# 0.3-d13 : Split extract procedure, rewrote procedures to process files in archive order
# 0.3     : Removed default test values for release, added list of known issues
# 0.4.1   : Set last modification time to value found on original file
# 0.4.2   : Corrected hanging with reducedirs if all files in source have different names
# 0.4.3   : Some re-factoring for pathfilter, change path regex to filename regex in event
# 0.4.4   : Fixed today's timestamp on joined files
# 0.5.1   : First version of log analysis with patterns
# 0.5.2   : Implementation of event save, display on match, display on change, timestamp
# 0.5.3   : Consolidation of ElementTree patch, fix of CDATA removal while saving, re-factoring
# 0.5.4   : Re-factoring of many elements, chronological order, newline in HMI
# 0.5.5   : Support of multiline patterns, improvement of timestamp pattern and logic
# 0.5.6   : Case sensitive option, hide timestamp option
# 0.5.7   : Fixed timestamp in ipsec.log, implemented global source
# 0.5.8   : Lookup functions, re-factoring search function, change of syntax to {field}-style
# 0.5.9   : XML/CSV export
# 0.6.0   : Python execution
# 0.6.1   : HMI changes following new ScriptInterface, widgets location, re-wording
# 0.6.2   : Python execution consolidation, displayIfChanged introduced
# 0.6.3   : Introduced checkPathFilter, named groups in path filter now create dirs
# 0.6.4   : Reducedir separately in each output dir, avoid duplicate files in tar
# 0.6.5   : Moved code to new EventSet class with lookup, corrections in scanPath
# 0.6.6   : Added Python functions, fully pseudo-path in _source_path, fixed multiline issue
# 0.6.7   : Further Python functions (e.g. get_fields), __str__ for Event, _user_fields

__version__ = "0.6.7"

# TODO support cascaded event types (Parent, ParentFile), with includes of patterns in other files
# TODO prevent changing XML structure and comments when saving event type
# TODO improve events search performance
{'fraction_non_alphanumeric': 0.06632499246306904, 'fraction_numerical': 0.04622650989850266, 'mean_word_length': 3.5422181652213602, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3888724', 'n_tokens_mistral': 3029, 'n_tokens_neox': 2705, 'n_words': 1337}