<commit_before>This is a re-write of pychecker using traditional parse tree
techniques, rather than the bytecode.

There are some other design changes, related to Warning objects and
Option handling.  In general, pychecker2 tries to distribute checks to
separate classes.  These classes are more self-contained: they hold
onto the option variables and define the Warning objects they support.

ToDo:
        All missing stuff
        Always returning None
        More path analysis
            used before set for if/else try/except
        Warn about unpacking args:
             def f(a, (b, c)): pass

Items still missing:
        Error suppression (errors may be suppressed globally)

        Not calling base class __init__
        Using return value which is always None
        Wrong # of args on function (including ctor)
        Function arguments, kwargs
        Explicit/implicit returns
        Abstract methods
        Detail of exception is an exception
        Local used after deleted, already deleted, deleted before set
        exec may use locals/globals

        Modifying a default parameter
        Tuple access to list list[1, 2]
        List .append() takes one argument
        Return None from __getattr__
        string iteration
        Inconsistent return type
        Object has no attr
        Method read as attribute, but not called:
        Function always returns None (use empty return)
        Unpack wrong size
        Unpack wrong type

        Exec warnings
        Security warning: input()

        Module imports itself
        Complexity
        Doc strings
        New python features:
             slots
             properties
             integer division
        Warning for string raising

Bugs:
        More member not used
        Speed
        Is __file__ a builtin?

        Aliases:
            class C:
                  def f(a, b): pass
                  def g(a, b): pass
                  h = f
                  h = g			# redef of h, not caught now

        foo = 1
        del foo	                        # should not be unused

        from __future__ import produces unused warnings
        import of readline has side effect on input/raw_input, and should be
           marked as used

Important stuff:

If you want to add new checks, there are some parts to be aware of:

        File
                A stupid data structure to hold everything we know about 
                a file to be checked.

                Members:
                   name
                        The file name.

                   parseTree
                        The root of the Abstract Syntax Tree returned from
                        compiler.parseFile.  Will be None if the file
                        won't even parse.

                   scopes
                        A dictionary of { node, scope } computed with
                        from compiler.symbols. For convenience, scopes
                        also refer back to the node and to the
                        enclosing parent scope (node and parent,
                        respectively).

                    root_scope
                        Convience for scopes[parseTree]

        Check
                Basic step for checking a file.  Usually contributes
                Warnings or additional information to File.  New checks
                will subclass this, and be added to the chain of checks
                given in main().

        Warning
                A description of a warning emitted by pychecker.

        CheckList
                Contains "global" check information: cached module
                data and the list of Check objects.

        Command line options are pulled from each check.
        A Warning is also turned into a boolean-style command line option.

<commit_msg>warn about implicit None<commit_after>This is a re-write of pychecker using traditional parse tree
techniques, rather than the bytecode.

There are some other design changes, related to Warning objects and
Option handling.  In general, pychecker2 tries to distribute checks to
separate classes.  These classes are more self-contained: they hold
onto the option variables and define the Warning objects they support.

ToDo:
        All missing stuff
        Always returning None
        More path analysis
            used before set for if/else try/except
        Warn about unpacking args:
             def f(a, (b, c)): pass

Items still missing:
        Error suppression (errors may be suppressed globally)

        Detail of exception is an exception
        Local used after deleted, already deleted, deleted before set
        Not calling base class __init__
        Using return value which is always None
        Wrong # of args on function (including ctor)
        Function arguments, kwargs
        Abstract methods
        exec may use locals/globals

        Modifying a default parameter
        Tuple access to list list[1, 2]
        List .append() takes one argument
        Return None from __getattr__
        string iteration
        Inconsistent return type
        Object has no attr
        Method read as attribute, but not called:
        Function always returns None (use empty return)
        Unpack wrong size
        Unpack wrong type

        Exec warnings
        Security warning: input()

        Module imports itself
        Complexity
        Doc strings
        New python features:
             slots
             properties
             integer division
        Warning for string raising

Bugs:
        More member not used
        Speed
        Is __file__ a builtin?

        Aliases:
            class C:
                  def f(a, b): pass
                  def g(a, b): pass
                  h = f
                  h = g			# redef of h, not caught now

        foo = 1
        del foo	                        # should not be unused

        from __future__ import produces unused warnings
        import of readline has side effect on input/raw_input, and should be
           marked as used

Important stuff:

If you want to add new checks, there are some parts to be aware of:

        File
                A stupid data structure to hold everything we know about 
                a file to be checked.

                Members:
                   name
                        The file name.

                   parseTree
                        The root of the Abstract Syntax Tree returned from
                        compiler.parseFile.  Will be None if the file
                        won't even parse.

                   scopes
                        A dictionary of { node, scope } computed with
                        from compiler.symbols. For convenience, scopes
                        also refer back to the node and to the
                        enclosing parent scope (node and parent,
                        respectively).

                    root_scope
                        Convience for scopes[parseTree]

        Check
                Basic step for checking a file.  Usually contributes
                Warnings or additional information to File.  New checks
                will subclass this, and be added to the chain of checks
                given in main().

        Warning
                A description of a warning emitted by pychecker.

        CheckList
                Contains "global" check information: cached module
                data and the list of Check objects.

        Command line options are pulled from each check.
        A Warning is also turned into a boolean-style command line option.


{'fraction_non_alphanumeric': 0.027438624130235, 'fraction_numerical': 0.0010502822633582775, 'mean_word_length': 1.5050970075633014, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6846668', 'n_tokens_mistral': 1714, 'n_tokens_neox': 1574, 'n_words': 859}