<commit_before>
Configuration

Configuration is documented on a per-use case: if a class uses a certain
value from the configuration object, it has to define its name and what the
value is used for.  This means decentralized configuration declarations that
are nevertheless error checking and a centralized configuration object.

Directives are divided into namespaces, indicating the major portion of
functionality they cover (although there may be overlaps.  Please consult
the documentation in ConfigDef for more information on these namespaces.

Since configuration is dependent on context, most of the internal classes
require a configuration object to be passed as a parameter.  However, a few
make this optional: they will supply a default configuration object if none
are passed.  These classes are: HTMLPurifier::*, Generator::generateFromTokens
and Lexer::tokenizeHTML.  However, whenever a valid configuration object
is defined, that object should be used.
<commit_msg>Detail structural changes involving Config and *Definition<commit_after>
Configuration

Configuration is documented on a per-use case: if a class uses a certain
value from the configuration object, it has to define its name and what the
value is used for.  This means decentralized configuration declarations that
are nevertheless error checking and a centralized configuration object.

Directives are divided into namespaces, indicating the major portion of
functionality they cover (although there may be overlaps.  Please consult
the documentation in ConfigDef for more information on these namespaces.

Since configuration is dependent on context, most of the internal classes
require a configuration object to be passed as a parameter.  However, a few
make this optional: they will supply a default configuration object if none
are passed.  These classes are: HTMLPurifier::*, Generator::generateFromTokens
and Lexer::tokenizeHTML.  However, whenever a valid configuration object
is defined, that object should be used.

In relation to HTMLDefinition and CSSDefinition, there are going to be some
major structural changes to enable the easy configuration of these objects.
Due to the intricacy of these objects, it's not feasible to ask an average
user to twiddle around with an element and its 20 other dependencies.  However,
these objects are the only possible point where change could occur in the
context of configuration.

The solution is to introduce a special class of directives that influence the
*construction* of the Definition object.  A standard call pattern would look
like:

1. Client calls Config->getHTMLDefinition()
2. Config calls HTMLDefinition->createNew(this)
3. HTMLDefinition constructs itself with base configuration
4. HTMLDefinition calls Config->get('HTMLDefinition')
5. Config returns array of directives that later construction
6. HTMLDefinition performs operations and changes specified by directives
7. HTMLPurifier returns constructed definition
8. Config caches definition so it doesn't have to be generated again
9. Config returns definition

You could also override Config's copy of the definition with your own
custom copy, which OVERRIDES all directives.  Only the base, vanilla copy
is the Singleton, the object actually interfaced with is a operated-upon
clone of that object.  Also, if an update to the directives would update
the definition, you'd have to force reconstruction.

{'fraction_non_alphanumeric': 0.03141514973576042, 'fraction_numerical': 0.0032295948326482677, 'mean_word_length': 5.262867647058823, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4774717', 'n_tokens_mistral': 750, 'n_tokens_neox': 719, 'n_words': 481}