<commit_before>#* This file is part of the MOOSE framework
#* https://www.mooseframework.org
#*
#* All rights reserved, see COPYRIGHT for full restrictions
#* https://github.com/idaholab/moose/blob/master/COPYRIGHT
#*
#* Licensed under LGPL 2.1, please see LICENSE for details
#* https://www.gnu.org/licenses/lgpl-2.1.html
"""Module for defining Executioner objects, which are helpers for tokenization and rendering."""
import sys
import os
import copy
import time
import logging
import traceback
import multiprocessing
import mooseutils
import random
import platform

import MooseDocs
from ..tree import pages
from ..common import mixins, exceptions

LOG = logging.getLogger('MooseDocs.Executioner')

class Executioner(mixins.ConfigObject, mixins.TranslatorObject):
    """
    Base class for executing the Markdown (MooseDown) to HTML translation procedure.

    A Translator object is responsible for managing the conversion of MooseDown content to the final
    HTML result via tokenization and rendering. However, how this process executes can vary given
    the variety of strategies offered by the Python multiprocessing package, of which several have
    been deemed suitable for MooseDocs. Thus, this class was created to provide an organized and
    well-defined API for executing the translation procedure in different ways and serve as the main
    driver of MooseDocs.

    This is an internal object that should not be used directly. The Translator object should
    provide the necessary interface for all low-level reader, renderer, or extension operations. If
    access to this object is needed for any purpose except to reference shared memory containers or
    processor locks, which should be avoided, then the Translator is not designed correctly.

    ***TRANSLATION PROCESSING***
    Significant effort has been made to correctly handle the data involved with translation using
    the multiprocessing package. Throughout the development of MooseDocs, the algorithm used has
    changed in an attempt to optimize performance. There has not been one method that has been ideal
    for all cases, so rather than be restricted to a single algorithm the Executioner system was
    created to allow for the algorithm to be changed as well as new ones to be developed.

    The exact translation process varies between Executioner objects and further depends on the
    inputs they are constructed with, but all proceed through the following basic steps:

        1. Execute preExecute methods
        2. Read raw MooseDown content from pages.Source objects
           2.1. Execute preRead methods
           2.2. Perform read using Reader.read method
           2.3. Execute postRead methods
        3. Tokenize pages.Source objects
           3.1. Execute preTokenize methods
           3.2. Perform tokenization using Reader.tokenize method
           3.3. Execute postTokenize methods
        4. Render pages.Source objects
           4.1. Execute preRender methods
           4.2. Perform rendering using Renderer.render method
           4.3. Execute postRender methods
        5. Write pages.Source objects
           5.1. Execute preWrite methods
           5.2. Perform write, using Renderer.write method
           5.3. Execute postWrite methods
        6. Finalize remaining content, i.e., copy pages.File objects to Translator.destination
        7. Execute postExecute methods

    Many MooseDocs objects like readers, renderers, and extensions define pre/post methods like
    postTokenize() or preRender(). The Translator object is responsible for invoking all matching
    occurrences of these methods upon request by this class. The read(), tokenize(), render(), and
    write() methods belong only to either the Reader or Renderer object, and tokenize() and render()
    perform the especially important task of invoking (via a trace of stack entries) the key
    functions of the Component objects defined in extension modules, such as createToken() or
    createHTML(). All of these distinct methods provide a lot of flexibility about where, when, and
    how content is received and translated.

    The various methods of this class and all those of its subclasses are designed to be invoked any
    number of times while the MooseDocs application is running and on any subset of Page objects.
    The main purpose of this design is to support translation of existing pages edited, or new pages
    added, during a live reload (see the MooseDocsWatcher class in 'commands/build.py'). This design
    also makes it possible to call the methods to perform specific steps of the translation process,
    e.g., to perform only the read and tokenize steps. The rule, though, is that successive
    invocations must follow the basic procedure: read -> tokenize -> render -> write. For example,
    if only the read step has been performed, the Executioner must be called to tokenize before it
    can be called to render. The progress checks that enforce this rule are reset during the write
    step, and the entire procedure must be repeated if called upon thereafter.

    ***PERFORMANCE REVIEW, SEPTEMBER 2021***
    The table below presents the results of a procedure to benchmark each of the Executioner classes
    provided by this module. The methodology consisted of executing the MooseDocs application
    multiple times per class: A dry run intended to initialize the Python JIT compilers and prepare
    buffer/cache memory for the write step, followed by two consecutive control runs for which the
    peak amount of RAM consumed by, and duration of, translation steps were recorded. The results
    reported are the mean averages of the two control runs. For parallel executioners, the procedure
    was conducted with both 4 and 8 CPUs. Prior to each test case, the system was rebooted to help
    control for potential interference by accumulated memory leaks or background processes.

    The specimen involved was the primary configuration of the MOOSE website, which includes the
    content for all physics modules but not any of the slideshows (e.g., the workshop). For example,
    a single run of the "ParallelBarrier [x8]" test case was executed using the following Bash
    commands from the root directory of the MOOSE repository:

        ```
        cd modules/doc
        ./moosedocs.py build -j8 --config config.yml --executioner MooseDocs.base.ParallelBarrier
        ```

    The apparatus was a Dell XPS 8940 with Ubuntu 20.04.3 LTS (64-bit) as the OS. The relevant
    system specifications are as follows:
        - Processor: Intel Core i7-11700K (8-Core, 16 CPUs, 16M Cache, 3.6-5.0GHz)
        - Memory: 32GB (16Gx2), DDR4, 2933MHz
        - Disk: 1TB M.2 PCIe NVME SSD

    It is important to note that the 'build' command disabled the SQA, Civet, and GitUtils
    extensions in all runs in accordance with its default behavior. Also note that, with the
    exception of the Serial class, the executioners randomize the order of the Page objects in the
    input list of those to be translated. However, this task was not performed in any of the test
    cases (by temporarily removing the corresponding lines of code) for the sake of fairness.

                           +---------------------+---------------------+----------+
                           |    READ/TOKENIZE    |    RENDER/WRITE     |  TOTAL*  |
    +----------------------+----------+----------+----------+----------+----------+
    |EXECUTIONER [CPUs]    |  RAM (GB)|  TIME (s)|  RAM (GB)|  TIME (s)|  TIME (s)|
    +----------------------+----------+----------+----------+----------+----------+
    |Serial                |       1.2|      75.2|       3.7|     161.2|     243.4|
    +----------------------+----------+----------+----------+----------+----------+
    |ParallelQueue [x4]    |       2.3|      35.4|       9.8|      76.7|     119.0|
    |ParallelQueue [x8]    |       2.7|      33.4|      11.7|      63.4|     103.9|
    +----------------------+----------+----------+----------+----------+----------+
    |ParallelBarrier [x4]  |       1.9|      53.8|       7.4|     121.0|     182.0|
    |ParallelBarrier [x8]  |       2.8|      47.7|       8.9|     113.1|     168.0|
    +----------------------+----------+----------+----------+----------+----------+
    |ParallelPipe [x4]     |       1.4|      48.5|       6.9|      99.8|     155.2|
    |ParallelPipe [x8]     |       1.5|      41.2|       9.5|      88.0|     136.2|
    +----------------------+----------+----------+----------+----------+----------+
        *Includes the durations of the preExecute, finalize, and postExecute steps.

    The data given above benchmarks the complete content of the main MOOSE website consisting of
    2,780 pages.Source objects as of September 8, 2021. The mean execution times will vary and most
    likely increase indefinitely over time as more content is added. However, it seems reasonable to
    assume that the difference in execution times between any two executioners would scale
    proportionally to those indicated here. The fastest executioner for any given configuration
    will depend on the amount and content being translated, as well as perhaps the computer running
    the application. But if the main MOOSE website content is assumed to represent the typical build
    job, then the data suggests that the ParallelQueue executioner is the highest performing one
    and, thus, should be the default.

    Each Executioner class has its advantages. For example, although ParallelPipe is not as fast as
    ParallelQueue in this study, it consumes less RAM, which becomes a precious resource as the
    amount required for the render and write steps approaches the usual amount available on economic
    workstations of 16 GB. ParallelBarrier seems to be the slowest parallel type, but it performs
    garbage collection of shared memory containers during runtime - a task that other executioners
    spend an amount of time doing not shown here while MooseDocs shuts down. As yet another example,
    the Reveal type renderer currently combines all slides into a single HTML page, and so it makes
    sense to use the Serial executioner for these configurations. Thus, the default executioner
    should be determined on a case-by-case basis, and benchmarking for all should be done following
    periods of significant content additions and/or updates in MooseDocs.

    TODO: As mentioned, execution times will most likely increase in the future. The build job for
    the main MOOSE website will only become more and more expensive to the point where the current
    infrastructure is no longer practical. For this imminent scenario, two solutions are proposed:
        1) Develop an executioner that makes use of the concurrent.futures Python module (see
           https://docs.python.org/3/library/concurrent.futures.html), which may provide
           capabilities and performance levels that are superior to the multiprocessing package.
        2) Cache the resulting site folder, either on a repository or from the initial build job a
           user runs (as it already does this but for EACH build job), and develop a change tracking
           system based on diffs or timestamps of the content files so it is known which need to be
           re-translated. Then, build only those Page objects as well as any of their dependencies.
    """

    @staticmethod
    def defaultConfig():
        config = mixins.ConfigObject.defaultConfig()
        config['profile'] = (False, "Run the executioner with profiling.")
        return config

    def __init__(self, **kwargs):
        mixins.ConfigObject.__init__(self, **kwargs)
        mixins.TranslatorObject.__init__(self)
        self._page_objects = dict()
        self._total_time = 0
        self._clear_progress()

        # The method for spawning processes changed to "spawn" for macOS in python 3.9. Currently,
        # MooseDocs does not work with this combination.
        if (platform.python_version() >= '3.9') and (platform.system() == 'Darwin'):
            self._ctx = multiprocessing.get_context('fork')
        else:
            self._ctx = multiprocessing.get_context()

        # A lock used prior to caching items or during directory creation to prevent race conditions<commit_msg>Fixup MooseDocs executioner class for 3.10 on Darwin (#21767)<commit_after>#* This file is part of the MOOSE framework
#* https://www.mooseframework.org
#*
#* All rights reserved, see COPYRIGHT for full restrictions
#* https://github.com/idaholab/moose/blob/master/COPYRIGHT
#*
#* Licensed under LGPL 2.1, please see LICENSE for details
#* https://www.gnu.org/licenses/lgpl-2.1.html
"""Module for defining Executioner objects, which are helpers for tokenization and rendering."""
import sys
import os
import copy
import time
import logging
import traceback
import multiprocessing
import mooseutils
import random
import platform

import MooseDocs
from ..tree import pages
from ..common import mixins, exceptions

LOG = logging.getLogger('MooseDocs.Executioner')

class Executioner(mixins.ConfigObject, mixins.TranslatorObject):
    """
    Base class for executing the Markdown (MooseDown) to HTML translation procedure.

    A Translator object is responsible for managing the conversion of MooseDown content to the final
    HTML result via tokenization and rendering. However, how this process executes can vary given
    the variety of strategies offered by the Python multiprocessing package, of which several have
    been deemed suitable for MooseDocs. Thus, this class was created to provide an organized and
    well-defined API for executing the translation procedure in different ways and serve as the main
    driver of MooseDocs.

    This is an internal object that should not be used directly. The Translator object should
    provide the necessary interface for all low-level reader, renderer, or extension operations. If
    access to this object is needed for any purpose except to reference shared memory containers or
    processor locks, which should be avoided, then the Translator is not designed correctly.

    ***TRANSLATION PROCESSING***
    Significant effort has been made to correctly handle the data involved with translation using
    the multiprocessing package. Throughout the development of MooseDocs, the algorithm used has
    changed in an attempt to optimize performance. There has not been one method that has been ideal
    for all cases, so rather than be restricted to a single algorithm the Executioner system was
    created to allow for the algorithm to be changed as well as new ones to be developed.

    The exact translation process varies between Executioner objects and further depends on the
    inputs they are constructed with, but all proceed through the following basic steps:

        1. Execute preExecute methods
        2. Read raw MooseDown content from pages.Source objects
           2.1. Execute preRead methods
           2.2. Perform read using Reader.read method
           2.3. Execute postRead methods
        3. Tokenize pages.Source objects
           3.1. Execute preTokenize methods
           3.2. Perform tokenization using Reader.tokenize method
           3.3. Execute postTokenize methods
        4. Render pages.Source objects
           4.1. Execute preRender methods
           4.2. Perform rendering using Renderer.render method
           4.3. Execute postRender methods
        5. Write pages.Source objects
           5.1. Execute preWrite methods
           5.2. Perform write, using Renderer.write method
           5.3. Execute postWrite methods
        6. Finalize remaining content, i.e., copy pages.File objects to Translator.destination
        7. Execute postExecute methods

    Many MooseDocs objects like readers, renderers, and extensions define pre/post methods like
    postTokenize() or preRender(). The Translator object is responsible for invoking all matching
    occurrences of these methods upon request by this class. The read(), tokenize(), render(), and
    write() methods belong only to either the Reader or Renderer object, and tokenize() and render()
    perform the especially important task of invoking (via a trace of stack entries) the key
    functions of the Component objects defined in extension modules, such as createToken() or
    createHTML(). All of these distinct methods provide a lot of flexibility about where, when, and
    how content is received and translated.

    The various methods of this class and all those of its subclasses are designed to be invoked any
    number of times while the MooseDocs application is running and on any subset of Page objects.
    The main purpose of this design is to support translation of existing pages edited, or new pages
    added, during a live reload (see the MooseDocsWatcher class in 'commands/build.py'). This design
    also makes it possible to call the methods to perform specific steps of the translation process,
    e.g., to perform only the read and tokenize steps. The rule, though, is that successive
    invocations must follow the basic procedure: read -> tokenize -> render -> write. For example,
    if only the read step has been performed, the Executioner must be called to tokenize before it
    can be called to render. The progress checks that enforce this rule are reset during the write
    step, and the entire procedure must be repeated if called upon thereafter.

    ***PERFORMANCE REVIEW, SEPTEMBER 2021***
    The table below presents the results of a procedure to benchmark each of the Executioner classes
    provided by this module. The methodology consisted of executing the MooseDocs application
    multiple times per class: A dry run intended to initialize the Python JIT compilers and prepare
    buffer/cache memory for the write step, followed by two consecutive control runs for which the
    peak amount of RAM consumed by, and duration of, translation steps were recorded. The results
    reported are the mean averages of the two control runs. For parallel executioners, the procedure
    was conducted with both 4 and 8 CPUs. Prior to each test case, the system was rebooted to help
    control for potential interference by accumulated memory leaks or background processes.

    The specimen involved was the primary configuration of the MOOSE website, which includes the
    content for all physics modules but not any of the slideshows (e.g., the workshop). For example,
    a single run of the "ParallelBarrier [x8]" test case was executed using the following Bash
    commands from the root directory of the MOOSE repository:

        ```
        cd modules/doc
        ./moosedocs.py build -j8 --config config.yml --executioner MooseDocs.base.ParallelBarrier
        ```

    The apparatus was a Dell XPS 8940 with Ubuntu 20.04.3 LTS (64-bit) as the OS. The relevant
    system specifications are as follows:
        - Processor: Intel Core i7-11700K (8-Core, 16 CPUs, 16M Cache, 3.6-5.0GHz)
        - Memory: 32GB (16Gx2), DDR4, 2933MHz
        - Disk: 1TB M.2 PCIe NVME SSD

    It is important to note that the 'build' command disabled the SQA, Civet, and GitUtils
    extensions in all runs in accordance with its default behavior. Also note that, with the
    exception of the Serial class, the executioners randomize the order of the Page objects in the
    input list of those to be translated. However, this task was not performed in any of the test
    cases (by temporarily removing the corresponding lines of code) for the sake of fairness.

                           +---------------------+---------------------+----------+
                           |    READ/TOKENIZE    |    RENDER/WRITE     |  TOTAL*  |
    +----------------------+----------+----------+----------+----------+----------+
    |EXECUTIONER [CPUs]    |  RAM (GB)|  TIME (s)|  RAM (GB)|  TIME (s)|  TIME (s)|
    +----------------------+----------+----------+----------+----------+----------+
    |Serial                |       1.2|      75.2|       3.7|     161.2|     243.4|
    +----------------------+----------+----------+----------+----------+----------+
    |ParallelQueue [x4]    |       2.3|      35.4|       9.8|      76.7|     119.0|
    |ParallelQueue [x8]    |       2.7|      33.4|      11.7|      63.4|     103.9|
    +----------------------+----------+----------+----------+----------+----------+
    |ParallelBarrier [x4]  |       1.9|      53.8|       7.4|     121.0|     182.0|
    |ParallelBarrier [x8]  |       2.8|      47.7|       8.9|     113.1|     168.0|
    +----------------------+----------+----------+----------+----------+----------+
    |ParallelPipe [x4]     |       1.4|      48.5|       6.9|      99.8|     155.2|
    |ParallelPipe [x8]     |       1.5|      41.2|       9.5|      88.0|     136.2|
    +----------------------+----------+----------+----------+----------+----------+
        *Includes the durations of the preExecute, finalize, and postExecute steps.

    The data given above benchmarks the complete content of the main MOOSE website consisting of
    2,780 pages.Source objects as of September 8, 2021. The mean execution times will vary and most
    likely increase indefinitely over time as more content is added. However, it seems reasonable to
    assume that the difference in execution times between any two executioners would scale
    proportionally to those indicated here. The fastest executioner for any given configuration
    will depend on the amount and content being translated, as well as perhaps the computer running
    the application. But if the main MOOSE website content is assumed to represent the typical build
    job, then the data suggests that the ParallelQueue executioner is the highest performing one
    and, thus, should be the default.

    Each Executioner class has its advantages. For example, although ParallelPipe is not as fast as
    ParallelQueue in this study, it consumes less RAM, which becomes a precious resource as the
    amount required for the render and write steps approaches the usual amount available on economic
    workstations of 16 GB. ParallelBarrier seems to be the slowest parallel type, but it performs
    garbage collection of shared memory containers during runtime - a task that other executioners
    spend an amount of time doing not shown here while MooseDocs shuts down. As yet another example,
    the Reveal type renderer currently combines all slides into a single HTML page, and so it makes
    sense to use the Serial executioner for these configurations. Thus, the default executioner
    should be determined on a case-by-case basis, and benchmarking for all should be done following
    periods of significant content additions and/or updates in MooseDocs.

    TODO: As mentioned, execution times will most likely increase in the future. The build job for
    the main MOOSE website will only become more and more expensive to the point where the current
    infrastructure is no longer practical. For this imminent scenario, two solutions are proposed:
        1) Develop an executioner that makes use of the concurrent.futures Python module (see
           https://docs.python.org/3/library/concurrent.futures.html), which may provide
           capabilities and performance levels that are superior to the multiprocessing package.
        2) Cache the resulting site folder, either on a repository or from the initial build job a
           user runs (as it already does this but for EACH build job), and develop a change tracking
           system based on diffs or timestamps of the content files so it is known which need to be
           re-translated. Then, build only those Page objects as well as any of their dependencies.
    """

    @staticmethod
    def defaultConfig():
        config = mixins.ConfigObject.defaultConfig()
        config['profile'] = (False, "Run the executioner with profiling.")
        return config

    def __init__(self, **kwargs):
        mixins.ConfigObject.__init__(self, **kwargs)
        mixins.TranslatorObject.__init__(self)
        self._page_objects = dict()
        self._total_time = 0
        self._clear_progress()

        # The method for spawning processes changed to "spawn" for macOS in python 3.9. Currently,
        # MooseDocs does not work with this combination.
        # Ints are used for the second comparison because simple string comparison is inadequate against 3.9 for >= 3.10
        if (platform.python_version_tuple()[0] >= '3') and (int(platform.python_version_tuple()[1]) >= 9) and (platform.system() == 'Darwin'):
            self._ctx = multiprocessing.get_context('fork')
        else:
            self._ctx = multiprocessing.get_context()

        # A lock used prior to caching items or during directory creation to prevent race conditions
{'fraction_non_alphanumeric': 0.09053646775165762, 'fraction_numerical': 0.017319670484227445, 'mean_word_length': 2.587429724664841, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 13, 'https://': 8, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1672534', 'n_tokens_mistral': 6238, 'n_tokens_neox': 5886, 'n_words': 3093}