<commit_before>﻿.. _Configuration:

================================
The OpenDSA Configuration System
================================

----------
Motivation
----------

* Allows content to be environment-independent

  * Configuration file contains all environment-dependent settings such as paths and target URLs
  * Example: If developers want to point their front-end code at different backend systems, they simply make the change in their own config file.  They can share all OpenDSA content, but when they build the book, it will be built using their personalized settings

* Allows easy replication

  * Collects all settings and preferences required to configure an instance of OpenDSA in a single, portable file that can be easily shared among instructors.
  * Once a configuration has been created, instructors can make identical copies without going through the configuration process

* Allows fine grain control

  * Existing configuration files provide sensible defaults, but allow instructors to control aspects such as how many points a specific exercise is worth or whether it is required for module proficiency

* Configuration file can be generated by a simple front-end or created by hand (tedious, but possible)


---------------------
Configuration Process
---------------------

Please see the comment at the beginning of configure.py for more information about how the script works.  Keeping the information in the script itself ensures the script is well-documented and makes it easier to maintain the documentation.

**Note**: The OpenDSA root directory must be web-accessible as there are many supplemental directories and files (AV/, Exercises/, lib/, SourceCode/, etc) which must be referenced.  These files are identical for all books and not copying them reduces the footprint of each book.


Module and Exercise Removal
===========================

* Only the modules listed in the configuration file will be included.  To remove a module from the book, simply remove the module object from the configuration file.
* To remove an exercise from a module, set the "remove" attribute to true.  Exercises that do not appear in the configuration file will still be included in the book using the default configuration options.  During configuration, a list will be printed of any exercises which were encountered in the modules but not present in the configuration file.


Future Features
===============

* Implement independent source locations for various files rather than basing them all off the OpenDSA root directory

  * Implement support for hosting AVs and exercises on a different domain than modules

* Ability to optionally include subsections in an RST file - similar to how we include / exclude exercises


------
Format
------

* OpenDSA configuration files are stored using JSON

  * Easily readable and writeable by both humans and machines

    * Easy to create and manipulate using Python and JavaScript
    * Easy to manually edit

  * Efficient

    * Less verbose than XML resulting in a smaller file size

  * Abundance of online JSON validators and formatters

* Book name

  * The name of the configuration file will be used to uniquely identify this instance of OpenDSA in the backend database


Settings (all are required unless otherwise specified)
======================================================

* **title** - the title of the OpenDSA textbook

* **build_dir** - (optional) the directory where the configured book directory will be created, defaults to 'Books' if omitted

  * A new directory, named after the configuration file, will be created at this location and serve as the output directory for the configuration process.  Files required to compile the book will be copied / written to the output directory, including modified version of the source RST files

    * Example: Assume "build_dir": "Books" and the name of the configuration file used is "CS3114.json", the output directory would be "Books/CS3114/"

  * The compiled textbook will appear in ``[build_dir]/[book name]/html``
  * This directory must be web accessible

* **code_dir** - specifies the directory which contains another directory whose name matches ``code_lang`` (see below) which contains the source code used in examples

  * Ex: If ``{"code_dir": "SourceCode/", "code_lang": "python"}`` then the book would look for example Python source code in ``~OpenDSA/SourceCode/python``

* **code_lang** - specifies the programming language used for examples and exercises throughout the book

* **module_origin** - the protocol and domain where the module files are hosted

  * Used by embedded exercises as the target of HTML5 post messages which send information to the parent (module) page
  * Ex: "module_origin": "http://algoviz.org",

* **av_root_dir** - (optional) allows the user to change the default location where the AV/ directory can be found, defaults to OpenDSA root directory if omitted

  * This can point to another location on the same machine that hosts the module files (as long as it is web-accessible) or it can point to a remote location (not fully supported yet)
  * **Note**: This should not point to the AV/ directory itself, but the directory containing the AV/ directory (to avoid breaking the relative paths in the RST files)
  * If this attribute references a remote location, 'av_origin' must be present and the value must be a prefix of the remote location
  * If this attribute is not present, the OpenDSA root directory will be used as the default
  * Ex: "av_root_dir": "/home/algoviz/OpenDSA/test/",
  * Ex: "av_root_dir": "http://algoviz.org/OpenDSA/", // This directory contains an AV/ directory

* **av_origin** - (optional, unless **av_root_dir** is defined) the protocol and domain where the AV files are hosted, defaults to match ``module_origin`` if omitted

  * Used on module pages to allow HTML5 post messages from this origin, allows embedded AVs to communicate with the parent module page
  * Ex: "av_origin": "http://algoviz.org",

* **glob_mod_options** - (optional) an object containing options applied to every module in the book, allow settings specific to a certain module to be passed to that module alone, can be overridden by module-specific options

* **glob_exer_options** - (optional) an object containing options applied to every exercise in the book, can be used to control the behavior of the exercise, can be overridden by exercise-specific options

* **exercises_root_dir** - (optional) allows the user to change the default location where the Exercises/ directory can be found, defaults to OpenDSA root directory if omitted

  * This can point to another location on the same machine that hosts the module files (as long as it is web-accessible) or it can point to a remote location (not fully supported yet)
  * **Note**: This should not point to the Exercises/ directory itself, but the directory containing the Exercises/ directory (to avoid breaking the relative paths in the RST files)
  * If this attribute references a remote location, 'exercise_origin' must be present and the value must be a prefix of the remote location
  * If this attribute is not present, the OpenDSA root directory will be used as the default
  * Ex: "exercises_root_dir": "/home/algoviz/OpenDSA/test/",
  * Ex: "exercises_root_dir": "http://algoviz.org/OpenDSA/", // This directory contains an Exercises/ directory

* **exercise_origin** - (optional, unless **exercises_root_dir** is defined) the protocol and domain where the Exercises files are hosted, defaults to match ``module_origin`` if omitted

  * Used on module pages to allow HTML5 post messages from this origin, allows embedded exercises to communicate with the parent module page
  * Ex: "exercise_origin": "http://algoviz.org",

* **backend_address** - (optional) the protocol and domain (and port number, if different than the protocol default) of the backend server which supports centralized user scoring and interaction data collection, defaults to an empty string (backend server disabled) if omitted

  * Trailing '/' is optional
  * Ex: "backend_address": "https://opendsa.cc.vt.edu/",

* **build_JSAV** - (optional) a boolean controlling whether or not the JSAV library should be rebuild every time the configuration file is run, defaults to ``false`` if omitted

  * This value should be set to false for development
  * Instructors may wish to set this to true for production environments when configuration is run infrequently and JSAV is likely to have changed since the last time configuration occurred

* **allow_anonymous_credit** - (optional) a boolean controlling whether credit for exercises completed anonymously (without logging in) will be transferred to the next user to log in, defaults to ``true`` if omitted

* **req_full_ss** - (optional) a boolean controlling whether students must view every step of a slideshow in order to obtain credit, defaults to ``true`` if omitted

* **start_chap_num** - (optional) specifies at which number to start numbering chapters, defaults to 0 if omitted

* **suppress_todo** - (optional) a boolean controlling whether or not the TODO directives are removed from the RST source files, if this attribute is not present, defaults to ``false`` if omitted

  * **Note**: When changing from ``false`` to ``true``, you must run ``make clean`` to completely remove any references to ``ToDo``

* **assumes** - (optional) a string containing a semi-colon delimited list of topics with which the book assumes students are familiar

* **chapters** - this object contains a hierarchy of chapters, sections, subsections, modules and exercises

  * Each key in "chapters" represents a chapter name, any key values in the associated object represent sections within that chapter.  This concept is applied recursively until a module object is reached.  A module object is one whose key matches the name of an RST file in the ~OpenDSA/RST/source/ directory and which contains the key "exercises".

  * **hidden** - This is an optional field to signal the preprocessor to not display the content of the chapter in the TOC. The configuration script will add the new directive ``odsatoctree``. The flagged chapters entries in the TOC will be of class ``hide-from-toc``, and will be removed by a CSS rule in odsaMOD.css file.

  * Modules

    * The key relating to each module object must correspond with a path to an RST file found in ~OpenDSA/RST/source/ or a sub-directory
    * If a module contains no exercises, it should still contain a key "exercises" with a value of an empty object

    * **long_name** - a long form, human-readable name used to identify the module in the GUI

    * **dispModComp** - (optional) a flag which if set to "true" will force the "Module Complete" message to appear even if the module contains no required exercises, if set to "false", the "Module Complete" message will not appear even if the module DOES contain required exercises

    * **mod_options** - (optional) overrides ``glob_mod_options``, allows modules to be configured independently from one another.  Can be used to override the options set using ``glob_mod_options``. Options that should be stored in ``JSAV_OPTIONS`` should be prepended with ``JOP-`` and options that should be stored in ``JSAV_EXERCISE_OPTIONS`` should be prepended with ``JXOP-`` (can be used to override the defaults set in ``odsaUtils.js``).  All other options will be made directly available to modules in the form of a parameters object created automatically by the client-side framework (specifically whn ``parseURLParams()`` is called in ``odsaUtils.js``)

    * **exercises** - a collection of exercise objects representing the exercises found in the module's RST file

      * Omitting an exercise from the module's "exercises" object will cause the exercise to be removed from the configured module
      * Each exercise object contains required information about that exercise including:

        * **exer_options** - (optional) an object containing exercise-specific configuration options for JSAV.  Can be used to override the options set using ``glob_exer_options``. Options that should be stored in ``JSAV_OPTIONS`` should be prepended with ``JOP-`` and options that should be stored in ``JSAV_EXERCISE_OPTIONS`` should be prepended with ``JXOP-`` (can be used to override the defaults set in ``odsaUtils.js``).  All other options will be made directly available to exercises in the form of a parameters object created automatically by the client-side framework (specifically whn ``parseURLParams()`` is called in ``odsaUtils.js``)

        * **long_name** - (optional) a long form, human-readable name used to identify the exercise in the GUI, defaults to short exercise name if omitted
        * **points** - (optional) the number of points the exercise is worth, defaults to ``0`` if omitted
        * **remove** - (optional) if set to true, the exercise will not be included in the module

          * This option can only be used with exercises embedded using the 'avembed' directive.  Slideshows and diagrams created using the 'inlineav' directive are considered content and cannot be removed via the configuration file.

        * **required** - (optional) whether the exercise is required for module proficiency, defaults to ``false`` if omitted
        * **showhide** - (optional) controls whether or not the exercises is displayed and a Show / Hide button created

          * Setting this attribute to "show" or "hide" will create the corresponding button and if set to "show", it will display the embedded page
          * Setting this attribute to "none" will cause the exercise to be embedded without a button
          * If this attribute is not present in the config file, the default behavior is 'hide'
          * This option is only valid for exercises embedded using the 'avembed' directive.  Slideshows and diagrams created using the 'inlineav' directive are considered content and cannot be removed via the configuration file.

        * **threshold** - (optional) the percentage a user needs to score on the exercise to obtain proficiency, defaults to 100% (1 on a 0-1 scale) if omitted

      * JSAV-based diagrams do not need to be listed
<commit_msg>Updated configuration documentation to include the new 'lang' option<commit_after>﻿.. _Configuration:

================================
The OpenDSA Configuration System
================================

----------
Motivation
----------

* Allows content to be environment-independent

  * Configuration file contains all environment-dependent settings such as paths and target URLs
  * Example: If developers want to point their front-end code at different backend systems, they simply make the change in their own config file.  They can share all OpenDSA content, but when they build the book, it will be built using their personalized settings

* Allows easy replication

  * Collects all settings and preferences required to configure an instance of OpenDSA in a single, portable file that can be easily shared among instructors.
  * Once a configuration has been created, instructors can make identical copies without going through the configuration process

* Allows fine grain control

  * Existing configuration files provide sensible defaults, but allow instructors to control aspects such as how many points a specific exercise is worth or whether it is required for module proficiency

* Configuration file can be generated by a simple front-end or created by hand (tedious, but possible)


---------------------
Configuration Process
---------------------

Please see the comment at the beginning of configure.py for more information about how the script works.  Keeping the information in the script itself ensures the script is well-documented and makes it easier to maintain the documentation.

**Note**: The OpenDSA root directory must be web-accessible as there are many supplemental directories and files (AV/, Exercises/, lib/, SourceCode/, etc) which must be referenced.  These files are identical for all books and not copying them reduces the footprint of each book.


Module and Exercise Removal
===========================

* Only the modules listed in the configuration file will be included.  To remove a module from the book, simply remove the module object from the configuration file.
* To remove an exercise from a module, set the "remove" attribute to true.  Exercises that do not appear in the configuration file will still be included in the book using the default configuration options.  During configuration, a list will be printed of any exercises which were encountered in the modules but not present in the configuration file.


Future Features
===============

* Implement independent source locations for various files rather than basing them all off the OpenDSA root directory

  * Implement support for hosting AVs and exercises on a different domain than modules

* Ability to optionally include subsections in an RST file - similar to how we include / exclude exercises


------
Format
------

* OpenDSA configuration files are stored using JSON

  * Easily readable and writeable by both humans and machines

    * Easy to create and manipulate using Python and JavaScript
    * Easy to manually edit

  * Efficient

    * Less verbose than XML resulting in a smaller file size

  * Abundance of online JSON validators and formatters

* Book name

  * The name of the configuration file will be used to uniquely identify this instance of OpenDSA in the backend database


Settings (all are required unless otherwise specified)
======================================================

* **title** - the title of the OpenDSA textbook

* **build_dir** - (optional) the directory where the configured book directory will be created, defaults to 'Books' if omitted

  * A new directory, named after the configuration file, will be created at this location and serve as the output directory for the configuration process.  Files required to compile the book will be copied / written to the output directory, including modified version of the source RST files

    * Example: Assume "build_dir": "Books" and the name of the configuration file used is "CS3114.json", the output directory would be "Books/CS3114/"

  * The compiled textbook will appear in ``[build_dir]/[book name]/html``
  * This directory must be web accessible

* **code_dir** - specifies the directory which contains another directory whose name matches ``code_lang`` (see below) which contains the source code used in examples

  * Ex: If ``{"code_dir": "SourceCode/", "code_lang": "python"}`` then the book would look for example Python source code in ``~OpenDSA/SourceCode/python``

* **lang** (optional) - specifies the native language of the book using the official ISO 639-1 or 639-2 standard abbreviation, defaults to ``en`` if omitted

* **code_lang** - specifies the programming language used for examples and exercises throughout the book

* **module_origin** - the protocol and domain where the module files are hosted

  * Used by embedded exercises as the target of HTML5 post messages which send information to the parent (module) page
  * Ex: "module_origin": "http://algoviz.org",

* **av_root_dir** - (optional) allows the user to change the default location where the AV/ directory can be found, defaults to OpenDSA root directory if omitted

  * This can point to another location on the same machine that hosts the module files (as long as it is web-accessible) or it can point to a remote location (not fully supported yet)
  * **Note**: This should not point to the AV/ directory itself, but the directory containing the AV/ directory (to avoid breaking the relative paths in the RST files)
  * If this attribute references a remote location, 'av_origin' must be present and the value must be a prefix of the remote location
  * If this attribute is not present, the OpenDSA root directory will be used as the default
  * Ex: "av_root_dir": "/home/algoviz/OpenDSA/test/",
  * Ex: "av_root_dir": "http://algoviz.org/OpenDSA/", // This directory contains an AV/ directory

* **av_origin** - (optional, unless **av_root_dir** is defined) the protocol and domain where the AV files are hosted, defaults to match ``module_origin`` if omitted

  * Used on module pages to allow HTML5 post messages from this origin, allows embedded AVs to communicate with the parent module page
  * Ex: "av_origin": "http://algoviz.org",

* **glob_mod_options** - (optional) an object containing options applied to every module in the book, allow settings specific to a certain module to be passed to that module alone, can be overridden by module-specific options

* **glob_exer_options** - (optional) an object containing options applied to every exercise in the book, can be used to control the behavior of the exercise, can be overridden by exercise-specific options

* **exercises_root_dir** - (optional) allows the user to change the default location where the Exercises/ directory can be found, defaults to OpenDSA root directory if omitted

  * This can point to another location on the same machine that hosts the module files (as long as it is web-accessible) or it can point to a remote location (not fully supported yet)
  * **Note**: This should not point to the Exercises/ directory itself, but the directory containing the Exercises/ directory (to avoid breaking the relative paths in the RST files)
  * If this attribute references a remote location, 'exercise_origin' must be present and the value must be a prefix of the remote location
  * If this attribute is not present, the OpenDSA root directory will be used as the default
  * Ex: "exercises_root_dir": "/home/algoviz/OpenDSA/test/",
  * Ex: "exercises_root_dir": "http://algoviz.org/OpenDSA/", // This directory contains an Exercises/ directory

* **exercise_origin** - (optional, unless **exercises_root_dir** is defined) the protocol and domain where the Exercises files are hosted, defaults to match ``module_origin`` if omitted

  * Used on module pages to allow HTML5 post messages from this origin, allows embedded exercises to communicate with the parent module page
  * Ex: "exercise_origin": "http://algoviz.org",

* **backend_address** - (optional) the protocol and domain (and port number, if different than the protocol default) of the backend server which supports centralized user scoring and interaction data collection, defaults to an empty string (backend server disabled) if omitted

  * Trailing '/' is optional
  * Ex: "backend_address": "https://opendsa.cc.vt.edu/",

* **build_JSAV** - (optional) a boolean controlling whether or not the JSAV library should be rebuild every time the configuration file is run, defaults to ``false`` if omitted

  * This value should be set to false for development
  * Instructors may wish to set this to true for production environments when configuration is run infrequently and JSAV is likely to have changed since the last time configuration occurred

* **allow_anonymous_credit** - (optional) a boolean controlling whether credit for exercises completed anonymously (without logging in) will be transferred to the next user to log in, defaults to ``true`` if omitted

* **req_full_ss** - (optional) a boolean controlling whether students must view every step of a slideshow in order to obtain credit, defaults to ``true`` if omitted

* **start_chap_num** - (optional) specifies at which number to start numbering chapters, defaults to 0 if omitted

* **suppress_todo** - (optional) a boolean controlling whether or not the TODO directives are removed from the RST source files, if this attribute is not present, defaults to ``false`` if omitted

  * **Note**: When changing from ``false`` to ``true``, you must run ``make clean`` to completely remove any references to ``ToDo``

* **assumes** - (optional) a string containing a semi-colon delimited list of topics with which the book assumes students are familiar

* **chapters** - this object contains a hierarchy of chapters, sections, subsections, modules and exercises

  * Each key in "chapters" represents a chapter name, any key values in the associated object represent sections within that chapter.  This concept is applied recursively until a module object is reached.  A module object is one whose key matches the name of an RST file in the ~OpenDSA/RST/source/ directory and which contains the key "exercises".

  * **hidden** - This is an optional field to signal the preprocessor to not display the content of the chapter in the TOC. The configuration script will add the new directive ``odsatoctree``. The flagged chapters entries in the TOC will be of class ``hide-from-toc``, and will be removed by a CSS rule in odsaMOD.css file.

  * Modules

    * The key relating to each module object must correspond with a path to an RST file found in ~OpenDSA/RST/source/ or a sub-directory
    * If a module contains no exercises, it should still contain a key "exercises" with a value of an empty object

    * **long_name** - a long form, human-readable name used to identify the module in the GUI

    * **dispModComp** - (optional) a flag which if set to "true" will force the "Module Complete" message to appear even if the module contains no required exercises, if set to "false", the "Module Complete" message will not appear even if the module DOES contain required exercises

    * **mod_options** - (optional) overrides ``glob_mod_options``, allows modules to be configured independently from one another.  Can be used to override the options set using ``glob_mod_options``. Options that should be stored in ``JSAV_OPTIONS`` should be prepended with ``JOP-`` and options that should be stored in ``JSAV_EXERCISE_OPTIONS`` should be prepended with ``JXOP-`` (can be used to override the defaults set in ``odsaUtils.js``).  All other options will be made directly available to modules in the form of a parameters object created automatically by the client-side framework (specifically whn ``parseURLParams()`` is called in ``odsaUtils.js``)

    * **exercises** - a collection of exercise objects representing the exercises found in the module's RST file

      * Omitting an exercise from the module's "exercises" object will cause the exercise to be removed from the configured module
      * Each exercise object contains required information about that exercise including:

        * **exer_options** - (optional) an object containing exercise-specific configuration options for JSAV.  Can be used to override the options set using ``glob_exer_options``. Options that should be stored in ``JSAV_OPTIONS`` should be prepended with ``JOP-`` and options that should be stored in ``JSAV_EXERCISE_OPTIONS`` should be prepended with ``JXOP-`` (can be used to override the defaults set in ``odsaUtils.js``).  All other options will be made directly available to exercises in the form of a parameters object created automatically by the client-side framework (specifically whn ``parseURLParams()`` is called in ``odsaUtils.js``)

        * **long_name** - (optional) a long form, human-readable name used to identify the exercise in the GUI, defaults to short exercise name if omitted
        * **points** - (optional) the number of points the exercise is worth, defaults to ``0`` if omitted
        * **remove** - (optional) if set to true, the exercise will not be included in the module

          * This option can only be used with exercises embedded using the 'avembed' directive.  Slideshows and diagrams created using the 'inlineav' directive are considered content and cannot be removed via the configuration file.

        * **required** - (optional) whether the exercise is required for module proficiency, defaults to ``false`` if omitted
        * **showhide** - (optional) controls whether or not the exercises is displayed and a Show / Hide button created

          * Setting this attribute to "show" or "hide" will create the corresponding button and if set to "show", it will display the embedded page
          * Setting this attribute to "none" will cause the exercise to be embedded without a button
          * If this attribute is not present in the config file, the default behavior is 'hide'
          * This option is only valid for exercises embedded using the 'avembed' directive.  Slideshows and diagrams created using the 'inlineav' directive are considered content and cannot be removed via the configuration file.

        * **threshold** - (optional) the percentage a user needs to score on the exercise to obtain proficiency, defaults to 100% (1 on a 0-1 scale) if omitted

      * JSAV-based diagrams do not need to be listed

{'fraction_non_alphanumeric': 0.0758731153591699, 'fraction_numerical': 0.001601727079633692, 'mean_word_length': 4.279411764705882, 'pattern_counts': {'":': 22, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6116935', 'n_tokens_mistral': 6932, 'n_tokens_neox': 6641, 'n_words': 4001}