<commit_before>Testing
=======

Overview
--------

We maintain three kinds of tests: unit tests, integration tests, and
acceptance tests.

Overall, you want to write the tests that **maximize coverage** while
**minimizing maintenance**. In practice, this usually means investing
heavily in unit tests, which tend to be the most robust to changes in
the code base.

.. figure:: test_pyramid.png
   :alt: Test Pyramid

   Test Pyramid

The pyramid above shows the relative number of unit tests, integration
tests, and acceptance tests. Most of our tests are unit tests or
integration tests.

Unit Tests
~~~~~~~~~~

-  Each test case should be concise: setup, execute, check, and
   teardown. If you find yourself writing tests with many steps,
   consider refactoring the unit under tests into smaller units, and
   then testing those individually.

-  As a rule of thumb, your unit tests should cover every code branch.

-  Mock or patch external dependencies. We use voidspace
   `Mock Library <http://www.voidspace.org.uk/python/mock/>`__.

-  We unit test Python code (using
   `unittest <http://docs.python.org/2/library/unittest.html>`__) and
   Javascript (using `Jasmine <http://jasmine.github.io/>`__)

Integration Tests
~~~~~~~~~~~~~~~~~

-  Test several units at the same time. Note that you can still mock or
   patch dependencies that are not under test! For example, you might
   test that ``LoncapaProblem``, ``NumericalResponse``, and
   ``CorrectMap`` in the ``capa`` package work together, while still
   mocking out template rendering.

-  Use integration tests to ensure that units are hooked up correctly.
   You do not need to test every possible input--that's what unit tests
   are for. Instead, focus on testing the "happy path" to verify that
   the components work together correctly.

-  Many of our tests use the `Django test
   client <https://docs.djangoproject.com/en/dev/topics/testing/overview/>`__
   to simulate HTTP requests to the server.

UI Acceptance Tests
~~~~~~~~~~~~~~~~~~~

-  Use these to test that major program features are working correctly.

-  We use `lettuce <http://lettuce.it/>`__ to write BDD-style tests.
   Most of these tests simulate user interactions through the browser
   using `splinter <http://splinter.cobrateam.info/>`__.

-  We use `Bok
   Choy <http://bok-choy.readthedocs.org/en/latest/tutorial.html>`__ to
   write end-user acceptance tests directly in Python, using the
   framework to maximize reliability and maintainability.

Test Locations
--------------

-  Python unit and integration tests: Located in subpackages called
   ``tests``. For example, the tests for the ``capa`` package are
   located in ``common/lib/capa/capa/tests``.

-  Javascript unit tests: Located in ``spec`` folders. For example,
   ``common/lib/xmodule/xmodule/js/spec`` and
   ``{cms,lms}/static/coffee/spec`` For consistency, you should use the
   same directory structure for implementation and test. For example,
   the test for ``src/views/module.coffee`` should be written in
   ``spec/views/module_spec.coffee``.

-  UI acceptance tests:

   -  Set up and helper methods, and stubs for external services:
      ``common/djangoapps/terrain``
   -  Lettuce Tests: located in ``features`` subpackage within a Django
      app. For example: ``lms/djangoapps/courseware/features``
   -  Bok Choy Acceptance Tests: located under ``common/test/acceptance/tests``
   -  Bok Choy Accessibility Tests: located under ``common/test/acceptance/tests`` and tagged with ``@attr("a11y")``
   -  Bok Choy PageObjects: located under ``common/test/acceptance/pages``

Factories
---------

Many tests delegate set-up to a "factory" class. For example, there are
factories for creating courses, problems, and users. This encapsulates
set-up logic from tests.

Factories are often implemented using
`FactoryBoy <https://readthedocs.org/projects/factoryboy/>`__

In general, factories should be located close to the code they use. For
example, the factory for creating problem XML definitions is located in
``common/lib/capa/capa/tests/response_xml_factory.py`` because the
``capa`` package handles problem XML.

Running Tests
=============

You can run all of the unit-level tests using this command.

::

    paver test

This includes python, javascript, and documentation tests. It does not,
however, run any acceptance tests.

Note -
`paver` is a scripting tool. To get information about various options, you can run the this command.
::
	paver -h
Running Python Unit tests
-------------------------

We use `nose <https://nose.readthedocs.org/en/latest/>`__ through the
`django-nose plugin <https://pypi.python.org/pypi/django-nose>`__ to run
the test suite.

For example, this command runs all the python test scripts.

::

    paver test_python

It also runs ``collectstatic``, which prepares the
static files used by the site (for example, compiling CoffeeScript to
JavaScript).

You can re-run all failed python tests by running this command (see note at end of
section).

::

    paver test_python --failed

To test lms python tests use this command.

::

    paver test_system -s lms

To test cms python tests use this command.

::

    paver test_system -s cms

To run these tests without ``collectstatic``, which is faster, append the following argument.

::

    paver test_system -s lms --fasttest

To run cms python tests without ``collectstatic`` use this command.

::

    paver test_system -s cms --fasttest

To run a single django test class use this command.

::

    paver test_system -t lms/djangoapps/courseware/tests/tests.py:ActivateLoginTest

When developing tests, it is often helpful to be able to really just run
one single test without the overhead of PIP installs, UX builds, etc. In
this case, it is helpful to look at the output of paver, and run just
the specific command (optionally, stripping away coverage metrics). At
the time of this writing, the command is the following.

::

    python ./manage.py lms test --verbosity=1 lms/djangoapps/courseware/tests/test_courses.py   --traceback --settings=test


To run a single test format the command like this.

::

    paver test_system -t lms/djangoapps/courseware/tests/tests.py:ActivateLoginTest.test_activate_login

To re-run all failing django tests from lms or cms, use the
``--failed``,\ ``-f`` flag (see note at end of section).

::

    paver test_system -s lms --failed
    paver test_system -s cms --failed

There is also a ``--fail_fast``, ``-x`` option that will stop nosetests
after the first failure.

common/lib tests are tested with the ``test_lib`` task, which also
accepts the ``--failed`` and ``--fail_fast`` options.

::

    paver test_lib -l common/lib/calc
    paver test_lib -l common/lib/xmodule --failed

For example, this command runs a single nose test file.

::

    nosetests common/lib/xmodule/xmodule/tests/test_stringify.py

This command runs a single nose test within a specified file.

::

    nosetests common/lib/xmodule/xmodule/tests/test_stringify.py:test_stringify


This is an example of how to run a single test and get stdout, with proper env config.

::

    python manage.py cms --settings test test contentstore.tests.test_import_nostatic -s

These are examples of how to run a single test and get stdout and get coverage.

::

    python -m coverage run --rcfile=./common/lib/xmodule/.coveragerc which ./manage.py cms --settings test test --traceback --logging-clear-handlers --liveserver=localhost:8000-9000 contentstore.tests.test_import_nostatic -s # cms example
    python -m coverage run --rcfile=./lms/.coveragerc which ./manage.py lms --settings test test --traceback --logging-clear-handlers --liveserver=localhost:8000-9000  courseware.tests.test_module_render -s # lms example

Use this command to generate coverage report.

::

    coverage report --rcfile=./common/lib/xmodule/.coveragerc

Use this command to generate an HTML report.

::

    coverage html --rcfile=./common/lib/xmodule/.coveragerc

The report is then saved in reports/common/lib/xmodule/cover/index.html

To run tests for stub servers, for example for `YouTube stub
server <https://github.com/edx/edx-platform/blob/master/common/djangoapps/terrain/stubs/tests/test_youtube_stub.py>`__,
you can run one of these commands.

::

    paver test_system -s cms -t common/djangoapps/terrain/stubs/tests/test_youtube_stub.py
    python -m coverage run --rcfile=cms/.coveragerc `which ./manage.py` cms --settings test test --traceback common/djangoapps/terrain/stubs/tests/test_youtube_stub.py

Very handy: if you pass the ``--pdb`` flag to a paver test function, or
uncomment the ``pdb=1`` line in ``setup.cfg``, the test runner
will drop you into pdb on error. This lets you go up and down the stack
and see what the values of the variables are. Check out `the pdb
documentation <http://docs.python.org/library/pdb.html>`__

Note: More on the ``--failed`` functionality

* In order to use this, you must run the tests first. If you haven't already
  run the tests, or if no tests failed in the previous run, then using the
  ``--failed`` switch will result in **all** of the tests being run. See more
  about this in the `nose documentation
  <http://nose.readthedocs.org/en/latest/plugins/testid.html#looping-over-failed-tests>`__.

* Note that ``paver test_python`` calls nosetests separately for cms and lms.
  This means that if tests failed only in lms on the previous run, then calling
  ``paver test_python --failed`` will run **all of the tests for cms** in
  addition to the previously failing lms tests. If you want it to run only the
  failing tests for lms or cms, use the ``paver test_system -s lms --failed``
  or ``paver test_system -s cms --failed`` commands.

Running Javascript Unit Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We use Jasmine to run JavaScript unit tests. To run all the JavaScript
tests::

    paver test_js

To run a specific set of JavaScript tests and print the results to the
console, run these commands.

::

    paver test_js_run -s lms
    paver test_js_run -s lms-coffee
    paver test_js_run -s cms
    paver test_js_run -s cms-squire
    paver test_js_run -s xmodule
    paver test_js_run -s common
    paver test_js_run -s common-requirejs

To run JavaScript tests in a browser, run these commands.

::

    paver test_js_dev -s lms
    paver test_js_dev -s lms-coffee
    paver test_js_dev -s cms
    paver test_js_dev -s cms-squire
    paver test_js_dev -s xmodule
    paver test_js_dev -s common
    paver test_js_dev -s common-requirejs

To debug these tests on devstack in a local browser:

 * first run the appropriate test_js_dev command from above which will open a browser using XQuartz
 * open the same URL in your browser but change the IP address to 192.168.33.10, e.g.
    http://192.168.33.10:TEST_PORT/suite/cms
 * this will run all the tests and show you the results including details of any failures
 * you can click on an individually failing test and/or suite to re-run it by itself
 * you can now use the browser's developer tools to debug as you would any other JavaScript code

Note: the port is also output to the console that you ran the tests from if you find that easier.

These paver commands call through to a custom test runner. For more
info, see `js-test-tool <https://github.com/edx/js-test-tool>`__.

Running Bok Choy Acceptance Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We use `Bok
Choy <http://bok-choy.readthedocs.org/en/latest/tutorial.html>`__ for
acceptance testing. Bok Choy is a UI-level acceptance test framework for
writing robust `Selenium <http://docs.seleniumhq.org/>`__ tests in
`Python <https://www.python.org/>`__. Bok Choy makes your acceptance
tests reliable and maintainable by utilizing the Page Object and Promise
design patterns.

**Prerequisites**:

These prerequisites are all automatically installed and available in `Devstack
<https://github.com/edx/configuration/wiki/edX-Developer-Stack>`__, the
supported development enviornment for the edX Platform.

* Chromedriver and Chrome (see Running Lettuce Acceptance Tests below for
  the latest tested versions)

* Mongo

* Memcache

* mySQL

To run all the bok choy acceptance tests run this command.

::

    paver test_bokchoy

Once the database has been set up and the static files collected, you
can use the 'fast' option to skip those tasks. This option can also be
used with any of the test specs below.

::

    paver test_bokchoy --fasttest

For example to run a single test, specify the name of the test file.

::

    paver test_bokchoy -t lms/test_lms.py

Notice the test file location is relative to
common/test/acceptance/tests. This is another example.

::

    paver test_bokchoy -t studio/test_studio_bad_data.py

To run a single test faster by not repeating setup tasks us the ``--fasttest`` option.

::

    paver test_bokchoy -t studio/test_studio_bad_data.py --fasttest

To test only a certain feature, specify the file and the testcase class.

::

    paver test_bokchoy -t studio/test_studio_bad_data.py:BadComponentTest

To execute only a certain test case, specify the file name, class, and
test case method.

::

    paver test_bokchoy -t lms/test_lms.py:RegistrationTest.test_register

During acceptance test execution, log files and also screenshots of
failed tests are captured in test\_root/log.

Use this command to put a debugging breakpoint in a test.

::

    from nose.tools import set_trace; set_trace()

By default, all bokchoy tests are run with the 'split' ModuleStore. To
override the modulestore that is used, use the default\_store option.
The currently supported stores are: 'split'
(xmodule.modulestore.split\_mongo.split\_draft.DraftVersioningModuleStore)
and 'draft' (xmodule.modulestore.mongo.DraftMongoModuleStore). This is an example
for the 'draft' store.

::

    paver test_bokchoy --default_store='draft'

Running Bok Choy Accessibility Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We use Bok
Choy for `automated accessibility testing
<http://bok-choy.readthedocs.org/en/latest/accessibility.html>`__.
Bok Choy, a UI-level acceptance test framework for writing robust
`Selenium <http://docs.seleniumhq.org/>`__
tests in `Python <https://www.python.org/>`__, includes the ability to perform
accessibility audits on web pages using `Google Accessibility Developer Tools
<https://github.com/GoogleChrome/accessibility-developer-tools/>`__ or
`Deque's aXe Core <https://github.com/dequelabs/axe-core/>`__.
For more details about how to write accessibility tests, please read
the `Bok Choy documentation <http://bok-choy.readthedocs.org/en/latest/accessibility.html>`__
and the Automated Accessibility Tests `openedx Confluence page
<https://openedx.atlassian.net/wiki/display/TE/Automated+Accessibility+Tests>`__.

**Prerequisites**:

These prerequisites are all automatically installed and available in `Devstack
<https://github.com/edx/configuration/wiki/edX-Developer-Stack>`__ (since the Cypress release), the supported development enviornment for the edX Platform.

* Mongo

* Memcache

* mySQL

To run all the bok choy accessibility tests use this command.

::

    paver test_bokchoy --extra_args="-a 'a11y'"

To run specific tests, use the ``-t`` flag to specify a nose-style test spec
relative to the ``common/test/acceptance/tests`` directory. This is an example for it.

::

    paver test_bokchoy --extra_args="-a 'a11y'" -t test_lms_dashboard.py:LmsDashboardA11yTest.test_dashboard_course_listings_a11y

Options for Faster Development Cycles in Bok-Choy Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The following are ways in which a developer could shorten the development
cycle for faster feedback. The options below can often be used together.

**Multiprocessing Mode**

Bok-choy tests can be threaded using the `-n` switch.  Using 2 threads generally
reduces test cycles by 33%.  The recommendation is to make sure the
number of threads is no more than the number of processors available. For
example, the Cypress release of devstack is provisioned by default with 2
processors. In that case, to run tests in multiprocess mode::

    paver test_bokchoy -n 2

*Caveat*: Not all tests have been designed with multiprocessing in mind; some
testcases (approx 10%) will fail in multiprocess mode for various reasons
(e.g., shared fixtures, unexpected state, etc). If you have tests that fail
in multiprocessing mode, it may be worthwhile to run them in single-stream mode
to understand if you are encountering such a failure. With that noted, this
can speed development for most test classes.

**Leave Your Servers Running**

There are two additional switches available in the `paver test_bokchoy` task.
Used together, they can shorten the cycle between test runs. Similar to above,
there are a handful of tests that won't work with this approach, due to insufficient
teardown and other unmanaged state.

1. Start your servers in one terminal/ssh session::

    paver test_bokchoy --serversonly

Note if setup has already been done, you can run::

    paver test_bokcyoy --serversonly --fasttest

2. Run your tests only in another terminal/ssh session::

    paver test_bokchoy --testsonly --fasttest

You must run BOTH `--testsonly` and `--fasttest`.

3. When done, you can kill your servers in the first terminal/ssh session with
Control-C. *Warning*: Only hit Control-C one time so the nose test framework can
properly clean up.

Running Lettuce Acceptance Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We use `Lettuce <http://lettuce.it/>`__ for acceptance testing. Most of<commit_msg>Fix typo<commit_after>Testing
=======

Overview
--------

We maintain three kinds of tests: unit tests, integration tests, and
acceptance tests.

Overall, you want to write the tests that **maximize coverage** while
**minimizing maintenance**. In practice, this usually means investing
heavily in unit tests, which tend to be the most robust to changes in
the code base.

.. figure:: test_pyramid.png
   :alt: Test Pyramid

   Test Pyramid

The pyramid above shows the relative number of unit tests, integration
tests, and acceptance tests. Most of our tests are unit tests or
integration tests.

Unit Tests
~~~~~~~~~~

-  Each test case should be concise: setup, execute, check, and
   teardown. If you find yourself writing tests with many steps,
   consider refactoring the unit under tests into smaller units, and
   then testing those individually.

-  As a rule of thumb, your unit tests should cover every code branch.

-  Mock or patch external dependencies. We use voidspace
   `Mock Library <http://www.voidspace.org.uk/python/mock/>`__.

-  We unit test Python code (using
   `unittest <http://docs.python.org/2/library/unittest.html>`__) and
   Javascript (using `Jasmine <http://jasmine.github.io/>`__)

Integration Tests
~~~~~~~~~~~~~~~~~

-  Test several units at the same time. Note that you can still mock or
   patch dependencies that are not under test! For example, you might
   test that ``LoncapaProblem``, ``NumericalResponse``, and
   ``CorrectMap`` in the ``capa`` package work together, while still
   mocking out template rendering.

-  Use integration tests to ensure that units are hooked up correctly.
   You do not need to test every possible input--that's what unit tests
   are for. Instead, focus on testing the "happy path" to verify that
   the components work together correctly.

-  Many of our tests use the `Django test
   client <https://docs.djangoproject.com/en/dev/topics/testing/overview/>`__
   to simulate HTTP requests to the server.

UI Acceptance Tests
~~~~~~~~~~~~~~~~~~~

-  Use these to test that major program features are working correctly.

-  We use `lettuce <http://lettuce.it/>`__ to write BDD-style tests.
   Most of these tests simulate user interactions through the browser
   using `splinter <http://splinter.cobrateam.info/>`__.

-  We use `Bok
   Choy <http://bok-choy.readthedocs.org/en/latest/tutorial.html>`__ to
   write end-user acceptance tests directly in Python, using the
   framework to maximize reliability and maintainability.

Test Locations
--------------

-  Python unit and integration tests: Located in subpackages called
   ``tests``. For example, the tests for the ``capa`` package are
   located in ``common/lib/capa/capa/tests``.

-  Javascript unit tests: Located in ``spec`` folders. For example,
   ``common/lib/xmodule/xmodule/js/spec`` and
   ``{cms,lms}/static/coffee/spec`` For consistency, you should use the
   same directory structure for implementation and test. For example,
   the test for ``src/views/module.coffee`` should be written in
   ``spec/views/module_spec.coffee``.

-  UI acceptance tests:

   -  Set up and helper methods, and stubs for external services:
      ``common/djangoapps/terrain``
   -  Lettuce Tests: located in ``features`` subpackage within a Django
      app. For example: ``lms/djangoapps/courseware/features``
   -  Bok Choy Acceptance Tests: located under ``common/test/acceptance/tests``
   -  Bok Choy Accessibility Tests: located under ``common/test/acceptance/tests`` and tagged with ``@attr("a11y")``
   -  Bok Choy PageObjects: located under ``common/test/acceptance/pages``

Factories
---------

Many tests delegate set-up to a "factory" class. For example, there are
factories for creating courses, problems, and users. This encapsulates
set-up logic from tests.

Factories are often implemented using
`FactoryBoy <https://readthedocs.org/projects/factoryboy/>`__

In general, factories should be located close to the code they use. For
example, the factory for creating problem XML definitions is located in
``common/lib/capa/capa/tests/response_xml_factory.py`` because the
``capa`` package handles problem XML.

Running Tests
=============

You can run all of the unit-level tests using this command.

::

    paver test

This includes python, javascript, and documentation tests. It does not,
however, run any acceptance tests.

Note -
`paver` is a scripting tool. To get information about various options, you can run the this command.
::
	paver -h
Running Python Unit tests
-------------------------

We use `nose <https://nose.readthedocs.org/en/latest/>`__ through the
`django-nose plugin <https://pypi.python.org/pypi/django-nose>`__ to run
the test suite.

For example, this command runs all the python test scripts.

::

    paver test_python

It also runs ``collectstatic``, which prepares the
static files used by the site (for example, compiling CoffeeScript to
JavaScript).

You can re-run all failed python tests by running this command (see note at end of
section).

::

    paver test_python --failed

To test lms python tests use this command.

::

    paver test_system -s lms

To test cms python tests use this command.

::

    paver test_system -s cms

To run these tests without ``collectstatic``, which is faster, append the following argument.

::

    paver test_system -s lms --fasttest

To run cms python tests without ``collectstatic`` use this command.

::

    paver test_system -s cms --fasttest

To run a single django test class use this command.

::

    paver test_system -t lms/djangoapps/courseware/tests/tests.py:ActivateLoginTest

When developing tests, it is often helpful to be able to really just run
one single test without the overhead of PIP installs, UX builds, etc. In
this case, it is helpful to look at the output of paver, and run just
the specific command (optionally, stripping away coverage metrics). At
the time of this writing, the command is the following.

::

    python ./manage.py lms test --verbosity=1 lms/djangoapps/courseware/tests/test_courses.py   --traceback --settings=test


To run a single test format the command like this.

::

    paver test_system -t lms/djangoapps/courseware/tests/tests.py:ActivateLoginTest.test_activate_login

To re-run all failing django tests from lms or cms, use the
``--failed``,\ ``-f`` flag (see note at end of section).

::

    paver test_system -s lms --failed
    paver test_system -s cms --failed

There is also a ``--fail_fast``, ``-x`` option that will stop nosetests
after the first failure.

common/lib tests are tested with the ``test_lib`` task, which also
accepts the ``--failed`` and ``--fail_fast`` options.

::

    paver test_lib -l common/lib/calc
    paver test_lib -l common/lib/xmodule --failed

For example, this command runs a single nose test file.

::

    nosetests common/lib/xmodule/xmodule/tests/test_stringify.py

This command runs a single nose test within a specified file.

::

    nosetests common/lib/xmodule/xmodule/tests/test_stringify.py:test_stringify


This is an example of how to run a single test and get stdout, with proper env config.

::

    python manage.py cms --settings test test contentstore.tests.test_import_nostatic -s

These are examples of how to run a single test and get stdout and get coverage.

::

    python -m coverage run --rcfile=./common/lib/xmodule/.coveragerc which ./manage.py cms --settings test test --traceback --logging-clear-handlers --liveserver=localhost:8000-9000 contentstore.tests.test_import_nostatic -s # cms example
    python -m coverage run --rcfile=./lms/.coveragerc which ./manage.py lms --settings test test --traceback --logging-clear-handlers --liveserver=localhost:8000-9000  courseware.tests.test_module_render -s # lms example

Use this command to generate coverage report.

::

    coverage report --rcfile=./common/lib/xmodule/.coveragerc

Use this command to generate an HTML report.

::

    coverage html --rcfile=./common/lib/xmodule/.coveragerc

The report is then saved in reports/common/lib/xmodule/cover/index.html

To run tests for stub servers, for example for `YouTube stub
server <https://github.com/edx/edx-platform/blob/master/common/djangoapps/terrain/stubs/tests/test_youtube_stub.py>`__,
you can run one of these commands.

::

    paver test_system -s cms -t common/djangoapps/terrain/stubs/tests/test_youtube_stub.py
    python -m coverage run --rcfile=cms/.coveragerc `which ./manage.py` cms --settings test test --traceback common/djangoapps/terrain/stubs/tests/test_youtube_stub.py

Very handy: if you pass the ``--pdb`` flag to a paver test function, or
uncomment the ``pdb=1`` line in ``setup.cfg``, the test runner
will drop you into pdb on error. This lets you go up and down the stack
and see what the values of the variables are. Check out `the pdb
documentation <http://docs.python.org/library/pdb.html>`__

Note: More on the ``--failed`` functionality

* In order to use this, you must run the tests first. If you haven't already
  run the tests, or if no tests failed in the previous run, then using the
  ``--failed`` switch will result in **all** of the tests being run. See more
  about this in the `nose documentation
  <http://nose.readthedocs.org/en/latest/plugins/testid.html#looping-over-failed-tests>`__.

* Note that ``paver test_python`` calls nosetests separately for cms and lms.
  This means that if tests failed only in lms on the previous run, then calling
  ``paver test_python --failed`` will run **all of the tests for cms** in
  addition to the previously failing lms tests. If you want it to run only the
  failing tests for lms or cms, use the ``paver test_system -s lms --failed``
  or ``paver test_system -s cms --failed`` commands.

Running Javascript Unit Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We use Jasmine to run JavaScript unit tests. To run all the JavaScript
tests::

    paver test_js

To run a specific set of JavaScript tests and print the results to the
console, run these commands.

::

    paver test_js_run -s lms
    paver test_js_run -s lms-coffee
    paver test_js_run -s cms
    paver test_js_run -s cms-squire
    paver test_js_run -s xmodule
    paver test_js_run -s common
    paver test_js_run -s common-requirejs

To run JavaScript tests in a browser, run these commands.

::

    paver test_js_dev -s lms
    paver test_js_dev -s lms-coffee
    paver test_js_dev -s cms
    paver test_js_dev -s cms-squire
    paver test_js_dev -s xmodule
    paver test_js_dev -s common
    paver test_js_dev -s common-requirejs

To debug these tests on devstack in a local browser:

 * first run the appropriate test_js_dev command from above which will open a browser using XQuartz
 * open the same URL in your browser but change the IP address to 192.168.33.10, e.g.
    http://192.168.33.10:TEST_PORT/suite/cms
 * this will run all the tests and show you the results including details of any failures
 * you can click on an individually failing test and/or suite to re-run it by itself
 * you can now use the browser's developer tools to debug as you would any other JavaScript code

Note: the port is also output to the console that you ran the tests from if you find that easier.

These paver commands call through to a custom test runner. For more
info, see `js-test-tool <https://github.com/edx/js-test-tool>`__.

Running Bok Choy Acceptance Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We use `Bok
Choy <http://bok-choy.readthedocs.org/en/latest/tutorial.html>`__ for
acceptance testing. Bok Choy is a UI-level acceptance test framework for
writing robust `Selenium <http://docs.seleniumhq.org/>`__ tests in
`Python <https://www.python.org/>`__. Bok Choy makes your acceptance
tests reliable and maintainable by utilizing the Page Object and Promise
design patterns.

**Prerequisites**:

These prerequisites are all automatically installed and available in `Devstack
<https://github.com/edx/configuration/wiki/edX-Developer-Stack>`__, the
supported development enviornment for the edX Platform.

* Chromedriver and Chrome (see Running Lettuce Acceptance Tests below for
  the latest tested versions)

* Mongo

* Memcache

* mySQL

To run all the bok choy acceptance tests run this command.

::

    paver test_bokchoy

Once the database has been set up and the static files collected, you
can use the 'fast' option to skip those tasks. This option can also be
used with any of the test specs below.

::

    paver test_bokchoy --fasttest

For example to run a single test, specify the name of the test file.

::

    paver test_bokchoy -t lms/test_lms.py

Notice the test file location is relative to
common/test/acceptance/tests. This is another example.

::

    paver test_bokchoy -t studio/test_studio_bad_data.py

To run a single test faster by not repeating setup tasks us the ``--fasttest`` option.

::

    paver test_bokchoy -t studio/test_studio_bad_data.py --fasttest

To test only a certain feature, specify the file and the testcase class.

::

    paver test_bokchoy -t studio/test_studio_bad_data.py:BadComponentTest

To execute only a certain test case, specify the file name, class, and
test case method.

::

    paver test_bokchoy -t lms/test_lms.py:RegistrationTest.test_register

During acceptance test execution, log files and also screenshots of
failed tests are captured in test\_root/log.

Use this command to put a debugging breakpoint in a test.

::

    from nose.tools import set_trace; set_trace()

By default, all bokchoy tests are run with the 'split' ModuleStore. To
override the modulestore that is used, use the default\_store option.
The currently supported stores are: 'split'
(xmodule.modulestore.split\_mongo.split\_draft.DraftVersioningModuleStore)
and 'draft' (xmodule.modulestore.mongo.DraftMongoModuleStore). This is an example
for the 'draft' store.

::

    paver test_bokchoy --default_store='draft'

Running Bok Choy Accessibility Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We use Bok
Choy for `automated accessibility testing
<http://bok-choy.readthedocs.org/en/latest/accessibility.html>`__.
Bok Choy, a UI-level acceptance test framework for writing robust
`Selenium <http://docs.seleniumhq.org/>`__
tests in `Python <https://www.python.org/>`__, includes the ability to perform
accessibility audits on web pages using `Google Accessibility Developer Tools
<https://github.com/GoogleChrome/accessibility-developer-tools/>`__ or
`Deque's aXe Core <https://github.com/dequelabs/axe-core/>`__.
For more details about how to write accessibility tests, please read
the `Bok Choy documentation <http://bok-choy.readthedocs.org/en/latest/accessibility.html>`__
and the Automated Accessibility Tests `openedx Confluence page
<https://openedx.atlassian.net/wiki/display/TE/Automated+Accessibility+Tests>`__.

**Prerequisites**:

These prerequisites are all automatically installed and available in `Devstack
<https://github.com/edx/configuration/wiki/edX-Developer-Stack>`__ (since the Cypress release), the supported development enviornment for the edX Platform.

* Mongo

* Memcache

* mySQL

To run all the bok choy accessibility tests use this command.

::

    paver test_bokchoy --extra_args="-a 'a11y'"

To run specific tests, use the ``-t`` flag to specify a nose-style test spec
relative to the ``common/test/acceptance/tests`` directory. This is an example for it.

::

    paver test_bokchoy --extra_args="-a 'a11y'" -t test_lms_dashboard.py:LmsDashboardA11yTest.test_dashboard_course_listings_a11y

Options for Faster Development Cycles in Bok-Choy Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The following are ways in which a developer could shorten the development
cycle for faster feedback. The options below can often be used together.

**Multiprocessing Mode**

Bok-choy tests can be threaded using the `-n` switch.  Using 2 threads generally
reduces test cycles by 33%.  The recommendation is to make sure the
number of threads is no more than the number of processors available. For
example, the Cypress release of devstack is provisioned by default with 2
processors. In that case, to run tests in multiprocess mode::

    paver test_bokchoy -n 2

*Caveat*: Not all tests have been designed with multiprocessing in mind; some
testcases (approx 10%) will fail in multiprocess mode for various reasons
(e.g., shared fixtures, unexpected state, etc). If you have tests that fail
in multiprocessing mode, it may be worthwhile to run them in single-stream mode
to understand if you are encountering such a failure. With that noted, this
can speed development for most test classes.

**Leave Your Servers Running**

There are two additional switches available in the `paver test_bokchoy` task.
Used together, they can shorten the cycle between test runs. Similar to above,
there are a handful of tests that won't work with this approach, due to insufficient
teardown and other unmanaged state.

1. Start your servers in one terminal/ssh session::

    paver test_bokchoy --serversonly

Note if setup has already been done, you can run::

    paver test_bokchoy --serversonly --fasttest

2. Run your tests only in another terminal/ssh session::

    paver test_bokchoy --testsonly --fasttest

You must run BOTH `--testsonly` and `--fasttest`.

3. When done, you can kill your servers in the first terminal/ssh session with
Control-C. *Warning*: Only hit Control-C one time so the nose test framework can
properly clean up.

Running Lettuce Acceptance Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We use `Lettuce <http://lettuce.it/>`__ for acceptance testing. Most of
{'fraction_non_alphanumeric': 0.09520280477724254, 'fraction_numerical': 0.0033634523843456944, 'mean_word_length': 3.9151022695432895, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 26, 'lorem ipsum': 0, 'www.': 6, 'xml': 2}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4987867', 'n_tokens_mistral': 10519, 'n_tokens_neox': 10112, 'n_words': 4340}