<commit_before>
                 A Subversion Testing Framework
                ================================


The three goals of Subversion's automated test-suite:

      1.  It must be easy to run.
      2.  It must be easy to understand the results.
      3.  It must be easy to add new tests.


What automake gives us
----------------------

Automake defines a variable TESTS (in a `Makefile.am') which is a list
of test-programs to run in a subdirectory.  `make check' runs all
these tests, examines the return value of each, and finally states how
many passed.  When run from the top of the source tree, `make check'
works recursively and reports the results from each subdir.

i.e., if a line in Makefile.am looks like:

      TESTS = test1 test2 test3 test4

the `make check' command will run each program and ultimately print
something like:

    ===================
    1 of 4 tests failed
    ===================



The definition of an SVN "test"
-------------------------------

Define a "Subversion test program" as any executable (binary or
script) with this interface:

1.  Requires a single numeric argument -- an internal test number to
    run.

2.  Print standardized, human-readable output after finishing a test.

      (argv[0]) test (N):  (description) ............... (PASS | FAIL)


For example,

[sussman@newton:~] ./frobtest 2

 frobtest test 2: order-N frobbing on even parity data........... PASS




Integrating an SVN test app into Makefile.am
--------------------------------------------

In the example above, we'd like the automake TESTS variable to run
frobtest four times -- each time with a different argument.  However,
this won't work:
     
     TESTS = frobtest 1 frobtest 2 frobtest 3 frobtest 4

`make check' sees that as eight different programs to run.

Instead, then, a magic Makefile rule creates mini-scripts on the fly.
Each mini-script executes a test program with a particular numeric
argument, redirects the output to a logfile, and then deletes itself.
(Look in a Makefile.am for details.)



Practical wisdom:  when to write new tests
------------------------------------------

  The world of CVS development, people have noticed that the same bugs
tend to recur over and over.  Thus the CVS community has adopted a
hard-and-fast rule that whenever somebody fixes a bug, a *new* test is
added to the suite to specifically check for it.  It's a common case
that in the process of fixing a bug, several old bugs are accidentally
resurrected... and then quickly revealed by the test suite.

  This same rule should apply to Subversion development:  ** If you
fix a bug, write a test for it. **

  (However, we should note that this rule is somewhat relaxed until
Subversion hits 1.0.  A majority of pre-1.0 bugs are due to the code
being in the "initial growth" stage.  The code is changing so fast
that any fixed bug will probably never recur, simply because features
and dependencies will be different a week later.)



Back to the 3 goals
-------------------

How well have we met the original three goals of a test-suite?


  1.  It must be easy to run.

      * run `make check' at any level of the tree


  2.  It must be easy to understand the results.

      * each test prints a standardized self-description and result
      * all test output is logged
      * `make check' summarizes statistical results


  3.  It must be easy to add new tests.

      * add a test to an existing app, and then add it to the TESTS
        line in the appropriate Makefile.am
<commit_msg>Done with test-suite spec.<commit_after>
                 A Subversion Testing Framework
                ================================


The three goals of Subversion's automated test-suite:

      1.  It must be easy to run.
      2.  It must be easy to understand the results.
      3.  It must be easy to add new tests.


Our Makefile.am
---------------

Automake defines a TESTS variable which is a list of tests to run
whenever someone types `make check'.

However, we're not using this facility -- it's too inflexible.  It
doesn't give us good control of results reporting.  (Specifically, we
can't stop it from spamming the terminal.)

Instead, our Makefile.am's contain a SVN_TEST_PROGRAMS variable (a
list of programs to run), and a home-made `check' target.

A `make check` will run each program in SVN_TEST_PROGRAMS and only
print a description *if a test fails*.  All test results are logged to
a file called `tests.log'.


Definition of an SVN "test program"
---------------------------------------

A Subversion test program is any executable that contains a number of
sub-tests it can run.  It has a standard interface:

1.  If run with a numeric argument N, the program runs sub-test N.

2.  If run with no arguments, the program runs *all* sub-tests.

3.  The program returns either 0 (success) or 1 (if any sub-test failed).

4.  The program prints standardized, human-readable output after
    finishing a test.

      (argv[0]) test (N):  (description) ............... (PASS | FAIL)

For example,

  [sussman@newton:~] ./frobtest 2

  frobtest test 2: frobnicating fragile data..................... PASS



How to write a new sub-test
---------------------------

Write a new self-contained routine in an existing test-program; make
sure the routine returns 0 or 1.  Then add this routine and its
description to the global arrays "test_funcs[]" and "descriptions[]".
That's all there is to it!


How to write an entirely new test program
-----------------------------------------

Use an existing test program as a "template" (such as stringtest.c) --
the standardization has already been written for you.  After the
program works, add it to the SVN_TEST_PROGRAMS variable in
Makefile.am.


When to write new tests
-----------------------

The world of CVS development, people have noticed that the same bugs
tend to recur over and over.  Thus the CVS community has adopted a
hard-and-fast rule that whenever somebody fixes a bug, a *new* test is
added to the suite to specifically check for it.  It's a common case
that in the process of fixing a bug, several old bugs are accidentally
resurrected... and then quickly revealed by the test suite.

This same rule should apply to Subversion development: ** If you fix a
bug, write a test for it. **

(However, we should note that this rule is somewhat relaxed until
Subversion hits 1.0.  A majority of pre-1.0 bugs are due to the code
being in the "initial growth" stage.)



Back to the 3 goals
-------------------

How well have we met the original three goals of a test-suite?


  1.  It must be easy to run.

      * run `make check` at any level of the tree

  2.  It must be easy to understand the results.

      * test programs output standardized messages
      * all messages are logged
      * `make check` only displays errors (not successes!)

  3.  It must be easy to add new tests.

      * add your own routine to an existing test program, or
      * add a test program to Makefile.am, using our template .c code

{'fraction_non_alphanumeric': 0.11633522727272727, 'fraction_numerical': 0.006818181818181818, 'mean_word_length': 2.566869300911854, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6510651', 'n_tokens_mistral': 1924, 'n_tokens_neox': 1795, 'n_words': 1041}