<commit_before>﻿Recording Command Tests with VCR.py
========================================

Azure CLI uses the VCR.py library to record the HTTP messages exchanged during a program run and play them back at a later time, making it useful for creating command level tests. These tests can be replayed at a later time without any network activity, allowing us to detect regressions in the handling of parameters and in the compatability between AzureCLI and the PythonSDK.

##Overview

Each command module has a `tests` folder with the following files: `command_specs.py` and `test_commands.py`. The second is largely boilerplate; you will add your test definitions to `command_specs.py`. These files contain two important definitions: `TEST_DEF` which lists the commands you want to test and gives them each a name, and `ENV_VAR` where you can set any environment variables you might need when replaying on TravisCI.

The structure of `TEST_DEF` is straightforward:

```
TEST_DEF = [
  {
    'test_name': 'name1',
    'command': 'command1'
  },
  {
    'test_name': 'name2',
    'command': 'command2'
  }
]
```

Simply add your new entries and run all tests. The test driver will automatically detect the new tests and run the command, show you the output, and query if it is correct. If so, it will save the HTTP recording into a YAML file, as well as the raw output into a dictionary called `expected_results.res`. When the test is replayed, as long as the test has an entry in `expected_results.res` and a corresponding .yaml file, the test will proceed automatically. If either the entry or the .yaml file are missing, the test will be re-recorded.

If the tests are run on TravisCI, any tests which cannot be replayed will automatically fail. 

##Recording Tests

Many tests, for example those which simply retrieve information, can simply be played back, verified and recorded.

Other tests, such as create and delete scenarios, may require additional commands to set up for recording, or may require additional commands to verify the proper operation of a command. For example, several create commands output nothing on success. Thus, you'll find yourself staring at nothing with a prompt asking if that is the expected response.

For these scenarios, I recommend having a second shell open, from which you can run any setup commands and then run any commands you need to verify the proper operation of the command in order to answer the prompt.

I don't recommend trying to structure your tests so that one test sets up for another, because in general you cannot guarantee the order in which the tests will run. Also, I don't recommend attempting to record large batches of tests at once. I generally add one to three tests at a time and leave the remaining new tests commented out. Running `testall.bat` will let me record these. Then I uncomment a few more and so on, until they are all recorded.

##Limitations

The current system saves time, but has some limitations.

+ Certain commands require manual steps to setup or verify
+ You can't test for things like 'this input results in an exception'. It simply tests that the response equals an expected response.
+ This system does not work with long running operations. While it technically does, the resulting recording takes as long as the original call, which negates some of the key benefits of automated testing.<commit_msg>Update test authoring help doc.<commit_after>﻿Recording Command Tests with VCR.py
========================================

Azure CLI uses the VCR.py library to record the HTTP messages exchanged during a program run and play them back at a later time, making it useful for creating command level tests. These tests can be replayed at a later time without any network activity, allowing us to detect regressions in the handling of parameters and in the compatability between AzureCLI and the PythonSDK.

##Overview

Each command module has a `tests` folder with the following files: `command_specs.py` and `test_commands.py`. The second is largely boilerplate; you will add your test definitions to `command_specs.py`. These files contain two important definitions: `TEST_DEF` which lists the commands you want to test and gives them each a name, and `ENV_VAR` where you can set any environment variables you might need when replaying on TravisCI.

The structure of `TEST_DEF` is straightforward:

```
TEST_DEF = [
  {
    'test_name': 'name1',
    'command': 'command1'
  },
  {
    'test_name': 'name2',
    'script': 'script_class()'
  }
]
```

Simply add your new entries and run all tests. The test driver will automatically detect the new tests and run the commands, show you the output, and query if it is correct. If so, it will save the HTTP recording into a YAML file, as well as the raw output into a dictionary called `expected_results.res`. When the test is replayed, as long as the test has an entry in `expected_results.res` and a corresponding .yaml file, the test will proceed automatically. If either the entry or the .yaml file are missing, the test will be re-recorded.

If the tests are run on TravisCI, any tests which cannot be replayed will automatically fail. 

##Types of Tests

The system currently accepts individual commands and script test objects. Individual commands will always display the output and query the user if the results are correct. These are the "old" style tests.

To allow for more complex testing scenarios involving creating and deleting resources, long-running operations, or automated verification, use the script object method. To do so, simply create a class in the `command_specs.py` file with the following structure:

```Python
class MyScenarioTest(CommandTestScript):
  def __init__(self):
    super(MyScenarioTest, self).__init__(self.set_up, self.test_body, self.tear_down)
  
  def set_up(self):
    # Setup logic here
    pass
    
  def test_body(self):
    # Main test logic
    pass
    
  def tear_down(self):
    # clean up logic here
    pass
```

The `set_up` and `tear_down` methods are optional and can be omitted. A number of helper methods are available for structuring your script tests.

####run(command_string)

This method executes a given command and returns the output. The results are not sent to the display or to expected results. Use this for:

- running commands that produce no output (the next statement will usually be a test)
- running commands that are needed for conditional logic or in setup/cleanup logic

####rec(command_string)

This method runs a given command and records its output to the display for manual verification. Using this command will force the user to verify the output via a Y/N prompt. If the user accepts the output, it will be saved to `expected_results.res`.

####test(command_string, checks)

This method runs a given command and automatically validates the output. The results are saved to `expected_results.res` if valid, but nothing is display on the screen. Valid checks include: `bool`, `str` and `dict`. A check with a `dict` can be used to check for multiple matching parameters (and logic). Child `dict` objects can be used as values to verify properties within nested objects.

####set_env(variable_name, value)

This method is a wrapper around `os.environ` and simply sets an environment variable to the specified value.

####pop_env(variable_name)

Another wrapper around `os.environ` this pops the value of the indicated environment variable.

##Long Running Operations (LRO)

The system now allows the testing of long running operations. Regardless of the time required to record the test, playback will truncate the long running operation to finish very quickly. However, because re-recording these actions can take a very long time, it is recommended that each LRO scenario be individually tested (possibly in tandem with a delete operation) rather than as part of a larger scenario.

##Limitations

The current system saves time, but has some limitations.

+ You can't test for things like 'this input results in an exception'. It simply tests that the response equals an expected response.
{'fraction_non_alphanumeric': 0.05941814595660749, 'fraction_numerical': 0.0008629191321499013, 'mean_word_length': 4.4743589743589745, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '209998', 'n_tokens_mistral': 1972, 'n_tokens_neox': 1916, 'n_words': 1215}