<commit_before>.. retesteth_ref:

#######################
Using Retesteth
#######################


Command Line Options
========================

.. note::

   There has to be a double dash (**-\\-**) between the **-t** option that sets the
   suite and all the other options.
   

Set The Suite
-----------------------
============================================= ===================================
**Option**                                    **Meaning**
============================================= ===================================
-t <TestSuite>                                Run all the tests in that suite

-t <TestSuite>/<Test Case Folder>             Run a specific test case folder
                                              (for **GeneralStateTests**)

-t <TestSuite>/<Test Type>/<Test Case Folder> Run a specific test case folder
                                              (for **BlockchainTests**)
============================================= ===================================

.. note:: 

   In **BlockchainTests** there are three possible values for the test type:
                                              
   - **ValidBlocks**  
   - **InvalidBlocks**
   - **TransitionTests**


Retesteth Options
-----------------------

================================= ======================================================
**Option**                        **Meaning**                                        
================================= ======================================================
 -j <ThreadNumber>                Run test execution using threads                    
 -\\-clients `client1, client2`   Use following configurations from                     
                                  datadir path (default: ~/.retesteth)                 
 -\\-datadir                      Path to configs (default: ~/.retesteth)            
 -\\-nodes                        List of client tcp ports ("addr:ip, addr:ip")      
 -\\-help -h                      Display list of command arguments                  
 -\\-version -v                   Display build information                          
 -\\-list                         Display available test suites                      
 -\\-testfolder                   Use to create a new test folder inside the suite.
                                  Not compatible with using a test case folder
================================= ======================================================


.. note::

   Setting **-\\-nodes** overrides the **socketAddress** section of the **config** file,
   `documented here 
   <https://ethereum-tests.readthedocs.io/en/latest/config-dir.html#socketaddress>`_.


Setting the Test Suite and Test
----------------------------------
========================================= ===================================================
Option                                    Meaning
========================================= ===================================================
-\\-testpath <PathToTheTestRepo>          Set path to the test repo
-\\-testfile <TestFile>                   Run tests from a file. Requires -t <TestSuite>
-\\-singletest <TestName>                 Run on a single test. `Testname` is the filename 
                                          without Filler.<type> (either **json** or **yml**)
-\\-singletest <TestName>/<Subtest>       `Subtest` is a test name inside the file.
========================================= ===================================================

.. note::

   **<Subtest>** is only relevant in **BlockchainTests**. Other test suites
   do not support files with multiple test names.


Debugging
----------------------------------

============================= ===================================================
Option                        Meaning
============================= ===================================================
-d <index>                    Set the transaction data array index when running 
                              GeneralStateTests
-g <index>                    Set the transaction gas array index when running 
                              GeneralStateTests
-v <index>                    Set the transaction value array index when running 
                              GeneralStateTests
-\\-vmtraceraw                Trace transaction execution
-\\-vmtrace                   Trace transaction execution, simplified version
-\\-limitblocks <num>         Limit the block exectuion in blockchain tests for 
                              debugging to the first <num> blocks
-\\-limitrpc                  Limit the rpc exectuion in tests for debug
-\\-verbosity <level>         Set logs verbosity. 0 - silent, 1 - only errors, 
                              2 - informative, >2 - detailed
-\\-exectimelog               Output execution time for each test suite
-\\-stderr                    Redirect ipc client stderr to stdout
-\\-travisout                 Output \`.\` to stdout
============================= ===================================================


Additional Tests
----------------------------------

======================================= ===================================
Option                                  Meaning
======================================= ===================================
-\\-all                                 Enable all tests
======================================= ===================================

This setting enables the following test suites:

* `GeenralStateTests/stTimeConsuming <https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stTimeConsuming>`_
* `GeenralStateTests/stQuadraticComplexityTest <https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stQuadraticComplexityTest>`_
* `GeneralStateTests/VMTests/vmPerformance <https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/VMTests/vmPerformance>`_
* `BlockchainTests/ValidBlocks/bcExploitTest <https://github.com/ethereum/tests/tree/develop/src/BlockchainTestsFiller/ValidBlocks/bcExploitTest>`_
* `BlockchainTests/ValidBlocks/bcWalletTest <https://github.com/ethereum/tests/tree/develop/src/BlockchainTestsFiller/ValidBlocks/bcWalletTest>`_
* `BlockchainTests/InvalidBlocks/bcExpectSection <https://github.com/ethereum/tests/tree/develop/src/BlockchainTestsFiller/InvalidBlocks/bcExpectSection>`_


Test Generation
----------------------------------

=============================== ===================================
Option                          Meaning
=============================== ===================================
-\\-filltests                   Run test fillers
-\\-fillchain                   When filling the state tests, fill 
                                tests as blockchain instead
-\\-showhash                    Show filler hash debug information
-\\-poststate                   Show post state hash or fullstate
-\\-fullstate                   Do not compress large states to hash
=============================== ===================================
<commit_msg>feat(retestheth-ref): Add the folder option for `--vmtraceraw` and `--poststate`<commit_after>.. retesteth_ref:

#######################
Using Retesteth
#######################


Command Line Options
========================

.. note::

   There has to be a double dash (**-\\-**) between the **-t** option that sets the
   suite and all the other options.
   

Set The Suite
-----------------------
============================================= ===================================
**Option**                                    **Meaning**
============================================= ===================================
-t <TestSuite>                                Run all the tests in that suite

-t <TestSuite>/<Test Case Folder>             Run a specific test case folder
                                              (for **GeneralStateTests**)

-t <TestSuite>/<Test Type>/<Test Case Folder> Run a specific test case folder
                                              (for **BlockchainTests**)
============================================= ===================================

.. note:: 

   In **BlockchainTests** there are three possible values for the test type:
                                              
   - **ValidBlocks**  
   - **InvalidBlocks**
   - **TransitionTests**


Retesteth Options
-----------------------

================================= ======================================================
**Option**                        **Meaning**                                        
================================= ======================================================
 -j <ThreadNumber>                Run test execution using threads                    
 -\\-clients `client1, client2`   Use following configurations from                     
                                  datadir path (default: ~/.retesteth)                 
 -\\-datadir                      Path to configs (default: ~/.retesteth)            
 -\\-nodes                        List of client tcp ports ("addr:ip, addr:ip")      
 -\\-help -h                      Display list of command arguments                  
 -\\-version -v                   Display build information                          
 -\\-list                         Display available test suites                      
 -\\-testfolder                   Use to create a new test folder inside the suite.
                                  Not compatible with using a test case folder
================================= ======================================================


.. note::

   Setting **-\\-nodes** overrides the **socketAddress** section of the **config** file,
   `documented here 
   <https://ethereum-tests.readthedocs.io/en/latest/config-dir.html#socketaddress>`_.


Setting the Test Suite and Test
----------------------------------
========================================= ===================================================
Option                                    Meaning
========================================= ===================================================
-\\-testpath <PathToTheTestRepo>          Set path to the test repo
-\\-testfile <TestFile>                   Run tests from a file. Requires -t <TestSuite>
-\\-singletest <TestName>                 Run on a single test. `Testname` is the filename 
                                          without Filler.<type> (either **json** or **yml**)
-\\-singletest <TestName>/<Subtest>       `Subtest` is a test name inside the file.
========================================= ===================================================

.. note::

   **<Subtest>** is only relevant in **BlockchainTests**. Other test suites
   do not support files with multiple test names.


Debugging
----------------------------------

============================= ===================================================
Option                        Meaning
============================= ===================================================
-d <index>                    Set the transaction data array index when running 
                              GeneralStateTests
-g <index>                    Set the transaction gas array index when running 
                              GeneralStateTests
-v <index>                    Set the transaction value array index when running 
                              GeneralStateTests
-\\-vmtraceraw [<folder>]     Trace transaction execution (see note)
-\\-vmtrace                   Trace transaction execution, simplified version
-\\-limitblocks <num>         Limit the block exectuion in blockchain tests for 
                              debugging to the first <num> blocks
-\\-limitrpc                  Limit the rpc exectuion in tests for debug
-\\-verbosity <level>         Set logs verbosity. 0 - silent, 1 - only errors, 
                              2 - informative, >2 - detailed
-\\-exectimelog               Output execution time for each test suite
-\\-stderr                    Redirect ipc client stderr to stdout
-\\-travisout                 Output \`.\` to stdout
============================= ===================================================


.. note::

   Normally the **-\\-vmtraceraw** output goes to standard output. However, you could specify a directory
   name and it would get written there (under a different file name for every fork, data value, etc.).
   If you are using docker that directory is in the folder, so it is easiest to use a directory such as
   **/tests/results** (because **/tests** on the docker is the tests repository on a file system outside
   the docker).



Additional Tests
----------------------------------

======================================= ===================================
Option                                  Meaning
======================================= ===================================
-\\-all                                 Enable all tests
======================================= ===================================

This setting enables the following test suites:

* `GeenralStateTests/stTimeConsuming <https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stTimeConsuming>`_
* `GeenralStateTests/stQuadraticComplexityTest <https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stQuadraticComplexityTest>`_
* `GeneralStateTests/VMTests/vmPerformance <https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/VMTests/vmPerformance>`_
* `BlockchainTests/ValidBlocks/bcExploitTest <https://github.com/ethereum/tests/tree/develop/src/BlockchainTestsFiller/ValidBlocks/bcExploitTest>`_
* `BlockchainTests/ValidBlocks/bcWalletTest <https://github.com/ethereum/tests/tree/develop/src/BlockchainTestsFiller/ValidBlocks/bcWalletTest>`_
* `BlockchainTests/InvalidBlocks/bcExpectSection <https://github.com/ethereum/tests/tree/develop/src/BlockchainTestsFiller/InvalidBlocks/bcExpectSection>`_


Test Generation
----------------------------------

=============================== ===================================
Option                          Meaning
=============================== ===================================
-\\-filltests                   Run test fillers
-\\-fillchain                   When filling the state tests, fill 
                                tests as blockchain instead
-\\-showhash                    Show filler hash debug information
-\\-poststate [<folder>]        Show post state hash or fullstate
                                Normally goes to output, but if a folder is specified written to that folder.
                                If you use Docker, those are on the image, so it's best to use **/test/...**.

-\\-fullstate                   Do not compress large states to hash
=============================== ===================================


.. note::

   Normally the **-\\-poststate** output goes to standard output. However, you could specify a directory
   name and it would get written there (under a different file name for every fork, data value, etc.).
   If you are using docker that directory is in the folder, so it is easiest to use a directory such as
   **/tests/results** (because **/tests** on the docker is the tests repository on a file system outside
   the docker).

{'fraction_non_alphanumeric': 0.28699287907493304, 'fraction_numerical': 0.0007839550532436141, 'mean_word_length': 2.1215334420880914, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 35, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '475939', 'n_tokens_mistral': 3341, 'n_tokens_neox': 2997, 'n_words': 1003}