<commit_before># ddf-test
Test suite for DDF

### Running test suite

1. Change the engine in [global] and queries for your engine in [your engine] in ddf-conf/ddf_spec.ini.
There are two ways to add the ddf-on-x jars:

* Add ddf-on-x implementation jar to `lib` directory. Make sure that there are no duplicate jars present. Remove
scalatest and junit jars if added from ddf-on-x. Do not remove the asm-all-4.0.jar from the lib directory as it
contains some essential classes required for the scalatest to generate a html report.

* Execute the shell script which takes the location of a directory which has the ddf-on-x jars.

2. Running the tests in sbt console

```
$ sbt

//to run all the tests
sbt> test
```

3. Running the tests using the shell script

```
$ bash bin/DDFTestGUI.sh

//Enter the required jar files and java Options in the command prompt.

Hello, welcome to ddf-test. This script will ask you choose an engine and run the tests in a GUI.

Choose your java options for tests (required for spark engine: -Dhive.metastore.warehouse.dir=/tmp/hive/warehouse )

Enter your java options and press [ENTER]:

Enter your ddf-on-x jar's destination and press [ENTER]:

```

To open the Html Reports of the tests generated, open index.html from target/test-reports directory in a web browser.
Clicking on the DDFSpec suite, will show the test reports.
<commit_msg>Changes in README<commit_after># ddf-test
Test suite for DDF

### Running test suite

1. Change the engine in [global] and queries for your engine in [your engine] in ddf-conf/ddf_spec.ini.
There are two ways to add the ddf-on-x jars:

* Add ddf-on-x implementation jar to `lib` directory. Make sure that there are no duplicate jars present. Do not include
scalatest and junit jars . Do not remove the asm-all-4.0.jar from the lib directory as it
contains some essential classes required for the scalatest to generate a html report.

* Execute the shell script which takes the location of a directory which has the ddf-on-x jars. Do not include scalatest and junit jars . 

2. Running the tests in sbt console

```
$ sbt

//to run all the tests
sbt> test
```

3. Running the tests using the shell script

```
$ bash bin/DDFTestRunner.sh

//Enter the required jar files and java Options in the command prompt.

Hello, welcome to ddf-test. This script will ask you choose an engine and run the tests in a GUI.

Choose your java options for tests (for spark: -Dhive.metastore.warehouse.dir=/tmp/hive/warehouse )

Enter your java options and press [ENTER]:

Enter your ddf-on-x jar's destination and press [ENTER]:

```

To open the Html Reports of the tests generated, open index.html from target/test-reports directory in a web browser.
Clicking on the DDFSpec suite, will show the test reports.

{'fraction_non_alphanumeric': 0.0675578034682081, 'fraction_numerical': 0.0036127167630057803, 'mean_word_length': 3.8072916666666665, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1231141', 'n_tokens_mistral': 838, 'n_tokens_neox': 807, 'n_words': 423}