<commit_before>= Tests Generated from Unicode Files
:toc:

== Instructions
The `UNIDATA` folder referred to below can be obtained by running `tools/UCD-download.p6`
from MoarVM's repository.

.Please list these data points if possible:
* Script path (should be the heading)
* Steps: list of steps to run for each Unicode update. Sometimes there is just one,
  other times there are multiple things to do. Please list the command and command's
  arguments (including the data file)
* Version: The Unicode version that was used for generating the tests
* Date: The date the file was regenerated (YYYY.MM.DD)
* Generated File(s): a list of files which this script generates

== Todo

Nice things to add: ensure all tests generate files which have a heading that
shows which Unicode version was used to generate them. Some tests do this already,
but many do not.

== S15-nfg/emoji-test-gen.p6

.Steps
* Run `S15-nfg/emoji-test-gen.p6 UNIDATA/emoji-xx.x/emoji-test.txt`

.Version
* Emoji v12.0

.Date
* Regenerated 2019.07.28

.Generated File(s)
* Program generates S15-nfg/emoji-test.t

== S15-nfg/mass-roundtrip-test-gen.p6

.Steps
* Run `S15-nfg/mass-roundtrip-test-gen.p6 --test UNIDATA/NormalizationTest.txt`
  to test normalization if you don't get any failures, all is ok.
* The --test option is needed to test all options, as not all tests will be put into
  the roast .t file
* Run `S15-nfg/mass-roundtrip-test-gen.p6 UNIDATA/NormalizationTest.txt`
  to generate the tests.

.Version
* Unicode v12.1

.Date
* Regenerated 2019.07.28

.Generated File(s)
* S15-nfg/mass-roundtrip-nfc.t
* S15-nfg/mass-roundtrip-nfd.t
* S15-nfg/mass-roundtrip-nfkc.t
* S15-nfg/mass-roundtrip-nfkd.t


== S32-str/CollationTestGen.p6

Note: you will have to be in S32-str before running this, so it puts files in the
correct folder.

S32-str/CollationTestGen.p6 --folder=$UNIDATA/UCA/CollationTest --test-only --should-test
MAKE SURE TO RUN `perl6 CollationTestGen.p6 --folder=MoarVM/UNIDATA/UCA/CollationTest --test-only --should-test ` to ensure there are ~74 failures only!

Then paste those failures (if they're ok into CollationTestGen.p6 in the %todo variable.

.Version
* Unicode v12.1

.Date
* Regenerated 2019.07.28

.Generated File(s)
* S32-str/CollationTest_NON_IGNORABLE-0.t
* S32-str/CollationTest_NON_IGNORABLE-1.t
* S32-str/CollationTest_NON_IGNORABLE-2.t
* S32-str/CollationTest_NON_IGNORABLE-3.t

== S15-normalization/test-gen.p6

.Steps
* Run `S15-normalization/test-gen.p6 UNIDATA/NormalizationTest.txt`

.Version
* Unicode v12.0

.Date
* Regenerated 2019.07.28

.Generated File(s)
* S15-normalization/nfc-X.t
* S15-normalization/nfc-sanity.t
* S15-normalization/nfd-X.t
* S15-normalization/nfd-sanity.t
* S15-normalization/nfkc-X.t
* S15-normalization/nfkc-sanity.t
* S15-normalization/nfkd-X.t
* S15-normalization/nfc-concat.t

== S15-nfg/GraphemeBreakTest.t

.How
* Copy file from UNIDATA to 3rdparty folder of roast:
cp UNIDATA/auxilary/GraphemeBreakTest.txt 3rdparty/Unicode/$version/ucd/auxilary/GraphemeBreakTest.txt

change the filename in the S15-nfg/GraphemeBreakTest.t

```
my IO::Path $repo-dir      = $?FILE.IO.parent(2).add("3rdparty/Unicode/11.0.0/ucd/auxiliary/GraphemeBreakTest.txt");
```
.Version

.Date

== External

=== Also you will have to do stuff in rakudo

Edit t/09-moar/UnipropCheck.pm6 folder to be set to the current version.

You will have to now copy:

mkdir -p t/3rdparty/Unicode/$version/extracted
cp $UNIDATA/LineBreak.txt $UNIDATA/UnicodeData.txt t/3rdparty/Unicode/$version
cp $UNIDATA/extracted/DerivedGeneralCategory.txt t/3rdparty/Unicode/extracted
<commit_msg>Update verbiage about generated tests<commit_after>= Tests Generated from Unicode Files
:toc:

== Instructions
The `UNIDATA` folder referred to below can be obtained by running `tools/UCD-download.p6`
from MoarVM's repository.

.Please list these data points if possible:
* Script path (should be the heading)
* Steps: list of steps to run for each Unicode update. Sometimes there is just one,
  other times there are multiple things to do. Please list the command and command's
  arguments (including the data file)
* Version: The Unicode version that was used for generating the tests
* Date: The date the file was regenerated (YYYY.MM.DD)
* Generated File(s): a list of files which this script generates

== Todo

Nice things to add: ensure all tests generate files which have a heading that
shows which Unicode version was used to generate them. Some tests do this already,
but many do not.

== S15-nfg/emoji-test-gen.p6

.Steps
* Run `S15-nfg/emoji-test-gen.p6 UNIDATA/emoji-xx.x/emoji-test.txt`

.Version
* Emoji v13.1

.Date
* Regenerated 2020.10.01

.Generated File(s)
* Program generates S15-nfg/emoji-test.t

== S15-nfg/mass-roundtrip-test-gen.p6

.Steps
* Run `S15-nfg/mass-roundtrip-test-gen.p6 --test UNIDATA/NormalizationTest.txt`
  to test normalization if you don't get any failures, all is ok.
* The --test option is needed to test all options, as not all tests will be put into
  the roast .t file
* Run `S15-nfg/mass-roundtrip-test-gen.p6 UNIDATA/NormalizationTest.txt`
  to generate the tests.

.Version
* Unicode v13.0

.Date
* Regenerated 2020.10.01

.Generated File(s)
* S15-nfg/mass-roundtrip-nfc.t
* S15-nfg/mass-roundtrip-nfd.t
* S15-nfg/mass-roundtrip-nfkc.t
* S15-nfg/mass-roundtrip-nfkd.t


== S32-str/CollationTestGen.p6

Note: you will have to be in S32-str before running this, so it puts files in the
correct folder.

S32-str/CollationTestGen.p6 --folder=$UNIDATA/UCA/CollationTest --test-only --should-test
MAKE SURE TO RUN `perl6 CollationTestGen.p6 --folder=MoarVM/UNIDATA/UCA/CollationTest --test-only --should-test ` to ensure there are ~74 failures only!

Then paste those failures (if they're ok into CollationTestGen.p6 in the %todo variable.

.Version
* Unicode v13.0

.Date
* Regenerated 2020.10.01

.Generated File(s)
* S32-str/CollationTest_NON_IGNORABLE-0.t
* S32-str/CollationTest_NON_IGNORABLE-1.t
* S32-str/CollationTest_NON_IGNORABLE-2.t
* S32-str/CollationTest_NON_IGNORABLE-3.t

== S15-normalization/test-gen.p6

.Steps
* Run `S15-normalization/test-gen.p6 UNIDATA/NormalizationTest.txt`

.Version
* Unicode v13.0

.Date
* Regenerated 2020.10.01

.Generated File(s)
* S15-normalization/nfc-X.t
* S15-normalization/nfc-sanity.t
* S15-normalization/nfd-X.t
* S15-normalization/nfd-sanity.t
* S15-normalization/nfkc-X.t
* S15-normalization/nfkc-sanity.t
* S15-normalization/nfkd-X.t
* S15-normalization/nfc-concat.t

== S15-nfg/GraphemeBreakTest.t

.How
* Copy file from UNIDATA to 3rdparty folder of roast:
cp UNIDATA/auxilary/GraphemeBreakTest.txt 3rdparty/Unicode/$version/ucd/auxilary/GraphemeBreakTest.txt

change the filename in the S15-nfg/GraphemeBreakTest.t

```
my IO::Path $repo-dir      = $?FILE.IO.parent(2).add("3rdparty/Unicode/11.0.0/ucd/auxiliary/GraphemeBreakTest.txt");
```
.Version

.Date

== External

=== Also you will have to do stuff in rakudo

Edit t/09-moar/UnipropCheck.pm6 folder to be set to the current version.

You will have to now copy:

mkdir -p t/3rdparty/Unicode/$version/extracted
cp $UNIDATA/LineBreak.txt $UNIDATA/UnicodeData.txt t/3rdparty/Unicode/$version
cp $UNIDATA/extracted/DerivedGeneralCategory.txt t/3rdparty/Unicode/extracted

{'fraction_non_alphanumeric': 0.10825027685492802, 'fraction_numerical': 0.03737541528239203, 'mean_word_length': 4.106007067137809, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2476150', 'n_tokens_mistral': 2843, 'n_tokens_neox': 2662, 'n_words': 712}