<commit_before>
On the tar-file-with-no-zipping branch
there was a client test failing.
It failed because there was not tar-file compression (at the time)
and files tar-piped OUT of the container are not (yet) size-limited.
They should be.
How?

the create_text_file_tar_list.sh could
o) get the size of each file
   how?
   stat -c%s "${filename}"
     Alpine. ok
     Ubuntu. ok
     Debian. ok
o) if it is too large...
   - truncate the file
     how?
     truncate -s ${SIZE} "${filename}"
       Alpine. ok
       Ubuntu. ok
       Debian. ok
   - add the filename to a 2nd file holding truncated filenames
     This 2nd file would be added to the 1st list...


Then once the files have been read tar-piped back from the
container into the host process, the 2nd file could be
used to set the correct truncation values.<commit_msg>[ci skip] notes update<commit_after>
On the tar-file-with-no-zipping branch
there was a client test failing.
It failed because there was no tar-file compression (at the time)
and files tar-piped OUT of the container are not size-limited.
They should be?
How?

the create_text_file_tar_list.sh could
o) get the size of each file
   how?
   stat -c%s "${filename}"
     Alpine. ok
     Ubuntu. ok
     Debian. ok
o) if it is too large...
   - truncate the file
     how?
     truncate -s ${SIZE} "${filename}"
       Alpine. ok
       Ubuntu. ok
       Debian. ok
   - add the filename to a 2nd file holding truncated filenames
     This 2nd file would be added to the 1st list...


Then once the files have been read tar-piped back from the
container into the host process, the 2nd file could be
used to set the correct truncation values.

However, the tar-file is now compressed so there is less need
to truncate it before the tar-piping.
And the ulimits should limit the maximum size of an individual file.
{'fraction_non_alphanumeric': 0.06351791530944625, 'fraction_numerical': 0.004343105320304018, 'mean_word_length': 2.8556485355648538, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6252674', 'n_tokens_mistral': 576, 'n_tokens_neox': 537, 'n_words': 280}