<commit_before>## Lines beginning with '##' are for explaining the contents of this
## file, and should be removed if you copy the file to create a new
## (set of) tests.  Also note that the file must be named to match the
## scheme "test_[...].py", i.e., "test_tokens.py".
##
## I don't think I need to explain the copyright notice :)

# vim: tabstop=4 shiftwidth=4 softtabstop=4
# Copyright (c) 2011 OpenStack, LLC.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.


## This test suite uses the DTest framework, which can be found on
## PyPi ("pip install dtest").  DTest uses threading to perform tests
## in parallel; the threading can be limited by using dependencies,
## but the purpose of dependencies is really to say that some test
## must pass before other actions can be performed.  Within dtest, the
## util module contains a number of useful "assert_*()" functions, but
## if you want, you can just use the "assert" statement.
##<commit_msg>Just making sure leading whitespace is stripped if automated.<commit_after>## Lines beginning with '##' are for explaining the contents of this
## file, and should be removed if you copy the file to create a new
## (set of) tests.  Also note that the file must be named to match the
## scheme "test_[...].py", i.e., "test_tokens.py".
##
## I don't think I need to explain the copyright notice :)
##
# vim: tabstop=4 shiftwidth=4 softtabstop=4
# Copyright (c) 2011 OpenStack, LLC.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.


## This test suite uses the DTest framework, which can be found on
## PyPi ("pip install dtest").  DTest uses threading to perform tests
## in parallel; the threading can be limited by using dependencies,
## but the purpose of dependencies is really to say that some test
## must pass before other actions can be performed.  Within dtest, the
## util module contains a number of useful "assert_*()" functions, but
## if you want, you can just use the "assert" statement.
##
{'fraction_non_alphanumeric': 0.08564658090724442, 'fraction_numerical': 0.007447528774542992, 'mean_word_length': 4.220848056537102, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1759854', 'n_tokens_mistral': 783, 'n_tokens_neox': 762, 'n_words': 438}