<commit_before>#!/usr/bin/env perl

=head1 NAME

generate-names.pl - generate a global index of feature names

=head1 USAGE

  generate-names.pl                        \
      [ --out <output directory> ]         \
      [ --verbose ]

=head1 OPTIONS

=over 4

=item --out <directory>

Data directory to process.  Default 'data/'.

=item --tracks <trackname>[,...]

Comma-separated list of which tracks to include in the names index.  If
not passed, all tracks are indexed.

=item --locationLimit <number>

Maximum number of distinct locations to store for a single name.  Default 100.

=item --sortMem <bytes>

Number of bytes of RAM we are allowed to use for sorting memory.
Default 256000000 (256 MiB).  If you machine has enough RAM,
increasing this amount can speed up this script's running time
significantly.

=item --workdir <dir>

Use the given location for building the names index, copying the index
over to the destination location when fully built.  By default, builds<commit_msg>change --sortMem in generate-names.pl POD to --mem, but --sortMem still works<commit_after>#!/usr/bin/env perl

=head1 NAME

generate-names.pl - generate a global index of feature names

=head1 USAGE

  generate-names.pl                        \
      [ --out <output directory> ]         \
      [ --verbose ]

=head1 OPTIONS

=over 4

=item --out <directory>

Data directory to process.  Default 'data/'.

=item --tracks <trackname>[,...]

Comma-separated list of which tracks to include in the names index.  If
not passed, all tracks are indexed.

=item --locationLimit <number>

Maximum number of distinct locations to store for a single name.  Default 100.

=item --mem <bytes>

Number of bytes of RAM we are allowed to use for caching memory.
Default 256000000 (256 MiB).  If you machine has enough RAM,
increasing this amount can speed up this script's running time
significantly.

=item --workdir <dir>

Use the given location for building the names index, copying the index
over to the destination location when fully built.  By default, builds
{'fraction_non_alphanumeric': 0.08357771260997067, 'fraction_numerical': 0.01857282502443793, 'mean_word_length': 3.0534653465346535, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6138022', 'n_tokens_mistral': 629, 'n_tokens_neox': 566, 'n_words': 273}