<commit_before>.\" Copyright (c) 2011 <NAME> <<EMAIL>>
.\" All rights reserved.
.\" 
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\" 
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.Dd November 18, 2011
.Dt FPART 1
.Os
.Sh NAME
.Nm fpart
.Nd sort file trees and divide them into partitions
.Sh SYNOPSIS
.Nm
.Op Fl h
.Op Fl V
.Fl n Ar num | Fl f Ar files | Fl s Ar size
.Op Fl i Ar infile
.Op Fl a
.Op Fl o Ar outfile
.Op Fl d Ar depth
.Op Fl e
.Op Fl v
.Op Fl L
.Op Fl l
.Op Fl x
.Op Ar file or dir ...
.Sh DESCRIPTION
The
.Nm
utility helps you sort file trees and divide them into bags
(called "partitions").
.Pp
.Sh GENERAL OPTIONS
.Bl -tag -width indent
.It Fl h
Print help
.It Fl V
Print version
.El
.Sh PARTITION CONTROL
.Bl -tag -width indent
.It Ic -n Ar num
Cause
.Nm
to create
.Ar num
partitions and spread files over them by trying to generate partitions with
the same size and number of files. This option cannot be used in conjunction
with
.Fl L .
.It Ic -f Ar files
Cause
.Nm
to create partitions containing at most
.Ar files
files. This option can be used in conjunction with
.Fl s
and
.Fl L .
.It Ic -s Ar size
Cause
.Nm
to create partitions of maximum size
.Ar size .
With this option, partition 0 may be used to handle files that do not fit in a
regular partition, given the provided
.Ar size
limit. This option can be used in conjunction with
.Fl f
and
.Fl L .
.El
.Sh INPUT CONTROL
.Bl -tag -width indent
.It Ic -i Ar file
Read file or directory list from
.Ar file .
If
.Ar file
is
.Dq Li "-" ,
then list is read from stdin.
.It Fl a
Input contains arbitrary values; just sort them (do not examine file system).
Input must follow the
.Dq Li "size(blank)path"
scheme.
.El
.Sh OUTPUT CONTROL
.Bl -tag -width indent
.It Ic -o Ar file
Output partitions' content to
.Ar file
template. Multiple files will be
generated given that template. Each
.Ar file
will get partition number as
a suffix. If
.Ar file
is
.Dq Li "-" ,
then partitions will be printed to stdout, with partition number used as a
prefix (so you can grep partitions you are interested in, or do whatever you
want).
.It Ic -d Ar depth
After a certain
.Ar depth ,
output directories as file entries (directories themselves will be added to
partitions, instead of their content).
.It Fl e
When displaying directories (see 
.Fl d
option), add an ending
.Dq Li "/"
to each directory entry.
.It Fl v
Verbose mode (may be specified more than once).
.El
.Sh BEHAVIOUR CONTROL
.Bl -tag -width indent
.It Fl L
Live mode (default: disabled). When using this mode, partitions will be
generated while crawling filesystem. This option saves time and memory, but does
not give partition 0 a special meaning (see option
.Fl s
). As a consequence, it can generate partitions
larger than the size specified with option
.Fl s .
This option can be used in conjunction with options
.Fl f
and
.Fl s ,
but not with option
.Fl n .
.It Fl l<commit_msg>- Document -w and -W - Add missing options in summary header<commit_after>.\" Copyright (c) 2011 <NAME> <<EMAIL>>
.\" All rights reserved.
.\" 
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\" 
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.Dd November 18, 2011
.Dt FPART 1
.Os
.Sh NAME
.Nm fpart
.Nd sort file trees and divide them into partitions
.Sh SYNOPSIS
.Nm
.Op Fl h
.Op Fl V
.Fl n Ar num | Fl f Ar files | Fl s Ar size
.Op Fl i Ar infile
.Op Fl a
.Op Fl o Ar outfile
.Op Fl d Ar depth
.Op Fl e
.Op Fl v
.Op Fl L
.Op Fl w Ar cmd
.Op Fl W Ar cmd
.Op Fl l
.Op Fl x
.Op Fl p Ar num
.Op Fl q Ar num
.Op Fl r Ar num
.Op Fl m
.Op Ar file or dir ...
.Sh DESCRIPTION
The
.Nm
utility helps you sort file trees and divide them into bags
(called "partitions").
.Pp
.Sh GENERAL OPTIONS
.Bl -tag -width indent
.It Fl h
Print help
.It Fl V
Print version
.El
.Sh PARTITION CONTROL
.Bl -tag -width indent
.It Ic -n Ar num
Cause
.Nm
to create
.Ar num
partitions and spread files over them by trying to generate partitions with
the same size and number of files. This option cannot be used in conjunction
with
.Fl L .
.It Ic -f Ar files
Cause
.Nm
to create partitions containing at most
.Ar files
files. This option can be used in conjunction with
.Fl s
and
.Fl L .
.It Ic -s Ar size
Cause
.Nm
to create partitions of maximum size
.Ar size .
With this option, partition 0 may be used to handle files that do not fit in a
regular partition, given the provided
.Ar size
limit. This option can be used in conjunction with
.Fl f
and
.Fl L .
.El
.Sh INPUT CONTROL
.Bl -tag -width indent
.It Ic -i Ar file
Read file or directory list from
.Ar file .
If
.Ar file
is
.Dq Li "-" ,
then list is read from stdin.
.It Fl a
Input contains arbitrary values; just sort them (do not examine file system).
Input must follow the
.Dq Li "size(blank)path"
scheme.
.El
.Sh OUTPUT CONTROL
.Bl -tag -width indent
.It Ic -o Ar file
Output partitions' content to
.Ar file
template. Multiple files will be
generated given that template. Each
.Ar file
will get partition number as
a suffix. If
.Ar file
is
.Dq Li "-" ,
then partitions will be printed to stdout, with partition number used as a
prefix (so you can grep partitions you are interested in, or do whatever you
want).
.It Ic -d Ar depth
After a certain
.Ar depth ,
output directories as file entries (directories themselves will be added to
partitions, instead of their content).
.It Fl e
When displaying directories (see 
.Fl d
option), add an ending
.Dq Li "/"
to each directory entry.
.It Fl v
Verbose mode (may be specified more than once).
.El
.Sh BEHAVIOUR CONTROL
.Bl -tag -width indent
.It Fl L
Live mode (default: disabled). When using this mode, partitions will be
generated while crawling filesystem. This option saves time and memory, but does
not give partition 0 a special meaning (see option
.Fl s
). As a consequence, it can generate partitions
larger than the size specified with option
.Fl s .
This option can be used in conjunction with options
.Fl f
and
.Fl s ,
but not with option
.Fl n .
.It Ic -w Ar cmd
When using live mode, execute
.Ar cmd
when starting a new partition (before having opened output file, if any).
.Ar cmd
is run in a specific environment that provides several variables describing the
state of current partition :
.Ev FPART_PARTFILENAME
(current partition's output file name),
.Ev FPART_PARTNUMBER
(current partition number),
.Ev FPART_PARTSIZE
(current partition size),
.Ev FPART_PARTNUMFILES
(number of files in current partition). Note that variables may or may not be
defined, depending of requested options and current partition's state when the
hook is triggered. Also note that hooks are executed in a synchronous way while
crawling filesystem, so avoid executing commands that take a long time to
execute as it slows down filesystem crawling.
.It Ic -W Ar cmd
Same as
.Fl w ,
but executes
.Ar cmd
when finishing a partition (after having closed output file, if any).
.It Fl l
{'fraction_non_alphanumeric': 0.0688355057632231, 'fraction_numerical': 0.0032317138855973283, 'mean_word_length': 3.442105263157895, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1123858', 'n_tokens_mistral': 2854, 'n_tokens_neox': 2609, 'n_words': 1592}