<commit_before>AUR wrapper interface

This is a proposal on how rsync actions are translated into "events" that are
seen by the rsync daemon client.  There are four components to this system:

1. rsync itself

   It must be run with the --batch flag in order to generate a logfile of its
   processing.  We prefer rsync to APNIC's rcynic.

2. The AUR program

   This program operates on the rsync logfile generated by #1 and sends "events"
   to the rsync daemon.  Events are just a fancy way of saying "strings sent
   over a socket".  The AUR program should accept a command line argument specifying
   the port to which AUR messages are sent, e.g. -t4444 for TCP port 4444 or -u3937
   for UDP port 3937, for example.

3. A wrapper script

   This tiny convenience script calls #1 and #2 with the appropriate options.

4. The rsync client

   This client will be listening on a socket (port provided on the command line,
   as with #2).

Socket messages are unidirectional; there are no replies or acks from the rsync
client. The socket messages might have the following form:

	TAG whitespace VALUE CRLF

I envision the following tags and their meaning:

   B (begin).  This is sent when the AUR program starts. Its VALUE is the current
   date and time.

   E (end).  Sent when the AUR program is done. VALUE is the current date and time.
   AUR may close its end of the socket immediately after sending this message; it
   need not wait.

   A (add). Sent when a file is added to the repository. VALUE is the full, absolute
   path to the file.

   U (update). Sent when a file is updated in the repository, e.g. the contents change
   but the filename remains the same and is in the same directory.

   R (remove). Sent when a file is removed from the repository.  VALUE is the full path
   to the file.

   L (link). Sent when a link (hard or symbolic) is made between two files in the repository.
   VALUE is formed as follows: "filename1" SP filename2.  Filename1 is a full pathname in
   double quotes; it is followed by a single space, and then filename2 which is also a
   full pathname. The link direction is filename1 -> filename2.

   F (fatal error). Sent (if possible) when the AUR program detects an unrecoverable error
   occurs. VALUE is the error text. It is expected that AUR will immediately close its
   end of the socket when this happens (perhaps even without being able to send an E message).

   X (error). Sent when an error occurs. VALUE is error text.  This is an optional message.

   W (warning). Sent when a warning occurs. VALUE is warning text. Optional message.

   I (information). Sent to convey arbitrary information.  VALUE is the informational text.
   Optional information.

Note that for all messages, VALUE is allowed to contain any (8-bit) character in any character
set so long as it does not contain either literal CR or LF.  In the extremely rare case that
VALUE does contain CR or LF, they should be replaced by their corresponding escaped versions,
e.g. \013 and \010.

Note also that it is the responsibility of the rsync client to detect unexpected termination
(death of peer, e.g. AUR) on the socket.  AUR should send an F and/or Q is possible, but there
are clearly conditions in which AUR might be unable to do so.  It is expected that AUR will at
least arrange to handle ^C from the user, as well as other common signals, and attempt to terminate
gracefully.
<commit_msg>Added documentation on the 'C' message.<commit_after>AUR wrapper interface

This is a proposal on how rsync actions are translated into "events" that are
seen by the rsync daemon client.  There are four components to this system:

1. rsync itself

   It must be run with the --batch flag in order to generate a logfile of its
   processing.  We prefer rsync to APNIC's rcynic.

2. The AUR program

   This program operates on the rsync logfile generated by #1 and sends "events"
   to the rsync daemon.  Events are just a fancy way of saying "strings sent
   over a socket".  The AUR program should accept a command line argument specifying
   the port to which AUR messages are sent, e.g. -t4444 for TCP port 4444 or -u3937
   for UDP port 3937, for example.

3. A wrapper script

   This tiny convenience script calls #1 and #2 with the appropriate options.

4. The rsync client

   This client will be listening on a socket (port provided on the command line,
   as with #2).

Socket messages are unidirectional; there are no replies or acks from the rsync
client. The socket messages might have the following form:

	TAG whitespace VALUE CRLF

I envision the following tags and their meaning:

   B (begin).  This is sent when the AUR program starts. Its VALUE is the current
   date and time.

   E (end).  Sent when the AUR program is done. VALUE is the current date and time.
   AUR may close its end of the socket immediately after sending this message; it
   need not wait.

   C (cd). Send when the AUR program changes directories.

   A (add). Sent when a file is added to the repository. VALUE is the full, absolute
   path to the file.

   U (update). Sent when a file is updated in the repository, e.g. the contents change
   but the filename remains the same and is in the same directory.

   R (remove). Sent when a file is removed from the repository.  VALUE is the full path
   to the file.

   L (link). Sent when a link (hard or symbolic) is made between two files in the repository.
   VALUE is formed as follows: "filename1" SP filename2.  Filename1 is a full pathname in
   double quotes; it is followed by a single space, and then filename2 which is also a
   full pathname. The link direction is filename1 -> filename2.

   F (fatal error). Sent (if possible) when the AUR program detects an unrecoverable error
   occurs. VALUE is the error text. It is expected that AUR will immediately close its
   end of the socket when this happens (perhaps even without being able to send an E message).

   X (error). Sent when an error occurs. VALUE is error text.  This is an optional message.

   W (warning). Sent when a warning occurs. VALUE is warning text. Optional message.

   I (information). Sent to convey arbitrary information.  VALUE is the informational text.
   Optional information.

Note that for all messages, VALUE is allowed to contain any (8-bit) character in any character
set so long as it does not contain either literal CR or LF.  In the extremely rare case that
VALUE does contain CR or LF, they should be replaced by their corresponding escaped versions,
e.g. \013 and \010.

Note also that it is the responsibility of the rsync client to detect unexpected termination
(death of peer, e.g. AUR) on the socket.  AUR should send an F and/or Q is possible, but there
are clearly conditions in which AUR might be unable to do so.  It is expected that AUR will at
least arrange to handle ^C from the user, as well as other common signals, and attempt to terminate
gracefully.

{'fraction_non_alphanumeric': 0.04070995914917594, 'fraction_numerical': 0.010424003380757853, 'mean_word_length': 3.0617848970251718, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1642860', 'n_tokens_mistral': 2022, 'n_tokens_neox': 1815, 'n_words': 1174}