I have a binary file on a Linux server that is being actively appended by a process (written in C with a constantly open file handler and flushing non-ASCII buffer to this file). I would like to replicate this file to another server without locking the write (C process), preferably, and not copying the entire file every time (file size ~1+GB and replication frequency < 1 sec). I've explored the following:rsync: I believe rsync does a full replication, but not incremental. filebeat by elasticsearch: it requires ASCII text and newlines (I have neither). I would preferably like to leverage standard Linux tools, but I am open to any other 3rd party solution or creating a C program myself :).
Unidirectional syncing/replicate large file incrementally
files;rsync;sftp;inotify;replication
If it's only being appended to (and not modified in the middle), you could just run tail -f on it. It should wait for any newly appended data and print it, and you can tell it what position to start at:tail -c 0 -f datafile        # start at the current file endtail -c +123 -f datafile     # start at byte 123 To actually move the data somewhere, piping through to ssh should work:So if the remote end has the first 123456 bytes already:tail -c +123456 -f datafile | ssh user@somehost 'cat >> datafile.copy'(Though of course you need to go check the file size on the remote before starting the pipeline.)If, instead, you have modifications to the middle of the file, you're going to need some sort of a logging layer in the program itself. Filesystem snapshots might do, but the one second interval may be too hard, especially since you'd need to scan the file for the changes anyway.