A service on a linux server is only able to do full backups, where each backup is a .tar archive (no compression). Many contents of the archive do not change from day to day. Each .tar file size is about 3GB (slowly increasing from day to day).I want to transfer the backups to another server, which archives them. The transfer is done through the internet.A requirement is that the backups are not altered (the result is again a list of .tar files, whose md5 sum is still identical to the original files on the server).I'm currently using rsync to transfer the files, which works great, but all files are transferred with their full size. As far as I know rsync does some kind of deduplication on transfers, but only on a per-file level (right?). Is there any way to transfer few similar files through a SSH connection without retransmitting identical chunks of the files (so some kind of reduplication), thatdoes not require write access on the server (no unpacking of the tar files)is tolerant to connection losses (does not leave temp files on abortions and detects not correctly transmitted files)is able to resume the transfer after connection losses (do not retransmit all files if connection aborts)does not require any additional tools on the server (besides the standard unix toolchain including rsync)still uses a client-initiated SSH connection for the transfer
transfer many similar files over ssh
ssh;file transfer;deduplication
One thing you might do is to (on the receiving side) copy the last backup file to the new name before starting rsync.  Then it will transfer only the diffs between what you have and what you should have.If you do this, be careful if you have rsync -u (update only, based on timestamp) that you ensure that your copy is older than the new source file.