Q:

ignore files in use (being written to) when using rsync

I have an SFTP server where clients are constantly uploading large files. On a periodic basis, I want to copy all complete (fully uploaded) files to another machine for processing. I don't want to copy a file that is actively being written to. Is there a way to accomplish that? I am currently using rsync but I am open to switching to something else.

A:

To check if a file is currently open (if a file is currently written is for sure open by some process) the standard way is to use lsof:
if lsof /your/file > /dev/null; then echo "file currently open"; fi

You can use this snippet to filter find results for only not opened files and use them to feed rsync:
find . -type f -exec sh -c 'if ! lsof `readlink -f {}` > /dev/null; then echo `basename {}`; fi' \; | tr '\n' '\0' | rsync -avz --from0 --files-from=- ./ user@host:destination/

Some notes:

readlink -f is needed to have full path of a file, lsof accept only full path
tr '\n' '\0' emulate find -print0 

