I have a very large file (200GB). Apparently when I transfer it over it did not copy correctly. The sha1 hash on both are different. Is there a way I can divide the file up to blocks (like 1MB or 64MB) and output a hash for each block? Then compare/fix?I might just write a quick app to do it.
Hash a file by 64MB blocks?
hashsum
That quick app already exists, and is relatively common: rsync. Of course, rsync will do a whole lot more than that, but what you want is fairly simple:rsync -cvP --inplace user@source:src-path-to-file dest-path-to-file   # from the destinationrsync -cvP --inplace src-path-to-file user@dest:dest-path-to-file     # from the sourceThat will by default use ssh (or maybe rsh, on a really old version) to make the connection and transfer the data. Other methods are possible, too.Options I passed are:-c  skip based on checksums, not file size/mtime. By default rsync optimizes and skips transfers where the size & mtime match. -c forces it to compute the checksum (which is an expensive operation, in terms of I/O). Note this is a block-based checksum (unless you tell it to do whole files only), and it'll only transfer the corrupted blocks. The block size is automatically chosen, but can be overridden with -B (I doubt there is any reason to).-v  verbose, will give some details (which file it's working on)-P  turns on both partial files (so if it gets halfway through, it won't throw out the work) and a progress bar.--inplace  Update the existing file, not a temporary file (which would then replace the original file). Saves you from having a 200GB temporary file. Also implies partial files, so that -P is partially redundant.BTW: I'm not sure how you did the original transfer, but if it was sftp/scp, then something is very wrongthose fully protect from any corruption on the network. You really ought to track down the cause. Defective RAM is a relatively common one.