File transfer basics for developers

Post date: Dec 9, 2013 4:28:22 PM

All this should be very clear for everyone, but it unfortunately isn't. I have seen so many broken file transfer implementations. Even if it's not hard at all. I simply hate when things need to be manually fixed, because software written to take care of the task is so bad that it gets stuck with different simple things. I also hate programs which leave junk files everywhere which require manual purging every now and then.

• If transaction is already running go to connect remote host.
• Create temp filename
• Start local transaction containing source, temp and destination filenames and sizes
• Connect remote host
• Check if final filename exists locally check size
• If local file doesn't exist go to finish transaction
• Check if final filename exists remotely check size
• If destination file exists is right size go to rename ... step
• Check if remote temp file exists, check size
• If file is smaller than local file try to resume transfer if possible
• If file size matches, good, go to Temp file ...
• If resume is not possible, delete destination temp file and re-upload whole file
• Temp file transfer is complete
• Check temp file size
• If there's mismatch retry
• If size matches and you want to be extra sure and protocol allows you to do it, check local and remote file hash at this point
• Rename temp file to it's final destination filename
• Rename, delete, move, local file, what ever needs to be done when file had been successfully transferred
• Finish file transfer transaction
• Daily or so, purge temp files from destination which are older than N days, hours or so.

Of course if these things are running in parallel, you can use the transaction table to lock files so that multiple threads / processes aren't handling same file.