rsync: remote copy incrementally and on-line compressed

rsync is a UNIX command to synchronize files and folders between two computers. It is very useful to download files to and from the server. The key difference to more commonly used scp (SSH copy) is that rsync can do incremental copies: it only copies the modified files. This is especially useful if you sync a large source code tree between different computers. rsync should be in any developer’s toolbox who are working with UNIX based software and deployments.

rsync works over SSH by default (I think it was over telnet by default in long past, but nowadays it’s SSH based, someone please refresh my memory). rsync also supports SSH:s on-line compression, which greatly reduces bytes needed to transfer when copying textual content (source code, logs).

To copy a remote folder to your local computer:

rsync –compress-level=9 -av username@server:/home/user/yourlogfiles

Explanation

  • –compression-level: Use maximum on-line compression
  • -a: Archive. Keep file timestamps in intact and copy only modified files.
  • -v: Verbose. Show what files we are copying.

If the copy process is interrupted rsync will resume the copy from the last file it didn’t manage to copy. In the end rsync tells the copy speed and (incremental) speed-up stats:

sent 376 bytes  received 80696946 bytes  658753.65 bytes/sec
total size is 2604253766  speedup is 32.2

Note: incremental copy might be confused by continuously changing files (logs, databases).

See also a prior blog post about scp copy and compression level.

Buy open source friendly bitcoins  Subscribe to this blog in a reader Follow me on Twitter Follow me on Facebook Follow me Google+

4 thoughts on “rsync: remote copy incrementally and on-line compressed

  1. Actually, rsync does not only copy the modified files, it only copies the changed parts of the modified files. So if you change one byte in a 2GB file, one byte (plus some protocol overhead, hashes, etc.) is transferred.

  2. Rsync used to use rsh, not telnet. They’re both unencrypted protocols that let you access a shell. IIRC, telnet at least requires that you authenticated using a username and a password, while in rsh you’re authenticated by your IP address. Can you spell “OMG, seriously???”

    I could be misremembering, since I’ve never used — actually, never even seen — rsh in my life.

  3. Ah young guns, yes it was rsh and it was very easy to use back in the day.. I don’t mind rsh still on internal network and I am using it in my private labs (which ironically is used to test security treats)!

  4. Pingback: SSH key and passwordless login basics for developers

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>