Using Tar and SSH to improve SCP Speeds

Sep

2008

Submitted by Nick on Wed, 3rd Sep 2008, 11:06 and has been commented on 5 times.

I am currently in the process of migrating several dozen sites between two servers. I tried using scp command to copy the sites over however scp is a very slow command when transferring many small files.

I did a little research on how to use tar over an ssh connection and realised that you could specify the stdout on tar.

How?

Using this method effectively sends the compressed tarball to the terminal. You then pipe that into an ssh session which is running the extract version of the previous tar function along with the change directory argument. This, essentially, sends the compressed tarball into a decompression process at the other end over a secure ssh "pipe".

The result is a pretty quick file transfer which - as the data is being sent in a compressed GZIP form (of BZip2 if you replace the z with a j in the tar functions) you save on bandwidth too.

Here an an example of how to do this, assuming you are in (for example) /var/www/html/ and the website you want to transfer is the folder www.example.com.

tar czf - www.example.com/ | ssh joebloggs@otherserver.com tar xzf - -C ~/

This will send the entire www.example.com folder over to the home folder on your target server in compressed form over and encrypted connection.

5 Comments

The most recent comment was on Tue, 15th Jun 2010, 15:39

how to get directory over ssh :)

Mar

2009

Submitted by Vitalie (not verified) on Sun, 1st Mar 2009, 12:11.

If you need to get a directory from a server (you must to have an account on it) do such way:

ssh username@from_server "tar czf - directory_to_get" | tar xzvf - -C path_where_to_put

Enjoy :)

Nice!

Mar

2009

Submitted by Nick on Mon, 2nd Mar 2009, 10:14.

That's a really handy way of doing it...

It's like a "pull" (your script) compared to a "push" (my script).

The f - part is redundant;

Nov

2009

Submitted by Anonymous (not verified) on Wed, 18th Nov 2009, 17:33.

The f - part is redundant; Tar will use stdin/stdout b default. So the command becomes:

tar cz www.example.com/ | ssh joebloggs@otherserver.com tar xz -C ~/

Remote backup stream?

Jun

2010

Submitted by Mr. Nobody (not verified) on Tue, 15th Jun 2010, 08:56.

What if I wanted to 'pull' it, but leave it as a compressed tarball? I.E. Remote 'pull' backup.

In other words, create a tar/7z file by streaming the tar/7z output into ssh to another box where it resides as the compressed archive.

You would do that like this..

Jun

2010

Submitted by Nick on Tue, 15th Jun 2010, 15:39.

If you want to do a remote backup and then pull that to your system...

ssh <script type="text/javascript">eval(unescape('%64%6f%63%75%6d%65%6e%74%2e%77%72%69%74%65%28%27%3c%61%20%68%72%65%66%3d%22%6d%61%69%6c%74%6f%3a%75%73%65%72%40%65%78%61%6d%70%6c%65%2e%63%6f%6d%22%3e%75%73%65%72%40%65%78%61%6d%70%6c%65%2e%63%6f%6d%3c%2f%61%3e%27%29%3b'))</script> "tar czf - /path/to/backup/" > ~/backup.tar.gz