Imagine you are on lan party or on the road and quickly want to transfer a file or directory to another computer. Both computer owners are just to lazy to setup something like ftp, smb, nfs. A very simple and even cross platform solution is using netcat and in case of a directory in combination with tar like the following steps. I will just show you how to use it without compression for a directory. Fell free to play around. You can test it locally of course.
1. The sender
The sender has to call netcat in server mode and pipe content into it. The next line tells tar to build a tarball and write it to standard output which is redirected via a pipe to netcat. Netcat is told to start in server mode (-l), listen on port 7878 (-p 7878) and shutdown itself after waiting 10 seconds after having seen an end of file in standard input (-q 10):
$ tar c directory | nc -q 10 -l -p 7878
2. The receiver
The receiver has to call netcat and tell him to connect to the remote machine and the correct port and redirects the standard output to a file. For convenience he also sets a timeout parameter (-w 10):
$ nc -w 10 remotehost 7878 > nameoftar.tar
That’s all. You just setup a very fast file transfer. For testing purposes use localhost. Please note that on the receiver side you are completely free to choose a file name for the .tar file. If you use something like gzip or bzip2 compression you should choose something like .tar.gz or .tar.bz2 of course.
Thanks to mnemonik pointing this out.
Feel free to add your file transfer quick hack here.
Pingback: The Linux Index » Caspar Clemens Mierau: Using netcat and tar for network file transfer
tar stream is also much faster to copy files than scp „ping-pong“, especially when you have lots of files. to copy files from server to your machine, do this:
ssh some-server ‚cd /some/dir && tar cz dir‘ | tar xz
and the other direction:
tar cz dir | ssh some-server ‚cd /some/dir && tar xz‘
You can also clone a complete partition via netcat and dd:
sender:
dd if=/dev/sda | nc 192.168.0.100 1234
receiver:
nc -l -p 1234 | dd of=/dev/sda
@bog: That’s totally right. I prefer tar-ssh over scp in a lot of cases, too. In this setting here I just assumed you don’t have and don’t want to setup a ssh server.
@WebKid: Think you should the timeout flag as otherwise you have to ctrl-c after it’s complete. Correct me if I am wrong but with tar I had to do this.
Another cool trick, to copy over directory, minus the ssh.
On the „destination“ host where you want the files to end up:
nc -l 7000 | tar -xvf –
On the „source“ host, where the files are initially located:
tac c | nc 7000
Sorry, in the last comment itshould be:
tar c directory_to_copy | nc destination_host 7000
thanks for the tip.
another scenario where this is a lot useful is transferring file over a VPN connection.
this way you avoid the overhead – both in bandwidth usage and in CPU time – of double encypher/decypher and you can go as fast as you can. This can make a significant difference for slow connections and/or high load of data to transfer.
decompressing the tarfile on the fly on the destination machine is even better.
Pingback: nc (netcat) « boompty boomp
IMHO, scp with proper options are fast enough:
scp -c arcfour256 -C -4 user@remotehost:Downloads/bigfile.iso .
These options can be made implicit by using alias or ssh config file, too
Pingback: Simple File Transfers | eightbits.org
On Mac, netcat has no -q option nor -p seems to work as expected. The following solution (as „the sender“) seems to work fine:
tar c directory | nc -l localhost_or_ip 7878
found on: http://superuser.com/questions/115553/netcat-on-mac-os-x
Had 1000’s of files varying from a few bytes to several 10’s of GB. Found that the following command provided the fastest transfer:
tar cz SRC_DIR | ssh root@DEST_SVR ‚cd DEST_DIR && tar xz‘
Over a crossover cable between servers it hovered at around 930 mbps.
I have been using net cat for just a few days now.
My preferred way of transferring a file or a directory over the network is
Receiver: nc -l 3452 | bunzip2 -vvv | tar -x
Sender: tar -c folder/ | bzip2 -vv9 | nc 127.123.0.1 3452
The benefit is that I do not need to decompress the file on the receiving file, it decompresses by itself, I do not have to set a name or a folder name, tar will create those for me. I can just as easily omit the unzip and untag and save it.
I know tar -j does compression too, but I prefer to see the terminal „doing something“ instead of staring at a blank screen.
I have seen savings of 40% of transfer time using this method and I like it a lot, even more so when moving VMs from one server to another.
Nice post.
[root@localhost user]# tar c index.html | nc -q10 -l -7878
nc: invalid option — ‚q‘
usage: nc [-46CDdhklnrStUuvz] [-I length] [-i interval] [-O length]
[-P proxy_username] [-p source_port] [-s source] [-T ToS]
[-V rtable] [-w timeout] [-X proxy_protocol]
[-x proxy_address[:port]] [destination] [port]
Pingback: OpenWrt Sysupgrade | Gotoif.com
Pingback: Using netcat and tar for network file transfer | pdasite