Fast file transfer with netcat

May 22nd, 2009 netcat, file-transfer, ubuntu

Yesterday I wanted to move an 2.3 GiByte VMware image from one Ubuntu System to another. At first I tried SFTP but it's more than slow (1 MiByte/s on a 100 MBit/s connection). The next logical step would have been to use Samba or NFS but I don't like both of them…

This is where netcat comes in handy. netcat is a small Unix/Linux command line tool that lets you pipe data over a network connection (among many other things). The cool thing: it's quite simple.

First listen on the server ¶

netcat -l -w 2 1111 > vm.zip

The -l parameter makes netcat listen for incoming data while -w 2 tells netcat to automatically close the connection 2 seconds after data stopped coming in. 1111 at last is the port number where netcat actually listens. All data received is then stored in the file vm.zip.

Then send the data from the client ¶

netcat 192.168.0.1 1111 < vm.zip

This one's straight forward: give netcat the IP address and port where the other system's listening (192.168.0.1 1111) and feed it with data (< vm.zip).

This's it ¶

This little trick gave me a transfer rate from about 7 to 9 MiByte/s. Much better than SFTP.

Of course this netcat styles of data transfer has it's drawbacks. Theres no real protocol involved so it's somewhat vague. On the positive side attackers will have a hard time to figure out what the heck you're sending. It's also not possible to transfer multiple files with netcat alone but tar can help us here:

Server:

netcat -l -w 2 1111 | tar -xz

Client:

tar -cz * | netcat 192.168.0.1 1111

It's basically piping a tar archive over the network. -c for creating the archive and -x for extracting it. The z parameter of tar gives us data compression for free. Pretty nice and basic.

On the fly check with MD5 ¶

Some Unix piping goodes makes it possible to check the data transfer on the fly with MD5. For those who don't trust netcat. 🙂

Again, server:

netcat -l -w 2 1111 | tee >( md5sum > /dev/stderr ) | tar -xz

Then client:

tar -cz * | tee >( md5sum > /dev/stderr ) | netcat 127.0.0.1 1111

This transfers the data over the network and outputs an MD5 checksum afterwards. Just compare the checksums and you know your data wasn't hurt on the way.

These commands use tee and the bash process substitution feature to redirect the tar archive to the md5sum program and netcat on the same time (taken from Redirect output to multiple processes). It's a bit complex though so it's better suited for shell scripts I think.

#1 by Timothy Legg April 1st, 2011 22:15

Thank you very much! Excellent page and terrific usage of the tee command.

I tested it in a slightly different way.

Source machine:

netcat 192.168.10.170 1111 < test.txt

Destination machine:

netcat -l -w 2 1111 | tee >(md5sum > test.txt.md5) | cat > test.txt

What this does:

Source sends a file test.txt on port 1111
Destination reads from port 1111 and will tee the output to md5sum > test.txt.md5 and cat > test.txt

The destination will have test.txt and test.txt.md5 residing on it's filesystem.

Tim Legg

#2 by Thomas Lee July 11th, 2014 19:15

Have been trying to transfer files between Mininet VMs. This post helped me to do that. Ran netcat on H1 and H2 in miniet and could successfully transfer files across S1.

#3 by Elusiverite December 11th, 2017 5:57

That's about 2.3Gb in under 5 MINUTES??? WHOAAAA NELLY!!! nice!!!

First listen on the server ¶

Then send the data from the client ¶

This's it ¶

On the fly check with MD5 ¶

Comments