Welcome to Arkanis Development

Fast file transfer with netcat

Published

Yesterday I wanted to move an 2.3 GiByte VMware image from one Ubuntu System to another. At first I tried SFTP but it's more than slow (1 MiByte/s on a 100 MBit/s connection). The next logical step would have been to use Samba or NFS but I don't like both of them…

This is where netcat comes in handy. netcat is a small Unix/Linux command line tool that lets you pipe data over a network connection (among many other things). The cool thing: it's quite simple.

First listen on the server

netcat -l -w 2 1111 > vm.zip

The -l parameter makes netcat listen for incoming data while -w 2 tells netcat to automatically close the connection 2 seconds after data stopped coming in. 1111 at last is the port number where netcat actually listens. All data received is then stored in the file vm.zip.

Then send the data from the client

netcat 192.168.0.1 1111 < vm.zip

This one's straight forward: give netcat the IP address and port where the other system's listening (192.168.0.1 1111) and feed it with data (< vm.zip).

This's it

This little trick gave me a transfer rate from about 7 to 9 MiByte/s. Much better than SFTP.

Of course this netcat styles of data transfer has it's drawbacks. Theres no real protocol involved so it's somewhat vague. On the positive side attackers will have a hard time to figure out what the heck you're sending. It's also not possible to transfer multiple files with netcat alone but tar can help us here:

Server:

netcat -l -w 2 1111 | tar -xz

Client:

tar -cz * | netcat 192.168.0.1 1111

It's basically piping a tar archive over the network. -c for creating the archive and -x for extracting it. The z parameter of tar gives us data compression for free. Pretty nice and basic.

On the fly check with MD5

Some Unix piping goodes makes it possible to check the data transfer on the fly with MD5. For those who don't trust netcat. :)

Again, server:

netcat -l -w 2 1111 | tee >( md5sum > /dev/stderr ) | tar -xz

Then client:

tar -cz * | tee >( md5sum > /dev/stderr ) | netcat 127.0.0.1 1111

This transfers the data over the network and outputs an MD5 checksum afterwards. Just compare the checksums and you know your data wasn't hurt on the way.

These commands use tee and the bash process substitution feature to redirect the tar archive to the md5sum program and netcat on the same time (taken from Redirect output to multiple processes). It's a bit complex though so it's better suited for shell scripts I think.

2 comments for this post

leave a new one

#1 by
Timothy Legg
,

Thank you very much! Excellent page and terrific usage of the tee command.

I tested it in a slightly different way.

Source machine:

netcat 192.168.10.170 1111 < test.txt

Destination machine:

netcat -l -w 2 1111 | tee >(md5sum > test.txt.md5) | cat > test.txt

What this does: • Source sends a file test.txt on port 1111 • Destination reads from port 1111 and will tee the output to md5sum > test.txt.md5 and cat > test.txt

The destination will have test.txt and test.txt.md5 residing on it's filesystem.

Tim Legg

#2 by
Thomas Lee
,

Have been trying to transfer files between Mininet VMs. This post helped me to do that. Ran netcat on H1 and H2 in miniet and could successfully transfer files across S1.

Leave a new comment

Having thoughts on your mind about this stuff here? Want to tell me and the rest of the world your opinion? Write and post it right here. Be sure to check out the format help (focus the large text field) and give the preview button a try.

Format help

Please us the following stuff to spice up your comment.

An empty line starts a new paragraph. ---- print "---- lines start/end code" ---- * List items start with a * or -

or