Welcome to Arkanis Development

A short introduction to NNTP

Published

I'm currently working on a small web frontend for newsgroups (among other things) and while newsgroups have a reputation of old dusty technology I was impressed by the underlying protocol: NNTP.

Not that the protocol is very well done (I don't have the experience to judge that) but the fact that it's directly usable for a human impressed me. And because I really like such technology get ready for a small netcat session: we're going to read a newsgroup without a newsgroup reader. :)

Connect

First thing we need is a connection (it always starts like this nowadays, does it?). If you run Linux old netcat will do the trick:

netcat news.example.com 119

For encrypted newsgroups socat will help:

socat stdio openssl:news.example.com:563,verify=0

Usually newsgroup servers listen on port 119 and if they use SSL encryption on port 563. But that's all about these commands, there is no magic behind them. If you run them you'll get a plain TCP channel to the NNTP server and the server will send you a welcome line. Something like that:

$ netcat news.example.com 119
200 news.example.com InterNetNews NNRP server INN 2.5.0 ready (no posting)

If you newsgroup requires authentication (a login) you can send it using authinfo. If you don't know just skip authinfo. The server will complain if it needs to know who you are and the you can tell it.

authinfo user newone
381 Enter password
authinfo pass secret12345
281 Authentication succeeded

Dive in

To get an overview of available newsgroups just type list and hit return. The server will then send a list of all newsgroups you can read.

list
215 Newsgroups in form "group high low flags"
example.talk 0000005146 0000002417 y
example.announce 0000000123 0000000016 n

For each newsgroup we get the name, the numbers of the newest and oldest posts in the group and finally a flag. If the flag is y we are allowed to post in there and if it's n we are not. The numbers are also no real post IDs but more like indices. All posts in a group are numbered consecutively and the two numbers just say us where we can start to look and where we can stop. Also a post in a newsgroup can be deleted any time, therefore there can be holes in the list and it's more like a "watermark": we know there has been a message with the number 5146 in the example.talk group but it might have been deleted already.

Anyway, lets read the newest message in the example.talk group:

group example.talk
211 2617 2417 5146 example.talk
article 5146
…
From: Someone <someone@example.com>
Newsgroups: example.talk
Subject: Some demo message
Date: Thu, 27 Jan 2011 23:20:26 +0100
…

This is the content of the demo message. Usually just the same as with mails.
.

With the group command we can say which group we are interested in. The server confirms that with a 211 status code (the first line of a server reply always starts with the status code) an estimated post count for this group and the low and high watermarks again (this time in the reverse order).

With article we can now read every message we want. It returns the contents of the message with the given number followed by a line that just contains a dot. And this is the downside of newsgroups: all the mess of mails are in there, too. I cut a lot of headers out of the sample above and some clients use very funny ways to encode things like the subject and the body. Unfortunately this is a remnant of the pre UTF-8 era and there isn't much that can be done about that.

On the bright side NNTP makes browsing easy. Just type last and return to select the previous article. The server remembers which article you red last and last sets this pointer to the previous message. If you now type article without a number the server will send you the contents this message. next works the same way just into the opposite direction. However once you enter a newsgroup with the group command you first have to select a message with a number for this browsing to work.

Have your say

Posting a message works similar but the other way around. Just type post and write your stuff, followed by a line with just a dot. If you actually want to write a line with only a dot in your message just make two dots out of it. The server will decode that later on.

post
340 Ok, recommended message-ID <u4yuyzyw@news.example.com>
From: New one <newone@example.com>
Subject: Hello there
Content-Type: text/plain; charset=utf-8
Newsgroups: example.talk

Hi,

I'm posting this using just a raw TCP channel.
.
240 Article received <u4yuyzyw@news.example.com>

And that's basically it. You don't really need the Content-Type header if you don't use UTF-8 characters but if you don't type English text it's a good idea since most Linux terminals use UTF-8 today.

Some other nice stuff

To get a short overview of what's going on in a group you can ask for all subject headers of some messages:

hdr subject 5100-5146
225 Header or metadata information for subject follows (from overview)
5100 An older message
…
5146 Some demo message
.

The hdr command will show the specified header of all messages in the range. On some older newsgroup servers you might have to use xhdr instead of hdr since the command is kind of new.

In case you want to write your own newsgroup reader (just for fun of course) you might also find the over or xover command useful. NNTP has quite some more stuff to offer so a look into RFC 3977 can't hurt if you plan to spend some more time with NNTP.

Some thoughts

Even after some weeks and mixed experiences with NNTP I still like the fact that you can easily talk some NNTP yourself. Of course I don't use a terminal and netcat to read real newsgroups (if you want to see some obscurely formatted mails that might be interesting). I rather stick to Thunderbird or my own newsreader. However it's a very nice thought to being able to just ditch all news reading software if something goes wrong and see for yourself what's going on. This is a somewhat refreshing change to the newer REST style APIs which are more machine friendly but 10 KiByte of JSON code in one line tend to be a bit difficult to read…

One comment for this post

leave a new one

#1 by
Anonymus
,

Love it thanks! Did not know you can use netcat for this :)

Leave a new comment

Having thoughts on your mind about this stuff here? Want to tell me and the rest of the world your opinion? Write and post it right here. Be sure to check out the format help (focus the large text field) and give the preview button a try.

Format help

Please us the following stuff to spice up your comment.

An empty line starts a new paragraph. ---- print "---- lines start/end code" ---- * List items start with a * or -

or