I'm currently working on a small web frontend for newsgroups (among other things) and while newsgroups have a reputation of old dusty technology I was impressed by the underlying protocol: NNTP.
Not that the protocol is very well done (I don't have the experience to judge that) but the fact that
it's directly usable for a human impressed me. And because I really like such technology get ready
for a small
netcat session: we're going to read a newsgroup without a newsgroup reader. :)
First thing we need is a connection (it always starts like this nowadays, does it?). If you run Linux
netcat will do the trick:
netcat news.example.com 119
For encrypted newsgroups
socat will help:
socat stdio openssl:news.example.com:563,verify=0
Usually newsgroup servers listen on port 119 and if they use SSL encryption on port 563. But that's all about these commands, there is no magic behind them. If you run them you'll get a plain TCP channel to the NNTP server and the server will send you a welcome line. Something like that:
$ netcat news.example.com 119 200 news.example.com InterNetNews NNRP server INN 2.5.0 ready (no posting)
If you newsgroup requires authentication (a login) you can send it using
authinfo. If you
don't know just skip
authinfo. The server will complain if it needs to know who you are and the you
can tell it.
authinfo user newone 381 Enter password authinfo pass secret12345 281 Authentication succeeded
To get an overview of available newsgroups just type
list and hit return. The server will then
send a list of all newsgroups you can read.
list 215 Newsgroups in form "group high low flags" example.talk 0000005146 0000002417 y example.announce 0000000123 0000000016 n
For each newsgroup we get the name, the numbers of the newest and oldest posts in the group and
finally a flag. If the flag is
y we are allowed to post in there and if it's
n we are not. The numbers
are also no real post IDs but more like indices. All posts in a group are numbered consecutively and the
two numbers just say us where we can start to look and where we can stop. Also a post in a newsgroup
can be deleted any time, therefore there can be holes in the list and it's more like a "watermark": we
know there has been a message with the number
5146 in the
example.talk group but it might
have been deleted already.
Anyway, lets read the newest message in the
group example.talk 211 2617 2417 5146 example.talk article 5146 … From: Someone <email@example.com> Newsgroups: example.talk Subject: Some demo message Date: Thu, 27 Jan 2011 23:20:26 +0100 … This is the content of the demo message. Usually just the same as with mails. .
group command we can say which group we are interested in. The server confirms
that with a 211 status code (the first line of a server reply always starts with the status code) an
estimated post count for this group and the low and high watermarks again (this time in the reverse
article we can now read every message we want. It returns the contents of the message
with the given number followed by a line that just contains a dot. And this is the downside of newsgroups:
all the mess of mails are in there, too. I cut a lot of headers out of the sample above and some clients use
very funny ways to encode things like the subject and the body. Unfortunately this is a remnant of the
pre UTF-8 era and there isn't much that can be done about that.
On the bright side NNTP makes browsing easy. Just type
last and return to select the previous
article. The server remembers which article you red last and
last sets this pointer to the previous
message. If you now type
article without a number the server will send you the contents this message.
next works the same way just into the opposite direction. However once you enter a newsgroup
group command you first have to select a message with a number for this browsing to work.
Have your say
Posting a message works similar but the other way around. Just type
post and write your stuff,
followed by a line with just a dot. If you actually want to write a line with only a dot in your message
just make two dots out of it. The server will decode that later on.
post 340 Ok, recommended message-ID <firstname.lastname@example.org> From: New one <email@example.com> Subject: Hello there Content-Type: text/plain; charset=utf-8 Newsgroups: example.talk Hi, I'm posting this using just a raw TCP channel. . 240 Article received <firstname.lastname@example.org>
And that's basically it. You don't really need the
Content-Type header if you don't use UTF-8 characters
but if you don't type English text it's a good idea since most Linux terminals use UTF-8 today.
Some other nice stuff
To get a short overview of what's going on in a group you can ask for all
subject headers of some
hdr subject 5100-5146 225 Header or metadata information for subject follows (from overview) 5100 An older message … 5146 Some demo message .
hdr command will show the specified header of all messages in the range. On some older
newsgroup servers you might have to use
xhdr instead of
hdr since the command is kind of new.
In case you want to write your own newsgroup reader (just for fun of course) you might also find the
xover command useful. NNTP has quite some more stuff to offer so a look into
RFC 3977 can't hurt if you plan to spend some more time with NNTP.
Even after some weeks and mixed experiences with NNTP I still like the fact that you can easily
talk some NNTP yourself. Of course I don't use a terminal and
netcat to read real newsgroups
(if you want to see some obscurely formatted mails that might be interesting). I rather stick to
Thunderbird or my own newsreader. However it's a very nice thought to being able to just ditch
all news reading software if something goes wrong and see for yourself what's going on. This is a
somewhat refreshing change to the newer REST style APIs which are more machine friendly but
10 KiByte of JSON code in one line tend to be a bit difficult to read…