Welcome to Arkanis Development

I was just thinking about the tools one can use to write programs. Sure text editors are not especially great compared to professional IDEs like Visual Studio. But somehow I had to think about punch cards and that it probably required much more discipline to program with these. One hole punched at the wrong place… to bad, start over with a new card.

Though thought patters only known to unconscious regions of my brain I remembered a nice xkcd comic: Real Programmers. In there someone says "Real programmers use cat". Hm, that is as low level as I could imagine to go with software. But what if even cat isn't there to help you? What if you attack a server and for some unknown reason get a root shell with a full gcc environment but without cat?

gcc -x c -

This will instruct gcc to read a C programm (-x c) from standard input (the last -). Now you can write down your program line by line. Made an error in the line above? To bad, start over with a new program. :)

Well, it's not as cool as using a magnetized needle but I'm sure it's a nice toy for real programmers. Now, I think I just invented another discipline real programmers can compete in: line buffered programming. Thank good I'm not one of them…

ps.: The really real programmers can write line buffered Java.

pps.: I'm currently writing my bachelor thesis. So, please don't be surprised if some craziness lacks out of my skull here and there…

When the NNTP-Forum was first published two of my friends wanted to see it on GitHub. Well, it's there now: arkanis/nntp-forum.

The French developer Mr Xhark helped me to continue development on the project. Thanks to his relentless testing the NNTP-Forum is now internationalized. It supports three languages: English, German and French (by Mr Xhark). It's also quite easy to write your own language file.

Funny how the experience from the long gone Simple Localization Rails plugin help with this PHP project: The i18n code is about 30 lines of code. It always amazes me how little code is necessary if you know what you need. Ironcially the function that searches the HTTP headers for the users prefered language is about the same size. :D

Anyway, maybe you accidentally have an NNTP server running. In that case you can give the NNTP-Forum a try. Be sure to check out the setup and configuration guide.

LLVM is a very handy piece of software with a quite long manual for its intermediate language. Since I don't like reading long documents on screen I invested 3 hours to generate and print a nice PDF version of the manual. It's not perfect but still comfortable to read.

Thanks to PrinceXML all that is necessary is a small CSS style sheet with printing styles. I also used the Gentium fonts because they are easy to read on paper. Anyone interested in generating an up to date PDF can download the style sheet archive. It contains the style sheet and the Gentium fonts. Extract it into a directory and with PrinceXML installed this command will generate the PDF:

prince --no-author-style --style llvm-lang-ref.css -o llvm-lang-ref.pdf \
    http://www.llvm.org/docs/LangRef.html

Have fun reading. :)

Who doesn't know this situation: You made a note and some time later you need it again. And then it happens: "Where did I put that?!". And the search begins…

Having some stacks of notes flying around my desk I'm very prone to that problem. I've tried some software solutions to this over the last few years: offline wikis, address books (Thunderbird, Opera, etc.), some desktop notes (Tomboy, Opera), files sorted into different directory structurs, tagging, deskop search engines, etc. Nothing really helped me much. It's often quite easy to store data but finding it… well, it didn't work quite well for me.

Please, don't get me wrong. It's not that I can not find something I put into a directory 10 minutes ago. It's just the sheer amount of data that makes it difficult to organize. Many project, many ideas, notes and bookmarks. A small library of sorts, but well, everyone has that today I suppose.

Anyway, I finally took the time to write a little tool to help me organize my stuff:

The plain project

The core idea is the map metaphor. Put everything on a 2D space so it forms some recognizable patterns. Then your spatial sense of direction will help you to get around.

This works very well with classical maps. You know, these things on paper we all used before google maps and navigation systems were around. Once you've found a place on a map it's very easy to find it again. The same principal has been partially used in graphical user interfaces since 1968 (or maybe even earlier, don't know), albeit for other purposes. Well, usability principals is something for another post. Lets just say I've tested out one extreme end of the design space: put everything on a map.

The plain project is a small web application that allows you to put notes and ideas on a 2D space. You can group related stuff into new areas I somehow started to call "plains". Don't ask me why, the name just stuck. Basically it's just a different representation of files (notes and ideas) and directories (plains). However the content is shown directly and the user can move it around on the plains. Even if it sounds stupid, that is the important part: after a little while you build your own map. And since you know every spot it's pretty easy to find something. In an afterthought this reminds me of some memorization methods.

You can give it a try on the sample installation. The code is hosted on GitHub, but be aware: It's a tool for personal usage and therefore not very clean.

How well this map metaphor will scale with large amounts of information I don't know. Right now I've only moved the stuff of the last year or so into my personal plains. Over the next few weeks I'll move over my contacts and complete bookmarks. Especially the bookmarks contain some tricky to sort stuff but only time will tell how well that works. Maybe I can use symlinks in some cases to sort something into multiple categories.

Everything is stored in text files and directories right now. This is primary to keep the data "free" of any application. If the plain project should not work out I don't want to have a big SQLite file laying around. There's also the benefit that I can use the file manager to organize stuff in case the web application doesn't work well (e.g. for large documents nothing beats your text editor of choice).

And this is one of the major problems right now: I used CSS transformation for the panning and zooming. This causes bugs in pretty much any browser I uses. Opera has redraw issues, Firefox jumps around when you click the edit button and Chromium does not allow to scroll in textareas that are off the center. The performance is abysmal in every browser. Right now I use Chromium most of the time.

Further ideas

While I have something I can work with now the map metaphor has massive potential. Not only for small text snippets but for files, pictures, etc. I want to explore that form of usability a bit more but browsers might become to much of a bottleneck. I don't want to store everything on my server either.

Therefore the next iteration will probably be done as a native Linux application based on X11 and OpenGL. This should allow much better controls and other very interesting stuff like organizing hundreds of pictures and files. Maybe its even possible to add normal applications as entries to plains. Well, only time will tell.

Just a few hours ago a good friend of mine and I took CELT out for a test drive. CELT is a new audio codec created by the Xiph foundation. Well, actually we were discussing continuations over an XMPP audio chat and the quality was a bit bad. I mentioned CELT just as a joke but a few minutes later both of us somehow downloaded and compiled the CELT codec.

Basically the idea was the following:

• Get a sound stream from the microphone
• Pipe it through the CELT encoder
• Use netcat to transfer the encoded sound stream over the internet
• Pipe the received sound stream through the CELT decoder
• Play the decoded sound on the other computer

Well, it was just for fun but it did work extraordinary well. We grabbed the mic output with the parec command and pacat plays back such a recorded stream. So this command basically plays back whatever your mic receives:

parec --latency-msec=1 | pacat --latency-msec=1

The --latency-msec option makes sure the latency of pulse audio is not to high. If you omit it pulse audio will choose a very high latency (about 2 seconds in our case). Therefore we just set it to a very small value.

Now we put CELT into the game. Fortunately the CELT library form the Xiph foundation already contains a fully functional encoder and decoder program. celtenc in the tools directory of the library is the encoder, celtdec the decoder. We both just compiled the library (with ./configure and make) and used the encoder and decoder directly from the tools directory. With CELT in the game our local test command looked like that:

parec --latency-msec=1 | ./celtenc --stereo --bitrate 50 - - | \
./celtdec --stereo - - | pacat --latency-msec=1

Now we encode and directly decode the sound stream with CELT. The only thing to watch out for here is the --stereo parameter of the CELT tools. The pulse audio tools use 2 channels by default so the --stereo options is needed to match that. Alternatively you can use the --channels=1 parameter of parec and pacat to make pulse audio just use one channel and then omit the --stereo option.

Anyway, more important is the bitrate. We played around a bit and 50 seems to be quite ok. Lower bitrates still sound good but increased the latency in our case. Probably not because of the codec but because of the audio frame size. We tried to reduce this but this did not work. Anyway 50 KBit/s was a vaule we were happy with.

Now the most important part of all: move the data over the internet. Well, thanks to netcat that is a piece of cake. We first did it in one direction and later on extended it into a real bidirectional voice chat.

For the one who wants to listen:

netcat -l 50123 | \
./celtdec --stereo - - | pacat --latency-msec=1

And the one how likes to send its mic sound:

parec --latency-msec=1 | ./celtenc --stereo --bitrate 50 - - | \
netcat [target-ip] 50123

Basically we just put a netcat pair between the celtenc and celtdec programs. [target-ip] is the IP address of the listener computer. 50123 is just a random port number, you can use whatever port you like. Another idea here is, that the listener starts the netcat server and receives the encoded sound stream.

Now, lets make it two way. So it will actually be useful for something. An easy attempt would be to do the same two commands again. Just in the other direction. That is also the best solution we could come up with.

However it seems a bit wasted to use two connections when we just send one data stream and receive another one. We can use one netcat connection for that:

Computer A:

parec --latency-msec=1 | ./celtenc --stereo - - | \
netcat -l 127.0.0.1 50123 | \
./celtdec --stereo - - | pacat --latency-msec=1

Computer B:

parec --latency-msec=1 | ./celtenc --stereo - - | \
netcat 127.0.0.1 50123 | \
./celtdec --stereo - - | pacat --latency-msec=1

Now the commands get a bit more complex. We send one encoded sound stream into netcat and decode and play back the received sound stream. I couldn't resist to draw a nice little diagram of this setup:

The dashed lines represent CELT encoded data, while normal lines are raw uncompressed audio. The problem with this setup is the latency. As soon as computer A runs the command parec and celtenc will start to record and encode data. Even if no connection has been established yet. The pipe to netcat will buffer this data as long as possible and when computer B connects it will first receive all buffered data. For example if computer B runs the command 2 seconds after computer A there will be a 2 second delay in the sound stream from computer A to B (but only in that direction).

We stopped at that point. It was working quite well and the sound quality was absolutely marvellous. For something hacked together in about 4 hours it's quite well. Most of our time went into working round NAT problems of our routers anyway. Maybe I'll play a bit around with the library and write some basic UDP transport system. The API looks quite easy to work with. :)

A while ago I wrote a forum like frontend for NNTP newsgroups. Since that I was asked two times to make it publically availabe. Today I finally had the time to clean up the code and write some documentation. The result is a small project page with the download: NNTP Forum project page.

There you will find the download and a small installation guide. If something does not work or you need help on the setup just drop me a mail or leave a comment. Right now the procedue is anything but perfect because the NNTP Forum was not made to be portable. Instead it was made to make the best use of the available infrastructure. While this keeps the internals simple the setup suffers a bit. Sorry for that.

The NNTP forum is a modern frontend to newsgroups. It's not a real forum on its own but rather displays the messages of a newsgroup server like a forum does.

Here's the short overview:

• Presents a newsgroup server as a forum
  • First displays all newsgroups on a server
  • Then all topics in a newsgoup
  • And all messages in a topic
• Post new topics and messages
• Supports attachments
• The tree structure of the messages is preserved
  • You can answer to a specific message
  • You can hide all answers to a specific message (collapse a topic branch)
• Markdown is used to display a message as HTML
• Automatically collapses large quote blocks (e.g. the quote of the previous message)
• Highlights unread messages
  • The unread tracker data is stored on the server, one small file per user
  • The tracker data does not grow over time but only with the number of users (only the 50 newest messages in each group are tracked)
• Provides freely configurable newsfeeds for the newsgroups

This is the basic stuff. Right now the forum is optimized for the environment of my university. Therefore the authorization of users is handled in an unusual way:

• Uses HTTP authorization to get user credentials: The authorization method can be feely configured in the webserver, e.g. Apache. Right now we use an LDAP lookup. The downside of this is that the user is always asked for credentials. Guest access would need to be whitelisted in the virtual host config with a proper guest user. This guest user credentials can then be used by the NNTP forum to access the newsgroups.
• When posting a message the username is translated to a display name with an LDAP lookup.

Ok, now to the technical stuff:

• Written in PHP 5.3
• Messages are parsed per line so almost no memory is required (important for large attachments)
• Caching is used extensively to reduce load on the NNTP server
• The frontend does not copy the newsgroup content to a new database
• Almost no new data is stored in the frontend (except the unread tracker data and the cache)
• Supports encrypted NNTP connections
• URLs are rewritten to nice and short URLs using Apaches mod_rewrite

The NNTP forum is up and running at my university since serveral weeks. Unfotunately there is no demo page because you would need an account of my university. You can view some more screen shots on the projects page. If you're interested and want to play with the software just let me know. Leave a comment or drop me a mail. I usually make my projects publically available but the project is optimized for a specific deployment right now. I'm a bit ill from releasing projects without getting any feedback. Therefore I will only take the time to generalize the code base if there is someone who wants it.

Why another NNTP frontend?

My university uses newsgroups to announce official information. These newsgroups can also be used by students to discuss different topics (pretty much everything). Unfortunately the acceptance among the students wasn't very high during the last years. Many students (me included) don't know newsgroups when starting to study. For some people it can be hard to configure the newsgroups in their mail client because our university uses certificates and its own certificate authority. Thanks to that Thunderbird just refuses to accept the news server until you import the certificate authority in a somewhat complicated way. Its a bit disappointing that the result after all that looks just like a bunch of ordinary mails.

People could use the Web-News frontend for the newsgroups as an alternative to a real mail client. However Web-News is not really the most up to date software and the internet changed a little bit since it received its last major update. This doesn't make Web-News a bad NNTP frontend. It's just not perfect for new students that are used to internet forums and facebook. The userbility just isn't as good as it could be for young students.

Because of that not much students checked the newsgroups regulary. Important news are not noticed, lectures are not visited because only a few know they exist… well, I exeggerate quite a bit but this was the direction we were heading towards. These acceptance problems lead to a discussion about the future of newsgroups on our university.

One idea was to replace the newsgroups with a forum. However newsgroups are already a forum. There are some newsgroups (forums) with threads and messages. It's just the presentation in mail clients and Web-News that make newsgroups look “old”. To test that I created a prototype of the NNTP forum within two days. It was read only and did only very simple and inefficient message parsing. But it was already quite intuitive to handle and much more like forums people a used to today.

It's finally done! I submitted my practical term report yesterday. The 6 months at the super computing center Stuttgart (HLRS) were a very interesting time. My task there was to port a computational fluid dynamics (CFD) code to graphics processors. The CDF code was from the Institute of Aerodynamics and Gas Dynamics of the University of Stuttgart.

The people there were very helpful and tried to explain CFD to me. This topic was (and largely still is) totally new to me. It's very interesting to be a programmer in a world of physics and mathematics. I can highly recommend spending 6 month in a field totally new to you. It's a great experience to see the world and problems from different point of views. But just one hint: Don't start with fluid dynamics. ;)

The part of porting the code to GPUs turned out to become a major bug hunt. I hunted some bugs for up to 3 weeks. However I wasn't desperate enough to give them names… so it wasn't as bad as in the IE 6 days.

The CFD code called “Strukti” is written in Fortran and was not multithreaded when the project started. One of the super computers of the HLRS contains some GPUs as calculation accelerators. The basic idea of the project was to use these for Strukti. Since the GPUs are nVidia Tesla chips CUDA was the main focus on the GPU side of things. So for 6 months I tried to understand CFD, learned Fortran, CUDA and a whole lot of other stuff.

Fortran is a very interesting language. Despite its somewhat dusty reputation is's quite alive. Some ideas behind the language really fascinated me (modules, optimization, pointer handling, …). While the ideas are nice the language itself shows the long tradition of backward compatibility and compromises. Basically its like C with a lot nicer syntax and with its ugly corners at other places. But thanks to its syntax it sometimes even felt like writing Ruby (but only for a very short time). I know some people will scream when they read this but well, I'm tired of detailed language comparisons.

Learning CUDA was interesting because of its focus on the hardware. The language itself is not very impressive in my opinion but the GPU architectures are. It's very interesting how they evolved parallel to the CPUs but share some of the same thoughts. I wrote about that in my background studies section of the report: GPGPU origins and GPU hardware architecture. I was somewhat tired of the topic when I wrote the report so it's unfortunately not my usual level of writing. But I don't have the time to rewrite it in a proper way.

If you're interested in what I really did feel free to look into my report: Evaluation of CUDA Fortran for the CFD code “Strukti”. Its a bit better since I wrote it first. It covers how we mixed CUDA and Fortran with the CUDA Fortran PGI compiler extension. The compiler bugs that consumed most of the projects time are also listed and explained in detail.

I presented the project at the institute some weeks ago. You can watch the video here or download it. A high quality version is available at events.mi.

It's a basic introduction into general purpose computations on the GPU as well as CUDA Fortran. It also covers the bugs encountered during the project. The discussion towards the end is more about how one can manage the the complexity of the current hardware. This is the more interesting part in my opinion. GPUs and CPUs are mixing on other platforms like notebooks and desktop systems, too. Therefore this might be interesting for other programmers, not only for high performance computing people.

Looking back it was a very interesting time. I learned much, gained insight into the scientific world and worked on some very interesting super computers. I don't regret taking that choice. :)

I'm currently working on a small web frontend for newsgroups (among other things) and while newsgroups have a reputation of old dusty technology I was impressed by the underlying protocol: NNTP.

Not that the protocol is very well done (I don't have the experience to judge that) but the fact that it's directly usable for a human impressed me. And because I really like such technology get ready for a small netcat session: we're going to read a newsgroup without a newsgroup reader. :)

Connect

First thing we need is a connection (it always starts like this nowadays, does it?). If you run Linux old netcat will do the trick:

netcat news.example.com 119

For encrypted newsgroups socat will help:

socat stdio openssl:news.example.com:563,verify=0

Usually newsgroup servers listen on port 119 and if they use SSL encryption on port 563. But that's all about these commands, there is no magic behind them. If you run them you'll get a plain TCP channel to the NNTP server and the server will send you a welcome line. Something like that:

$ netcat news.example.com 119
200 news.example.com InterNetNews NNRP server INN 2.5.0 ready (no posting)

If you newsgroup requires authentication (a login) you can send it using authinfo. If you don't know just skip authinfo. The server will complain if it needs to know who you are and the you can tell it.

authinfo user newone
381 Enter password
authinfo pass secret12345
281 Authentication succeeded

Dive in

To get an overview of available newsgroups just type list and hit return. The server will then send a list of all newsgroups you can read.

list
215 Newsgroups in form "group high low flags"
example.talk 0000005146 0000002417 y
example.announce 0000000123 0000000016 n

For each newsgroup we get the name, the numbers of the newest and oldest posts in the group and finally a flag. If the flag is y we are allowed to post in there and if it's n we are not. The numbers are also no real post IDs but more like indices. All posts in a group are numbered consecutively and the two numbers just say us where we can start to look and where we can stop. Also a post in a newsgroup can be deleted any time, therefore there can be holes in the list and it's more like a "watermark": we know there has been a message with the number 5146 in the example.talk group but it might have been deleted already.

Anyway, lets read the newest message in the example.talk group:

group example.talk
211 2617 2417 5146 example.talk
article 5146
…
From: Someone <someone@example.com>
Newsgroups: example.talk
Subject: Some demo message
Date: Thu, 27 Jan 2011 23:20:26 +0100
…

This is the content of the demo message. Usually just the same as with mails.
.

With the group command we can say which group we are interested in. The server confirms that with a 211 status code (the first line of a server reply always starts with the status code) an estimated post count for this group and the low and high watermarks again (this time in the reverse order).

With article we can now read every message we want. It returns the contents of the message with the given number followed by a line that just contains a dot. And this is the downside of newsgroups: all the mess of mails are in there, too. I cut a lot of headers out of the sample above and some clients use very funny ways to encode things like the subject and the body. Unfortunately this is a remnant of the pre UTF-8 era and there isn't much that can be done about that.

On the bright side NNTP makes browsing easy. Just type last and return to select the previous article. The server remembers which article you red last and last sets this pointer to the previous message. If you now type article without a number the server will send you the contents this message. next works the same way just into the opposite direction. However once you enter a newsgroup with the group command you first have to select a message with a number for this browsing to work.

Have your say

Posting a message works similar but the other way around. Just type post and write your stuff, followed by a line with just a dot. If you actually want to write a line with only a dot in your message just make two dots out of it. The server will decode that later on.

post
340 Ok, recommended message-ID <u4yuyzyw@news.example.com>
From: New one <newone@example.com>
Subject: Hello there
Content-Type: text/plain; charset=utf-8
Newsgroups: example.talk

Hi,

I'm posting this using just a raw TCP channel.
.
240 Article received <u4yuyzyw@news.example.com>

And that's basically it. You don't really need the Content-Type header if you don't use UTF-8 characters but if you don't type English text it's a good idea since most Linux terminals use UTF-8 today.

Some other nice stuff

To get a short overview of what's going on in a group you can ask for all subject headers of some messages:

hdr subject 5100-5146
225 Header or metadata information for subject follows (from overview)
5100 An older message
…
5146 Some demo message
.

The hdr command will show the specified header of all messages in the range. On some older newsgroup servers you might have to use xhdr instead of hdr since the command is kind of new.

In case you want to write your own newsgroup reader (just for fun of course) you might also find the over or xover command useful. NNTP has quite some more stuff to offer so a look into RFC 3977 can't hurt if you plan to spend some more time with NNTP.

Some thoughts

Even after some weeks and mixed experiences with NNTP I still like the fact that you can easily talk some NNTP yourself. Of course I don't use a terminal and netcat to read real newsgroups (if you want to see some obscurely formatted mails that might be interesting). I rather stick to Thunderbird or my own newsreader. However it's a very nice thought to being able to just ditch all news reading software if something goes wrong and see for yourself what's going on. This is a somewhat refreshing change to the newer REST style APIs which are more machine friendly but 10 KiByte of JSON code in one line tend to be a bit difficult to read…

It's a funny thing, while organizing something I almost never remember to write about it on my own blog. Well, today I finally remembered, so here it is:

Coming Friday (21st of January) the 8th GamesDay takes place at my university. The three talks will cover digital distribution of games (e.g. downloadable games), game development on the Wii console and artificial intelligence in games. A detailed schedule is available on the website. During the breaks visitors can play with some consoles (XBox 360, Wii, PS3) and my brother will show some PC gaming and graphics stuff.

As always the talks will be recorded and a live stream is available at events.mi.

ps.: Why I wrote this post in English even if everyone interested surely can read German? I have no idea. I just felt to exhausted to write German… please don't ask me why my brain works like that, I have no idea.