Performance bug in kestrel 1.1

18 May 2009

If you’re using (or have tried to use) kestrel 1.1, you should upgrade to 1.1.1 (in github) for a performance fix.

Kestrel uses naggati, an actor-adapter for mina, to handle I/O and session management. Mina is a java library for handling thousands of open connections across NIO, and naggati converts mina events from java-style “listener method calls” to actor messages. I wrote all about mina & naggati in another article.

One feature added to naggati 0.6 was the ability to filter out mina events you didn’t care about. So if, for example, you didn’t care to receive an event message every time mina finished transmitting a block of data, you could filter out those messages:

IoHandlerActorAdapter.filter(session) -= classOf[MinaMessage.MessageSent]

Potentially this saves you a lot of unwanted actor messages that you’d just ignore anyway, causing fewer context switches. If the feature works correctly… aye, there’s the rub.

Naggati 0.6 would accidentally overwrite any filter changes that happened during an actor’s constructor. Kestrel’s session actors changed their filters in the constructor, and so their filter changes were lost. This caused kestrel actors to receive reams of unwanted messages like MessageSent, which by itself would only be a minor annoyance, and would have the same performance behavior as kestrel 1.0.

But, in actors, if you don’t pattern match on a message type, and you receive a message of that type, the message just sits in your queue forever. Long-lived connections would eventually have thousands of these unwanted MessageSent messages, which had to be linearly traversed to get to interesting messages. Scanning this queue on every I/O event eventually slowed down a long-lived session to a crawl – and it consumed memory in the process.

This bug is now fixed in naggati 0.7 and kestrel 1.1.1. Kudos to John Kalucki, who found the bug during his own stress tests and persevered when I couldn’t reproduce it myself.

« Back to article list

Please do not post this article to Hacker News.

Permission to scrape this site or any of its content, for any purpose, is denied, regardless of your personal beliefs or desire to design a novel opt-out method.