If you’re using (or have tried to use) kestrel 1.1, you should upgrade to 1.1.1 (in github) for a performance fix.
Kestrel uses naggati, an actor-adapter for mina, to handle I/O and session management. Mina is a java library for handling thousands of open connections across NIO, and naggati converts mina events from java-style “listener method calls” to actor messages. I wrote all about mina & naggati in another article.
One feature added to naggati 0.6 was the ability to filter out mina events you didn’t care about. So if, for example, you didn’t care to receive an event message every time mina finished transmitting a block of data, you could filter out those messages:
IoHandlerActorAdapter.filter(session) -= classOf[MinaMessage.MessageSent]
Potentially this saves you a lot of unwanted actor messages that you’d just ignore anyway, causing fewer context switches. If the feature works correctly… aye, there’s the rub.
Naggati 0.6 would accidentally overwrite any filter changes that happened during an actor’s
constructor. Kestrel’s session actors changed their filters in the constructor, and so their filter
changes were lost. This caused kestrel actors to receive reams of unwanted messages like
MessageSent, which by itself would only be a minor annoyance, and would have the same performance
behavior as kestrel 1.0.
But, in actors, if you don’t pattern match on a message type, and you receive a message of that
type, the message just sits in your queue forever. Long-lived connections would eventually have
thousands of these unwanted MessageSent messages, which had to be linearly traversed to get to
interesting messages. Scanning this queue on every I/O event eventually slowed down a long-lived
session to a crawl – and it consumed memory in the process.
This bug is now fixed in naggati 0.7 and kestrel 1.1.1. Kudos to John Kalucki, who found the bug during his own stress tests and persevered when I couldn’t reproduce it myself.