Fanout queues in kestrel 1.2

10 Oct 2009

I bumped kestrel to version 1.2 this week and there are a few new features:

Thanks to Clement, Zachary Kim, Blair Zajac, and Frederic Jean for submitting patches!

Special thanks to Justin Azoff, who’s responsible for the coolest new feature: fanout queues. My coworkers had been asking about for this feature for a while, and he posted a branch on github which added them in a way that makes sense to me, so I pulled it in with minor changes. Here’s how they work:

A fanout queue is a “parent” queue that has “children” queues. Whenever an item is added to the parent, the same item is added automatically to each child. The children each have their own journal files, and are essentially independent queues. An item removed from one child doesn’t affect any other children. In fact, you can even add items directly to a child if you want, bypassing the fanout. The primary distinction is that SETs to the parent are repeated to each child automatically.

A child queue is created automatically by referring to a queue with a + in its name. For example, a queue named “orders+audit” is considered to be a child of the “orders” queue. Once a child is created, it starts receiving new items immediately, but existing items in the parent are not copied over. (The child’s history begins now.) Fanout stops when a child is DELETEd.

Children use the same configuration as their parent, to make things easier. There’s no specific configuration for a child. Also, you can expect fanout queues to use more disk space and memory, since items are being copied around muliple times.

Hopefully the simplicity of the implementation makes it easy to reason about, which I think is more important than having a lot of clever tricks or magic.

Kestrel is, as always, hosted here: http://github.com/robey/kestrel

« Back to article list

Please do not post this article to Hacker News.

Permission to scrape this site or any of its content, for any purpose, is denied, regardless of your personal beliefs or desire to design a novel opt-out method.