On 23rd September, a colleague and I attended the PubSub Huddle in London. It was a one-day conference/hack-afternoon hosted by SkillsMatter.

The day was a good event, there were quite a few people around in the morning and I had some particularly interesting conversations with people responsible for messaging software in hedge funds and banks. The basic structure of the day was talks in the morning which was then followed by a self-organised ‘hack-afternoon’. The morning was excellent and had a large number of attendees. The afternoon was also good fun, but things did tail off as people started disappearing.

The morning started off with Martin Sústrik discussing his vision for ‘the future of messaging’. Martin is the founder of zeromq, but he was talking more about his higher-level goals to simplify messaging by pushing handling of basic messaging patterns into infrastructure. Basically, he was arguing for messaging APIs to be provided as layer-6 (presentation-layer) protocol. The basic points were:

• We need to be able to handle messaging in B2B and B2C scenarios, whereas messaging is today something that generally stays within the corporate firewall.
• Standard messaging patterns need to be simple to code
• Coding messaging should be as easy as coding against a standard Socket abstraction (as demonstrated by his work on zeromq)
• A major challenge of providing this kind of infrastructure is that multiple patterns would need to be supported (he mentioned push-pull, pub-sub, request-response), the difficulty being that the logical topology changes with each pattern
• He mentioned there would be even more challenges with security etc but didn’t discuss them.
• His current plans are to form a proposal for a new protocol – currently named Scalability Protocol – and as a starting point towards this, there is a mailing list http://groups.google.com/group/sp-discuss-group which is apparently an interesting read.
• In terms of practical stuff, he sees zeromq as being a step-towards this (in that it demonstrates the simple APIs he’d like to push for). Other practical steps forward are that people are thinking about adding zeromq to the linux kernel, and considering how vendors might tackle messaging-aware hardware.

The second talk of the day was from Andy Piper from IBM Research. He was talking about MQTT, an open standard (with an IBM reference implementation) for doing simple pub-sub messaging. MQTT was designed to be a low-footprint messaging library for sensors and he showed examples of it running on Android, and also on an Arduino prototype board. It has some nice features, including low resource requirements, and support for handling different types of retry-handling. The other nice feature was support for registering an action to be performed if a client disappears. There is an open source implementation named Mosquito which looked quite interesting. I think a lot of people were a bit suspicious of the ‘IBM sell’, but it looked like a good fit for collecting data from many sources.

The next two talks moved into the territory of doing pub-sub between web clients and servers.

We heard from Marek Majkowski (RabbitMq) about the difficulties of building a generic solution to server-browser messaging. He discussed a project that had attempted to do that, and how it had become really challenging to solve issues around client presence and identity. I found Marek’s talk quite interesting, as he discussed the web-socket standard and showed lots of examples of how the different implementations behaved differently. It really drove home how much the websocket standard is in it’s infancy. He also went on to discuss how to approach building a load-balanced server that could handle messaging-enabled clients, and talked about other products that help make a non-messaging application built on a standard MVC platform (he kept mentioning Django) to become messaging enabled. Some of the example services he mentioned were: AppEngine’s channels API, PubNub, and Pusher. The majority of these services are designed to support server push rather than client pull. And while all of these solutions offer some useful features, he concluded by arguing that they would never be able to provide a simple solution for two-way messaging. He saw the issues around identity and presence as being too difficult to solve in the generic. In the end he basically said that you should build your application around messaging from day 1, and then went on to talk about using NodeJS and SockJS to provide an asynchronous server-side platform.

The fourth talk was from Julien Genestoux about SuperFeedr which allows websites to turn standard RSS feeds into pub-sub channels. He demoed the PubSubHubhub website that basically acts as messaging broker. Sites (publishers) host references to rss feeds and notify the Hub when their feed changes. Supported clients who have subscribed can then be notified by the hub. In a demo he showed how people could attach arbitrary filters to their subscriptions so they only get notified about certain changes. He also talked about how their infrastructure could be used to augment a chain of rss feeds.

After lunch we had a series of lightning talks about various projects… the two that stood out for me were:

• the use of pub-sub messaging to implement the BBC’s Radio1 homepage using XMPP.
• a discussion of the architecture behind datasift.com – which is a stream-based data marketplace where you can subscribe (any pay) for streams of tweets and other social events containing a given criteria. They were using a lot of zeromq and kafka to do their processing (4000 messages a second, I think he said)

All in all, the day was definitely worth attending, and although I didn’t come away with loads of new info, I did come away with a feeling that investing in learning in more pub-sub technologies is a good thing to do.

It was also good to get some validation about how people are using zeromq and comparing it to the work I’ve been doing at Talis. Chatting to Lorenzo Alberton after his talk, it was interesting to hear how they approach failover and scaling of zeromq. It was nice to hear that we following very similar approaches to using and tuning it – particularly since they are getting quite impressive throughput.