Publish-Subscribe Messaging in Bash

In the last article, I have explained how to implement the Point-to-Point Channel and Competing Consumers patterns with toolbox to process messages in parallel. When employing the Competing Consumers pattern, every message on the channel will be received by only one process. But what if we wanted to broadcast messages so that each process receives a copy? This is where Publish-Subscribe Channels enter the stage.

When a process sends a message to a publish-subscribe channel (or topic, as they are called in toolbox), each of the processes that are subscribed to the channel will receive a copy of the message. This means that receiving processes need to subscribe to a channel before messages are being published (one could thus argue that this pattern should better be called Subscribe-Publish Messaging instead). This pattern can be used when the messages that are sent by a process are interesting for many processes, such as data from sensors or other kinds of notifications. What makes this pattern extremely powerful is that you can add new processes to a system without having to change the messaging architecture. All the new process has to do is subscribe to the publish-subscribe channel. This allows you to build very flexible and decoupled architectures.

Pub-Sub with toolbox

Publish-subscribe messaging is so useful, I use it more than point-to-point messages. And because it gets a lot of my attention, it is very straightforward to use with toolbox.

To publish messages, one first needs to open a new IPC endpoint. Then you can publish messages using ipc_endpoint_publish().

Note: To use the ipc_ family of functions, you need to include either the uipc or ipc module. The latter uses GnuPG for message signing and verification, which makes it a bit more complicated to use. If you don’t exchange messages with other users, or you just want to get quick results, you should use the uipc module.

. toolbox.sh
include "uipc"

ep=$(ipc_endpoint_open)
ipc_endpoint_publish "$ep" "topic" "Hello"

A subscriber that loops forever, printing received messages isn’t much longer. Like the publisher, it first needs an endpoint that is consequently subscribed to the topic with ipc_endpoint_subscribe(). Pub-sub messages can be received like point-to-point messages, using ipc_endpoint_recv().

. toolbox.sh
include "uipc"
include "log"

ep=$(ipc_endpoint_open)
ipc_endpoint_subscribe "$ep" "topic"

while true; do
    msg=$(ipc_endpoint_recv "$ep")
    ipc_msg_get_data "$msg" | log_highlight "Message"
done

Another interesting thing about pub-sub messaging in toolbox is that it can be easily combined with the Competing Consumers pattern. All one has to do is pass a name to ipc_endpoint_open() to obtain a public endpoint that can be shared between processes.

As a real-life example, let’s have a look at foundry. A script called watchbot periodically checks a set of git repositories if they have changed. When watchbot detects a change, it sends a message to the commits topic. Another script called buildbot subscribes to this topic and waits for messages. Once it has received a message, it builds Debian packages from the git repository that is mentioned in the message. Buildbot builds packages only for the CPU architecture that it is running on, meaning that one needs instances running on i386 and amd64 hosts if one wants to build packages for those two architectures. If there is only one buildbot instance per architecture, this will work well with naive pub-sub messaging. But what if there are a lot of packages waiting to be built, and one lonely buildbot per architecture can’t catch up with the workload? If we use naive pub-sub messaging and start multiple buildbots for the same architecture, they all will receive any notifications from watchbot, meaning they all will build the same versions of the same packages for the same architecture, leading to many duplicate package builds and no performance gain whatsoever. We prevent this using the Competing Consumers pattern on top of the pub-sub messaging. By having all buildbots for a particular architecture share the same endpoint, we can make sure that only one buildbot on that endpoint will receive a notification, thus avoiding duplicate package builds.

To implement this using toolbox, all we have to do is change the competing consumers example from the last time so that the shared endpoint is subscribed to the topic that we’re interested in, as shown below.

. toolbox.sh
include "uipc"

consumer() {
	local endpoint_name="$1"
	local topic="$2"

	local endpoint
	local message

	endpoint=$(ipc_endpoint_open "$endpoint_name")
	ipc_endpoint_subscribe "$endpoint" "$topic"

	while true; do
		message=$(ipc_endpoint_recv "$endpoint")
		handle_message "$message"
	done
}

endpoint_name="foo"
topic="bar"
num_cpus=$(nproc)

for (( i = 0; i < num_cpus; i++ )); do
	( consumer "$endpoint_name" "$topic" ) &
done
wait

There is one catch, though. Because toolbox uses the filesystem to send and receive messages, communication between processes on different machines only works if the machines share a network filesystem, making toolbox better suited for smaller systems. But it’s not like anyone would use Bash to implement large-scale geographically-distributed systems anyway, right?

If I managed to pique your interest, don’t forget to have a look at foundry, my proof-of-concept for messaging-based distributed systems in Bash.