Toolbox

I spend a lot of time with shell scripts, which can be a veritable pain in the butt because in shell you can’t just include "some_module" and start using functions from it. After writing what felt like my hundredth command line parser, I asked myself, “would it be possible to write my own include function?”

Turns out it is! Say hello to toolbox, my module framework for Bash. To use it, just source toolbox.sh and load any module you need.

. toolbox.sh
include "ipc"

If you are in the mood for some logging, all you have to do is include "log" and you can use functions such as log_error(), log_stacktrace() or log_highlight() to log to your heart’s content.

Maybe now you want to accept parameters on the command line? No problem! include "opt" and you have a flexible command line parser for every occasion:

• Long and short parameter names
• Optional and required parameters
• Parameters with and without values
• Usage (-h / --help) text
• Validation of values with regular expressions
• Default values
• Callbacks

Say your script has a required argument -i (or --ingredient) that is followed by either “salt”, “pepper”, or “cumin”. What’s more, you want to call add_ingredient() whenever this argument was passed. How many lines do you think this takes? Toolbox does it in two (well, sort of).

include "opt"

# Adds "-i" / "--ingredient" option, (r)equired,
# followed by a (v)alue, no default value
opt_add_arg "i"                      \
            "ingredient"             \
            "rv"                     \
            ""                       \
            "Ingredient to be added" \
            '^(salt|pepper|cumin)$'  \
            add_ingredient

if ! opt_parse "$@"; then
	# Invalid or unrecognized option, invalid
	# value, or a callback returned an error
fi

ingredient=$(opt_get "ingredient")

But there’s more! Maybe you need a daemon really really quick. This is how you do it with the inst module.

include "inst"

_daemon_main() {
	while inst_running; do
		sleep 5
	done

	return 0
}

inst_start _daemon_main

You want the daemon to be a singleton? Swap inst_start for inst_singleton - done!

That’s still not everything. I bet you never even thought about writing a distributed system in Bash, did you? The ipc module lets you do just that, and it’s not even hard (assuming the machines are sharing an NFS mount they can communicate over). Create an IPC endpoint with ipc_endpoint_open(), and you can use it to send and receive messages in a point-to-point fashion using ipc_endpoint_send() and ipc_endpoint_recv().

# create a named endpoint
receiver=$(ipc_endpoint_open "my_endpoint")

# create an anonymous endpoint
sender=$(ipc_endpoint_open)

# Send on sender endpoint, receive on receiver endpoint
ipc_endpoint_send "$sender" "my_endpoint" "Hello world"
msg=$(ipc_endpoint_recv "$receiver")

# Get the source address and send a response
source=$(ipc_msg_get_source "$msg")
ipc_endpoint_send "$receiver" "$source" "Hello to you, too!"

You’d prefer communicating in a pub-sub fashion? ipc won’t let you down! The following is a minimal publisher.

endp=$(ipc_endpoint_open)
ipc_endpoint_publish "$endp" "mytopic" "Hello world"

And this is the corresponding subscriber.

endp=$(ipc_endpoint_open)
ipc_endpoint_subscribe "$endp" "mytopic"

if msg=$(ipc_endpoint_recv "$endp"); then
	data=$(ipc_msg_get_data "$msg")

	echo "Received message: $data"
fi

Because ipc uses the file system for message exchange, you can send messages even if your receiver is not running (assuming you’re using named endpoints). This means you won’t lose messages if a script crashes, and you can handle outages very gracefully. And since named endpoints can be shared between processes, you get built-in load-balancing on top of any point-to-point or pub-sub communication. Pretty awesome, isn’t it?

Ready to give it a try? Installation using apt

If you are using a Debian-based distribution, you can install toolbox through apt. First, import the GPG key used to sign packages in the repository and make sure you have apt-transport-https installed.

# wget -O - -- https://deb.m10k.eu/deb.m10k.eu.gpg.key | apt-key add -
# apt-get install apt-transport-https

Then add the following line to your /etc/apt/sources.lst.

deb https://deb.m10k.eu stable main

If you prefer a more recent (and maybe slightly more unstable version), use the unstable suite instead.

deb https://deb.m10k.eu unstable main

Next, update your package index using the following command.

# apt-get update

Now you can install and update toolbox as you’re used to.

# apt-get install toolbox

The packages in the repository are automatically built from the stable and unstable branches, so the Debian packages are usually no more than a few minutes older than the sources. The build system that makes this possible was the proof-of-concept for distributed systems in Bash with toolbox/ipc. The code will be released once the documentation is done. :)

Installation from the sources

If you would prefer to install toolbox from the sources, run the following commands.

$ git clone https://github.com/m10k/toolbox
$ cd toolbox
$ sudo make install

For the latest development version, switch to the unstable branch using git checkout unstable or specify the branch with -b when cloning the repository.