Toolbox

I spend a lot of time with shell scripts, which can be a veritable pain in the butt because in shell you can’t just include "some_module" and start using functions from it. After writing what felt like my hundredth command line parser, I asked myself, “would it be possible to write my own include function?”.

Turns out it is! Say hello to toolbox, my module framework for Bash. To use it, just source toolbox.sh and load any module you need.

. toolbox.sh
include "ipc"

If you are in the mood for some logging, all you have to do is include "log" and you can use functions such as log_error(), log_stacktrace() or log_highlight() to log to your heart’s content.

Maybe now you want to accept parameters on the command line? No problem! include "opt" and you have a flexible command line parser for every occasion:

Say your script has a required argument -i (or --ingredient) that is followed by either “salt”, “pepper”, or “cumin”. What’s more, you want to call add_ingredient() whenever this argument was passed. How many lines do you think this takes? Toolbox does it in two (well, sort of).

include "opt"

# Adds "-i" / "--ingredient" option, (r)equired,
# followed by a (v)alue, no default value
opt_add_arg "i"                      \
            "ingredient"             \
            "rv"                     \
            ""                       \
            "Ingredient to be added" \
            '^(salt|pepper|cumin)$'  \
            add_ingredient

if ! opt_parse "$@"; then
	# Invalid or unrecognized option, invalid
	# value, or the callback returned an error
fi

ingredient=$(opt_get "ingredient")

But there’s more! Maybe you need a daemon really really quick. This is how you do it with the inst module.

include "inst"

_daemon_main() {
	while inst_running; do
		sleep 5
	done

	return 0
}

inst_start _daemon_main

You want the daemon to be a singleton? Swap inst_start for inst_singleton - done!

But that’s still not everything. I bet you never even thought about writing a distributed system in Bash, did you? The ipc module lets you do just that, and it’s not even hard (assuming the machines are sharing an NFS mount they can commuicate over). Create an IPC endpoint with ipc_endpoint_open(), and you can use it to send and receive messages in a point-to-point fashion using ipc_endpoint_send() and ipc_endpoint_recv().

# create a named endpoint
receiver=$(ipc_endpoint_open "my_endpoint")

# create an anonymous endpoint
sender=$(ipc_endpoint_open)

# Send on sender endpoint, receive on receiver endpoint
ipc_endpoint_send "$sender" "my_endpoint" "Hello world"
msg=$(ipc_endpoint_recv "$receiver")

# Get the source address and send a response
source=$(ipc_msg_get_source "$msg")
ipc_endpoint_send "$receiver" "$source" "Hello to you, too!"

If you’d prefer communicating in a pub-sub fashion, ipc won’t let you down. The following is a minimal publisher.

endp=$(ipc_endpoint_open)
ipc_endpoint_publish "$endp" "mytopic" "Hello world"

And this is the corresponding subscriber.

endp=$(ipc_endpoint_open)
ipc_endpoint_subscribe "$endp" "mytopic"

if msg=$(ipc_endpoint_recv "$endp"); then
	data=$(ipc_msg_get_data "$msg")

	echo "Received message: $data"
fi

Because ipc uses the file system for message exchange, you can send messages even if your receiver is not running (assuming you’re using named endpoints). This means you won’t lose messages if a script crashes, and you can handle outages very gracefully. Pretty awesome, isn’t it?

Ready to give it a try? Installation using apt

If you are using a Debian-based distribution, you can install toolbox through apt. First, import the GPG key used to sign packages in the repository and make sure you have apt-transport-https installed.

# wget -O - -- https://deb.m10k.eu/deb.m10k.eu.gpg.key | apt-key add -
# apt-get install apt-transport-https

Then add the following line to your /etc/apt/sources.lst and run apt-get update.

deb https://deb.m10k.eu stable main

Now you can install toolbox using the following command.

# apt-get install toolbox

If you prefer a more recent (and maybe slightly more unstable version), use this repository instead.

deb https://deb.m10k.eu unstable main

The stable and unstable repositories are built automatically from the master and unstable branches, so the debian packages are usually no more than a few minutes older than the sources. The build system that makes this possible was the proof-of-concept for distributed systems with the ipc module!

Installation from the sources

If you would prefer to install toolbox from the sources, run the following commands.

$ git clone https://git.corax.cc/toolbox
$ cd toolbox
$ sudo make install

For the latest development version, switch to the unstable branch using git checkout unstable or specify the branch when cloning the repository.

$ git clone http://git.corax.cc/toolbox -b unstable