Foundry is a distributed build system that automatically builds Debian packages from git repositories and distributes built packages through apt repositories. I developed foundry to automate the distribution of toolbox and other packages published through deb.m10k.eu.
The build system consists of four workers, called bots, each of which performs one small step of the build process. The bots use pub-sub messaging for communication, reducing coupling between bots and increasing scalability.
|watchbot||Watch git repositories for changes|
|buildbot||Build Debian packages from a git repository|
|signbot||Sign Debian packages|
|distbot||Add signed packages to a Debian repository|
The pub-sub topics and their meaning is shown in the following table.
|commits||commit||A change was detected on a git repository|
|builds||build||A build has completed successfully|
|signs||sign||The packages from a build message have been successfully signed|
|dists||dist||The packages from a sign message have been successfully published|
The following diagram gives an overview how the individual foundry components interact. For understandability’s sake each bot is shown only once, but the foundry architecture does not impose a limit on the number of bots. The messaging system that foundry is based on (toolbox’s ipc module) performs automatic load-balancing, allowing bots to be added or removed on demand.
The architecture is explained in more detail in the sources.
Installation using apt
If you are using a Debian-based distribution, you can install foundry through apt.
First, import the GPG key used to sign packages in the repository and make sure you
# wget -O - -- https://deb.m10k.eu/deb.m10k.eu.gpg.key | apt-key add - # apt-get install apt-transport-https
Then add the following line to your
deb https://deb.m10k.eu stable main
If you prefer to use a development build, use the
unstable suite instead.
deb https://deb.m10k.eu unstable main
Next, update your package index using the following command.
# apt-get update
Now you can install and update foundry with apt.
# apt-get install foundry
Installation from the sources
If you would prefer to install foundry from the sources, first install toolbox, then run the following commands.
$ git clone https://github.com/m10k/foundry $ cd foundry $ sudo make install
For the latest development version, switch to the unstable branch using
git checkout unstable or specify the branch with
-b when cloning the repository.
Finally, create the foundry group and a directory for foundry to keep build contexts.
# groupadd foundry # mkdir -p /var/lib/foundry/contexts # chown -R root.foundry /var/lib/foundry/contexts # chmod -R 770 /var/lib/foundry/contexts
The user that runs foundry has to be member of the toolbox, toolbox_ipc, and foundry groups. You can add a user to these groups using the following command.
# usermod -a -G toolbox,toolbox_ipc,foundry USER
The user further needs to have GPG keys for IPC messaging, package signing, and repository metadata signing. Ideally you should create three keypairs, the default key being used for IPC messaging. Keys can be generated using the following command.
$ gpg --full-generate-key
The following example shows how to build toolbox with foundry.
First, we will start a watchbot to watch the toolbox repository’s stable branch. This can be achieved with the following command.
$ watchbot --repository https://github.com/m10k/toolbox#stable
A single watchbot can watch an arbitrary number of repositories. If you want to watch more
than one repository, pass
--repository multiple times, as shown below.
$ watchbot --repository https://github.com/m10k/toolbox#stable \ --repository https://github.com/m10k/toolbox#unstable
Watchbot supports watching of remote repositories (via git’s dumb and smart HTTP transports) as well as local repositories.
This one is really simple.
By default, buildbot will use the IPC endpoint
pub/buildbot to subscribe to commits.
This means, by default build jobs will be balanced over all buildbots. If you would prefer
multiple buildbots to build the same job in parallel, for example because you are
building for multiple architectures, you need to pass the
--endpoint parameter. You can
think of the IPC endpoint as a load-balancing group. For example, consider you have started
four buildbots as shown below.
$ buildbot --endpoint pub/buildbot_i386 $ buildbot --endpoint pub/buildbot_i386 $ buildbot --endpoint pub/buildbot_amd64 $ buildbot --endpoint pub/buildbot_amd64
In this scenario, if a new commit is published, it is sent to either IPC endpoint. This means, one of the buildbots listening on pub/buildbot_i386 and one of the buildbots listening on pub/buildbot_amd64 will see the message.
Signbots are started as shown below.
$ signbot --gpg-key <keyid>
The purpose of signbot is to sign Debian packages, so you need to tell it which key to
use using the
--gpg-key option. The key must be a key-id from the default GPG key ring.
By default, signbots are load-balanced the same way as the buildbots. If you want to sign
packages going to one repository with a different key than packages going to a different
repository, you will have to start two signbots with differing
Distbot has a few more options, but most of them are fairly self-explanatory.
$ distbot --name deb.example.org \ --output /srv/www/deb \ --arch amd64 \ --arch i386 \ --gpg-key <keyid> \ --description "My repository"
Like signbot, distbot needs a GPG key id. Unlike the signbot key, which is used to sign
packages, this key is used to sign the metadata in the repository. You are strongly
encouraged not to use the same key for both. If the path passed via
contains an existing repository, distbot will reuse it. Otherwise, it will create a new
repository in that path.
Note: If you have multiple signbots publishing messages to more than one topic, you
can change the topic that a distbot watches by passing
--watch and the name of the
topic that your signbot is publishing messages on.
Herding the bots
If you want to see a list of the running bots, pass
--list to the bot you’re interested
in. For example, the following command will list all running watchbots.
$ watchbot --list
If you want to stop a particular bot, use
$ watchbot --stop 12345
That’s all you need to know to get started. If everything went well, you should be seeing
built packages in your Debian repository. If not, the logs within
might tell you what’s going on. This is also a good opportunity to mention that you can
tell each of the bots to be more talkative by passing
--verbose one or more times.