Debian Mini Buildd

The problem: I am maintainer of several pieces of software, especially libgeier and taxbird, and I'm regularly asked for Debian packages for either older Debian releases like etch or for Ubuntu. Unfortunately the current distribution intrepid ships an outdated version of Taxbird. Since regularly build the packages for etch, lenny, hard, intrepid and jaunty for i386 as well as amd64 is painful, I finally decided to setup some semi-automatic build environment, i.e. a Debian buildd.

After googling I decided to go for rebuildd, which turned out to be powerful and lightweight. I have two machines acting as the build hosts:

• wanderlust.metafnord.de is the i386 build host
• waldlichtung.metafnord.de is the amd64 box

First of all you need to install the Debian package rebuildd on all the involved machines.

Creating the database

Rebuildd needs a database where it stores the list of packages and jobs it knows of. By default a SQLite database is used, which however is problematic if you use several build hosts. Since rebuildd uses Python's SQL abstraction you can easily use a MySQL database, therefore just create a user and a database on some TCP-accessible MySQL server (reachable by all the involved build hosts).

Configuring rebuildd

Here is my /etc/rebuildd/rebuilddrc from waldlichtung. On the other host there is a similiar one, i.e. just the hostnames replaced :-)

[mail]
mailto = stesie@brokenpipe.de
from = rebuildd@waldlichtung.metafnord.de
subject_prefix = [rebuildd]

[http]
logfile = /var/log/rebuildd/httpd.log
log_lines_nb = 30
ip = 0.0.0.0
cache = 1
templates_dir = /usr/share/rebuildd/templates
port = 9998

[build]
max_jobs = 10
max_threads = 1
check_every = 300
more_archs = any
kill_timeout = 90
source_cmd = apt-get -q --download-only -t ${d} source ${p}=${v}
work_dir = /var/cache/rebuildd/build
post_build_cmd = /usr/local/bin/buildd-update ${d} ${a}
dists = etch lenny sid hardy intrepid jaunty
build_cmd = pbuilder build --buildresult /var/cache/pbuilder/result/${d}/ --basetgz /var/cache/pbuilder/${d}-${a}.tgz ${p}_${v}.dsc
database_uri = mysql://rebuildd:********@spielplatz.metafnord.de/rebuildd
build_more_recent = 1

[telnet]
ip = 127.0.0.1
prompt = rebuildd@waldlichtung ->
motd = Connected on rebuildd on waldlichtung.metafnord.de
port = 9999

[log]
time_format = %d-%m-%Y %H:%M:%S
logs_dir = /var/log/rebuildd/build_logs
mail_failed = 1
mail_successful = 0
file = /var/log/rebuildd/rebuildd.log

• The database_uri-option tells rebuildd to connect to the MySQL-server. The syntax is mysql://username:password@hostname/databasename
• post_build_cmd specifies a shell script that is to be run after a successful build. I use it to refresh the Packages file below /var/cache/pbuilder/result/${d}/, which is accessible through lighttpd (http://waldlichtung.metafnord.de/buildd/ in that case). Furthermore it refreshes the basetgz of pbuilder, i.e. runs apt-get update, which makes freshly build packages available to consecutive buildd runs. That is, if I first build libgeier-dev it's afterwards available for the taxbird build.
• Keep in mind that rebuildd has to be configured on every build host (:mrgreen:)

Here is the content of the /usr/local/lib/buildd-update file:

#! /bin/bash

if [ $# -ne 2 ]; then
  echo Usage:   $0 SUITE ARCH
  exit 1
fi

[ -x /var/cache/pbuilder/result/$1 ] || {
  echo no such suite $1
  exit 1
}

[ -e /var/cache/pbuilder/$1-$2.tgz ] || {
  echo no such suite $1, arch $2 pair ..
  exit 1
}

set -xe
cd /var/cache/pbuilder/result/$1/
dpkg-scanpackages  . > Packages
gzip -c Packages > Packages.gz

pbuilder --update --basetgz /var/cache/pbuilder/$1-$2.tgz

Initializing the database

To initialize the database once, you have to run

rebuildd init

from one of the boxes.

Creating the basetgz-images for pbuilder

Last but not least the tedious part: You have to create basetgz-images for pbuilder, one for every distribution and architecture you want to build packages for. Luckily there is the rebuildd-init-build-system which makes life easier, but there's still a lot of work left :-)

First let's create the basetgz-images for building for Debian systems. Therefore add something like the following to /etc/default/rebuildd:

PBUILDER_MIRROR=http://ftp.de.debian.org/debian
DISTS="etch lenny sid"

and run rebuildd-init-build-system.

If that worked, let's do this for Ubuntu. Remove the debian lines just added to /etc/default/rebuildd and use something like

PBUILDER_MIRROR=http://www.ftp.uni-erlangen.de/pub/mirrors/ubuntu/
DISTS="hardy intrepid jaunty"

and run rebuildd-init-build-system again.

If you just want to rebuild packages that are in the official feeds you don't need to add individual lines to /etc/apt/sources.list. If you want to, you need to make pbuilder's result directory http-accessible, log into each and every basetgz, using

pbuilder --login --basetgz /var/cache/pbuilder/etch-i386.tgz --save-after-login

, modify /etc/apt/sources.list accordingly and run apt-get update.

Using it

Now that you're done with the initialization time has come to enjoy it (:mrgreen:)

Just enter something like

root@waldlichtung:~# rebuildd-job add
taxbird 0.12-1 low etch i386
libgeier 0.10-1 high jaunty amd64
taxbird 0.12-1 low jaunty amd64

and rebuildd should start working :-)

In order to see what's currently going on, you can use rebuildd-job list:

root@waldlichtung:~# rebuildd-job list
  id  |     package name     |    version     |    status    |  host  | dist  |  arch  
------+----------------------+----------------+--------------+--------+-------+--------
   11 |   kopete-silc-plugin |          0.5-1 |     BUILD_OK | wander |   sid |   i386
   12 |             libgeier |         0.10-1 |     BUILD_OK | wander | lenny |   i386
   13 |              taxbird |         0.12-1 |     BUILD_OK | wander | lenny |   i386
   14 |             libgeier |         0.10-1 |     BUILD_OK | wander |  etch |   i386
   15 |             libgeier |         0.10-1 |     BUILD_OK | waldli | jaunt |  amd64
   16 |              taxbird |         0.12-1 |     BUILDING | waldli | jaunt |  amd64
   17 |              taxbird |         0.12-1 |     BUILD_OK | wander |  etch |   i386