Distributed compilation with distcc on Arch

c linux arch distcc

2019-06-13

Sometimes, when I'm working, I'll prefer to sit on the couch with my laptop, which is a not-very-powerful i5/8GB Thinkpad. Sometimes I want to install some packages from source, or compile some software on my laptop. It's far from ideal for this job, mainly because it takes some time, and the thermal changes in the laptop make it kind of uncomfortable to actually perch it on my lap.

I also have a very powerful desktop machine on my network (i9/64GB), and another slightly less powerful NAS (i5/8BG). Naturally, I'd like to delegate as much processing to these machines as possible when compiling. Fortunately, distcc makes this nearly trivial.

Before we begin, it's worth noting that my network is ipv4 only and everything lives on 192.168.1.X. All the machines involved run Arch.

First, on all your hosts, install distcc.

$ sudo pacman -S distcc # on your local machine
$ ssh desktop.local sudo pacman -S distcc # repeat for each remote machine.

Now, on each machine in your cluster, amend the distccd configuration file to allow connections from your network. You should end up with something like this:

$ cat /etc/conf.d/distccd
#
# Parameters to be passed to distccd
#
# You must explicitly add IPs (or subnets) that are allowed to connect,
# using the --allow switch.  See the distccd manpage for more info.
#
DISTCC_ARGS="--allow 192.168.1.0/24 --log-level error --log-file /tmp/distccd.log"

Finally, on each machine, enable the service and start it:

$ sudo systemctl enable --now distccd # local
$ ssh desktop.local sudo systemctl enable --now distccd # repeat for remotes.

Now your cluster is ready, but you need to modify your /etc/makepkg.conf to tell makepkg to use the cluster. First, unbang the distcc in BUILDENV:

BUILDENV=(distcc color !ccache check !sign)

Then, enumerate your hosts, with the number of cores you wish to make available on each:

DISTCC_HOSTS="192.168.1.100/10 192.168.1.101/4"

Finally, change your MAKEFLAGS to use your total number of cores. In this care, I've for 10 cores on .100, 4 cores on .101, and 4 cores on the local laptop for a total of 18:

MAKEFLAGS="-j18"

That's it. When you compile anything with makepkg, it'll spread the compilation load around your hosts. If you want to check on the status of a compilation job, you can run the useful distccmon-text to get streaming updates from the job distributor.

$ distccmon-text 2 # change this number of faster/slower updates