The Grumpy Troll

Ramblings of a grumpy troll.

Pangolin update

One of this troll's two "homes" on the network is a Xen VM from prgmr, to provide a West Coast USA presence to augment my (FreeBSD) colo box in Amsterdam NL. It was running Ubuntu Oneiric (11.10) and is now running Ubuntu Pangolin (12.04). This mostly went smoothly.
The two exceptions: certificates/subversion and bind.


I had set up my system to just check out of svn the same layout used on my FreeBSD machine, which has a set of certificates under my control, some adjusted c_* scripts (mostly c_rehash which generates hash symlinks for both OpenSSL pre-1 and 1, with the two different hash schemes used, so applications using either system can still find the certs) and a Makefile, so that updates are just a "make".

I know of the update-ca-certificates system used on Debian (and thus Ubuntu) and have used it in the past, putting my own certs into my own directory, but I've opted to nuke certain CA certs from the defaults and prefer to use my own definitions.

Of course the upgrade stomped all over this. It did leave my certs alone, but svn is (now?) using GnuTLS instead of OpenSSL, ssl-authority-files in ~/.subversion/servers is ignored and the default certificate location is undocumented. strace(1) to the rescue, it's ‘/etc/ssl/certs/ca-certificates.crt’. Appended ‘globnixCA3.pem’ to that, svn could now trust my server and I did not have to compromise identity verification to run ‘svn up’ and get my certs back. I did, however, end up deciding to blow away all symlinks in /etc/ssl/certs to get rid of the symbolic links created by the Debian tools, so that svn would stop complaining:
svn: Error converting entry in directory 'certs' to UTF-8
svn: Can't convert string from native encoding to 'UTF-8':

There were too many UTF-8 filenames unloved by svn; I assume that I need to configure svn somehow to know that the local filesystem is in UTF-8, but declined to chase down yet another rat-hole at this time.
After an svn update and a make I had everything working once more; I replaced the default ca-certificates file with a symlink to my main generated one and will update my Makefile to do so.


With that in hand, I could look into why my chroot'd install of bind was not working. Errors:
May 25 05:52:14 barrow named[4784]: initializing DST: openssl failure
May 25 05:52:14 barrow named[4784]: exiting (due to fatal error)
When I set up bind, I couldn't spot where modern Linux was shoving chroots these days and the setup wasn't using a chroot, so I tinkered myself. /var/chroot/bind using another svn checkout and a build script, which gets a similar layout to my primary nameserver's setup, just slightly different configuration so as to secondary zones from the primary.

This: {% codeblock lang:console %} $ strace -o fred -ff -v -s 512 named -t /var/chroot/bind -u bind -f -g -d 9 -E "" {% endcodeblock %} got me a bunch of files; an fgrep for DST over them identified the path being used for finding the OpenSSL engine libraries. Also, some searching via Google revealed that apparmor is now being used and some updates would be needed there.

If it's not already obvious, I keep copious plain-text notes of the install for every OS of my own, and really need to get around to the overkill engineering of setting up puppet for my two boxes. :)

I edited /etc/apparmor.d/usr.sbin.named to gain the rules:
{% codeblock %} # phil /var/chroot/bind/** r, /var/chroot/bind/var/** rw, /var/chroot/bind/dev/null rw, /var/chroot/bind/etc/bind/secondary/** rw, {% endcodeblock %} and ran: {% codeblock lang:console %} # mkdir -p /var/chroot/bind/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines # mount --bind /usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines \ /var/chroot/bind/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines # mount -o remount,ro /var/chroot/bind/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines {% endcodeblock %} That worked; some minor tinkering of start-up scripts later, to mount and umount the bind-mount automatically and I have a functioning nameserver once more.

All this, so I can install the default libgnutls28 packages to test GnuTLS 3 on Ubuntu and see if there is anything useful I can adjust in Exim's src/EDITME suggestion for Local/Makefile to encourage GnuTLS 3 usage.

-The Grumpy Troll


Steven Plunkett
Thanks! I tried to adopt it somewhat from standard init.d scripts with a little artistic license thrown in.
Phil Pennock
My style would be different, but hey, looks good. :)
Steven Plunkett
To pay back the knowledge submitted here, I've written a shell script for the mount/unmount of this special directory. Phil, if you have something drastically better let me know. Otherwise it's for all to share.


Steven Plunkett
Fantastic, thank you very much. This solved my problem exactly.
Thanks for your reply ;-)

But, not so simple :

# mount -o remount,ro /var/bind9/chroot/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines
mount: /var/bind9/chroot/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines is busy

# lsof +D /var/bind9/chroot/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines

So, lsof doesn't find any file…

Ok, no time this afternoon, but I will search this evening.

Good afternoon ;-)
Phil Pennock
lsof(8) is your friend. "LiSt Open Files". If you run { lsof +D /var/bind9/chroot/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines } then you should see what's using it.

The use of a bind mount confuses lsof, so by default it shows more than it should. +D will work around that.
Thanks for this very usefull page !
But I have a little problem.
Using Ubuntu 12.10

mount is ok,
but remount ro always give an error "… is busy" :
(I change the folder name)

# mount –bind /usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines /var/bind9/chroot/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines
# mount -o remount,ro /var/bind9/chroot/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines
mount: /var/bind9/chroot/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines is busy

How can I find why the "remount,ro" is busy ?

Phil Pennock
Anonymous, where does that "mkdir -p" in the middle of the second command, the mount command, come from?

The second command comes in 4 whitespace-separated parts; the "mount" command, the "–bind" argument, the original existing path with content, and finally the path where that content needs to also now appear.

So "mkdir -p" is only in the first command, to make sure that last path exists.

And the third command just tinkers with how the mount-point is set up.
root@server01:/#mkdir -p /var/lib/named/etc/bind/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines

root@server01:/# mount –bind /usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines <br />> mkdir -p /var/lib/named/etc/bind/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines
mount: argument to -p or –pass-fd must be a number

US 12.04
why does not work?

I won't comment on Ubuntu vs Debian as the right distribution for servers :-)

I am setting up a Ubuntu (12.04) server to run BIND9 and I encountered this problem. Based on your indications, I get it working but I also had to add to AppArmor:
/usr/lib/** rm

I am new to AppArmor but I understand from the manual (yes I partially did RTFM :-) it is the correct way to solve the issue as reported in the AppArmor log (no right to use mmap on one of the libraries).

Hope this is relevant and can help other persons

Phil P
Almost a coin-toss. I want to be able to test in the environment which more people are using, for tracking down issues. Debian users tend to be clueful enough to solve problems readily. Ubuntu has a broader user-base.
Why do you run Ubuntu on a server?

Genuine question - it seems to be more and more pushed towards desktop and single user only, and I don't see any reason to favour it over, say, Debian, if you're primarily talking to it over the network. So what's the thing I'm missing?

Categories: exim bind GnuTLS Ubuntu svn certificates