The Grumpy Troll

Ramblings of a grumpy troll.

Four miscellaneous things

Four small things, none on their own worthy of a blog post; the first three are debugging notes from the past week or so and the last is … stunned admiration for PR skill.

First up: FreeBSD Jails and nullfs and ZFS

ZFS is very handy in FreeBSD 10, where you can now boot from ZFS. Note though that zfs maintains its own internal mapping of where names should be mounted, used via zfs mount -a in /etc/rc.d/zfs. This is surprisingly useful, when you start creating extra volumes which can be independently snapshotted. For some service jails, I have the base jail OS image but also a service area which holds the service data which can be maintained independently of the OS and if desired just rolled over to be used with a new userland install in a jail.

When you start sharing some areas, perhaps for administrative reasons or perhaps to give user accounts in one jail and web access in another, the easy solution then is to use a nullfs mount in /etc/fstab, which Linux folks should just think of as a bind mount. There’s a tiny problem with this though.

# cd /etc/rc.d
# rcorder *
[...]
mountcritlocal
zfs
[...]
mountcritremote
[...]
SERVERS
[...]

What this means is that long before daemons/servers start, the ordering is to mount local filesystems, then mount via ZFS, then mount remote filesystems.

The impact is that by default, upon reboot, you will drop to single user mode because mounting can not complete successfully because the nullfs mounts fail because the source of their filesystem data is not yet mounted into the system.

The workaround is easy: in /etc/rc.conf set extra_netfs_types="nullfs" – this is a mechanism used in the mount scripts to tell them which other filesystems, not already known to them, are “remote”. By declaring nullfs to be in the same camp as NFS, the ordering problem is resolved, while everything wanted is still fixed long before jail is started (because jail requires LOGIN which requires DAEMON which requires SERVERS which requires mountcritremote).

If you were to be using nullfs to provision parts of the filesystem required for networking to start, then (1) this would break and (2) you would be crazy.

Simple.


Next up: bazaar and Ubuntu and Apache

% bzr branch lp:ubuntu/trusty/apache2
[...]
bzr: ERROR: Revision {martin@piware.de-20120716080223-qxn97446e5mjot2f}
 not present in "Graph(StackedParentsProvider(bzrlib.repository.
 _LazyListJoin(([CachingParentsProvider(None)], []))))".

That error message (line-breaks inserted) means that you can not branch (“git clone”) the repository which holds the packaging information for Apache2 for Ubuntu. Unless:

echo 'launchpad.packaging_verbosity = off' >> ~/.bazaar/bazaar.conf

Yes, that’s right: a message verbosity control affects whether or not bazaar can run to completion. Simply by telling it to be less verbose, it doesn’t encounter an error which causes the cloning to abort.

Classy.


Next up: Synology and two-factor authentication and tracking

After this troll’s home storage server (or NAS), a Synology 413j, was upgraded to version 5+ of the vendors OS image (called “DSM”), it suddenly started sending out emails as configured. I vaguely recall an inability to handle authentication or TLS correctly, before. In any case, upon availability of new OS images, I now receive a helpful email (together with weekly reports and other nice things).

After the upgrade to version 5, I had taken advantage of a new feature of the system: two factor authentication support via TOTP: ie, get a code, scan it into an Authenticator app on your phone, get regularly changing passcodes. It all had seemed to work, but most of my access to the system is via SSH or NFS. When I went to upgrade, after entering the usercode and password, I was being told: “Wrong verification code. Please try again.

It wasn’t clear whether or not this was coming from two-factor, but I did the usual checks: was time correct? (No, 7 seconds fast, so much for using NTP, folks) But no joy. Finally, in frustration, I cleared my browser cookies.

Success!

Wait, really? Yes. The Synology login system is integrating two-factor in via cookies which can get into a state where it won’t actually prompt you for the two-factor auth, just immediately tell you that the code was wrong.

So I installed the DSM update (Linux kernel security fix) and went back in for a double-check. Failure. Same problem. Clear cookies for host: failure still. Clear cookies for IP address, which adds a Google Analytics _ga cookie: success.

Wait, really? Yes. To be able to sign into the NAS with two-factor, I need to be prepared to clear cookies for hostname and IP, including a Google tracking cookie which really should not be on a home appliance.

Why the hell is my access to my own device inside my own home being tracked in a way which sends metrics to Google?

For now, I am leaving two-factor on as the occasional inconvenience is worth the added security around the fact that I need passwords left enabled for some administrative tasks. But this single factor and the sudden loss of faith that a vendor of home appliances even understands the privacy implications of their actions mean that my next home storage solution will not be a Synology device, no matter how convenient they otherwise are.


Finally, today I have been taught a new lesson in the fine art of PR management around systems services and availability.

Last night, I ended up having to suspend monitoring of one of my employer’s services because it became clear that Amazon were having network problems reaching eu-west-1 (in Ireland). The world according to http://status.aws.amazon.com/ was just perfectly fine. Nothing unusual in vendor service status pages there.

In checking the aftermath today, I went to the status page and was unsurprised to see that the world was still full of green checkboxes, until I looked more closely. Imagine looking at a page with multiple tabs, each with columns for the past week, and dozens of rows. Then, when you really start to scan carefully:

screen capture 2014-06-07

See that tiny blue square box with an italic i in it? The way it weights in against the green check? The way that a cynic might suggest would result in people only discovering the problem existed if they knew to go looking for it?

7:05 PM PDT We are investigating connectivity issues in the EU-WEST-1 Region.

7:28 PM PDT Between 6:36 PM and 7:09 PM PDT we experienced connectivity issues to the Internet for some instances in the EU-WEST-1 Region. The issue has been resolved and the service is operating normally.

No amber status or red status for Amazon. No, network problems which result in the service being unreachable, with traceroutes dying inside Amazon’s network, are apparently all green checkboxy, with just a minor informative overlay.

I bow to the PR masters.

-The Grumpy Troll


Categories: FreeBSD mount jails nullfs ZFS hacks bazaar synology authentication 2factor Amazon status history