Reclaiming Blurb

Reclaiming the term “troll” from the idiots. Abusers and bullies not welcome.

2013-04-27

MySQL, SSL/TLS and Ubuntu

Some notes, from having set up a MySQL server on Ubuntu and worked to make sure it offered SSL (TLS) for the connections. In this case, Ubuntu 12.04.2 LTS running inside a VMWare Fusion 5 virtual machine. (I write TLS for the generic protocol support, SSL as it pertains to MySQL in particular because that's the term the MySQL documentation uses). I installed MySQL 5.5.

This is purely for testing purposes, so there was no MySQL performance tuning done for the VM; the only relevant tuning was existing tuning, just making sure that the Linux kernel did not try and treat the virtual disk, provided from a backing store as a file in the outer filesystem, as a spinning disk but instead as just a dumb backend (even more appropriate since the laptop has an SSD):
# /etc/rc.local
echo 0 > /sys/block/sda/queue/rotational
echo deadline > /sys/block/sda/queue/scheduler


Getting MySQL installed on Ubuntu is easy, thanks to the Debian heritage:
$ sudo apt-get install mysql-server mysql-client
[ supply a MySQL root password when prompted ]


Getting MySQL using SSL was far from easy. I assume for this post that the reader can create an RSA private key and an X.509v3 certificate and install them to be accessible to the MySQL runtime user. The problem is the pathetically dire diagnostics from MySQL, the lack of clarity over OpenSSL vs yaSSL as an SSL provider (and the impact on the ssl-cipher control statement) and the settings which used to work, are documented to work, but which cause SSL to not work anymore when set. I'd last set up MySQL with SSL back in the 2005/2006 timeframe, on FreeBSD, and still had my install notes, so am confident in my assertion of what used to work.

Ubuntu uses “/etc/mysql/my.cnf” for global MySQL configuration.

First gotcha: the need to remember that AppArmor is filtering access to files by MySQL. So to install a certificate, there's an apparently undocumented requirement to install the key and cert in a particular location with particular names: they must match /etc/mysql/*.pem. Sod that. If you have TLS set up for various services, with keys and certs sensibly managed instead of scattered about, and want to actually enable certificate verification, then your first step is to edit “/etc/apparmor.d/usr.sbin.mysqld” to grant read-access as appropriate. Note: most configuration options for MySQL's SSL support do not result in meaningful error messages when access to a file is blocked. It took setting ssl-ca to get a hint appear in “/var/log/mysql/error.log”.

Useful points to bear in mind: use \s in the mysql client to check connection status and use:
mysql> show variables like '%ssl%';
to get variables and settings. Note that a value of DISABLED means that the server has been built with support for SSL but the run-time configuration has not yet been sufficiently blindly poked with goat-blood infused wands.

Put this into “/etc/mysql/my.cnf”:
[client]
ssl
ssl-ca = /path/to/certauthority.pem
#tempdisable# ssl-verify-server-cert

[mysqld]
ssl-ca=/path/to/certauthority.pem
ssl-cert=/path/to/services/mysql/server.crt
ssl-key=/path/to/services/mysql/server.key
ssl-cipher=DHE-RSA-AES256-SHA


In particular: use of [mysqld]ssl-capath is not only insufficient, its use appears to break SSL setup. Use ssl-ca instead. You must set [mysqld]ssl-cipher these days, there's no reasonable default. The accepted values depend upon the backend and, even when OpenSSL is in use, appears to not accept cipher specs per the ciphers(1) manual page, as accepted by every other user of OpenSSL. Instead, you have to manually provide the specific ciphers allowed. I repeatedly attempted to use something as simple as ALL or DEFAULT in disbelief that these could have been disabled. Failures here will result in an error message of “SSL error: SSL_CTX_new failed” and no further information.

Note that by default, MySQL does not verify the server certificate identity, as hinted at by the “[client]” section above. Alas, there is also the problem that the client code appears to ignore the X.509v3 subjectAltNames extension, so the only identity from the certificate used for checking is that of the CN field in the certificate Subject.

Further, the default rules only allow the MySQL root user to connect from localhost, even after granting access to other IPs. So enabling certificate validation means that you need a certificate with CN=localhost in the Subject.

So first leave [client]ssl-verify-server-cert commented out. Then run this, where you should replace foobar.local with the hostname corresponding to the public IP of the machine; this allows you to connect from the local machine to the local machine, via the machine's hostname.

$ mysql -u root -p
[ supply the root password you used at install time ]
mysql> GRANT USAGE ON *.* TO ''@'foobar.local';
mysql> FLUSH PRIVILEGES;


After this, you can uncomment the ssl-verify-server-cert in the client section, provided that you remember that this breaks localhost access and root can still only connect from localhost, so you'll have to comment it out for major administrative tasks.

If you enable validation, a connection to localhost will yield:
ERROR 2026 (HY000): SSL connection error: SSL certificate validation failure

So use “mysql -h foobar.local” thereafter.

After all this, I can't believe that folks still perpetuate the myth that PostgreSQL is harder to set up than MySQL. PostgreSQL actually documents the settings accurately (instead of telling you about directives that apparently have broken configuration for several releases now but none of the maintainers appear to care, ssl-capath), provides reasonable diagnostics and lets you get up and running with TLS enabled connections in a matter of minutes, the only potential issue being how you configure pg_hba.conf. I've lost interest in making sure that MySQL can use multiple ciphers for compatibility with various clients, as is default behaviour in every other application.

You should now be able to do something like:
mysql> GRANT ALL PRIVILEGES ON foo.*
  TO 'fooadmin'@'adminhost.tld'
  IDENTIFIED BY 'raw_password_here'
  REQUIRE SSL;


-A grumpy troll

2013-04-19

Administrivia: comment system & Google (Plus)

Administrivia: I had the new Blogger/Google+ integrated comment system turned on for about 24 hours but have now reverted.

My policy is that I want the minimum barriers to commenting consistent with limiting spam. For a while, I was open commenting relying upon Google's excellent spam detection systems and cleaning up the little that slipped past. I do not want to require that folks submit their data into a particular fiefdom to be able to talk with me. Echo chambers are bad, even if the chambers are spectacularly large.

The non-Google+ blogger commenting system allows for federated identity systems. For the sufficiently paranoid, a federation member which has only one participant works. Federation works well as an organising structure on the Internet: email, XMPP, more, all uses it, leveraging DNS as the inter-member addressing system and identifier.

Alas, the Google+-integrated comments system supports showing old non-Google+ comments mixed in with the Google+ ones, but if there's a way to continue to allow non-Google+ commenting after the switch, neither myself nor a friend can spot it. Thus the carefully limited claims made by Google about the system: they don't actually say that you can continue accepting comments as before and their phrasing suggests that it's just a better experience for Google+ users. I had to read the description twice though to confirm that they were leaving a lot unsaid, then embarked on a 24-hour test to confirm that I wasn't reading too much into what was missing, but that instead Google have become really good at PR-deception press pieces that misdirect.

This is disappointing. Google used to win based on the best engineering. I may finally be kicked into just finding a new blog platform to escape an environment where the writers are treated as potential targets for manipulation. This may also be the wake up call that I should use Google+ less: this incident just does not bode well for the future.

-A dejected troll

2013-03-30

ISP liability and BCP 38

A couple of days ago, I came up with a thought about the ideal way to get ISPs to actually deploy BCP 38 (aka "don't let out traffic with source addresses that are spoofed to be from elsewhere").

The good news is that it's appropriately evil enough that folks I mentioned it to last night really appreciated it.

The bad news is that it involves legislation, and it impacts folks with deep pockets, so will never get passed without being corrupted to, at the very least, have severe side-effects, and more likely accomplish the opposite of that which was intended. Note that this is a consequence of the flawed legislative system in place, not a consequence of the proposal.

The Proposal: ISP carrier liability immunity is only applicable to traffic which is entirely within their network, or entering their network from outside, or leaving their network with a source address, as directly visible to their routers, which is allocated to them or for which appropriate contracts are in place.

You leak traffic with spoofed addresses? Well, you have no immunity for anything that traffic is doing.

Guess what? If you implement BCP 38 filtering, you're back to being safe.

VPNs and tunnels? You're only responsible for the outer layer.

Multi-homed customers? That's fine, you have contracts.

Second Proposal:

Want to make things nicer for operational cleanliness and help reduce the likelihood of more hijacking such as Pakistan messing up Youtube?

Proposal 2: ISP carrier liability immunity for traffic leaving their network is limited to traffic for which appropriate operational routing databases have correct information about the source address blocks.

-The Grumpy Troll

2013-03-29

Arms control in civil society

[I wrote this in a Google+ post on December 24th, 2012. I am reposting to my blog, for discoverability.]

In the aftermath of tragedy, people reach out for solutions. Sometimes the obvious approach is very wrong for reasons which are not immediately obvious. When you're upset, taking the time to understand those reasons can be difficult. Part of being adult is taking a deep breath and working to understand them anyway before forcing changes on everyone. Unscrupulous people will tell you the solution is easy. Just ban X, where X is currently firearms.

Reality is not so accommodating

We are entering an era of easy item creation by the unskilled, as manufacturing becomes digitized. 3D printing, with schematics that can be flawlessly copied. That copying can be in public where it can be seen, or "underground" with sneakernets of USB sticks.

If your solution is based on restricting access to something digital then you're going to fail. Badly. Just ask the RIAA.

Restricting access to additive manufacturing will kill the economy and just give criminals a new revenue stream, even as the world seems to be moving towards removing their drugs-based revenue through legalization: you don't need to approve of recreational drug use to accept that prohibition doesn't work. Same will happen for 3D printing if outlawed or heavily restricted, more so since there are so many uses that are positive for society.

Prohibition doesn't work, it just creates more criminals. Regulating access to firearms needs to be approached incredibly carefully, to minimize the population percentage that will grow to sympathize with underground gun printers.

It's not (just) that gun extremists think banning any kind of firearm is wrong. We're a couple of decades too late for the approach to have any chance of working.

So we have to work on the underlying problems of mental health instead. It's not just firearms that can be printed.

I support higher taxes to fund a comprehensive overhaul of mental health care in this country.

I don't see a workable alternative.

From the comments on the original

feedback response #1: I meant to write something on other things that can be made, without scare-mongering, but apparently forgot. Yes, the things which come later might make guns look like an irrelevance. All the more reason to work on the mental health problems now!

feedback response #2: What stands out in many of the most horrifying pre-planned atrocities is how the mental health care systems totally failed in dealing with people to whom attention had already been drawn. I'm not talking about someone who suddenly snaps, nor about launching witch hunts which stigmatize some group, but being able to respond adequately when a parent asks for help, keep better track of folks who had been receiving some care but suddenly stopped, having enough funded professionals that too many people just get given pills (until they're 18, the pills are stopped and folks who'd never learnt to deal adequately with the stress that most people learn to handle during puberty suddenly have to, and snap).

2013-03-07

Household Infrastructure

So, I need to get one of these plug computers up and running, so it can be my monitoring server for my household.

This morning, DNS resolution broke at home. My router, running OpenWRT, is using unbound, so I can get DNSSEC validation. DNSSEC validation broke, on being unable to validate the keys for the root zone, so I could only get DNS service back by disabling DNSSEC.

Turns out, the clock on the router said it was November 27th, 2012. Yet I had ntpd set up. My installation notes record the steps I took to set NTP up. Bonus feature of keeping a logbook of system installation, even if the logbook is just a text file somewhere: you can prove to yourself that if you are barking mad, it's not affecting your recollection of machine setup.

Alas, when I set that up, I hadn't yet set up off-device syslog of system logs, so have no data about when things broke. Since then, I've set up the syslog service on my home NAS (Synology 413j) and directed the router to send logs there, so if time fails again, I should at least be able to dig out logs of the incident (even if the two systems don't quite agree on syslog format and include a little garbage in each entry).

Since my laptop clearly uses Apple's time instead of trusting me to make sure the NTP server issued via DHCP is correct, I clearly need to actually set up home-based monitoring. It doesn't need its own long-term store, just needs to be able to give a web-page and send email. Needs to monitor itself, my router, the available routes, the services on the router, the services on the NAS, and perhaps reachability of a machine or two so I can charts and graphs to back up my experiences with occasional wedged network from my ISP.

And hey, perhaps even run arpwatch, so I can track the appearance of machines on my network.

Sounds like I've a project for, oh, next year sometime. :-/ Heck, my home setup might even start to appear professional ... nope, no centralised configuration management.

2012-09-13

TLS, CRIME, BEAST and you the programmer

Before continuing: I am not a cryptographer; merely someone who, on a good day, can use cryptographic libraries with what might pass for intelligence. I write this post with 20/20 hindsight.

One of the security maxims I hold dear is to not freely mix data from different security contexts. Programmers often see this with “don't mix code and data”, and guidance to prepare parameterised SQL queries and then execute them with untrusted data in parameters, rather than try to sanitise the data into an SQL query. Many website security flaws arise from a failure to follow this maxim.

Both the BEAST attack, and this new CRIME (Compression Ratio Info-leak Made Easy) attack, impressively manage to attack how common use of HTTP within TLS abuses this maxim.

The core problem is that folks combine highly sensitive data, such as authentication cookies or Authorization: HTTP headers with OAuth2 bearer tokens, with a POST payload (or anything with a client payload, PUT, PATCH, etc; [EDIT: actually, this time GET works, the attack can be in the path]) that might contain untrusted data. TLS protects one stream of data assumed to be of a homogenous security level, and is not designed to protect the "cleartext" against itself. So in practice, with browsers with retained authorisation information and web-sites protecting against cross-domain attacks, HTTPS is using HTTP over TLS in a way which TLS was not designed to protect. Nonetheless, TLS does a fair job of it: so much so, that folks are lulled into believing they're safe.

So both BEAST and now CRIME work by seeing how varying a later part of one stream can reveal information about an earlier part of the stream. [EDIT: actually, CRIME can reveal later parts of a stream too, it is bi-directional]

In BEAST's case, the unfortunate way that TLS historically specified choosing the initialisation vector for each block in the CBC mode of using a block cipher to protect a stream led to disclosure. Using a different cipher mode, or using a modern version of TLS, helps with that. Fortunately, the dominant SSL framework (OpenSSL) now has a stable release out which supports said modern versions of TLS, so folks have been busily upgrading to protect themselves and their users. See Wikipedia's “Block cipher modes of operation” for an introduction to modes, CBC, and my favourite demonstration image of why naive block cipher usage (“ECB mode”, aka no mode) is a really bad idea.

In CRIME's case, compression is used to attack TLS. Normally, folks have been advised to use compression to increase the per-octet entropy of the cleartext, which may help make cryptographic attacks harder. This still holds true, if the cleartext is all of one security context. If instead the payload can be modified by an attacker, across repeated requests, then when a pattern in the payload matches a pattern in the sensitive authorisation data, earlier in the stream, then compression will do what it's designed to do, and produce a smaller stream. That change in size is visible after encryption, so someone who can observe the TLS session can adjust their payload and gradually discern more of what the authorisation data must be, without directly decrypting it.

As long as sites use HTTPS with authorisation data inside the cleartext, problems like this will recur. They'll be band-aided, often by reducing the functionality of TLS, but they'll occur. The authorisation data needs to move out of the cleartext and instead be expressed in some other way. Perhaps issuing a short-lived client TLS certificate which will be accepted by the site. This affects cookies, OAuth2 with bearer tokens, and more.

This does not affect stand-alone client applications which do not let their credentials be used for all remote access, including from untrusted sources; it does affect them if they register a URL handler in a client's system and they suddenly start taking untrusted requests.

This does not affect protocols which do not mix authentication or authorisation data with untrusted data, to result in data flows which will not be visible to the user of the application.

CRIME does not affect TLS sessions in which compression is not used. Compression is part of the core protocol, but is optional. Programs written in the Go programming language, for instance, are likely not vulnerable simply because the only compression mechanism supported by crypto/tls is (currently) None. If, though, they use a Go wrapper for an external security library, then this is not relevant and they may still be vulnerable.

Browsers have been disabling compression, which prevents this attack but across many types of network path will result in slower https access.

I suspect that the longer-term work-around for CRIME will be to require the sender of mixed-security-level content to coerce a compression block end after the HTTP headers and before the untrusted payload. If the DOM API permits requests to be sent with arbitrary headers under the attacker's control, then move those headers to the end of the header and insert an extra block switch there. For DEFLATE compression (RFC 1951), the coding tree for each block is independent; if a compression scheme maintains a coding tree across blocks, then they'll also need a reset of the coding tree, since that's what really matters. [EDIT: or rather, move those untrusted headers to the start of the headers, because the attack can be in a GET line, not just in a payload]

Before now, I've not looked too closely at how compression and TLS libraries interact, so this paragraph is based on some quick API and code analysis. For ZLib itself, it appears that Z_FULL_FLUSH will be needed for calls to deflate(). Both OpenSSL and GnuTLS normally call deflate(..., Z_SYNC_FLUSH) to let a compression block span multiple network-layer sends.

For OpenSSL, this full flushing requirement boils down to a call to BIO_flush() between security contexts. Note that BIO_flush() is not guaranteed to end a compression block, but in practice does. Ideally, there would be a way to cork the actual network-layer send across the two or three upper-level flushes, before uncorking to send.

I do not see a way to force a block end in GnuTLS (checking current git master branch).

An earlier analysis of mine into how BEAST affects SMTP and Exim in particular is available in the exim-announce archives. The same principles apply to CRIME.

See also: Thomas Mornin on the Security StackExchange

Regards,
-The Grumpy Troll

2012-09-04

Go error construction loop

One of the really nice things about the Go programming language is that you have the complete source to the compiler and the standard library, as very readable code which can be used to investigate problems.

When merging a feature branch in a git repo I, of course, rebuilt my test frontend server to be sure everything still worked, as a check before committing. So I found that a client call was never returning, while my program was now chewing a whole CPU. (MacOS top(1) will sort by CPU usage; type ‘o’, then “-cpu” and Enter).

The first easiest pass to figure out what's going on does not even involve opening a debugger. Programs written in Go will dump out a stack trace for all Go-routines (on the way to exiting) when the program receives a SIGQUIT. The “stty -a” command will show the terminal settings for sending the signal, but it's normally control-backslash.

In one of the routines, I saw this repeated a lot (sorry for the long lines wrapping in this blog post, I only edited for names, not to re-layout):

fmt.Sprintf(0x24e08c, 0x656d614e00000012, 0xe12a4208, 0x100000001, 0xf87b255d20, ...)
  /usr/local/go/src/pkg/fmt/print.go:228 +0x5d
github.com/$employer/$repo/storage.NameNotFound.Error(0xf848d0afb8, 0x4, 0x1c3530, 0xf87b25b6e0, 0x32692, ...)
  /Users/$grumpytroll/src/go/src/github.com/$employer/$repo/storage/storage.go:60 +0x9f
github.com/$employer/$repo/storage.(*NameNotFound).Error(0xf87b255d00, 0x1eaa78, 0xf87b255d00, 0xf800000073)
  /Users/$grumpytroll/src/go/src/github.com/$employer/$repo/storage/storage.go:0 +0x6d
fmt.(*pp).handleMethods(0xf87b25c840, 0x73, 0x0, 0xf87b250100, 0x0, ...)
  /usr/local/go/src/pkg/fmt/print.go:698 +0x33a
fmt.(*pp).printField(0xf87b25c840, 0x1eaa78, 0xf87b255d00, 0x73, 0x0, ...)
  /usr/local/go/src/pkg/fmt/print.go:737 +0x275
fmt.(*pp).doPrintf(0xf87b25c840, 0x24e08c, 0x12, 0xe12a46b8, 0x100000001, ...)
  /usr/local/go/src/pkg/fmt/print.go:1078 +0x10ae
fmt.Sprintf(0x24e08c, 0x656d614e00000012, 0xe12a46b8, 0x100000001, 0xf87b255d00, ...)
  /usr/local/go/src/pkg/fmt/print.go:228 +0x5d
github.com/$employer/$repo/storage.NameNotFound.Error(0xf848d0afb8, 0x4, 0x1c3530, 0xf87b25b690, 0x32692, ...)
  /Users/$grumpytroll/src/go/src/github.com/$employer/$repo/storage/storage.go:60 +0x9f
github.com/$employer/$repo/storage.(*NameNotFound).Error(0xf87b255ce0, 0x1eaa78, 0xf87b255ce0, 0xf800000073)
  /Users/$grumpytroll/src/go/src/github.com/$employer/$repo/storage/storage.go:0 +0x6d


This is clearly a recursion loop. Note though the standard library errors refer to the source files on disk for the standard library. This meant that I could go read the source for the fmt package and understand why the loop was happening.

src/pkg/fmt/print.go in handleMethods() looks to see if a type matches the Error interface, before it checks if it's a string, and calls Error() to get the value, so an Error() method which calls fmt.Sprintf() with itself as a parameter to a %s expansion item will enter a recursion loop.

In this case the code, which had been merged to master but which my code was exercising, was:


type NameNotFound string
func (n NameNotFound) Error() string {
  return fmt.Sprintf("Name not found: %s", n)
}


The simplest fix for the merge was to use ‘string(n)” instead of “n” for the parameter.

-The Grumpy Troll