Bug #1721: AWS PEM keys cause OpenSSH to complain "Not a RSA1 key file"

Just noticed this nasty little issue here, which interplays with the way Mac OS X caches authentication credentials.

The PEM format keys provided by Amazon don't work as you would expect when passed in to SSH (via ssh -i path/to/key). Running SSH with multiple -v switches to see what's happening you'll try to connect:

debug1: Connection established.
debug3: Not a RSA1 key file path/to/key
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace

Later on in the output you'll see that it goes trying all the keys it knows about (ie. everything in ~/.ssh plus any other key you told it about on the command line with the -i switch) and eventually gets in with the AWS-supplied key:

debug1: Authentications that can continue: publickey
debug1: Offering public key: path/to/some/other/key/which/will/not/work
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: path/to/key
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey
debug2: we sent a publickey packet, wait for reply
debug1: Authentication succeeded (publickey).

The ugly little problem which I just ran into is that I recently added another key file to my ~/.ssh directory, raising the number of keys just high enough that I ran into the MaxAuthTries limit (defaults to 6) before ssh got around to re-trying my AWS PEM key.

This effectively locked me out of the system. Worst of all, moving keys out of ~/.ssh to reduce the number of keys tried before hitting the AWS key doesn't work on Mac OS X, as it continues offering keys even after they don't exist anymore. It is evidently caching credentials in some way, possibly via ssh-agent; it appears that ssh-agent is spawned by launchd when initiating a login session.

I didn't try killing the agent (launchd would probably just restart it) and instead elected to reboot. Having removed one of the keys from the ~/.ssh directory, authentication now succeeded, but I'd still like to get to the bottom of why the PEM format key is being rejected by OpenSSH, and the -i switch is effectively useless. I also tried DER format, and PEM-with-encryption, but there are still a couple of options that I haven't tried yet.