Q:

Is webkey a reasonably secure authentication system?

I just created Webkey in the last couple days. It's built to do client-certificate auth without using the (in my opinion unusable) client-cert functionality built into browsers. It would allow single-password auth just like an ssh key does, but for the web.
The user's password is never sent over the internet (even in encrypted form). This utility can be hosted entirely statically (and is currently being hosted at http://webkey-auth.github.io/test/guest.html ). 
The requesting application loads webkey in an iframe over https. On-request of auth, if the user hasn't created an RSA keypair, webkey does that in the iframe and saves it in local storage (encrypted with aes using a password the user creates). The user also saves an email to be used as their ID. 
Here are the auth steps in more detail from the perspective of the application requesting auth:

On the client, request the user's acceptance (requestAcceptance command) and send the resulting email and public key to the server
On the server, if that email is not already in the system with that public key, send them a verification email (if you want) and once verified, associate the email and public key in your database and skip to step 7
if that email is in the system with that public key, continue to step 3
On the server, generate a 3-20 character token (in string form) and send it to the client
On the client, send the token into webkey's iframe using the 'auth' postMessage command
On the client, receive the signed token (called the proof) from webkey and send that proof to the server.
On the server, verify that the token with the given public key (the server should still have the original token - do NOT trust any tokens sent to the server by the client)
Profit! Your user's now authenticated!

Is what I'm doing here theoretically secure? What are potential problems with how this works?

A:

If the service is meant to be statically hosted, there are several problems with this.
The most important one would be that you are using JavaScript in a browser for crypto. That is, as the link explains in detail, not even practically secure.
Discussing theoretical security may just as well stop  right there.
Do not use JavaScript for crypto. You just cannot trust the user on anything.

