Some comments on this:
1. There may be speed issues; I can stop your radius server
by making more requests for authentication than you can handle. I
may even do this legitamately.
2. The use of RSA is incorrect; see Anderson's "Robustness
Principles" paper from Crypto 95. You need to sign before encrypting,
not afterwards. ("This public key is used to encrypt the entire
authentication packet along with a dummy secret key, the resulting
encrypted packet is signed with the private key of the server.")
Anderson's paper can be found at http://www.cl.cam.ac.uk/users/rja14/
3. Since the code uses RSAref, its probably vulnerable to a
timing attack. (See Kocher's paper in Crypto 96;
www.cryptography.com)
The use of signing an encrypted message leads me to worry
substantially about the implementation. I haven't spent time looking
to see if there are other problems, but with one that large, I'd be
suprised if its the only one.
Adam
--
"It is seldom that liberty of any kind is lost all at once."
-Hume