Late night deployment and randomness
During deployment of Keepassa , our password manager, last night I had an interesting issue.
The server process (embedded Jetty), hanged on startup, just before building the SSL Context Factory.
As we have migrated from OpenBSD, to CentOS on our staging and production servers, I first thought that it was a CentOS issue as the code was running fine on my Ubuntu workstation.
As CentOS uses an old version of OpenSSL, and that was the only difference in the stack, I upgraded the OpenSSL library on the staging server.
AFAIK, nothing changed. then I tried changing the Linux kernel options (sysctl.conf), but still nothing happened. The Bootstrap process was just hanging on startup.
jstack to the rescue.
After some head scratching I fired jstack, to trace the running threads, as it was obviously, that there was a deadlock.
jstack, showed an interesting output. All the jetty threads, were waiting for a call to SecureRandom, the java pseudorandom generator.
Java, uses /dev/random by default on Linux, and it is a blocking random generator, so I tried switching to /dev/urandom for the experiment.
/usr/bin/java -Djava.security.egd=file:/dev/./urandom -jar /opt/keepassa/KeepassaServer.jar
It worked. But /dev/urandom isn’t secure enough for crypto usage, and we are making and Keepassa is a crypto app.
So a friend of mine suggested haveged. I hadn’t used it before, but the setup was painless. Just enter :
yum install haveged
It is available from the EPEL repo.
A word of caution – it is best to have the deamon, installed on the virtualization host as well.
After installing haveged, the server started as fast as when using /dev/urandom.