Today one of our production machine (in Amazon EC2) was down and I couldn’t bring-up the instance due to unavailability of SSH in PORT 22, connection was refused and I had no clue what to do, after sometime I was able to bring-up the instance – (ok short and sweet).

Though I got it working after some work-around, I wanted to make sure what I did was correct, hence I asked experts in stackoverflow.com – got good advice/suggestion and I hope all set for a fix. BTW, the answer on the question also helped in learning a new stuff called auditd on the disk – a must have tool for system admin.

Okay, the reason why I wanted to pin-down this blog is to show what I did to fix /dev/urandom and how did I get the server up and running. Let me go straight to there.

HOW!!!?

I recommend any further reading require an insight knowledge of Amazon AWS – I will write some blogs about it very soon.

Luckily (I knew sometime I had to face such problems) I had an EBS instance for the production server – which allowed me to start/stop the instance. (I will follow the point based explanation)

• I stopped the production machine.
• Detached the root-disk (it is connected to /dev/sda1).
• Created a micro-instance with same zone as the above volume.
• Attached #2 volume into #3 created micro-instance.
• Verified the logs (messages/syslog)
• Found a problem in SSH (since I badly wanted to SSH) – which gave me an hint on sshPRNG (yes – ie. PseudoRandomNumberGenerator)
• That helped me trace the actual problem – /dev/urandom file was missing.
• So wrote a quick shell script which will create the above file if missing – please see below for the script.
• Added the above script in an existing init-script – since any existing script will be triggered on system boot.
• Detached #4 volume and attached to production machine.
• Started the production machine - voila.
• I can SSH now without any problem.
• Do not forget to terminate the micro-instance

if [ -f /dev/urandom ] ;

then

    cd /dev; /sbin/MAKEDEV urandom; /etc/init.d/ssh start

fi