Amazon EC2 instance not starting
Mar 25, 2022

Amazon EC2 instance not starting

While working on a failed EC2 (ebs backed) instance recently, we were presented with an instance that would not start after reboot or stop/start.tl;dr: Create a snapshot of the existing EBS vol; remount and edit etc/fstab; re-attach and start the instance.The only symptom was in the console log:$ ec2-get-console-output i-nnnnnnnninit: console-setup main process (63) terminated with status 1%Ginit: plymouth main process (45) killed by SEGV signalinit: plymouth-splash main process (194) terminated with status 2cloud-init running: Sat, 29 Jan 2011 23:33:24 +0000. up 2.65 secondsmountall: Disconnected from PlymouthIt turned out this instance was running as a t1.micro instance, which do not have instance storage on /dev/sdb like all other instance types.The problem with this is the /etc/fstab entry contained and entry:/dev/sdb /mnt auto defaults,comment=cloudconfig 0 0The parameter "nobootwait" is missing!This was causing the instance to hang on reboot. The solution to this problem is as follows:

Gather instance info for reference

$ ec2-describe-instances i-brokeninst

Create a snapshot of the existing volume attached to the broken instance

$ ec2-create-snapshot vol-brokenvol$ ec2-create-volume --snapshot snap-fromcreateabove -z us-east-1dVOLUME vol-newvolume 15 snap-fromcreateabove us-east-1d creating 2011-01-30T00:01:30+0000

Attach this new volume to a temporary host, and edit /etc/fstab from newvol removing the /dev/sdb entry

$ ec2-attach-volume vol-newvol -i i-tempinstance -d /dev/sdX #use a free /dev/sd letterNote: You can remove the /dev/sdb entry, or use you should also be able to use an entry like:/dev/sdb /mnt auto defaults,nobootwait 0 0

detach this new volume from the temporary host

$ ec2-detach-volume vol-newvol

detach the original volume from the failed host

$ ec2-detach-volume vol-oldvolume

Attach the newly edited volume to the failed host (use ec2-describe-instances to determine device location)

$ ec2-attach-volume vol-newvol -i i-brokeninstance -d /dev/sda1

Related posts

Browse more
We haven't published any posts