Background
For over a year now, people have been able to build Amazon Machine Images out of their software appliances and derived distributions using rBuilder Online. As a part of the build process, the person building the image could specify where to mount a huge pile of volatile “instance storage”. This space wasn’t specified in /etc/fstab and, as such, wasn’t automatically mounted unless you specified in advanced options to mount it somewhere (e.g. /mnt or /scratch1).
Last October (2007), Amazon created the ability to super size your virtual environment. In addition to allowing beefier virtual instances to be launched, support was added for building and booting 64-bit x86 images using one of the larger environment types.
As with any change, we got slightly burned because we weren’t bundling our 64-bit images for the right environment. After having fixed that we thought we had taken care of all of the fallout from Amazon’s change.
We were wrong.
“Trouble at Mill!”
One of our support engineers complained that the image he created was “corrupted”. He launched the image—a 64-bit x86 build—which would boot but never come up. Upon asking Amazon for the console output, he found that the machine had dropped into a manual fsck, and was waiting on someone to tell it what to do (which would never happen, as no one is at the console to drive).
After poking around in the EC2 forums, I discovered that /dev/sda2 is invalid for large and extra-large instance types.2 Sadly, our /etc/fstab inside the image will hardcode in /dev/sda2 on the mountpoint of your choice, causing EPIC FAIL when attempting to boot, but only if you specify the mountpoint as an advanced option when building the image.
A Temporary Workaround
The workaround is easy: don’t specify the advanced option called “Mountpoint for scratch space [/dev/sda2] on AMI”. Just leave it blank. Do that, and all should be well. You will to mount your instance storage by hand, but that’s better than ending up with a dead image.
In the meantime, you’ll want to watch this issue.
1 Mounting the huge image on /tmp isn’t advisable due to the fact that the sticky bit is lost. This causes havoc with some things like, say, running MediaWiki backed with MySQL on your AMI. I’ll tell you about that over beer, someday.
2 Mea culpa. Mantra: BEWARE ASSUMPTIONS. In this case: assuming that /dev/sda2 would always be there.