tag:blogger.com,1999:blog-534724342082307142.post5618095312997451001..comments2023-02-22T18:30:17.187+02:00Comments on Cloud Developer Tips: Boot EC2 Instances from EBSShlomohttp://www.blogger.com/profile/10469902663120418195noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-534724342082307142.post-40199365995544502492009-09-26T05:43:27.034+03:002009-09-26T05:43:27.034+03:00Hi Shlomo,
Nice post regarding boot from EBS.
On...Hi Shlomo,<br /><br />Nice post regarding boot from EBS.<br /><br />On Error: fsck died with exit status 8<br /><br />I think one possible cause of this is when fsck found inconsistent block on EBS volume used.<br /><br />A possible solution could be ignoring filesystem check by editing the fstab setting pass to 0. As posted at http://developer.amazonwebservices.com/connect/thread.jspa?threadID=24091&start=30&tstart=0<br /><br />Or either attached that EBS volume to another instance and do fsck to that device then reattached it again to the pivot AMI instance.<br /><br />Just a thought.<br /><br />RodneyRodney Quillohttps://www.blogger.com/profile/12610437211480954197noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-65368944328824008312009-09-24T15:14:32.533+03:002009-09-24T15:14:32.533+03:00@Robert Staveley (Tom),
Yes, there are many cool ...@Robert Staveley (Tom),<br /><br />Yes, there are many cool use cases enabled by shared snapshots. I even wrote an article about some: http://clouddevelopertips.blogspot.com/2009/09/cool-things-you-can-do-with-shared-ebs.html<br /><br />I wish I could claim credit for "getting through to them" but I shouldn't. It takes more than one person to clamor, and AWS seem to be strongly influenced by customer clamor.Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-26816010666652651032009-09-24T13:02:27.093+03:002009-09-24T13:02:27.093+03:00Looks like you got through to them, Shlomo! Shared...Looks like you got through to them, Shlomo! Shared snapshots have been announced now. They are implemented in the AWS console http://aws.amazon.com/console/ and in the API and collaboration with shared DEV environments using your technique should now be able to cross Amazon account boundaries. :-)Rob Staveley (Tom)https://www.blogger.com/profile/12685264964545576535noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-5057388202433004702009-09-01T22:17:32.438+03:002009-09-01T22:17:32.438+03:00@rstaveley,
It's true - any distro can be use...@rstaveley,<br /><br />It's true - any distro can be used to build the boot AMI, and so the public AMI you mentioned (ami-2feb0f46 nimlabs/pivot-sdj-20080824.manifest.xml) can be used to boot from any EBS attached to /dev/sdj . Anyone who needs a boot AMI using a different attach point can follow the procedure in this article to build one from an Ubuntu AMI.<br /><br />I did identify that the difference between Ubuntu and Debian lies in the RAID support. As you say, there appears to be little practical value to discovering the underlying cause of failure.<br /><br />Thanks for your feedback!Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-11479246677718823512009-09-01T21:39:31.938+03:002009-09-01T21:39:31.938+03:00I'd only look into it if it satisfies an intel...I'd only look into it if it satisfies an intellectual itch, Shlomo. A virtue of this pivot_root AMI approach is that the AMI doesn't need to be the same distro as the bootable EBS volume. As long as the AMI can [reasonably] reliably get any bootable EBS distro you want up and running, you might as well use the same AMI... Unless I am mistaken.<br /><br />I hate to think that my brand loyalty has got you delving into the differences between Debian and Ubuntu or Fedora, when any distro will suffice for the pivot AMI.Rob Staveley (Tom)https://www.blogger.com/profile/12685264964545576535noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-79651631524567342392009-09-01T18:39:32.560+03:002009-09-01T18:39:32.560+03:00@rstaveley,
Yes, it turns out that the actual pro...@rstaveley,<br /><br />Yes, it turns out that the actual problems begin later in the boot sequence. The entire console output (which can take a while to show up, and sometimes only flushes completely at instance termination) shows more interesting stuff.<br /><br />Boot-time problems are notoriously hard to debug in EC2 where you can't get console access.<br /><br />Using the same AMI/AKI/ARI as you, I get this:<br />[skipping uninteresting stuff]<br />EXT3 FS on sda1, internal journal<br />Making pivot_root device node /new-root-dev<br />Attempting to mount pivot_root volume on /new-root-dev<br />1 scsi_mod: no version for "struct_module" found: kernel tainted.<br />SCSI subsystem initialized<br />register_blkdev: cannot get major 8 for sd<br />mount: /new-root-dev is not a valid block device<br />2 mount: /new-root-dev is not a valid block device<br />3 mount: /new-root-dev is not a valid block device<br />4 mount: /new-root-dev is not a valid block device<br />5 mount: /new-root-dev is not a valid block device<br />6 mount: /new-root-dev is not a valid block device<br />7 mount: /new-root-dev is not a valid block device<br />8 mount: /new-root-dev is not a valid block device<br />9 mount: /new-root-dev is not a valid block device<br />10 mount: /new-root-dev is not a valid block device<br />11 mount: /new-root-dev is not a valid block device<br />12 mount: /new-root-dev is not a valid block device<br />13 mount: /new-root-dev is not a valid block device<br />14 mount: /new-root-dev is not a valid block device<br />15 mount: /new-root-dev is not a valid block device<br />16 mount: /new-root-dev is not a valid block device<br />17 mount: /new-root-dev is not a valid block device<br />18 mount: /new-root-dev is not a valid block device<br /> sdp: unknown partition table<br />19 kjournald starting. Commit interval 5 seconds<br />EXT3-fs: mounted filesystem with ordered data mode.<br />Mounted pivot_root volume.<br />Mount failed for selinuxfs on /selinux: No such file or directory<br />INIT: version 2.86 booting<br />/proc is empty (not mounted ?)<br />hostname: the specified hostname is invalid<br />Starting the hotplug events dispatcher: udevd.<br />Synthesizing the initial hotplug events...done.<br />Waiting for /dev to be fully populated...input: PC Speaker as /class/input/input1<br />done.<br />Activating swap...Adding 917496k swap on /dev/sda3. Priority:-1 extents:1 across:917496k<br />done.<br />Checking root file system...fsck 1.41.3 (12-Oct-2008)<br />fsck.ext3: Device or resource busy while trying to open /dev/sdp<br />Filesystem mounted or opened exclusively by another program?<br />fsck died with exit status 8<br />[31mfailed (code 8).[39;49m<br />An automatic file system check (fsck) of the root filesystem failed. A manual fsck must be performed, then the system restarted. The fsck should be performed in maintenance mode with the root filesystem mounted in read-only mode. [31mfailed![39;49m<br />The root filesystem is currently mounted in read-only mode. A maintenance shell will now be started. After performing system maintenance, press CONTROL-D to terminate the maintenance shell and restart the system. [33m(warning).[39;49m<br />Press enter for maintenance<br />(or type Control-D to continue): <br /><br />The 19 retries happened while waiting for the volume to be attached to /dev/sdp. Then the boot continues OK but dies with:<br />Checking root file system...fsck 1.41.3 (12-Oct-2008)<br />fsck.ext3: Device or resource busy while trying to open /dev/sdp<br />Filesystem mounted or opened exclusively by another program?<br />fsck died with exit status 8<br /><br />Ugh. I'll look into this some more.Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-38079618056836471632009-09-01T14:23:23.040+03:002009-09-01T14:23:23.040+03:00I've been playing with this some time now this...I've been playing with this some time now this morning and I have noticed that the public pivot AMI also throws up the same scsi_mod error. I apologise that was misleading. The Debian boot was similar to your Ubuntu right the way up to:<br /><br />Checking root file system...fsck 1.41.3 (12-Oct-2008)<br />fsck.ext3: Device or resource busy while trying to open /lib/init/rw/rootdev<br />Filesystem mounted or opened exclusively by another program?<br />fsck died with exit status 8<br /><br />In 20 or so boots I found that the public pivot AMI also failed once, so the problem may be a subtle one.<br /><br />However... I made a happy discovery while I was playing with this, which probably seems obvious to you, but which may interest your readers.<br /><br />I (shortsightedly?) created a 1G EBS volume for my DEV environment, which was OK for experimenting but very soon filled up. I wished I made a 2G volume.<br /><br />I thought of going back to the drawing board, but then it occurred to me that I could do this:<br /><br />1. Create a snapshot of the 1G bootable EBS volume with my DEV system on it.<br />2. Create a 2G volume from that snapshot.<br />3. Attach the new volume as (say) /dev/sdk, but don't mount it.<br />4. Resize the filesystem to 2G<br /> e2fsck -f /dev/sdk<br /> resize2fs /dev/sdk<br />6. Snapshot the volume with the resized filesystem<br />7. Use my new snapshot henceforth as the bootable EBS volume.<br /><br />This quickly gives me a way to increase the size of the bootable volume. An elastic smile appeared on my face :-)Rob Staveley (Tom)https://www.blogger.com/profile/12685264964545576535noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-28723252468165447972009-09-01T11:37:51.324+03:002009-09-01T11:37:51.324+03:00I used the default AKI and ARI.I used the default AKI and ARI.Rob Staveley (Tom)https://www.blogger.com/profile/12685264964545576535noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-54479111454375974762009-09-01T09:59:01.846+03:002009-09-01T09:59:01.846+03:00@rstaveley,
The problem reported by scsi_mod may ...@rstaveley,<br /><br />The problem reported by scsi_mod may be an issue with the kernel modules included in the ramdisk. What AKI and ARI did you use? I'll try to reproduce with the default AKI and ARI and see what happens....<br /><br />The ability to create public, shared EBS volumes has been requested by many. It would allow some really cool sharing scenarios, such as the one you describe. I have submitted a question to the AWS folks who manage the public data set submissions:<br /><br />---<br />I'd like to share an EBS volume with the public in the same way I share an AMI. Public EBS snapshots would allow many really cool use cases, including bypassing the 10GB root-partition limit of an AMI (in combination with the pivot_root technique). Can I submit an EBS snapshot containing the linux boot code and startup sequence? Would you approve it? It's more than "data", it would also be the kind of stuff you'd burn into an AMI - apps, configs, etc....<br />---<br /><br />I'll report back what they say.Shlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-23240008217561309042009-09-01T08:26:15.348+03:002009-09-01T08:26:15.348+03:00After your tip-off in your excellent http://cloudd...After your tip-off in your excellent http://clouddevelopertips.blogspot.com/2009/08/mount-ebs-volume-created-from-snapshot.html, I was curious to see about putting the root file system onto EBS for development environments. When everything is on EBS, installing additional packages is less painful than it is if you only have your development application in persistent storage.<br /><br />I've been using the Debian AMIs for no better reason than brand loyalty, so my starting point was creating the EBS bootable volume from ami-ff46a796 alestic/debian-5.0-lenny-base-20090804.manifest.xml following your instructions here.<br /><br />However, when I booted from the pivot AMI, I saw this error in my console output:<br /><br />Making pivot_root device node /new-root-dev<br />Attempting to mount pivot_root volume on /new-root-dev<br />1 scsi_mod: no version for "struct_module" found: kernel tainted.<br />SCSI subsystem initialized<br />register_blkdev: cannot get major 8 for sd<br />mount: /new-root-dev is not a valid block device<br />2 mount: /new-root-dev is not a valid block device<br />3 mount: /new-root-dev is not a valid block device<br />4 mount: /new-root-dev is not a valid block device<br />5 mount: /new-root-dev is not a valid block device<br />6 mount: /new-root-dev is not a valid block device<br />7 mount: /new-root-dev is not a valid block device<br /> sdp: unknown partition table<br /><br />I tried going through your procedure a couple of times and repeated the same error.<br /><br />Google took me to a similar error reported in a thread, which also had a public pivot AMI, which works nicely, if I mount my volume on /dev/sdj:<br /><br />http://developer.amazonwebservices.com/connect/thread.jspa?threadID=24091&start=15&tstart=0#99041<br /><br />So the public pivot AMI ami-2feb0f46 nimlabs/pivot-sdj-20080824.manifest.xml works nicely for me with the Debian Lenny.<br /><br />The only substantial differences I can see between your init scripts - his is at http://developer.amazonwebservices.com/connect/servlet/JiveServlet/download/30-24091-99041-1961/init.sh - are:<br /><br />1. N. Martin uses /dev/sdj rather than /dev/sdp (yes... I did attach accordingly :-)<br />2. He doesn't invoke the pre-pivot script<br /><br />I suspect that following your instructions for setting up a boot AMI from Ubuntu would work fine with my Debian bootable EBS, but it makes sense to me to use a public AMI for the boot.<br /><br />This got me thinking - lazy person that I am - that there is a case now for public snapshots for use with public pivot AMIs. Get yourself started on your development project on a bootable EBS volume with a pre-configured set up for all of the usual suspects you see listed as public AMIs. I guess these would go into http://aws.amazon.com/publicdatasets/. You'd get yourself started with a public snapshot and then take private snapshots as your project evolves - i.e. carry the AMI paradigm through to EBS.Rob Staveley (Tom)https://www.blogger.com/profile/12685264964545576535noreply@blogger.comtag:blogger.com,1999:blog-534724342082307142.post-30809935416422953392009-08-31T23:22:32.116+03:002009-08-31T23:22:32.116+03:00are you kidding? you are a rock star for this po...are you kidding? you are a rock star for this post! Thank you!!!Unknownhttps://www.blogger.com/profile/02467572439553803578noreply@blogger.com