Justin's IT and Security Pages

OpenIndiana and XenServer

27 comments

Now that OpenSolaris is officially dead and the drama has died down a bit, I thought it might be time to figure out how to install OpenIndiana on my XenServer. I’ve heard some say that you can’t install the new distribution on Xen as a domU, but that is demonstrably incorrect (although as I understand it, dom0 is presently out of the question due to Oracle pulling out the relevant xVM bits).

Installing OpenIndiana 147 on XenServer 5.5 is nearly identical to installing OpenSolaris on the same (as would be expected considering OpenIndiana’s roots).

  1. Create a new VM using the “Other Install Media” profile within XenCenter. Set up the VM with 10GB of disk space and 1GB of RAM.
  2. Copy the /platform/i86xpv/kernel/amd64/unix and /platform/i86pc/amd64/boot_archive files from the OI install disk over to the XenServer host.
  3. On the XenServer, determine the UUID of the newly created OpenIndiana VM using xe vm-list; just note the first 3 or 4 characters and tab completion will enter the rest when necessary.
  4. Configure the following parameters:
    1. xe vm-param-set uuid=<vm uuid> PV-kernel=<full path to the 'unix' file on the XenServer>
    2. xe vm-param-set uuid=<vm uuid> PV-ramdisk=<full path to the 'boot_archive' file on the XenServer>
    3. xe vm-param-set uuid=<vm uuid> PV-args='/platform/i86xpv/kernel/amd64/unix -B console=ttya'
    4. xe vm-param-set uuid=<vm uuid> HVM-boot-policy=
    5. xe vm-param-set uuid=<vm uuid> PV-bootloader=
  5. Mount the OpenIndiana install CD in the appropriate drive (e.g., select the correct ISO in XenCenter)
  6. Boot the OpenIndiana VM. Log in as jack/jack when appropriate to do so.
  7. Configure basic networking if needed; my Windows 2008 DHCP server never works to assign addresses to OpenSolaris/OpenIndiana (for whatever reason) so this is generally a mandatory step.
  8. Connect to the OpenIndiana server with an SSH client with X-tunneling enabled using the jack/jack account.
  9. Execute: pfexec /usr/bin/gui-install. The graphical install process will begin. Complete the steps as requested.
  10. After the installation is completed (and before rebooting), change the PV-args on the XenServer to: xe vm-param-set uuid=<vm uuid> PV-args='/platform/i86xpv/kernel/amd64/unix -B console=ttya,zfs-bootfs=rpool/ROOT/openindiana,bootpath="/xpvd/xdf@51728:a"'. Note the two changes from the OpenSolaris instructions from an earlier blog post: the zfs-bootfs is openindiana, not opensolaris and the bootpath is 51728 instead of 51712. I have no idea why the latter change is necessary – I just know that there was no 51712 in my devices directory, only 51728 and a 51760.

Reboot, and you’re good to go! Remember to run bootadm update-archive after the first boot (and anytime you make changes that require that file to be updated) and copy out the updated /platform/i86pc/amd64/boot_archive to the XenServer before rebooting.

Here’s a screenshot; note that I don’t know what’s up with the savecore error. Considering the unstable nature of the code, those sorts of hiccoughs don’t surprise me.

OpenIndiana on XenServer

Written by justin

October 1st, 2010 at 7:47 pm

Posted in infrastructure

27 Responses to 'OpenIndiana and XenServer'

Subscribe to comments with RSS or TrackBack to 'OpenIndiana and XenServer'.

  1. In xencenter 5.6 security behavior changed, the kernel and ramdisk NEEDS to be placed in “/boot/guest/” if you don’t or the xen server will give error “illegal kernel path”.

    Also if it wasn’t apparent, you’ll need to create the rsa/dsa keys and start up sshd manually. 1. use “sudo su” to become full root. 2. “ssh-keygen -b 1024 -t dsa” save it to “/etc/ssh/ssh_host_dsa_key” 3. run “ssh-keygen -b 1024 -t rsa” save it to “/etc/ssh/ssh_host_rsa_key” 4. run “/usr/lib/ssh/sshd”

    Other than those two points, your guide is perfect with xenserver 5.6.1 and OpenIndiana 148. Also that savecore message no longer appears in this newer version of OI. Thanks for posting this!

    ac

    1 Feb 11 at 4:39 pm

  2. oi-dev-148-x86.iso

    jack@openindiana:/# pfexec /usr/bin/gui-install
    The application ‘gui-install’ lost its connection to the display localhost:11.0;
    most likely the X server was shut down or you killed/destroyed
    the application.

    Eric Snellman

    9 Feb 11 at 5:45 am

  3. I also did, not sure if its needed.
    echo “ForwardX11 yes” >> /etc/ssh/ssh_config

    Also for step 9, if your a newbie like me, this guide helps:
    http://bderzhavets.blogspot.com/2007/11/cygwinx-install-oracle-10.html

    Eric Snellman

    10 Feb 11 at 7:30 am

  4. Great guide, I followed it almost to the letter, except I’m on xenserver 5.6.0. But when I go to boot it (in step 6, before I install OpenIndiana) the VM stops and I get this in the Event Log:

    Error: Starting VM ‘openindiana’ – Internal error: XenguestHelper.Xc_dom_linux_build_failure(2, ” xc_dom_find_loader: no loader foundn”")

    I then tried a few tweaks by using the oi-dev-148
    -text-x86.iso (text installer) and using /platform/i86xpv/kernel/unix and /platform/i86pc/boot_archive instead of the amd64 ones. It boots and I can install just fine, but when I reboot after installation I get this:

    v3.4.2 chgset ‘trunk.455c6392747f.421ad87df521 (3.4.2 5.6.0.597.20014)’
    OpenIndiana Build oi_148 32-bit
    SunOS Release 5.11 – Copyright 1983-2010 Oracle and/or its affiliates.
    All rights reserved. Use is subject to license terms.
    NOTICE: zfs_domount: error 2
    Cannot mount root on rpool/ROOT/fileserve fstype zfs

    panic[cpu0]/thread=f502b720: vfs_mountroot: cannot mount root

    Warning – stack not written to the dumpbuf
    f505ac70 genunix:vfs_mountroot+319 (f506dc50, f50559c8,)
    f505ac80 genunix:main+105 ()

    skipping system dump – no dump device configured
    rebooting…

    Any Ideas?

    Sam Ashcraft

    19 Mar 11 at 9:23 pm

  5. Try setting the kernel parameters to point to “rpool/ROOT/openindiana” instead of “rpool/ROOT/fileserve.”

    Justin Thomas

    19 Mar 11 at 9:29 pm

  6. Thank you for replying, same result though:

    Cannot mount root on rpool/ROOT/openindiana fstype zfs

    btw fileserve is the host name I gave it during install

    Sam Ashcraft

    20 Mar 11 at 8:38 am

  7. I need to go back through my previous posts later (I’m on my phone now); there was one situation where I had to boot OSOL into maintenance mode and force import the zpool. I wonder if you might be running up against a similar problem.

    EDIT: Actually, it was Nexenta, not OSOL and was a result of having to do the install as an HVM (could not install from the PV kernel).

    Post the output from these commands and I’ll see if anything looks awry:

    xe vm-param-get uuid=OI_VM_UUID param-name=PV-args
    xe vm-param-get uuid=OI_VM_UUID param-name=PV-kernel
    xe vm-param-get uuid=OI_VM_UUID param-name=PV-ramdisk

    Justin Thomas

    20 Mar 11 at 9:51 am

  8. heres the output:

    param-name=PV-args
    /platform/i86xpv/kernel/unix -B console=ttya,zfs-bootfs=rpool/ROOT/fileserve,bootpath=”/xpvd/xdf@51728:a”

    param-name=PV-kernel
    /boot/guest/openindiana/unix

    param-name=PV-ramdisk
    /boot/guest/openindiana/boot_archive

    I tried booting into recovery mode, but that leaves me with a “No bootable device.” error. Its almost like the bootloader cant access the virtual drive.

    Sam Ashcraft

    20 Mar 11 at 6:04 pm

  9. Hey Justin,

    I’m pretty new to XenServer and Bash/SSH

    What’s the best way to copy the files (step 2) to the server from either the dvd drive in the server or from a remote computer (ie my desktop)?

    Cheers

    Simon

    18 Jul 11 at 6:15 am

  10. Hi Simon,

    I always use SSH from my desktop, so I just type something like “scp ./filename root@xenserver:” and enter in the root password to save “filename” to the /root directory on the XenServer.

    You could probably use the DVD drive, but that’s more effort for me (I don’t have an optical drive in my server).

    Hope that helps!

    Justin Thomas

    19 Jul 11 at 5:18 pm

  11. [...] 2. OpenIndianaのインストールについて OpenIndiana and XenServer « Justin’s IT and Security Pages [...]

    [XenServer, OpenIndiana] XenServer 5.9 (6.0 Boston Beta) への OpenIndianaのインストール | 殴り書きメモ

    1 Aug 11 at 4:35 am

  12. Just to play on the edge of this post I tried the above using XenServer 6 and Openindiana 151a.

    I formed the initial VM using the Solaris 10 (experimental) template then religiously followed this howto.

    Seems the pool label has changed again.

    I get the line
    NOTICE: Can not read the pool label from ‘/xpvd/xdf@51728:a’

    It does a kernel panic and shuts down.
    With any luck it is simply getting the right bootpath.

    How might I go about find what the alternatives could be?

    Thanks for the great post.

    Greg Stevenson

    19 Oct 11 at 9:44 pm

  13. Hi Greg, thanks for the note! I’ll try to take a look at this tomorrow; I know that 51712 was one variation, but I know there are others. I wish I knew the significance of those numbers, but unfortunately they seem rather arbitrary to me (although I’m sure they’re not).

    justin

    20 Oct 11 at 5:18 am

  14. Thanks Justin,

    How did you go about finding the numbers in the first place. Do you have to import the zfs pool somehow?

    Cheers
    Greg

    Greg Stevenson

    20 Oct 11 at 9:51 pm

  15. I have had to force import the pool when switching from HVM to PV in the past. I believe the command was something like ‘zpool import -f’ but it’s been a while. Generally, if the system was installed as PV, that shouldn’t be necessary.

    When I look in the /devices directory on my SXCE VM, I have entries for 51712, 51728, 51744 and 51760.

    justin

    22 Oct 11 at 5:43 am

  16. Thanks Justin,

    For everyones reference it appears the right number for Xenserver 6 and Openindiana 151a is 51712.

    Working as expected.

    Time to hack Xenserver to give direct access to all the local disks to the OpenIndiana VM.

    http://forums.citrix.com/thread.jspa?threadID=292035

    Then setup storage-server to share a zfs target via iscsi to all my other VMs

    Greg Stevenson

    22 Oct 11 at 8:38 am

  17. Im stuck. xenserver 6.1 oi 151a Followed guide re vm-params, inserted oi151a iso, vm starts but then just stops loading before even a console shows so i cannot seem to install. I have unix and boot_archive in /boot/guest. wah! Im trying to go where Greg went above – xenserver hosting a virtualised nas for a pool as shared storage.

    john

    2 Apr 12 at 12:53 pm

  18. Finally did it. Great guide thanks heaps. OI 1.51a on xenserver 6.1
    To install from windows without cygwin
    Download xming and install. start Xming (tray icon X appears)
    Download plink.exe; open command prompt in same directory
    plink.exe -X jack@ xterm
    gives you the terminal window
    pfexec /usr/bin/gui-install

    Gui comes up

    john

    7 Apr 12 at 5:05 am

  19. Thanks John! I’m glad you got it up and running.

    justin

    9 Apr 12 at 5:18 pm

  20. Hi Justin,

    I’m getting the exact same error that Sam Ashcraft had:

    NOTICE: zfs_domount: error 2
    Cannot mount root on rpool/ROOT/openindiana fstype zfs

    panic[cpu0]/thread=fffffffffbc60aa0: vfs_mountroot: cannot mount root

    Warning – stack not written to the dump buffer
    fffffffffbcb5180 genunix:vfs_mountroot+75 ()
    fffffffffbcb51b0 genunix:main+136 ()
    fffffffffbcb51c0 unix:_locore_start+7e ()

    I’m using XenServer 5.6 and OI 151a5. I used 51712 otherwise I get hte “Can not read the pool label” error. My details are as follows:

    xe vm-param-get uuid=$VM param-name=PV-args

    /platform/i86xpv/kernel/amd64/unix -B console=ttya,zfs-bootfs=rpool/ROOT/openindiana,bootpath=”/xpvd/xdf@51712:a”

    xe vm-param-get uuid=$VM param-name=PV-kernel

    /boot/guest/unix

    xe vm-param-get uuid=$VM param-name=PV-ramdisk

    /boot/guest/boot_archive

    Did you have any insights?

    Appreciate

    Mark C

    30 Jul 12 at 3:50 pm

  21. Maybe add “-s” to the end of your pv-args line (to boot to single-user mode) and then try to force an import of the pool using:

    zpool import -f rpool

    I’m honestly just guessing, though – it’s been a long time since I’ve looked at this.

    justin

    30 Jul 12 at 9:13 pm

  22. Isnt working for me, I cant partially boot without all the -B stuff
    but I cant do nothing

    It ried with all the snapshot i got rpool openindiana and nappit which is my last and and my xcp vm just poweroff alone

    HellMind

    5 Apr 13 at 4:28 am

  23. unt: error 19
    Cannot mount root on rpool/ROOT/napp-it-0.8h_update_06.28-1 fstype zfs

    panic[cpu0]/thread=fffffffffbc60aa0: vfs_mountroot: cannot mount root

    Warning – stack not written to the dump buffer
    fffffffffbcb5180 genunix:vfs_mountroot+75 ()
    fffffffffbcb51b0 genunix:main+136 ()
    fffffffffbcb51c0 unix:_locore_start+7e ()

    skipping system dump – no dump device configured
    rebooting…

    HellMind

    5 Apr 13 at 9:01 am

  24. pseudo0 at root
    pseudo0 is /pseudo
    scsi_vhci0 at root
    scsi_vhci0 is /scsi_vhci
    xdf@51712, xdf0
    xdf0 is /xpvd/xdf@51712
    /xpvd/xdf@51712 (xdf0) online
    xdf@51712: 41943040 blocksNOTICE: zfs_domount: error 19
    Cannot mount root on rpool/ROOT/napp-it-0.8h_update_06.28-1 fstype zfs

    panic[cpu0]/thread=fffffffffbc60aa0: vfs_mountroot: cannot mount root

    Warning – stack not written to the dump buffer
    fffffffffbcb5180 genunix:vfs_mountroot+75 ()
    fffffffffbcb51b0 genunix:main+136 ()
    fffffffffbcb51c0 unix:_locore_start+7e ()

    skipping system dump – no dump device configured
    rebooting…

    HellMind

    5 Apr 13 at 9:05 am

  25. A few notes from my experience on OI 151a7 on XenServer 6.1:

    * This works on XenServer 6.1 – however, the kernel and ramdisk must be under /boot/guests, or else you will get an error that says “Internal error: xenopsd internal error: Domain.Could_not_read_file(“”)”. Took me awhile to debug that one, but if you look through xensource.log, you will see messages that the path to /opt/osol is not whitelisted, and see the actual domain construction passing through a blank kernel and bootloader.
    * /xpvd/xdf@51712:a worked for me with OpenIndiana 151a7
    * Keyboard console doesn’t work for me; I ended up adding the ‘livessh=enable’ kernel parameter while installing, and set up a DHCP server to give the VM a static lease, then SSH’d in as jack to do the install.

    Nate Carlson

    8 Apr 13 at 9:01 pm

  26. ..type-o in my comment above; ‘/boot/guests’ should be ‘/boot/guest’, as noted in comments way up above. Just a different error message for me. ;)

    In addition, keyboard works after restarting XenCenter. Stupid XenCenter.

    I’m also trying to get this working with pv-grub so that updates in the guest will be reflected.. got it to the point where pv-grub boots, but doesn’t find menu.lst; when I paste in the contents of the stock menu.lst from openindiana it boots.

    Nate Carlson

    8 Apr 13 at 9:32 pm

  27. ..and pv-grub works great (you can select the boot menu option, change it, etc, just like native hardware); just need to specify the following ‘PV-args’:

    (hd0,0,a)/boot/grub/menu.lst

    This is on a OpenIndiana install with one virtual disk, and I told it to use the whole disk.

    pv-grub is built as described here, with ZFS patches:
    https://github.com/aszeszo/pv-grub

    Nate Carlson

    9 Apr 13 at 1:41 pm

Leave a Reply