Sunday, January 08, 2017

Freenas: Migrating from VirtualBox to Bhyve

I, like many people, use the VirtualBox template jail on Freenas.  I've been using this for about a year to run Crashplan.  It's generally worked good, but the last couple of 9.10 maintenance updates have caused some problems (ex: The virtual machine 'xxxx' has terminated unexpectedly during startup with exit code 1).  See here for more on how 9.10.2 broke the VirtualBox template.

From what I've been able to read on the Freenas forums, this isn't going to get fixed and you have two options.  Don't upgrade and keep using VirtualBox or upgrade and use Bhyve.  Since I only have one VM and it's pretty simple, I opted for the latter solution.  This post is going to cover the steps that I had to do to convert from using VirtualBox to using Bhyve and iohyve without reinstalling the VM OS.

Convert the Image

The first step is to convert your old VirtualBox image to a raw disk image.  There's no direct way to import the VDI or directly use it with iohyve, but you can convert the VDI to a raw disk image and we can use that.  To convert the disk, do the following.
  1. SSH to your VirtualBox jail.
  2. Install the qemu-devel pkg with the command `pkg install qemu-devel`.
  3. Run `qemu-img convert -f vdi -O raw /path/to/virtualbox/vm.vdi /path/to/raw/image.img`
That's it.  You now have a raw image file of your old VirtualBox VM.

Setup Iohyve

There is some setup that's needed before you can use iohyve on your Freenas.  This only needs to be done once, so if you've used iohyve before on your Freenas you can skip these steps.
  1. Initialize iohyve by running `iohyve setup pool=main kmod=1 net=igb0`.  Note that you need to replace `main` with the name of your ZFS pool and `igb0` with the name of your network device.
  2. Run `ln -s /mnt/iohyve /iohyve`.  I don't know exactly why you need this symlink, but just make sure the symlink exists so that you comply with the requirement from the iohyve README.
Create a New VM

The next step is to create our new VM using iohyve.  If you're not familiar with it, iohyve is a convenient wrapper script around Bhyve which makes managing your VMs easier.  In addition to it being convenient, it's also installed out of the box on Freenas 9.10.2, which makes it a logical choice to use.

To create a new VM with iohyve, run the following commands:
  1. Create the VM, run `iohyve create crashplan 8G`.  The `8G` should be the maximum size of your disk.
  2. Configure the VM, run `iohyve set crashplan loader=grub-bhyve os=ubuntu ram=512M cpu=1`.  We set `loader=grub-bhyve` because this is a Linux VM being booted by Grub.  Set the OS to whatever you're using, my VM is using Ubuntu (if you're using LVM with Debian or Ubuntu, use `d8lvm` instead).  Then set the RAM and CPU limits, I picked 512M of RAM and one VCPU.
  3. You can now run `iohyve getall crashplan` to confirm all the settings of your VM.
Your VM is now created.  At this point, you would normally proceed to install an OS, however we've done that already in VirtualBox and so we don't want to do it again.

Import the Raw Image

The next step is to import the raw image that we exported previously.  This is simply done with `dd`.  When using iohyve, you'll see that you have a ZFS dataset for each VM.  For example, my ZFS pool is named `main` and my VM is named `crashplan`, so I have a disk at `main/iohyve/crashplan`.  In addition to that, there is a dataset for the disk.  In my case there's only one disk, so it's `disk0`.  To import the disk, we just need to `dd` the raw disk image to `disk0`.  That can be done with the following command.
dd if=/mnt/main/jails/virtualbox_jail/home/vbox/VirtualBox\ VMs/CrashPl
anBackup/CrashPlanBackup.img of=/dev/zvol/main/iohyve/crashplan/disk0
Make sure you replace `if=...` with the path to your raw image file and replace `of=...` to the path to your iohyve VM disk.  Once `dd` finishes, the VM should be ready to run.

Run Your VM

To start your VM run `iohyve start crashplan`.  Then run `iohyve console crashplan` to connect the console to the VM.  The console will display the output from the VM as it starts up.  When done you'll even be able to login to the VM through the console, but you'll probably want to connect via SSH instead.

Start VM at Boot

At this point your VM should be up and running, however if you were to restart your Freenas the VM would not be running on restart.  Fortunately it's pretty easy to make the VM start at boot.  The following instructions will make your VM start at boot.

  • In the Freenas UI, go to System -> Tunables.
  • Add a tunable.  Variable is `iohyve_enable`, value is `YES`, type is `rc.conf`. Make sure enabled is checked.
  • Add a tunable.  Variable is `iohyve_flags`, value is `kmod=1 net=igb0` (where `igb0` is the name of your network device), type is `rc.conf`.  Also make sure enabled is checked.
These settings will enable iohyve and get it started.  From there, we need to tell iohyve to start our VM.  That can be done with the following command.

iohyve set crashplan boot=1
As with all the commands I've listed, replace `crashplan` with the name of your VM.

Clean Up

While not strictly necessary, you can now clean up the old VirtualBox template jail.  Simply delete the jail a that should be it.


With any luck you should now have your VirtualBox VM functional and running on Bhyve instead.  More work may be necessary if you've got a more complicated VirtualBox VM.  For example, if you have multiple disks, multiple NICs, a GUI or were using VirtualBox specific features like shared folders.  In that case, you may want to stay on VirtualBox until official support for Bhyve is ready in Freenas 10.


Anonymous said...

Hi Daniel, cool post... anny Ideas on how to migrate a Virtualbox Machine from FN 9.10.1-U4 to bhyve on FN10.0?
Or, how to Import a VM (anny given vmdk file) into FN 10? In FN 10 i can only finde the preeconfigured VM`s. But how do i integrate my own vmdk files in a VM on FN10?
Thanx a lott!!!
Greetings Helge

zohaib said...

Thanks for finding this out. I tested it too, and if I manually hash my password using HMACSHA256,

Dan's Blog said...

I think these instructions should work to convert Virtualbox VMs. The trick is using `qemu-img convert` to convert them to a raw image that you can `dd` to the mount that's created by `iohyve`. If you're just using bhyve, I think you could do something similar. `iohyve` is just some convenience wrappers on top of bhyve. I'm not totally sure how you'd go about that though. I have just stuck with `iohyve`.

Rik Bignell said...


I tried this in freenas 11. Created the raw file, dd into a new zvol, create a VM with grub and select the new zvol disk.

I tried to boot but get an error immediately saying There is no boot disk for vm. If i select UEFI instead i just get a shell when it tries to boot.

I read some posts about doing a grub repair with an Ubuntu live disk but that didnt work either.

Any ideas?

Dan's Blog said...

Sorry @Rik Bignell, I haven't tried this on FreeNAS 11. It might have different versions of iohyve. I would suggest double checking that the path to your zvol is correct so it can be found. Also, check iohyve docs to see if there are any flags you might need for your particular OS. Maybe something changed here.