Freenas: Migrating from VirtualBox to Bhyve

I, like many people, use the VirtualBox template jail on Freenas.  I've been using this for about a year to run Crashplan.  It's generally worked good, but the last couple of 9.10 maintenance updates have caused some problems (ex: The virtual machine 'xxxx' has terminated unexpectedly during startup with exit code 1).  See here for more on how 9.10.2 broke the VirtualBox template.

From what I've been able to read on the Freenas forums, this isn't going to get fixed and you have two options.  Don't upgrade and keep using VirtualBox or upgrade and use Bhyve.  Since I only have one VM and it's pretty simple, I opted for the latter solution.  This post is going to cover the steps that I had to do to convert from using VirtualBox to using Bhyve and iohyve without reinstalling the VM OS.

Convert the Image

The first step is to convert your old VirtualBox image to a raw disk image.  There's no direct way to import the VDI or directly use it with iohyve, but you can convert the VDI to a raw disk image and we can use that.  To convert the disk, do the following.

1. SSH to your VirtualBox jail.
2. Install the qemu-devel pkg with the command `pkg install qemu-devel`.
3. Run `qemu-img convert -f vdi -O raw /path/to/virtualbox/vm.vdi /path/to/raw/image.img`

That's it.  You now have a raw image file of your old VirtualBox VM.

Setup Iohyve

There is some setup that's needed before you can use iohyve on your Freenas.  This only needs to be done once, so if you've used iohyve before on your Freenas you can skip these steps.

1. Initialize iohyve by running `iohyve setup pool=main kmod=1 net=igb0`.  Note that you need to replace `main` with the name of your ZFS pool and `igb0` with the name of your network device.
2. Run `ln -s /mnt/iohyve /iohyve`.  I don't know exactly why you need this symlink, but just make sure the symlink exists so that you comply with the requirement from the iohyve README.

Create a New VM

The next step is to create our new VM using iohyve.  If you're not familiar with it, iohyve is a convenient wrapper script around Bhyve which makes managing your VMs easier.  In addition to it being convenient, it's also installed out of the box on Freenas 9.10.2, which makes it a logical choice to use.

To create a new VM with iohyve, run the following commands:

1. Create the VM, run `iohyve create crashplan 8G`.  The `8G` should be the maximum size of your disk.
2. Configure the VM, run `iohyve set crashplan loader=grub-bhyve os=ubuntu ram=512M cpu=1`.  We set `loader=grub-bhyve` because this is a Linux VM being booted by Grub.  Set the OS to whatever you're using, my VM is using Ubuntu (if you're using LVM with Debian or Ubuntu, use `d8lvm` instead).  Then set the RAM and CPU limits, I picked 512M of RAM and one VCPU.
3. You can now run `iohyve getall crashplan` to confirm all the settings of your VM.

Your VM is now created.  At this point, you would normally proceed to install an OS, however we've done that already in VirtualBox and so we don't want to do it again.

Import the Raw Image

The next step is to import the raw image that we exported previously.  This is simply done with `dd`.  When using iohyve, you'll see that you have a ZFS dataset for each VM.  For example, my ZFS pool is named `main` and my VM is named `crashplan`, so I have a disk at `main/iohyve/crashplan`.  In addition to that, there is a dataset for the disk.  In my case there's only one disk, so it's `disk0`.  To import the disk, we just need to `dd` the raw disk image to `disk0`.  That can be done with the following command.

dd if=/mnt/main/jails/virtualbox_jail/home/vbox/VirtualBox\ VMs/CrashPl
anBackup/CrashPlanBackup.img of=/dev/zvol/main/iohyve/crashplan/disk0

Make sure you replace `if=...` with the path to your raw image file and replace `of=...` to the path to your iohyve VM disk.  Once `dd` finishes, the VM should be ready to run.

Run Your VM

To start your VM run `iohyve start crashplan`.  Then run `iohyve console crashplan` to connect the console to the VM.  The console will display the output from the VM as it starts up.  When done you'll even be able to login to the VM through the console, but you'll probably want to connect via SSH instead.

Start VM at Boot

At this point your VM should be up and running, however if you were to restart your Freenas the VM would not be running on restart.  Fortunately it's pretty easy to make the VM start at boot.  The following instructions will make your VM start at boot.

• In the Freenas UI, go to System -> Tunables.
• Add a tunable.  Variable is `iohyve_enable`, value is `YES`, type is `rc.conf`. Make sure enabled is checked.
• Add a tunable.  Variable is `iohyve_flags`, value is `kmod=1 net=igb0` (where `igb0` is the name of your network device), type is `rc.conf`.  Also make sure enabled is checked.

These settings will enable iohyve and get it started.  From there, we need to tell iohyve to start our VM.  That can be done with the following command.

iohyve set crashplan boot=1

As with all the commands I've listed, replace `crashplan` with the name of your VM.

Clean Up

While not strictly necessary, you can now clean up the old VirtualBox template jail.  Simply delete the jail a that should be it.

Summary

With any luck you should now have your VirtualBox VM functional and running on Bhyve instead.  More work may be necessary if you've got a more complicated VirtualBox VM.  For example, if you have multiple disks, multiple NICs, a GUI or were using VirtualBox specific features like shared folders.  In that case, you may want to stay on VirtualBox until official support for Bhyve is ready in Freenas 10.