My Illumos Development Workflow

Lately, I’ve been thinking about how to setup a development workspace to tinker with Illumos builds as they come rolling hot off github.  Ideally, I’d prefer to code on my laptop (macbook), build on my server (orion), and test on a virtual system (oi-dev-box).  Luckily, being mostly finished with our big move westward has afforded me some time to sit down and play with Illumos again.

I’ve been using OpenIndiana KVM for a little while now as my primary hyperviser at home and I’ve been very pleased with its ease of use and stability. (You know you’re a nerd if you ever use the phrase “primary hyperviser at home” seriously.)  I used to run systems in VirtualBox but the overhead associated with VirtualBox VMs was a little annoying.  KVM + ZFS is quick and I don’t have to bother with a lot of the VirtualBox-isms to manipulate and test simple VMs.

Yesterday, I browsed over to the Illumos build document to see if much had changed since the last time I logged in.  I have to say, “Wow!”  What a difference a few months makes.  The build document now contains an excellent write up on how to build Illumos on one system and test it on a different system.  This is a much easier method to test with than building on a local system, applying the update with onu, and rebooting.  Especially if you’re like me and your build server is the fastest machine in the house and you’ve got it doing more things than just building Illumos.

I setup my basic development environment like this:

1. [Orion] Clone Illumos source tree from github with git.
2. [Orion] Create an experimental branch on the build machine’s Illumos git repository.
3. [Macbook] Clone the experimental branch from the build server to the laptop.
4. [Macbook] Edit Illumos source.
5. [Macbook] Push the Illumos source back into the build system.
6. [Orion] Build the source.
7. [Orion] Make sure the package depot for your build is functional.
8. [OI-dev-box] Update with your latest changes.

I know that seems like a lot to create a build environment but much of it is only setup that just needs to be done once.  The actual workflow goes much faster once you’re familiar with the tools and process.  So, lets tackle each one of these points separately.

1. [Orion] Clone Illumos source tree from github with git.

This one is easy to take care of.  Simply download the Illumos source to a system you’re going to build with.  The Illumos build document has more information on how to prep and configure a machine to build Illumos.

orion$ git clone git://github.com/illumos/illumos-gate.git

2. [Orion] Create an experimental branch on the build machine’s Illumos git repository.

With git, branching and merging is dead simple and your coding life much easier.  It’s also easy to make patches of code you’ve just modified if you work out of your own branch. In fact, I generally create branches most of the time I am writing new code so I can either merge the branch or just delete it.  The first thing I do after cloning Illumos is create my own branch to tinker with:

orion$ git checkout -b experimental

You should now have two branches. A master and experimental. You can double check which one you’re currently using by executing:

orion$ git branch
* experimental
  master

3. [Macbook] Clone the experimental branch from the build server to the laptop.

Now that I’ve created an experimental branch on my build system, I need to clone the experimental branch to the system I plan to edit the source on.  Use an SSH key between the two systems since you’ll likely be committing changes to your build system often.  The following command will pull the experimental branch to your local system:

macbook$ git clone -b experimental ssh://orion/code/illumos-gate

4. [Macbook] Edit Illumos source.

I can now modify the illumos source code on my laptop with an editor of my choosing (macvim) and push the changes into my build system.  Here’s a trivial example that’s easy to test:

macbook$ cd illumos-gate
macbook$ vim usr/src/cmd/find/find.c

Add some silliness to find.c that’s easy to test. Something at the beginning of main() like:

fprintf(stderr, "I found find!\n");

5. [Macbook] Push the Illumos source back into the build system.

Now you need to add your modification to your repositories’ staged changes, check the status of the change, commit the change, and push it into your build machine.

macbook$ git add -u
macbook$ git status
macbook$ git commit -m 'a simple change to find'
macbook$ git push

The change has now been pushed into your build system on the experimental branch. To verify, log into your build machine and make sure you’re in the experimental branch.

orion$ cd illumos-gate
orion$ git checkout experimental
orion$ vim usr/src/cmd/find/find.c

Hopefully, your modification made it over to the build server. Now it’s time to build Illumos and test the change on a VM.

6. [Orion] Build the source.

I’m not going to go over setting up a build system for Illumos since the process is well documented in the Illumos build document.  You’ll want to make sure you’re on the experimental branch of the build system’s Illumos repository before starting, however.

orion$ cd illumos-gate
orion$ git checkout experimental
orion$ ./nightly.sh -i illumos.sh

7. [Orion] Make sure the package depot for your build is functional.

The Illumos build process packages the files you just compiled and linked into IPS images.  These images are then copied into a file repository on your build system.  If you share this repository across a network, any system using Illumos and IPS can update from it.

The easiest way to setup a network repository is to start the pkg.depotd listener manually and point it at your new Illumos file repository.  (If you’re going to do this a lot, you’ll want to setup the repo server in SMF as documented in the Illumos build document.)

orion$ sudo /usr/lib/pkg.depotd \ 
-d /code/illumos-gate/packages/i386/nightly/repo.redist/ -p 8151 \
--proxy-base http://"`hostname`.`domainname`":8151/on-nightly

In this example, I started the depotd listener on port 8151 and named the repository on-nightly.

8. [OI-dev-box] Update with your latest changes.

Next log into your test VM and create a new boot environment:

oi-dev-box$ sudo beadm create nightly

Now mount the boot environment to /a in the VM:

oi-dev-box$ sudo beadm mount nightly /a

Set your new repository as the primary publisher for your new boot environment:

oi-dev-box$ sudo pkg -R /a set-publisher -P -e --no-refresh \
-O http://orion:8151/on-nightly on-nightly

Downgrade the current publisher for the new boot environment:

oi-dev-box$ sudo pkg -R /a set-publisher --no-refresh \
--non-sticky openindiana.org

Refresh the packages in your boot environment:

oi-dev-box$ sudo pkg -R /a refresh --full

Uninstall the entire image so that you’re able to update piece parts of the Illumos system:

oi-dev-box$ sudo pkg -R /a uninstall entire

Now, update Illumos on the new boot environment!

oi-dev-box$ sudo pkg -R /a image-update

Once the new boot environment is done updating, activate it with:

oi-dev-box$ sudo beadm activate nightly

Reboot the machine. Hopefully, it boots! Now attempt to run your modified find command and see the results:

As long as you’re using this boot environment in your VM, you don’t have to follow all the steps above every time you want to make a change to your Illumos source. You only need to push the changes into your build machine, rebuild, and run pkg image-update on the VM.

This blog post will probably be updated in the future as I learn new ways to improve my development process.  Let me know if you have any trouble or questions with this process by either leaving comments or tweets at @Graftolistic.

Have fun with Illumos!