Child pages
  • Advanced - Split-root installation

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3


Another note: For other enhanced OI setup ideas see Advanced - Creating an rpool manually, Advanced - Manual installation of OpenIndiana from LiveCD mediaAdvanced - ZFS Pools as SMF services and iSCSI loopback mounts, Zones as SMF services or Using host-only networking to get from build zones and test VMs to the Internet. Not all of these articles are applicable and limited just to only what it says on the label (wink)


Code Block
:; cd "$BENEW_MPT"/var && \
   for D in adm cores crash log mail spool/clientmqueue spool/mqueue ; do \
     mkdir -p "$D" && /bin/chmod S+ci "$D"; \
     zfs create -o canmount=on "$RPOOL_SHARED/var/$D"; \
### Verify success of the previous operation(s) before proceeding
:; /bin/df -k | grep " $BENEW_MNT" 
:; for D in cores crash ; do \
     zfs set quota=5G "$RPOOL_SHARED/var/$D" ; \
     zfs set com.sun:auto-snapshot=false "$RPOOL_SHARED/var/$D" ; \
:; for D in spool/clientmqueue spool/mqueue ; do \
     zfs set quota=2G "$RPOOL_SHARED/var/$D" ; done

NOTE: Don't blindly split off /var/tmp like this, at least not unless you are ready to test this as much as you can. It was earlier known to fail, though it may work better now dependent on the distribution features, SMF dependency order and other such variables. It actually works on my system, but I am not ready to "guarantee" this for others. Since the problem was that in legacy setups some services wrote into this directory before the dedicated dataset was mounted (thus either blocking the mount, or losing access to written files), now there should be no problem since mounting is done before other services as enforced by SMF dependencies – unless you store your /var/tmp on a non-root pool and then that pool import fails at boot. If you do find that the temporary directories over dedicated ZFS datasets (whether as /var/tmp or in some differently-named paths perhaps stored on a separate user-data pool) work well for you, consider adding some security and performance options into the mix, for example:

Code Block
:; mkdir "$BENEW_MPT"/var/tmp && /bin/chmod S+ci "$BENEW_MPT"/var/tmp
:; zfs create -o canmount=on -o setuid=off -o devices=off -o sync=disabled -o atime=off "$RPOOL_SHARED"/var/tmp
:; chmod 1777 "$RPOOL_SHARED"/var/tmp
### Set quota or don't set it, as you see fit

The example above creates the immutable mountpoint directories in the rootfs hierarchy's version of /var, then creates and mounts the datasets into the new hierarchy's tree. Afterwards some typically acceptable quotas (YMMV) are set to protect the root file system from overfilling with garbage. Also, zfs/auto-snapshot service is forbidden to make autosnaps of the common space-hogs /var/cores and /var/crash, so that deletion of files from there to free up rpool can proceed unhindered.


  • -x – single-filesystem traversal (only copy objects from source filesystem, don't dive into sub-mounts – verify – you should manually verify and ensure that mountpoints like /tmp or /proc should ultimately exist on targets);
  • -avPHK – typical recursive replication with respect for soft- and hard-links and verbose reports;
  • -z – if you copy over a slow network link, this would help by applying compression to the transferred data (not included in examples below);
  • The rsync program is executed in a loop, so if something breaks (i.e. out of memory on LiveCD environment) it would pick up and proceed until success.


In order for your OS updates to enjoy the disk-space savings, the compression attributes should be appropriately applied to the cloned datasets. Unfortunately, current beadm clone does not take care of that. The most simple approach is to create and fix the new BE first, then use it as a target for the package upgrades.Note that

the two procedures below (scripted or explicitly manual) only process the global zone root filesystems, and does not clone the local zones for an upgrade.The three mini-chapters below go from the most-automated to the most-manual description of essentially the same procedure (inverse order of evolution as examples from this page got scripted). Generally the first snippet should be used in practice, while the others are more of interest for further development of audit of the procedure. The concluding mini-chapter covers destruction of such BEs as they become un-needed, because it also becomes slightly more complicated.

The environment variables involved in the procedures are or scripts below are similar to ones used in the manual above, but there are less of them set, since we are playing within one rpool (no networked copies are implied).

Script-automated rootfs/BE cloning and upgrading

The most automated help can be received from which automates BE cloning with (described below) and then issues IPS and PKGSRC package updates in the new BE:

  • For a fully-automatic job, download the scripts:

    Code Block
    :; wget -O /root/ "" && \
       chmod +x /root/
    :; wget -O /root/ "" && \
       chmod +x /root/
  • Run the upgrader (optionally pre-set and export the envvars described all around this text); the script prints the variables it is going to use and pauses before proceeding (press ENTER to go on):

    Code Block
    :; /root/ 

    If all was ok – activate (copy-paste the BE name from last lines of output of and gracefully reboot:

    Code Block
    :; beadm activate "$BENEW" && \
       init 6
Script-automated rootfs/BE cloning

The attached script (Git master: automates most of the logic described in the text below, and uses the same environment variables. You can execute it as a shell script as well as just "source" it into your current (root) shell – but beware that it can exit upon errors; execution requires that you "export" the envvars you need, while "sourcing" would set whatever remains to guesswork in the shell context which remains current and would not redefine them in subsequent runs.


In a second layer of usability it may suffice that you only set BEOLD and/or BENEW and it should guess the rest.

For just the BE cloning with the script do:

  • Download the script:

    Code Block
    :; wget -O /root/ "httphttps://wikigithub.openindiana.orgcom/download/attachments/27230229/jimklimov/illumos-splitroot-scripts/raw/master/bin/" && \
       chmod +x /root/
  • Source it into the current shell so it sets all the variables as it goes (by default it will propose a new BE name based on the first token of the current BE before a separator such as the dash character, and suffix it with current timestamp); the script prints the variables it is going to use and pauses before proceeding (press ENTER to go on):

    Code Block
    :; . /root/ 

    Alternately, don't source but rather run the script and copy-paste the reported variable values into your shell.

  • When the script is done cloning and has reported no errors, copy-paste the suggestions from the end of its output, i.e.:

    Code Block
    :; pkg -R "$BENEW_MNT" image-update --deny-new-be --no-backup-be && \
       touch "$BENEW_MNT/reconfigure" && \
       bootadm update-archive -R "$BENEW_MNT" && \
       beadm umount "$BENEW"
    :; TS="`date -u "+%Y%m%dZ%H%M%S"`" && \
       zfs snapshot -r "$RPOOL_SHARED@postupgrade-$TS" &&\
       zfs snapshot -r "$BENEW_DS@postupgrade-$TS"
  • If all was ok – activate and gracefully reboot:

    Code Block
    :; beadm activate "$BENEW" && \
       init 6

    Hopefully, everything goes up nicely and quickly, and a `df -k /` would show the new root dataset (wink)


  • The official method is beadm activate which updates the GRUB menu and possibly does other housekeeping; when it is done, you should gracefully(warning) reboot (when time comes):

    Code Block
    :; beadm activate "$BENEW"
    Activated successfully
    :; init 6

    In particular, the updated GRUB menu entries allow you to easily fall back and boot an older BE, without hacking at the console to enter the bootfs you want as active. 

  • Still, if you are oldschool and rely on default bootfs (referenced from GRUB menu as the default choice), just update it and reboot (when time comes), and hope that this suffices (smile) in the release du-jour:

    Code Block
    :; zpool set bootfs="$BENEW_DS" "$RPOOL"
    :; init 6

Good luck! (smile)

Removing a BE prepared by the above procedure

It is perfectly possible that you don't get everything the way you wanted on the first attempt, and would like to retry. An update attempt might not find any packages to update and the new BE is thus useless. 

In these or any similar cases you should use the "-s" flag to beadm destroy because after the procedure above (and/or after some life-time on a system with Time-Slider or equivalent technology), the new BE contains several snapshots which block "normal" removal:

Code Block
:; beadm destroy -s "$BENEW"

If an update was successful and well-tried in practice, so you no longer need an old BE... do be careful in its removal:


WARNING: Before doing recursive ZFS removals (which is what beadm destroy -s should be doing), remember that this action can impact all child datasets that are not blocked by being mounted and files open, by running zfs send sessions or by a zfs hold, for example. Beside the snapshots and sub-datasets in the hierarchy which you do intend to remove, such "children" may include ZFS clones such as newer BE's.

There is a difference between zfs destroy -r and zfs destroy -R commands that lies essentially just in this aspect – whether clones are also removed.

Do verify first what your particular OS distribution and version does to destroy old BEs, or resort to destruction of datasets snapshot-by-snapshot (and mind that beadm destroy dataset@snapshot syntax does offer a means to automate that). Alternately, consider using zfs promote to ensure that a newer clone is considered to be the master (inspect zfs list -o origin,name -r rpool/ROOT output to see the current relationships between datasets on your system).

On simply using "pkg upgrade"

Unfortunately, if you've issued a simple pkg upgrade call which results in a cloned BE automatically (due to package flags requiring a reboot), the new BE would currently have default compression and other per-dataset settings. Still, you have a chance to catch the new BE and ifx fix it as early as you can "in flight".