THIS ARTICLE IS A WORK IN PROGRESS.

You may consult it for ideas, but I strongly suggest you do not blindly follow it yet. Some of the scripts are not even the newest of their kind, I am searching my archives for their latest versions.

As my other setup options, the one described here is an "advanced" variant which may be cumbersome to set up, has its benefits and maybe drawbacks, and is not "required" for any and all usage scenarios (just for some).

What?

This article describes two related setups, the first one may be used by itself, and the second one builds on the first:

Please keep in mind that this page and example services are at this time posted from my notes and home-brewn hacks. In particular, the SMF integration can be remade in a prettier manner (i.e. each pool managed by a separate instance, etc.) I publish these "as is", and they will likely need customization for your systems. Maybe I (or other contributors) would later remake these pieces into a fine generalized redistributable work of art (smile)

Why?

There may be several reasons to pursue such a setup. I had a few to make it, in the first place:

Overall, one can point out several typical variants:

How?

Some of the steps below assume the following environment variables:

### Set the names for "physical" (or main for data) and "virtual" (stored in zvol on physical) pools
:; POOL_PHYS=pool
:; POOL_VIRT=dcpool

Again, note that at this moment the sample scripts and instructions are from a custom installation at my rig. Generalization may follow-up later. So far a curious reader would have to adapt the instructions.

Phase 1: ZFS pool as an SMF service

So, the first phase is rather trivial...

At this point, your data pool is imported not blindly by the OS, but by your service. Which you can disable in case of problems (at the moment this may require to boot into a livecd to create the block-file /etc/zfs/noimport-pool which would cancel the service startup, if for some reason an automatic creation and clearing of a block-file around the start/stop calls does not help).

Perhaps more importantly, you now have the main pool wrapped as an SMF resource on which other services can depend (or not depend) for orderly startup. If this pool takes very long to import and/or can fail in the process, it does not delay the startup of other services (like ssh), and you can monitor the state of the import as an SMF status with svcs or numerous remote-management tools.

Phase 2: iSCSI target (server)

Here we set up the zvol and share over iSCSI which would store "virtual" ZFS pool, named below dcpool for historical reasons (it was deduped inside and compressed outside on my test rig, so I hoped to compress only the unique data written).

TODO: find my notes on setup of the server – unfortunately, the HomeNAS itself is not currently available to look at... but there was (IIRC) not much different from usual COMSTAR iSCSI (with stmf). Enable services, create a backing store for a LU implemented as a zvol, allow localhost and/or remote hosts to access it.

One possible caveat is that the iSCSI server services should be made dependent on the main-pool-import service created above (assuming that it holds the zvol). If there are several physical pools, and others serve iSCSI too – a separate instance (or full service made as a replica) of iscsi/target may be in order, to wrap just the creation/teardown and sharing/unsharing of target(s) on the SMFized pool – see iscsi-lun-dcpool for an example (NOTE: hardcoded values would need adaptation for your systems). 

Phase 3: iSCSI initiator (client)

This phase involves wrapping of the iSCSI-mounted pool as an SMF service. Indeed, some readers who simply use remote iSCSI pools, might start reading the article here (wink)

First, there is the initiator part: the networking client. There is really no magic here, this service was needed just for peace of mind about not conflicting with system services (i.e. over OS upgrades) while I create specific dependency setups. It uses the same system logic as iscsi/initiator for actual work. Possibly, just one needs to be enabled at a time.

Second, prepare the mountpoint (also protected from modifications with immutability):

:; POOL=$POOL_VIRT
 
:; df -k /$POOL
:; ls -la /$POOL
### Make sure that the pool is exported and its mountpoint directory does not exist or is empty
 
:; mkdir /$POOL
:; /bin/chmod S+ci /$POOL

Third, set up the import of the pool over iSCSI (assuming that the COMSTAR initiator has been set up to query the needed target servers, and now zpool knows where to find iSCSI-backed vdevs):

Finally, I also had a watchdog service to actively monitor the viability of the "virtual" pool, as things tended to lock up once in a while, either due to internetworking faults or the experimental server problems. But that was too much of an in-house hack to publish at the moment (and relied on some currently proprietary status-testing methods).

HTH,
//Jim Klimov