In this writeup I'll try to document how to use CrossBow virtual networking and some other components to create a LAN-in-a-box.
We will set up internet access for local zones (LZ), such as when you're Building in zones, for cases when your corporate or home LAN does not provide for these zones' individual network addresses and only your machine's global zone (GZ) has an "external" IP address assigned. The same setup can be used for your testbed virtual machines, as well as for easier creation of networked local zones on an OI installation implemented as a VM in VirtualBox (which sets some limitations on networking choices).
Here are the major steps:
It is arguable whether the building local zones need the network access at all, but it might be needed for example if you maintain a golden source-code repository in the global zone and want to distribute the changesets to local zones' cloned workspaces (see Building in zones, Working on several bugs at once and Making a LAN mirror of global package repositories), or if you want to upload webrevs straight from your build zones. YMMV, this here is just a technology POC
It is also worth noting that some best-practice setups use a similar approach, with a dedicated local zone being the router/firewall/NAT and it has the external IP address, and the global zone "hypervisor" being one of the privately-addressed nodes which have the internet access through such local zone. In particular, this may be the only way to have
ip-type=shared local zones to access the external network using NAT through the same box. It is not difficult to transition from the setup described below to such a best-practice one by swapping some commands and VNICs (Virtual Network Interface Cards), though the adventure is left for the curious and won't be detailed in this page.
In particular, this setup will focus on:
in.dhcpd) with text-file storage format;
(So far) this document does not deal with configuration of naming services, though some pointers can be given anyway
One solution is to use a private DNS server on the host machine for the zones to know each other's names and private IP addresses, which would forward requests for other names to the LAN DNS servers. See Solaris DHCP Administration Guide Chapter 4 for details on enabling the registration of DHCP clients in the ISC BIND server by the Sun DHCP server, and optionally allowing the DHCP client to push its desired hostname via DHCP server to DNS.
An alternate useful configuration might be to use DNS for remote hostname look-ups, and allow the DHCP server to update its host's
/etc/hosts file with information about client hostnames, but this is not explored here either.
In this page it is assumed that the local zones receive their default hostnames via
zoneadm install (
`zonename` or ultimately
/etc/nodename), and can be configured to use external DNS servers for internet/LAN access, and they can address each other and their GZ via
/etc/hosts file if that is at all needed.
For the purposes of the setup described below:
192.168.127.0/24, on a dedicated etherstub, with
In many of the examples below, samples of commands to run as "
root" (or via
pfexec) are prefixed with a "
Basically this prepends a call of "
true" to the sample command and does nothing, unlike the typically used "hash" (
#) prompt which acts as a comment and precludes the copy-pasted command blocks from running.
This chapter focuses on creating the host-only network, which allows using a private IP address range for local zones to communicate with each other and with the global zone.
This by itself does not provide communications with the outside world, and you may have your own reasons to stop the network configuration at that point, and proceed to setting up the local zone(s). In that case the local zones would only be able to access each other and the global zone, using the VNICs connected to a common "etherstub".
This provides a host-only Ethernet segment, acting like a virtual switch, named "
The first line defines a VNIC for GZ named "
vnic127001" with a random MAC address, attached to the virtual switch created above.
The second line creates a VNIC for an LZ, attached to the same switch with a predefined MAC address, others can be defined similarly.
Here I use a simple naming scheme, with
127 standing for the subnet and
001 standing for the static IP address (
192.168.127.1) that will be assigned to the VNIC for the global zone.
101 characters in the
vnic127101 name and
00:12:71:27:01:01 in the MAC address stand for the IP address (
192.168.127.101) that will be assigned to the VNIC via DHCP – fixed addressing, even if distributed with dynamic methods, is useful for setting up
/etc/hosts and the publishing of services via NAT.
A simple construct like the one below would pre-create some 9 VNICs to this pattern quickly:
The first two lines take care of saving the VNIC addressing for the future. The last line brings up the new IPv4 interface and sets up its addressing.
Quick check (note that etherstub default MTU is 9000, equivalent of Jumbo Frames):
First of all, the DHCP server software should be installed. In this example I'll use Sun BOOTP/DHCP server software (alternately ISC DHCP can be used, for example).
binfilesstorage backend for DHCP configuration to be available, but I find inspecting and manipulating text files to be easier
The management of the server can be done in a couple of ways: properly via command-line tools including
/usr/sbin/dhtadm, or with the GUI applet
/usr/sadm/admin/bin/dhcpmgr (also symlinked as
/usr/sbin/dhcpmgr), and somewhat improperly by manipulating the config files (
/etc/inet/dhcpsvc.conf) and DHCP text-formatted database files
/var/dhcp/SUNW* directly – this is not encouraged, but is often faster, more straightforward, and "just plain works"TM.
First of all, to enable the DHCP server, its configuration file should exist and say that the server is enabled (for more details see init-script
/etc/init.d/dhcp which wraps the SMF service).
In the example above, the
-d option sets the local domain name, and the
-a option sets the DNS servers to be used (that would likely be your LAN's DNS server address, or
192.168.127.1 if you set up a DNS server instance on this box). We've also requested to use the less performant but more convenient text-formatted database files "resource".
The wizard also took some freedom to add a "server macro" based on
`hostname` and IP addressing of this zone, which may be acceptable or invalid for our uses – we'll see.
You would likely want the DHCP server to log its events, errors and other messages into
syslog to ease the debugging.
Enable logging on the DHCP server:
Note: You can use any local
syslog facility ranging from 0 to 7.
Add this line to
/etc/syslog.conf to enable saving of these messages into a particular file (NOTE: the two parts must be separated by TAB characters):
Note that other
local0 events would get into this log file, like
ipmon does on my system. Well, while debugging THIS setup, it is only a bonus to have the two logs co-located
Now you can touch the file, and restart the
syslog does not create log files by itself, and complains if one is not present at the moment of the daemon's startup or restart.
Ultimately, enable log rotation to restrain the disk space requirements:
This uses the currently default log rotation engine
logadm regularly called from
cron; if you use something else (
logrotate.d, etc.) – configure that engine appropriately.
Here are some sample
syslogged entries about DHCP events:
Now that we have a server running, we can define a subnet macro:
The macros are used to basically group some settings for a client, and can be nested, as we'll do below.
Under The Hood
Let's see under the hood – what have we defined in the previous step?
The first two macros were predefined by initial
dhcpconfig based on what it knows about the naming and networking setup of the host and on our command-line options, while the last one was added by us explicitly.
/var/dhcp/SUNWfiles1_192_168_127_0 file is currently empty (comments only), but it will soon contain the definitions which bind IP addresses, MAC addresses and DHCP Macros for particular clients.
Now, combine (nest) the macros into one DHCP profile which can be used to configure the clients:
NOTE: This implies the
Timeserv value to be the GZ public address, but since local zones don't have individual clocks – we don't care. This can be a problem however if the DHCP clients include different machines which have clocks and can't reach the router's external address, or are zones which host VMs; in this case the
`hostname`-based macro (the
openindiana above) should be redefined, or another similar one can be defined and
Included by our
localstub127 macro instead. In the latter case I'd make several macros – for time servers, for DNS servers, etc. and combine them all as suitable for a particular client profile.
More information about Sun DHCP macros can be found here:
Some client addresses can be predefined, following our IP/MAC-address naming pattern explained above. This simple construct would create some 9 entries for the VNICs made in examples above:
Here we define the following
-A– the client IP address (or hostname known via
-i– the client identifier (MAC address prefixed by the ARP code for the type of network, usually
-s– DHCP server responsible for the leases;
-m– the client macro for common settings;
-f– the flag describing the lease type (
01= PERMANENT lease; announced via DHCP and never expires to be recycled for other clients);
-c– an optional comment describing this client;
IMPORTANT NOTE about the OWNERIP construct used above and below: The Sun DHCP server is identified by the numeric IP address (i.e.
192.168.1.40 from the external network, for our example) which corresponds to the textual hostname (i.e.
openindiana), which in turn corresponds to this host's name set in
/etc/nodename. That is, even though you serve addresses on a
192.168.127.0/24 subnet, the "owner" of these leases must be
192.168.1.40 if that address corresponds to the nodename as resolvable via
/etc/hosts (or possibly DNS too). Otherwise this instance of the DHCP server will refuse to issue addresses "owned" by another server (even if the provided value resolves to this same machine).
Also keep this in mind if you ever rename the GZ
hostname and/or change its "primary" IP address.
You can also use these commands (likely with different options) to reserve some IP addresses for dynamic leasing for not-predefined hosts, i.e. to quickly provision VMs or test zones. You might use a different macro (i.e. with shorter
LeaseTim) in this case, as well.
For example, a range of
192.168.127.201-209 will be defined here, leasing addresses for 30 minutes to any client (except those who already have a reservation for the same MAC address):
You might want to use a local mirror of the OpenIndiana package repository hosted in the global zone, as detailed in Making a LAN mirror of global package repositories.
However, after completing the procedure described on that page, you might need to spawn another instance of the package depot service to listen on the more-specific IP address (than the default
0.0.0.0) and do HTTP-redirects to it, with these commands:
NOTE that the default listener
dev created in that procedure would now depend on this new service instance
dev127 can fail to start due to its own internal checks (technically:
pkg.depotd checks if it can connect to the specified
address:port first, and the default listener of
dev is considered a busy port by
dev127). Any restarts of
dev127 should cause shutdown and restart of
dev as well, for the same reasons.
These steps allow networking clients in the LZ to contact the outside world and allow networking servers in the LZ to be accessible from the outside world.
You can set up "forwarding" and "routing" in Solaris with
routeadm. Forwarding is the process of relaying packets not originated by nor destined to this host's IP addresses, from one interface to another. Routing encompasses special daemons which manage the kernel's routing table dynamically and/or announce this box as a router. For our purposes, we only need forwarding enabled (and we don't want to announce our host-only subnet to the external LAN).
Enable IPv4 packet forwarding and update the running configuration:
Here we can see that the interfaces got a
ROUTER flag enabled:
To enable NAT we need to set up translation rules in
/etc/ipf/ipnat.conf and allow connections on the packet filter in
/etc/ipf/ipf.conf. Rough-cut examples follow.
Also note that local zones with
ip-type=exclusive can define and use their own firewall settings, although it won't likely be required for the setup we're making here.
maprules above and
rdrrules should automagically substitute the interface's active address(es); this is a mode originally used for NATing by hosts with dial-up or otherwise dynamic external addresses.
group" rules at the end of the file. The default action of these groups is to
logthe packets (so you can trace them with
blockthem, denying the communication. However, the defined actions above simply allow all packets to enter and exit the external interface. If you further tune the firewall, these rules should be commented out and replaced by specific permissions – and don't forget to allow your SSH
Have your console access ready, just in case you forbid yourself the pleasures of networking with the host!
Alternately, you can use
screen or VNC to set a timer which would "
(sleep 20; svcadm disable ipfilter)&" – just in case...
First of all, reconfigure the
ipfilter SMF service to use traditional configuration methods (with the manually created files instead of rule snippets added by some services):
Apply the settings – but first see if
ipfilter service is running and enable or restart it accordingly:
Test if the settings got applied:
ipfstat -hionoutput can differ on your machine, it is the rule-hit count.
If you initially did just a host-only network and stopped at that, you might not have configured a router announcement in the macros, or the setting was/became invalid.
Make sure that this host (
192.168.127.1) is now announced as the router for the host-only network clients, per DHCP macro examples above.
The local zone setup is explained in more detail in other documents, such as Building in zones.
For the purposes of this document, we only overview the creation of the zone and the tests that it has network access both ways.
You might have set up some VNICs in the examples above. If you ran out of these addresses, follow the procedures above to add a VNIC and a DHCP reservation for its address.
/etc/ipf/ipnat.conf rules if you need to publish something (i.e. SSH access) as you see fit, and
svcadm restart ipfilter in this case.
NOTE: Be certain to use the VNIC name, zone name and zone path which you want and not the ones you've copy-pasted from this example
Beside installing the zone, we will use a
/etc/sysidcfg file to predefine the local zone's environmental configuration, so that the zone (and its possible clones) gets configured automatically.
Install the local zone:
NOTE about using a local package mirror
If you are Making a LAN mirror of global package repositories on this machine's global zone, you might want to use that for LZ installation and subsequent upgrades, and bypass an HTTP proxy server that you might have configured in your environment. In that case you'd use the command like this for zone installation, instead of the one given above:
... or likewise for the repo hosted on another box in your LAN – using the LAN's IP address (
192.168.1.40 in our example). The remainder of the procedures below remains as described.
/etc/sysidcfg file with environmental presets for the zone (for hands-free boots), boot it and monitor how it goes:
Unfortunately, I did not find a way for
sysidcfg to pick up DNS settings from DHCP, so they are to be either pushed explicitly with
name_service=DNS... configuration with a block like this:
... or with
name_service=NONE and a hack described below in order to allow local zones to use the DNS settings from DHCP properties.
It seems that the ability for a local zone to get DNS settings from DHCP is crippled explicitly in
Commenting away the block for
smf_is_nonglobalzone (in the local zone's copy of the script) and doing
svcadm restart svc://network/service:default does the trick for auto-generation of
/etc/resolv.conf and fixup of
/etc/nsswitch.conf, but further impact is unknown, so this is for now a "use at your own risk" kind of workaround. Comments requested on mailing list.
More information about
/etc/sysidcfg options can be found here:
Note that with the
/etc/sysidcfg file in place, the local zone proceeds to auto-configuration and reboot (which may take a minute of it seemingly hanging).
If you get errors like: "
Could not contact a dhcp server on the network interface vnic127101" or "
in.rdisc: No interfaces up", possibly after several minutes of waiting, double-check the DHCP server and firewall settings. In particular, check that the "owner" of the addresses is set correctly in the dhcptab (see details above).
If the local zone does not receive an IP address, you can try to see if it even tries:
There may be several repetitions of these lines in varied order. If the server does not give it the address, there would only be repetitions of the
DHCPREQUEST line every few seconds.
You may also want to check the services and configuration from inside the local zone.
While the zone has not yet been configured, direct attempts to
zlogin into it can fail, because the
root user is not yet activated:
In this case, use an administrative workaround:
Now you can see what the zone has configured for itself by using
ifconfig -a, what it is running with
ps -ef, the state of services with
svcs and so on.
Now that your local zone seems to be configured and running,
zlogin into it and see what networking settings it has grabbed:
Rough outline of the routine, detailed steps are in the chapters above:
ipnat.confin the GZ;
Test accessibility of the service from your external LAN host (different from the routing GZ):
If you rename the VNICs dedicated to a zone, such as when you clone a working zone and give it a new identity (in the example below I'll be changing 101 to 109), keep in mind the following configuration files:
/etc/hostname.vnic* – empty file, its existence notifies the OS that the named interface should get "plumbed" at startup.
If this file is not empty, it can contain additional parameters for
ifconfig, such as static addressing or MTU size – in case of DHCP we likely want it empty.
/etc/dhcp.vnic* – empty file, notifies the OS that the named interface should get its settings via DHCP.
/etc/hosts – matches IP addresses to textual names in absence of DNS or a different naming server; one of the lines should be the current host's name and address. The entry should be changed, i.e.
Without this things that reference the current host can fail, such as
sendmail and some
/etc/nodename – if you've also changed the local zone's name (i.e. while cloning it), you want to change its perception of its name.
SSH keys – to be pedantic, you might want each cloned zone to use different SSH server keys, though this is not technically required.
Now you can restart (
init 6) the cloned/renamed zone to apply the new settings. Simple
svcadm restart p
hysical:default can also suffice for networking setup.