Intro
In this writeup I'll try to document how to use CrossBow virtual networking and some other components to create a LAN-in-a-box.
We will set up internet access for local zones (LZ), such as when you're Building in zones, for cases when your corporate or home LAN does not provide for these zones' individual network addresses and only your machine's global zone (GZ) has an "external" IP address assigned. The same setup can be used for your testbed virtual machines, as well as for easier creation of networked local zones on an OI installation implemented as a VM in VirtualBox (which sets some limitations on networking choices).
Here are the major steps:
What are we talking about?
It is arguable whether the building local zones need the network access at all, but it might be needed for example if you maintain a golden source-code repository in the global zone and want to distribute the changesets to local zones' cloned workspaces (see Building in zones, Working on several bugs at once and Making a LAN mirror of global package repositories), or if you want to upload webrevs straight from your build zones. YMMV, this here is just a technology POC 
It is also worth noting that some best-practice setups use a similar approach, with a dedicated local zone being the router/firewall/NAT and it has the external IP address, and the global zone "hypervisor" being one of the privately-addressed nodes which have the internet access through such local zone. In particular, this may be the only way to have ip-type=shared local zones to access the external network using NAT through the same box. It is not difficult to transition from the setup described below to such a best-practice one by swapping some commands and VNICs (Virtual Network Interface Cards), though the adventure is left for the curious and won't be detailed in this page.
In particular, this setup will focus on:
- distributing local zones' IP addresses with a private DHCP server (so that you can quickly provision new zones); in this example I'll use Sun DHCP (aka
in.dhcpd) with text-file storage format; - getting outgoing communications NAT'ed from local zones' private IP addresses through the global zone using its public IP address in the external network (which can still be an "unroutable" one used by your physical LAN);
- it is also possible to publish certain services, such as SSH, using reverse NAT with the global zone's public IP address.
DHCP host-naming
(So far) this document does not deal with configuration of naming services, though some pointers can be given anyway 
One solution is to use a private DNS server on the host machine for the zones to know each other's names and private IP addresses, which would forward requests for other names to the LAN DNS servers. See Solaris DHCP Administration Guide Chapter 4 for details on enabling the registration of DHCP clients in the ISC BIND server by the Sun DHCP server, and optionally allowing the DHCP client to push its desired hostname via DHCP server to DNS.
An alternate useful configuration might be to use DNS for remote hostname look-ups, and allow the DHCP server to update its host's /etc/hosts file with information about client hostnames, but this is not explored here either.
In this page it is assumed that the local zones receive their default hostnames via zoneadm install (`zonename` or ultimately /etc/nodename), and can be configured to use external DNS servers for internet/LAN access, and they can address each other and their GZ via /etc/hosts file if that is at all needed.
The example network layout
For the purposes of the setup described below:
- the host-only network will be
192.168.127.0/24, on a dedicated etherstub, with - the global zone having an address of
192.168.127.1, and - the local zones use "exclusive" IP stacks with dedicated VNICs and individually manage their networking settings (via DHCP in this case);
- determinism in IP addressing (and private naming) will be provided by special setup of the VNICs' properties and the DHCP server;
- the host's assigned "external IP address" in the physical LAN is
192.168.1.40on ae1000g0interface; - the examples only deal with IPv4 addressing.
Typographic note
In many of the examples below, samples of commands to run as "root" (or via pfexec) are prefixed with a ":;" prompt.
Basically this prepends a call of "true" to the sample command and does nothing, unlike the typically used "hash" (#) prompt which acts as a comment and precludes the copy-pasted command blocks from running.
The host-only network setup
This chapter focuses on creating the host-only network, which allows using a private IP address range for local zones to communicate with each other and with the global zone.
This by itself does not provide communications with the outside world, and you may have your own reasons to stop the network configuration at that point, and proceed to setting up the local zone(s). In that case the local zones would only be able to access each other and the global zone, using the VNICs connected to a common "etherstub".
Create an etherstub (a virtual Ethernet switch)
:; dladm create-etherstub localstub127
This provides a host-only Ethernet segment, acting like a virtual switch, named "localstub127".
Create some virtual NICs attached to the etherstub
:; dladm create-vnic -l localstub127 vnic127001 :; dladm create-vnic -l localstub127 -m 00:12:71:27:01:01 vnic127101
The first line defines a VNIC for GZ named "vnic127001" with a random MAC address, attached to the virtual switch created above.
The second line creates a VNIC for an LZ, attached to the same switch with a predefined MAC address, others can be defined similarly.
Here I use a simple naming scheme, with 127 standing for the subnet and 001 standing for the static IP address (192.168.127.1) that will be assigned to the VNIC for the global zone.
Similarly the 101 characters in the vnic127101 name and 00:12:71:27:01:01 in the MAC address stand for the IP address (192.168.127.101) that will be assigned to the VNIC via DHCP – fixed addressing, even if distributed with dynamic methods, is useful for setting up /etc/hosts and the publishing of services via NAT.
A simple construct like the one below would pre-create some 9 VNICs to this pattern quickly:
:; for IP in 01 02 03 04 05 06 07 08 09; do \
dladm create-vnic -l localstub127 -m 00:12:71:27:01:${IP} vnic1271${IP}; done
Set up the GZ's VNIC
:; grep "192\.168\.127\." /etc/netmasks || \ echo "192.168.127.0 255.255.255.0" >> /etc/netmasks :; ipadm create-if vnic127001 && \ ipadm create-addr -T static -alocal=192.168.127.1 vnic127001/v4 ### Legacy ways: # echo "192.168.127.1" > /etc/hostname.vnic127001 # ifconfig vnic127001 plumb 192.168.127.1/24 up
The first two lines take care of saving the VNIC addressing for the future. The last line brings up the new IPv4 interface and sets up its addressing.
Quick check (note that etherstub default MTU is 9000, equivalent of Jumbo Frames):
:; ifconfig vnic127001
vnic127001: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9000 index 3
inet 192.168.127.1 netmask ffffff00 broadcast 192.168.127.255
ether 2:8:20:87:e4:43
Set up DHCP server in the GZ
Install Sun DHCP server software
First of all, the DHCP server software should be installed. In this example I'll use Sun BOOTP/DHCP server software (alternately ISC DHCP can be used, for example).
- Install Sun DHCP server package:
:; pkg install service/network/dhcp :; svccfg import /lib/svc/manifest/network/dhcp-server.xml :; svcs -a | grep -i dhcp disabled 16:51:37 svc:/network/dhcp-server:default
- Optionally, you might want the
binfilesstorage backend for DHCP configuration to be available, but I find inspecting and manipulating text files to be easier
Just in case, here's the optional command:
- Optionally, you might want the
:; pkg install service/network/dhcp/datastore/binfiles
- Optionally, if you'd prefer to use the X11 GUI to administer DHCP, there is a Java applet for that (and some further optional localizations which I skip):
:; pkg install service/network/dhcp network/dhcp/dhcpmgr
The management of the server can be done in a couple of ways: properly via command-line tools including /usr/sbin/dhcpconfig, /usr/sbin/pntadm and /usr/sbin/dhtadm, or with the GUI applet /usr/sadm/admin/bin/dhcpmgr (also symlinked as /usr/sbin/dhcpmgr), and somewhat improperly by manipulating the config files (/etc/inet/dhcpsvc.conf) and DHCP text-formatted database files /var/dhcp/SUNW* directly – this is not encouraged, but is often faster, more straightforward, and "just plain works"TM.
Configure Sun DHCP server software
First of all, to enable the DHCP server, its configuration file should exist and say that the server is enabled (for more details see init-script /etc/init.d/dhcp which wraps the SMF service).
- Create the initial configuration:
:; dhcpconfig -r SUNWfiles -p /var/dhcp -D -d `domainname` -a 8.8.8.8 -l 86400 Created DHCP configuration file. Created dhcptab. Added "Locale" macro to dhcptab. Added server macro to dhcptab - openindiana. DHCP server started. :; cat /etc/inet/dhcpsvc.conf DAEMON_ENABLED=TRUE RESOURCE=SUNWfiles RUN_MODE=server PATH=/var/dhcp CONVER=1
In the example above, the -d option sets the local domain name, and the -a option sets the DNS servers to be used (that would likely be your LAN's DNS server address, or 192.168.127.1 if you set up a DNS server instance on this box). We've also requested to use the less performant but more convenient text-formatted database files "resource".
The wizard also took some freedom to add a "server macro" based on `hostname` and IP addressing of this zone, which may be acceptable or invalid for our uses – we'll see.
- In our case we'll want to bind the server to only work for the host-only network (and the broadcasts, as seen below):
:; echo "INTERFACES=vnic127001" >> /etc/inet/dhcpsvc.conf
:; /etc/init.d/dhcp stop; /etc/init.d/dhcp start
:; svcs -p dhcp-server
STATE STIME FMRI
online 17:44:51 svc:/network/dhcp-server:default
17:44:51 26974 in.dhcpd
:; netstat -an | grep -w 67
255.255.255.255.67 Idle
192.168.127.0.67 Idle
192.168.127.1.67 Idle
Optional: Configure Sun DHCP server logging
You would likely want the DHCP server to log its events, errors and other messages into syslog to ease the debugging.
Enable logging on the DHCP server:
:; echo "LOGGING_FACILITY=0" >> /etc/inet/dhcpsvc.conf :; svcadm restart dhcp-server
Note: You can use any local
syslogfacility ranging from 0 to 7.Add this line to
/etc/syslog.confto enable saving of these messages into a particular file (NOTE: the two parts must be separated by TAB characters):local0.notice /var/log/dhcpsvc
Note that other
local0events would get into this log file, likeipmondoes on my system. Well, while debugging THIS setup, it is only a bonus to have the two logs co-locatedNow you can touch the file, and restart the
syslog::; touch /var/log/dhcpsvc :; svcadm restart system-log
Note that
syslogdoes not create log files by itself, and complains if one is not present at the moment of the daemon's startup or restart.Ultimately, enable log rotation to restrain the disk space requirements:
:; cat << EOF >> /etc/logadm.conf ### Rotate DHCP/ipmon logs /var/log/dhcpsvc -C 4 -s 1m -a '/usr/sbin/svcadm refresh system-log' EOF
This uses the currently default log rotation engine
logadmregularly called fromcron; if you use something else (newsyslog,logrotate.d, etc.) – configure that engine appropriately.
Here are some sample syslogged entries about DHCP events:
Example syslog entries for DHCP leases
Jun 6 23:48:26 openindiana in.dhcpd[6041]: [ID 771465 local0.notice] DHCP ASSIGN 1339012106 -000000001 192.168.127.101 192.168.1.40 001271270101 N/A 001271270101 Jun 7 14:24:05 openindiana in.dhcpd[6041]: [ID 771465 local0.notice] DHCP EXTEND 1339064645 -000000001 192.168.127.101 192.168.1.40 001271270101 N/A 001271270101
Define our DHCP subnet settings
Now that we have a server running, we can define a subnet macro:
:; dhcpconfig -N 192.168.127.0 -m 255.255.255.0 -t 192.168.127.1 Added network macro to dhcptab - 192.168.127.0. Created network table.
The macros are used to basically group some settings for a client, and can be nested, as we'll do below.
Under The Hood
Let's see under the hood – what have we defined in the previous step?
:; ls -la /var/dhcp/SUNW* -rw-r--r-- 1 root root 104 Jun 6 17:51 /var/dhcp/SUNWfiles1_192_168_127_0 -rw-r--r-- 1 root root 385 Jun 6 17:51 /var/dhcp/SUNWfiles1_dhcptab :; cat /var/dhcp/SUNWfiles1_dhcptab # SUNWfiles1_dhcptab # # Do NOT edit this file by hand -- use dhtadm(1M) or dhcpmgr(1M) instead # Locale|m|4785356079057862657|:UTCoffst=10800: openindiana|m|10017694421132247041|:Include=Locale:Timeserv=192.168.1.40:LeaseTim=86400:LeaseNeg:DNSdmain="domain.com":DNSserv=8.8.8.8: 192.168.127.0|m|2030560481990672385|:Subnet=255.255.255.0:Router=192.168.127.1:Broadcst=192.168.127.255:
The first two macros were predefined by initial dhcpconfig based on what it knows about the naming and networking setup of the host and on our command-line options, while the last one was added by us explicitly.
The /var/dhcp/SUNWfiles1_192_168_127_0 file is currently empty (comments only), but it will soon contain the definitions which bind IP addresses, MAC addresses and DHCP Macros for particular clients.
Now, combine (nest) the macros into one DHCP profile which can be used to configure the clients:
:; dhtadm -A -m localstub127 -d ":Include=192.168.127.0:Include=`hostname`:" :; tail -1 /var/dhcp/SUNWfiles1_dhcptab localstub127|m|593912200859484161|:Include=192.168.127.0:Include=openindiana:
NOTE: This implies the Timeserv value to be the GZ public address, but since local zones don't have individual clocks – we don't care. This can be a problem however if the DHCP clients include different machines which have clocks and can't reach the router's external address, or are zones which host VMs; in this case the `hostname`-based macro (the openindiana above) should be redefined, or another similar one can be defined and Included by our localstub127 macro instead. In the latter case I'd make several macros – for time servers, for DNS servers, etc. and combine them all as suitable for a particular client profile.
More information about Sun DHCP macros can be found here:
- Solaris 10 10/09 Installation Guide: Network-Based Installations: Chapter 3
- System Administration Guide: IP Services: Chapter 18 (DHCP Reference)
Define fixed client addresses
Some client addresses can be predefined, following our IP/MAC-address naming pattern explained above. This simple construct would create some 9 entries for the VNICs made in examples above:
:; OWNERIP=$(getent hosts `hostname` | awk '{print $1}')
:; for IP in 01 02 03 04 05 06 07 08 09; do \
pntadm -A 192.168.127.1${IP} -i 010012712701${IP} \
-s ${OWNERIP} -m localstub127 -f 01 \
-c "Build zone on localstub127" 192.168.127.0; done
Here we define the following pntadm options:
-A– the client IP address (or hostname known via/etc/hosts, technically);-i– the client identifier (MAC address prefixed by the ARP code for the type of network, usually01for Ethernet);-s– DHCP server responsible for the leases;-m– the client macro for common settings;-f– the flag describing the lease type (01= PERMANENT lease; announced via DHCP and never expires to be recycled for other clients);-c– an optional comment describing this client;- The final parameter is the subnet name (as it is known by the DHCP server).
IMPORTANT NOTE about the OWNERIP construct used above and below: The Sun DHCP server is identified by the numeric IP address (i.e. 192.168.1.40 from the external network, for our example) which corresponds to the textual hostname (i.e. openindiana), which in turn corresponds to this host's name set in /etc/nodename. That is, even though you serve addresses on a 192.168.127.0/24 subnet, the "owner" of these leases must be 192.168.1.40 if that address corresponds to the nodename as resolvable via /etc/hosts (or possibly DNS too). Otherwise this instance of the DHCP server will refuse to issue addresses "owned" by another server (even if the provided value resolves to this same machine).
Also keep this in mind if you ever rename the GZ hostname and/or change its "primary" IP address.
Optionally reserve random client addresses
You can also use these commands (likely with different options) to reserve some IP addresses for dynamic leasing for not-predefined hosts, i.e. to quickly provision VMs or test zones. You might use a different macro (i.e. with shorter LeaseTim) in this case, as well.
For example, a range of 192.168.127.201-209 will be defined here, leasing addresses for 30 minutes to any client (except those who already have a reservation for the same MAC address):
:; OWNERIP=$(getent hosts `hostname` | awk '{print $1}')
:; dhtadm -A -m localstub127rnd -d \
':Include=192.168.127.0:Include=Locale:LeaseTim=1800:LeaseNeg:DNSdmain="'"`domainname`"'":DNSserv=8.8.8.8:'
:; for IP in 01 02 03 04 05 06 07 08 09; do \
pntadm -A 192.168.127.2${IP} -i 00 \
-s ${OWNERIP} -m localstub127rnd -f 00 \
-c "Quick-test hosts on localstub127" 192.168.127.0; done
(Optional) Prepare a package repository listener on the private network
You might want to use a local mirror of the OpenIndiana package repository hosted in the global zone, as detailed in Making a LAN mirror of global package repositories.
However, after completing the procedure described on that page, you might need to spawn another instance of the package depot service to listen on the more-specific IP address (than the default 0.0.0.0) and do HTTP-redirects to it, with these commands:
:; svccfg -s pkg/server add dev127 select dev127 addpg pkg application setprop pkg/inst_root = astring: "/export/pkg/dev/" setprop pkg/port = count: 10002 setprop pkg/address = net_address: 192.168.127.1 setprop pkg/proxy_base = astring: http://192.168.127.1:10002 exit :; svccfg -s pkg/server select dev addpg dev dependency setprop dev/entities = fmri: svc:/application/pkg/server:dev127 setprop dev/grouping = astring: optional_all setprop dev/restart_on = astring: restart setprop dev/type = astring: service exit :; svcadm refresh pkg/server:dev127 pkg/server:dev :; svcadm disable -st pkg/server:dev127 pkg/server:dev :; svcadm enable pkg/server:dev127 pkg/server:dev
NOTE that the default listener
devcreated in that procedure would now depend on this new service instancedev127, otherwisedev127can fail to start due to its own internal checks (technically:pkg.depotdchecks if it can connect to the specifiedaddress:portfirst, and the default listener ofdevis considered a busy port bydev127). Any restarts ofdev127should cause shutdown and restart ofdevas well, for the same reasons.
Communications between host-only and external networks
These steps allow networking clients in the LZ to contact the outside world and allow networking servers in the LZ to be accessible from the outside world.
Enable IP packet forwarding through GZ
You can set up "forwarding" and "routing" in Solaris with routeadm. Forwarding is the process of relaying packets not originated by nor destined to this host's IP addresses, from one interface to another. Routing encompasses special daemons which manage the kernel's routing table dynamically and/or announce this box as a router. For our purposes, we only need forwarding enabled (and we don't want to announce our host-only subnet to the external LAN).
Enable IPv4 packet forwarding and update the running configuration:
:; routeadm -e ipv4-forwarding :; routeadm -u
Here we can see that the interfaces got a ROUTER flag enabled:
:; ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=1100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 1500 index 2
inet 192.168.1.40 netmask ffffff00 broadcast 192.168.1.255
ether 0:15:17:5b:d9:64
vnic127001: flags=1100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 9000 index 3
inet 192.168.127.1 netmask ffffff00 broadcast 192.168.127.255
ether 2:8:20:87:e4:43
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
Set up outgoing and incoming NAT through GZ
To enable NAT we need to set up translation rules in /etc/ipf/ipnat.conf and allow connections on the packet filter in /etc/ipf/ipf.conf. Rough-cut examples follow.
Also note that local zones with ip-type=exclusive can define and use their own firewall settings, although it won't likely be required for the setup we're making here.
- Define the network address translation rules:
ipnat.conf
### ipnat.conf for zone-builder host ### Interfaces: ### e1000g0 = physical net (192.168.1.40) ### vnic127001 = build zones in 192.168.127.0/24 ### Allow incoming access to local zones (incoming on e1000g0) rdr e1000g0 0.0.0.0/0 port 22101 -> 192.168.127.101 port 22 tcp ### Allow local zones to access external network (outgoing packets from e1000g0) map e1000g0 192.168.127.0/24 -> 0/32 proxy port ftp ftp/tcp age 600 map e1000g0 192.168.127.0/24 -> 0/32 portmap tcp/udp auto age 600 map e1000g0 192.168.127.0/24 -> 0/32 age 600
- NOTE: The use of
0/32in themaprules above and0/0inrdrrules should automagically substitute the interface's active address(es); this is a mode originally used for NATing by hosts with dial-up or otherwise dynamic external addresses.
- NOTE: The use of
- Define the packet filtering rules if needed (NOTE that by default everything is allowed, so if you trust your LAN and its protection – you might skip this part):
ipf.conf
# # ipf.conf for zone-builder host # # IP Filter rules to be loaded during startup # # See ipf(4) manpage for more information on # IP Filter rules syntax. ### Interfaces: ### e1000g0 = physical net ### vnic127001 = build zones in 192.168.127.0/24 ### Allow all on internal interfaces pass in quick on lo0 all keep state keep frags pass out quick on lo0 all keep state keep frags pass in quick on vnic127001 all keep state keep frags pass out quick on vnic127001 all keep state keep frags pass in quick on vnic127001 pass out quick on vnic127001 ### Allow PING pass out quick proto icmp from any to any icmp-type 8 code 0 pass in quick proto icmp from any to any icmp-type 0 code 0 pass in quick proto icmp from any to any icmp-type 8 code 0 pass out quick proto icmp from any to any icmp-type 0 code 0 ### Allow TRACEROUTE pass out quick proto icmp from any to any icmp-type 11 code 0 pass in quick proto icmp from any to any icmp-type 3 code 3 pass in quick proto icmp from any to any icmp-type 11 code 0 pass out quick proto icmp from any to any icmp-type 3 code 3 ### Default rule block in log all ### Start new rule groups for the physical interface, ### see man for the special syntax block out log quick on e1000g0 all head 100 block in log quick on e1000g0 all head 101 ##################################################################### ### Specific rules can follow here, for now we just allow anything ### NOTE: rules for NATed connections use internal/private addresses ##################################################################### ### Rules for outgoing packets pass out quick proto tcp from any to any flags S keep state keep frags group 100 pass out quick proto udp from any to any keep state group 100 pass out quick from any to any group 100 ### Rules for incoming packets pass in quick proto tcp from any to any flags S keep state keep frags group 101 pass in quick proto udp from any to any keep state group 101 pass in quick from any to any group 101
- This is a relatively complex configuration which should later allow you to add specific permissions and prohibitions for certain protocols and addresses by expanding the "
group" rules at the end of the file. The default action of these groups is tologthe packets (so you can trace them withipmon) andblockthem, denying the communication. However, the defined actions above simply allow all packets to enter and exit the external interface. If you further tune the firewall, these rules should be commented out and replaced by specific permissions – and don't forget to allow your SSH
- This is a relatively complex configuration which should later allow you to add specific permissions and prohibitions for certain protocols and addresses by expanding the "
Enable NAT/firewall settings
Have your console access ready, just in case you forbid yourself the pleasures of networking with the host!
Alternately, you can use screen or VNC to set a timer which would "(sleep 20; svcadm disable ipfilter)&" – just in case...
First of all, reconfigure the ipfilter SMF service to use traditional configuration methods (with the manually created files instead of rule snippets added by some services):
:; svccfg -s svc:/network/ipfilter:default setprop firewall_config_default/policy=custom :; svccfg -s svc:/network/ipfilter:default setprop firewall_config_default/custom_policy_file=/etc/ipf/ipf.conf :; svcadm refresh ipfilter
Apply the settings – but first see if ipfilter service is running and enable or restart it accordingly:
### If ipfilter was disabled -- start it: :; svcs ipfilter STATE STIME FMRI disabled May_03 svc:/network/ipfilter:default :; svcadm enable ipfilter ### If ipfilter was enabled -- restart it: :; svcs ipfilter STATE STIME FMRI online May_16 svc:/network/ipfilter:default :; svcadm restart ipfilter
Test if the settings got applied:
Check the active ipfilter settings
:; ipnat -l List of active MAP/Redirect filters: rdr e1000g0 0.0.0.0/0 port 22101 -> 192.168.127.101 port 22 tcp map e1000g0 192.168.127.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp age 600/600 map e1000g0 192.168.127.0/24 -> 0.0.0.0/32 portmap tcp/udp auto age 600/600 map e1000g0 192.168.127.0/24 -> 0.0.0.0/32 age 600/600 List of active sessions: :; ipfstat -hion 0 @1 pass out quick on lo0 all keep state keep frags 5 @2 pass out quick on vnic127001 all keep state keep frags 0 @3 pass out quick on vnic127001 all 0 @4 pass out quick proto icmp from any to any icmp-type echo code 0 1059 @5 pass out quick proto icmp from any to any icmp-type echorep code 0 0 @6 pass out quick proto icmp from any to any icmp-type timex code 0 0 @7 pass out quick proto icmp from any to any icmp-type unreach code 3 57 @8 block out log quick on e1000g0 all head 100 0 @1 pass out quick proto tcp from any to any flags S/FSRPAU keep state keep frags group 100 7 @2 pass out quick proto udp from any to any keep state group 100 127 @3 pass out quick from any to any group 100 0 @1 pass in quick on lo0 all keep state keep frags 4 @2 pass in quick on vnic127001 all keep state keep frags 0 @3 pass in quick on vnic127001 all 0 @4 pass in quick proto icmp from any to any icmp-type echorep code 0 1059 @5 pass in quick proto icmp from any to any icmp-type echo code 0 0 @6 pass in quick proto icmp from any to any icmp-type unreach code 3 0 @7 pass in quick proto icmp from any to any icmp-type timex code 0 205 @8 block in log all 205 @9 block in log quick on e1000g0 all head 101 0 @1 pass in quick proto tcp from any to any flags S/FSRPAU keep state keep frags group 101 79 @2 pass in quick proto udp from any to any keep state group 101 177 @3 pass in quick from any to any group 101
- NOTE: The first column in
ipfstat -hionoutput can differ on your machine, it is the rule-hit count.
Update DHCP server settings in the GZ
If you initially did just a host-only network and stopped at that, you might not have configured a router announcement in the macros, or the setting was/became invalid.
Make sure that this host (192.168.127.1) is now announced as the router for the host-only network clients, per DHCP macro examples above.
Spawn a local zone (typical procedure)
The local zone setup is explained in more detail in other documents, such as Building in zones.
For the purposes of this document, we only overview the creation of the zone and the tests that it has network access both ways.
Set up an LZ's VNIC
You might have set up some VNICs in the examples above. If you ran out of these addresses, follow the procedures above to add a VNIC and a DHCP reservation for its address.
In short:
:; dladm create-vnic -l localstub127 -m 00:12:71:27:02:34 vnic127234
:; OWNERIP=$(getent hosts `hostname` | awk '{print $1}')
:; pntadm -A 192.168.127.234 -i 01001271270234 \
-s ${OWNERIP} -m localstub127 -f 01 \
-c "Build zone on localstub127" 192.168.127.0
Update /etc/ipf/ipnat.conf rules if you need to publish something (i.e. SSH access) as you see fit, and svcadm restart ipfilter in this case.
Configure the local zone
:; zonecfg -z zone1 create -b set zonepath=/zones/build/zone1 set brand=ipkg set autoboot=true set ip-type=exclusive add net set physical=vnic127101 end verify commit exit
NOTE: Be certain to use the VNIC name, zone name and zone path which you want and not the ones you've copy-pasted from this example
Install and boot the local zone
Beside installing the zone, we will use a /etc/sysidcfg file to predefine the local zone's environmental configuration, so that the zone (and its possible clones) gets configured automatically.
Install the local zone:
:; zoneadm -z zone1 install ... Next Steps: Boot the zone, then log into the zone console (zlogin -C) to complete the configuration process. :; zfs snapshot -r rpool/zones/build/zone1@1NOTE about using a local package mirror
If you are Making a LAN mirror of global package repositories on this machine's global zone, you might want to use that for LZ installation and subsequent upgrades, and bypass an HTTP proxy server that you might have configured in your environment. In that case you'd use the command like this for zone installation, instead of the one given above:
:; http_proxy= https_proxy= zoneadm -z zone1 install -P openindiana.org=http://192.168.127.1:10002/ :; zfs snapshot -r rpool/zones/build/zone1@1
... or likewise for the repo hosted on another box in your LAN – using the LAN's IP address (
192.168.1.40in our example). The remainder of the procedures below remains as described.
Prepare the
/etc/sysidcfgfile with environmental presets for the zone (for hands-free boots), boot it and monitor how it goes::; zoneadm -z zone1 ready :; cat <<EOF > /zones/build/zone1/root/etc/sysidcfg terminal=vt100 network_interface=PRIMARY {dhcp protocol_ipv6=no} security_policy=none nfs4_domain=dynamic timezone=Europe/Moscow root_password=fto/dU8MKwQRI name_service=NONE EOF ### the encrypted root password shown here is: abc123 :; zoneadm -z zone1 halt :; zfs snapshot -r rpool/zones/build/zone1@2 :; zoneadm -z zone1 boot :; zlogin -C zone1Unfortunately, I did not find a way for
sysidcfgto pick up DNS settings from DHCP, so they are to be either pushed explicitly withname_service=DNS...configuration with a block like this:name_service=DNS { domain_name=myhome.com name_server=192.168.1.1,8.8.8.8 search=myhome.com,mywork.com}... or with
name_service=NONEand a hack described below in order to allow local zones to use the DNS settings from DHCP properties.It seems that the ability for a local zone to get DNS settings from DHCP is crippled explicitly in
/lib/svc/share/smf_include.shroutinesmf_netstrategy:... # The network boot strategy for a zone is always "none". # smf_netstrategy () { if smf_is_nonglobalzone; then _INIT_NET_STRATEGY="none" export _INIT_NET_STRATEGY return 0 fi ...Commenting away the block for
smf_is_nonglobalzone(in the local zone's copy of the script) and doingsvcadm restart svc://network/service:defaultdoes the trick for auto-generation of/etc/resolv.confand fixup of/etc/nsswitch.conf, but further impact is unknown, so this is for now a "use at your own risk" kind of workaround. Comments requested on mailing list.
More information about /etc/sysidcfg options can be found here:
Troubleshooting
Note that with the /etc/sysidcfg file in place, the local zone proceeds to auto-configuration and reboot (which may take a minute of it seemingly hanging).
If you get errors like: "Could not contact a dhcp server on the network interface vnic127101" or "in.rdisc: No interfaces up", possibly after several minutes of waiting, double-check the DHCP server and firewall settings. In particular, check that the "owner" of the addresses is set correctly in the dhcptab (see details above).
Sniff whether the zone tries to use DHCP
If the local zone does not receive an IP address, you can try to see if it even tries:
:; snoop -r -d vnic127001 Using device vnic127001 (promiscuous mode) OLD-BROADCAST -> BROADCAST DHCP/BOOTP DHCPREQUEST OLD-BROADCAST -> (broadcast) ARP C Who is 192.168.127.101, 192.168.127.101 ? 192.168.127.101 -> (broadcast) ARP C Who is 192.168.127.101, 192.168.127.101 ? OLD-BROADCAST -> (broadcast) RARP C Who is 0:12:71:27:1:1 ? 192.168.127.101 -> 192.168.127.255 BPARAM C WHOAMI? 192.168.127.101 192.168.127.101 -> 192.168.127.255 BPARAM C WHOAMI? 192.168.127.101 (retransmit)
There may be several repetitions of these lines in varied order. If the server does not give it the address, there would only be repetitions of the DHCPREQUEST line every few seconds.
Access the LZ shell
You may also want to check the services and configuration from inside the local zone.
While the zone has not yet been configured, direct attempts to zlogin into it can fail, because the root user is not yet activated:
:; zlogin zone1 [Connected to zone 'zone1' pts/3] Login incorrect [Connection to zone 'zone1' pts/3 closed]
In this case, use an administrative workaround:
:; zlogin -S zone1 [Connected to zone 'zone1' pts/3] @zone1:~$ bash bash-4.0#
Now you can see what the zone has configured for itself by using ifconfig -a, what it is running with ps -ef, the state of services with svcs and so on.
Test internet access from the local zone
Now that your local zone seems to be configured and running, zlogin into it and see what networking settings it has grabbed:
:; ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
vnic127101: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 9000 index 2
inet 192.168.127.101 netmask ffffff00 broadcast 192.168.127.255
ether 0:12:71:27:1:1
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
:; netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 192.168.127.1 UG 1 0 vnic127101
127.0.0.1 127.0.0.1 UH 6 56 lo0
192.168.127.0 192.168.127.101 U 2 0 vnic127101
Routing Table: IPv6
Destination/Mask Gateway Flags Ref Use If
--------------------------- --------------------------- ----- --- ------- -----
::1 ::1 UH 2 0 lo0
:; traceroute -nI www.ru
traceroute: unknown host www.ru
### Oops... here DNS settings were not defined explicitly nor via DHCP with the workaround above.
### If DNS config is applied in any manner, name resolution and internet should work:
:; ping -ns pkg.openindiana.org
PING pkg.openindiana.org (91.194.74.133): 56 data bytes
64 bytes from 91.194.74.133: icmp_seq=0. time=53.891 ms
64 bytes from 91.194.74.133: icmp_seq=1. time=52.602 ms
64 bytes from 91.194.74.133: icmp_seq=2. time=53.448 ms
^C
----pkg.openindiana.org PING Statistics----
3 packets transmitted, 3 packets received, 0% packet loss
round-trip (ms) min/avg/max/stddev = 52.602/53.314/53.891/0.655
:; traceroute -nI pkg.openindiana.org
traceroute to pkg.openindiana.org (91.194.74.133), 30 hops max, 40 byte packets
1 192.168.127.1 0.342 ms 0.086 ms 0.071 ms
2 192.168.1.1 11.741 ms 2.199 ms 1.384 ms
...
12 87.245.245.10 61.736 ms 95.524 ms 96.185 ms
13 195.72.129.157 50.344 ms 54.766 ms 49.147 ms
14 91.194.74.133 53.484 ms 52.423 ms 51.936 ms
Further evolution
(optional) Publish a TCPv4 service (SSH, HTTP)
Rough outline of the routine, detailed steps are in the chapters above:
- Start up an (SSH, HTTP, ...) server;
- Update the
ipnat.confin the GZ; - Apply
ipfiltersettings; Test accessibility of the service from your external LAN host (different from the routing GZ):
### Testing from another host :; (echo ""; sleep 1) | telnet 192.168.1.40 22101 Trying 192.168.1.40... Connected to 192.168.1.40. Escape character is '^]'. SSH-2.0-Sun_SSH_1.5 Protocol mismatch. Connection to 192.168.1.40 closed by foreign host.
Renaming the LZs/VNICs/IPs
If you rename the VNICs dedicated to a zone, such as when you clone a working zone and give it a new identity (in the example below I'll be changing 101 to 109), keep in mind the following configuration files:
/etc/hostname.vnic*– empty file, its existence notifies the OS that the named interface should get "plumbed" at startup.:; mv /etc/hostname.vnic127101 /etc/hostname.vnic127109
If this file is not empty, it can contain additional parameters for
ifconfig, such as static addressing or MTU size – in case of DHCP we likely want it empty./etc/dhcp.vnic*– empty file, notifies the OS that the named interface should get its settings via DHCP.:; mv /etc/dhcp.vnic127101 /etc/dhcp.vnic127109
/etc/hosts– matches IP addresses to textual names in absence of DNS or a different naming server; one of the lines should be the current host's name and address. The entry should be changed, i.e.-192.168.127.101 zone1 +192.168.127.109 build-gcc444
Without this things that reference the current host can fail, such as
sendmailand someSolaris_audittasks./etc/nodename– if you've also changed the local zone's name (i.e. while cloning it), you want to change its perception of its name.:; zonename > /etc/nodename
SSH keys – to be pedantic, you might want each cloned zone to use different SSH server keys, though this is not technically required.
:; rm -f /etc/ssh/ssh_host_*_key* && \ svcadm restart ssh
Now you can restart (init 6) the cloned/renamed zone to apply the new settings. Simple svcadm restart physical:default can also suffice for networking setup.
Overview
Content Tools