Child pages
  • Using host-only networking to get from build zones and test VMs to the Internet

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

Intro

In this writeup I'll try to document how to use CrossBow virtual networking and some other components to create a LAN-in-a-box.

We will set up internet access for local zones (LZ), such as when you're Building in zones, for cases when your corporate or home LAN does not provide for these zones' individual network addresses and only your machine's global zone (GZ) has an "external" IP address assigned. The same setup can be used for your testbed virtual machines, as well as for easier creation of networked local zones on an OI installation implemented as a VM in VirtualBox (which sets some limitations on networking choices).

Here are the major steps:

Table of Contents

What are we talking about?

It is arguable whether the building local zones need the network access at all, but it might be needed for example if you maintain a golden source-code repository in the global zone and want to distribute the changesets to local zones' cloned workspaces (see Building in zones, Working on several bugs at once and Making a LAN mirror of global package repositories), or if you want to upload webrevs straight from your build zones. YMMV, this here is just a technology POC (wink)

It is also worth noting that some best-practice setups use a similar approach, with a dedicated local zone being the router/firewall/NAT and it has the external IP address, and the global zone "hypervisor" being one of the privately-addressed nodes which have the internet access through such local zone. In particular, this may be the only way to have ip-type=shared local zones to access the external network using NAT through the same box. It is not difficult to transition from the setup described below to such a best-practice one by swapping some commands and VNICs (Virtual Network Interface Cards), though the adventure is left for the curious and won't be detailed in this page.

In particular, this setup will focus on:

  • distributing local zones' IP addresses with a private DHCP server (so that you can quickly provision new zones); in this example I'll use Sun DHCP (aka in.dhcpd) with text-file storage format;
  • getting outgoing communications NAT'ed from local zones' private IP addresses through the global zone using its public IP address in the external network (which can still be an "unroutable" one used by your physical LAN);
  • it is also possible to publish certain services, such as SSH, using reverse NAT with the global zone's public IP address.

DHCP host-naming

(So far) this document does not deal with configuration of naming services, though some pointers can be given anyway (smile)

One solution is to use a private DNS server on the host machine for the zones to know each other's names and private IP addresses, which would forward requests for other names to the LAN DNS servers. See Solaris DHCP Administration Guide Chapter 4 for details on enabling the registration of DHCP clients in the ISC BIND server by the Sun DHCP server, and optionally allowing the DHCP client to push its desired hostname via DHCP server to DNS.

An alternate useful configuration might be to use DNS for remote hostname look-ups, and allow the DHCP server to update its host's /etc/hosts file with information about client hostnames, but this is not explored here either.

In this page it is assumed that the local zones receive their default hostnames via zoneadm install (`zonename` or ultimately /etc/nodename), and can be configured to use external DNS servers for internet/LAN access, and they can address each other and their GZ via /etc/hosts file if that is at all needed.

The example network layout

For the purposes of the setup described below:

  • the host-only network will be 192.168.127.0/24, on a dedicated etherstub, with
  • the global zone having an address of 192.168.127.1, and
  • the local zones use "exclusive" IP stacks with dedicated VNICs and individually manage their networking settings (via DHCP in this case);
  • determinism in IP addressing (and private naming) will be provided by special setup of the VNICs' properties and the DHCP server;
  • the host's assigned "external IP address" in the physical LAN is 192.168.1.40 on a e1000g0 interface;
  • the examples only deal with IPv4 addressing.

Typographic note

In many of the examples below, samples of commands to run as "root" (or via pfexec) are prefixed with a ":;" prompt.
Basically this prepends a call of "true" to the sample command and does nothing, unlike the typically used "hash" (#) prompt which acts as a comment and precludes the copy-pasted command blocks from running.

The host-only network setup

This chapter focuses on creating the host-only network, which allows using a private IP address range for local zones to communicate with each other and with the global zone.

This by itself does not provide communications with the outside world, and you may have your own reasons to stop the network configuration at that point, and proceed to setting up the local zone(s). In that case the local zones would only be able to access each other and the global zone, using the VNICs connected to a common "etherstub".

Create an etherstub (a virtual Ethernet switch)

Code Block
:; dladm create-etherstub localstub127

This provides a host-only Ethernet segment, acting like a virtual switch, named "localstub127".

Create some virtual NICs attached to the etherstub

Code Block
:; dladm create-vnic -l localstub127 vnic127001
:; dladm create-vnic -l localstub127 -m 00:12:71:27:01:01 vnic127101

The first line defines a VNIC for GZ named "vnic127001" with a random MAC address, attached to the virtual switch created above.

The second line creates a VNIC for an LZ, attached to the same switch with a predefined MAC address, others can be defined similarly.

Here I use a simple naming scheme, with 127 standing for the subnet and 001 standing for the static IP address (192.168.127.1) that will be assigned to the VNIC for the global zone.

Similarly the 101 characters in the vnic127101 name and 00:12:71:27:01:01 in the MAC address stand for the IP address (192.168.127.101) that will be assigned to the VNIC via DHCP – fixed addressing, even if distributed with dynamic methods, is useful for setting up /etc/hosts and the publishing of services via NAT.

A simple construct like the one below would pre-create some 9 VNICs to this pattern quickly:

Code Block
:; for IP in 01 02 03 04 05 06 07 08 09; do \
   dladm create-vnic -l localstub127 -m 00:12:71:27:01:${IP} vnic1271${IP}; done

Set up the GZ's VNIC

Code Block
:; grep "192\.168\.127\." /etc/netmasks || \
   echo "192.168.127.0   255.255.255.0" >> /etc/netmasks

:; ipadm create-if vnic127001 && \
   ipadm create-addr -T static -alocal=192.168.127.1 vnic127001/v4
 
### Legacy ways:
#  echo "192.168.127.1" > /etc/hostname.vnic127001
#  ifconfig vnic127001 plumb 192.168.127.1/24 up

The first two lines take care of saving the VNIC addressing for the future. The last line brings up the new IPv4 interface and sets up its addressing.

Quick check (note that etherstub default MTU is 9000, equivalent of Jumbo Frames):

Code Block
:; ifconfig vnic127001
vnic127001: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9000 index 3
        inet 192.168.127.1 netmask ffffff00 broadcast 192.168.127.255
        ether 2:8:20:87:e4:43

Set up DHCP server in the GZ

Install Sun DHCP server software

First of all, the DHCP server software should be installed. In this example I'll use Sun BOOTP/DHCP server software (alternately ISC DHCP can be used, for example).

  • Install Sun DHCP server package:
Code Block
:; pkg install service/network/dhcp
:; svccfg import /lib/svc/manifest/network/dhcp-server.xml
:; svcs -a | grep -i dhcp
disabled       16:51:37 svc:/network/dhcp-server:default
    • Optionally, you might want the binfiles storage backend for DHCP configuration to be available, but I find inspecting and manipulating text files to be easier (wink)
      Just in case, here's the optional command:
Code Block
:; pkg install service/network/dhcp/datastore/binfiles
    • Optionally, if you'd prefer to use the X11 GUI to administer DHCP, there is a Java applet for that (and some further optional localizations which I skip):
Code Block
:; pkg install service/network/dhcp network/dhcp/dhcpmgr

The management of the server can be done in a couple of ways: properly via command-line tools including /usr/sbin/dhcpconfig, /usr/sbin/pntadm and /usr/sbin/dhtadm, or with the GUI applet /usr/sadm/admin/bin/dhcpmgr (also symlinked as /usr/sbin/dhcpmgr), and somewhat improperly by manipulating the config files (/etc/inet/dhcpsvc.conf) and DHCP text-formatted database files /var/dhcp/SUNW* directly – this is not encouraged, but is often faster, more straightforward, and "just plain works"TM.

Configure Sun DHCP server software

First of all, to enable the DHCP server, its configuration file should exist and say that the server is enabled (for more details see init-script /etc/init.d/dhcp which wraps the SMF service).

  • Create the initial configuration:
Code Block
:; dhcpconfig -r SUNWfiles -p /var/dhcp -D -d `domainname` -a 8.8.8.8 -l 86400
Created DHCP configuration file.
Created dhcptab.
Added "Locale" macro to dhcptab.
Added server macro to dhcptab - openindiana.
DHCP server started.

:; cat /etc/inet/dhcpsvc.conf 
DAEMON_ENABLED=TRUE
RESOURCE=SUNWfiles
RUN_MODE=server
PATH=/var/dhcp
CONVER=1

In the example above, the -d option sets the local domain name, and the -a option sets the DNS servers to be used (that would likely be your LAN's DNS server address, or 192.168.127.1 if you set up a DNS server instance on this box). We've also requested to use the less performant but more convenient text-formatted database files "resource".
The wizard also took some freedom to add a "server macro" based on `hostname` and IP addressing of this zone, which may be acceptable or invalid for our uses – we'll see.

  • In our case we'll want to bind the server to only work for the host-only network (and the broadcasts, as seen below):
Code Block
:; echo "INTERFACES=vnic127001" >> /etc/inet/dhcpsvc.conf
:; /etc/init.d/dhcp stop; /etc/init.d/dhcp start

:; svcs -p dhcp-server
STATE          STIME    FMRI
online         17:44:51 svc:/network/dhcp-server:default
               17:44:51    26974 in.dhcpd

:; netstat -an | grep -w 67
255.255.255.255.67                        Idle
192.168.127.0.67                          Idle
192.168.127.1.67                          Idle
Optional: Configure Sun DHCP server logging

You would likely want the DHCP server to log its events, errors and other messages into syslog to ease the debugging.

  • Enable logging on the DHCP server:

    Code Block
    :; echo "LOGGING_FACILITY=0" >> /etc/inet/dhcpsvc.conf
    :; svcadm restart dhcp-server

    Note: You can use any local syslog facility ranging from 0 to 7.

  • Add this line to /etc/syslog.conf to enable saving of these messages into a particular file (NOTE: the two parts must be separated by TAB characters):

    Code Block
    local0.notice                   /var/log/dhcpsvc

    Note that other local0 events would get into this log file, like ipmon does on my system. Well, while debugging THIS setup, it is only a bonus to have the two logs co-located (wink)

  • Now you can touch the file, and restart the syslog:

    Code Block
    :; touch /var/log/dhcpsvc
    :; svcadm restart system-log

    Note that syslog does not create log files by itself, and complains if one is not present at the moment of the daemon's startup or restart.

  • Ultimately, enable log rotation to restrain the disk space requirements:

    Code Block
    :; cat << EOF >> /etc/logadm.conf
    ### Rotate DHCP/ipmon logs
    /var/log/dhcpsvc -C 4 -s 1m -a '/usr/sbin/svcadm refresh system-log'
    EOF

    This uses the currently default log rotation engine logadm regularly called from cron; if you use something else (newsysloglogrotate.d, etc.) – configure that engine appropriately.

Here are some sample syslogged entries about DHCP events:

Code Block
titleExample syslog entries for DHCP leases
Jun  6 23:48:26 openindiana in.dhcpd[6041]: [ID 771465 local0.notice]
  DHCP ASSIGN 1339012106 -000000001 192.168.127.101 192.168.1.40 001271270101 N/A 001271270101

Jun  7 14:24:05 openindiana in.dhcpd[6041]: [ID 771465 local0.notice]
  DHCP EXTEND 1339064645 -000000001 192.168.127.101 192.168.1.40 001271270101 N/A 001271270101
Define our DHCP subnet settings

Now that we have a server running, we can define a subnet macro:

Code Block
:; dhcpconfig -N 192.168.127.0 -m 255.255.255.0 -t 192.168.127.1
Added network macro to dhcptab - 192.168.127.0.
Created network table.

The macros are used to basically group some settings for a client, and can be nested, as we'll do below.

Note
titleUnder The Hood

Let's see under the hood – what have we defined in the previous step?

Code Block
:; ls -la /var/dhcp/SUNW*
-rw-r--r--   1 root     root         104 Jun  6 17:51 /var/dhcp/SUNWfiles1_192_168_127_0
-rw-r--r--   1 root     root         385 Jun  6 17:51 /var/dhcp/SUNWfiles1_dhcptab

:; cat /var/dhcp/SUNWfiles1_dhcptab
# SUNWfiles1_dhcptab
#
# Do NOT edit this file by hand -- use dhtadm(1M) or dhcpmgr(1M) instead
#
Locale|m|4785356079057862657|:UTCoffst=10800:
openindiana|m|10017694421132247041|:Include=Locale:Timeserv=192.168.1.40:LeaseTim=86400:LeaseNeg:DNSdmain="domain.com":DNSserv=8.8.8.8:
192.168.127.0|m|2030560481990672385|:Subnet=255.255.255.0:Router=192.168.127.1:Broadcst=192.168.127.255:

The first two macros were predefined by initial dhcpconfig based on what it knows about the naming and networking setup of the host and on our command-line options, while the last one was added by us explicitly.
The /var/dhcp/SUNWfiles1_192_168_127_0 file is currently empty (comments only), but it will soon contain the definitions which bind IP addresses, MAC addresses and DHCP Macros for particular clients.

Now, combine (nest) the macros into one DHCP profile which can be used to configure the clients:

Code Block
:; dhtadm -A -m localstub127 -d ":Include=192.168.127.0:Include=`hostname`:"

:; tail -1 /var/dhcp/SUNWfiles1_dhcptab
localstub127|m|593912200859484161|:Include=192.168.127.0:Include=openindiana:

NOTE: This implies the Timeserv value to be the GZ public address, but since local zones don't have individual clocks – we don't care. This can be a problem however if the DHCP clients include different machines which have clocks and can't reach the router's external address, or are zones which host VMs; in this case the `hostname`-based macro (the openindiana above) should be redefined, or another similar one can be defined and Included by our localstub127 macro instead. In the latter case I'd make several macros – for time servers, for DNS servers, etc. and combine them all as suitable for a particular client profile.

More information about Sun DHCP macros can be found here:

Define fixed client addresses

Some client addresses can be predefined, following our IP/MAC-address naming pattern explained above. This simple construct would create some 9 entries for the VNICs made in examples above:

Code Block
:; OWNERIP=$(getent hosts `hostname` | awk '{print $1}')
:; for IP in 01 02 03 04 05 06 07 08 09; do \
   pntadm -A 192.168.127.1${IP} -i 010012712701${IP} \
   -s ${OWNERIP} -m localstub127 -f 01 \
   -c "Build zone on localstub127" 192.168.127.0; done

Here we define the following pntadm options:

  • -A – the client IP address (or hostname known via /etc/hosts, technically);
  • -i – the client identifier (MAC address prefixed by the ARP code for the type of network, usually 01 for Ethernet);
  • -s – DHCP server responsible for the leases;
  • -m – the client macro for common settings;
  • -f – the flag describing the lease type (01 = PERMANENT lease; announced via DHCP and never expires to be recycled for other clients);
  • -c – an optional comment describing this client;
  • The final parameter is the subnet name (as it is known by the DHCP server).
Warning

IMPORTANT NOTE about the OWNERIP construct used above and below: The Sun DHCP server is identified by the numeric IP address (i.e. 192.168.1.40 from the external network, for our example) which corresponds to the textual hostname (i.e. openindiana), which in turn corresponds to this host's name set in /etc/nodename. That is, even though you serve addresses on a 192.168.127.0/24 subnet, the "owner" of these leases must be 192.168.1.40 if that address corresponds to the nodename as resolvable via /etc/hosts (or possibly DNS too). Otherwise this instance of the DHCP server will refuse to issue addresses "owned" by another server (even if the provided value resolves to this same machine).
Also keep this in mind if you ever rename the GZ hostname and/or change its "primary" IP address.

Optionally reserve random client addresses

You can also use these commands (likely with different options) to reserve some IP addresses for dynamic leasing for not-predefined hosts, i.e. to quickly provision VMs or test zones. You might use a different macro (i.e. with shorter LeaseTim) in this case, as well.

For example, a range of 192.168.127.201-209 will be defined here, leasing addresses for 30 minutes to any client (except those who already have a reservation for the same MAC address):

Code Block
:; OWNERIP=$(getent hosts `hostname` | awk '{print $1}')
:; dhtadm -A -m localstub127rnd -d \
   ':Include=192.168.127.0:Include=Locale:LeaseTim=1800:LeaseNeg:DNSdmain="'"`domainname`"'":DNSserv=8.8.8.8:'

:; for IP in 01 02 03 04 05 06 07 08 09; do \
   pntadm -A 192.168.127.2${IP} -i 00 \
   -s ${OWNERIP} -m localstub127rnd -f 00 \
   -c "Quick-test hosts on localstub127" 192.168.127.0; done

(Optional) Prepare a package repository listener on the private network

You might want to use a local mirror of the OpenIndiana package repository hosted in the global zone, as detailed in Making a LAN mirror of global package repositories.

However, after completing the procedure described on that page, you might need to spawn another instance of the package depot service to listen on the more-specific IP address (than the default 0.0.0.0) and do HTTP-redirects to it, with these commands:

Code Block
:; svccfg -s pkg/server
add dev127
select dev127
addpg pkg application
setprop pkg/inst_root = astring: "/export/pkg/dev/"
setprop pkg/port = count: 10002
setprop pkg/address = net_address: 192.168.127.1
setprop pkg/proxy_base = astring: http://192.168.127.1:10002
exit
 
:; svccfg -s pkg/server
select dev
addpg dev dependency
setprop dev/entities = fmri: svc:/application/pkg/server:dev127
setprop dev/grouping = astring: optional_all
setprop dev/restart_on = astring: restart
setprop dev/type = astring: service
exit
 
:; svcadm refresh pkg/server:dev127 pkg/server:dev
:; svcadm disable -st pkg/server:dev127 pkg/server:dev
:; svcadm enable pkg/server:dev127 pkg/server:dev
  • NOTE that the default listener dev created in that procedure would now depend on this new service instance dev127, otherwise dev127 can fail to start due to its own internal checks (technically: pkg.depotd checks if it can connect to the specified address:port first, and the default listener of dev is considered a busy port by dev127). Any restarts of dev127 should cause shutdown and restart of dev as well, for the same reasons.

Communications between host-only and external networks

These steps allow networking clients in the LZ to contact the outside world and allow networking servers in the LZ to be accessible from the outside world.

Enable IP packet forwarding through GZ

You can set up "forwarding" and "routing" in Solaris with routeadm. Forwarding is the process of relaying packets not originated by nor destined to this host's IP addresses, from one interface to another. Routing encompasses special daemons which manage the kernel's routing table dynamically and/or announce this box as a router. For our purposes, we only need forwarding enabled (and we don't want to announce our host-only subnet to the external LAN).

Enable IPv4 packet forwarding and update the running configuration:

Code Block
:; routeadm -e ipv4-forwarding
:; routeadm -u

Here we can see that the interfaces got a ROUTER flag enabled:

Code Block
:; ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
e1000g0: flags=1100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 1500 index 2
        inet 192.168.1.40 netmask ffffff00 broadcast 192.168.1.255
        ether 0:15:17:5b:d9:64 
vnic127001: flags=1100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 9000 index 3
        inet 192.168.127.1 netmask ffffff00 broadcast 192.168.127.255
        ether 2:8:20:87:e4:43 
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128 

Set up outgoing and incoming NAT through GZ

To enable NAT we need to set up translation rules in /etc/ipf/ipnat.conf and allow connections on the packet filter in /etc/ipf/ipf.conf. Rough-cut examples follow.

Also note that local zones with ip-type=exclusive can define and use their own firewall settings, although it won't likely be required for the setup we're making here.

  • Define the network address translation rules:
Code Block
titleipnat.conf
### ipnat.conf for zone-builder host

### Interfaces:
###    e1000g0 = physical net (192.168.1.40)
###    vnic127001 = build zones in 192.168.127.0/24

### Allow incoming access to local zones (incoming on e1000g0)
rdr e1000g0 0.0.0.0/0 port 22101 -> 192.168.127.101 port 22 tcp

### Allow local zones to access external network (outgoing packets from e1000g0)
map e1000g0 192.168.127.0/24 -> 0/32 proxy port ftp ftp/tcp age 600
map e1000g0 192.168.127.0/24 -> 0/32 portmap tcp/udp auto age 600
map e1000g0 192.168.127.0/24 -> 0/32 age 600
    • NOTE: The use of 0/32 in the map rules above and 0/0 in rdr rules should automagically substitute the interface's active address(es); this is a mode originally used for NATing by hosts with dial-up or otherwise dynamic external addresses.
  • Define the packet filtering rules if needed (NOTE that by default everything is allowed, so if you trust your LAN and its protection – you might skip this part):
Code Block
titleipf.conf
#
# ipf.conf for zone-builder host
#
# IP Filter rules to be loaded during startup
#
# See ipf(4) manpage for more information on
# IP Filter rules syntax.

### Interfaces:
###    e1000g0 = physical net
###    vnic127001 = build zones in 192.168.127.0/24

### Allow all on internal interfaces
pass in quick on lo0 all keep state keep frags
pass out quick on lo0 all keep state keep frags
pass in quick on vnic127001 all keep state keep frags
pass out quick on vnic127001 all keep state keep frags
pass in quick on vnic127001
pass out quick on vnic127001

### Allow PING
pass out quick proto icmp from any to any icmp-type 8 code 0
pass in quick proto icmp from any to any icmp-type 0 code 0
pass in quick proto icmp from any to any icmp-type 8 code 0
pass out quick proto icmp from any to any icmp-type 0 code 0

### Allow TRACEROUTE
pass out quick proto icmp from any to any icmp-type 11 code 0
pass in quick proto icmp from any to any icmp-type 3 code 3
pass in quick proto icmp from any to any icmp-type 11 code 0
pass out quick proto icmp from any to any icmp-type 3 code 3

### Default rule
block in log all

### Start new rule groups for the physical interface,
### see man for the special syntax
block out log quick on e1000g0 all head 100
block in  log quick on e1000g0 all head 101

#####################################################################
### Specific rules can follow here, for now we just allow anything
### NOTE: rules for NATed connections use internal/private addresses
#####################################################################

### Rules for outgoing packets
pass out quick proto tcp from any to any flags S keep state keep frags group 100
pass out quick proto udp from any to any keep state group 100
pass out quick from any to any group 100

### Rules for incoming packets
pass in quick proto tcp from any to any flags S keep state keep frags group 101
pass in quick proto udp from any to any keep state group 101
pass in quick from any to any group 101
    • This is a relatively complex configuration which should later allow you to add specific permissions and prohibitions for certain protocols and addresses by expanding the "group" rules at the end of the file. The default action of these groups is to log the packets (so you can trace them with ipmon) and block them, denying the communication. However, the defined actions above simply allow all packets to enter and exit the external interface. If you further tune the firewall, these rules should be commented out and replaced by specific permissions – and don't forget to allow your SSH (wink)
Enable NAT/firewall settings
Warning

Have your console access ready, just in case you forbid yourself the pleasures of networking with the host! (wink)

Alternately, you can use screen or VNC to set a timer which would "(sleep 20; svcadm disable ipfilter)&" – just in case...

First of all, reconfigure the ipfilter SMF service to use traditional configuration methods (with the manually created files instead of rule snippets added by some services):

Code Block
:; svccfg -s svc:/network/ipfilter:default setprop firewall_config_default/policy=custom
:; svccfg -s svc:/network/ipfilter:default setprop firewall_config_default/custom_policy_file=/etc/ipf/ipf.conf
:; svcadm refresh ipfilter

Apply the settings – but first see if ipfilter service is running and enable or restart it accordingly:

Code Block
### If ipfilter was disabled -- start it:
:; svcs ipfilter
STATE          STIME    FMRI
disabled       May_03   svc:/network/ipfilter:default

:; svcadm enable ipfilter

### If ipfilter was enabled -- restart it:
:; svcs ipfilter
STATE          STIME    FMRI
online         May_16   svc:/network/ipfilter:default

:; svcadm restart ipfilter

Test if the settings got applied:

Code Block
titleCheck the active ipfilter settings
:; ipnat -l
List of active MAP/Redirect filters:
rdr e1000g0 0.0.0.0/0 port 22101 -> 192.168.127.101 port 22 tcp
map e1000g0 192.168.127.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp age 600/600
map e1000g0 192.168.127.0/24 -> 0.0.0.0/32 portmap tcp/udp auto age 600/600
map e1000g0 192.168.127.0/24 -> 0.0.0.0/32 age 600/600

List of active sessions:


:; ipfstat -hion
0 @1 pass out quick on lo0 all keep state keep frags
5 @2 pass out quick on vnic127001 all keep state keep frags
0 @3 pass out quick on vnic127001 all
0 @4 pass out quick proto icmp from any to any icmp-type echo code 0
1059 @5 pass out quick proto icmp from any to any icmp-type echorep code 0
0 @6 pass out quick proto icmp from any to any icmp-type timex code 0
0 @7 pass out quick proto icmp from any to any icmp-type unreach code 3
57 @8 block out log quick on e1000g0 all head 100
0 @1 pass out quick proto tcp from any to any flags S/FSRPAU keep state keep frags group 100
7 @2 pass out quick proto udp from any to any keep state group 100
127 @3 pass out quick from any to any group 100
0 @1 pass in quick on lo0 all keep state keep frags
4 @2 pass in quick on vnic127001 all keep state keep frags
0 @3 pass in quick on vnic127001 all
0 @4 pass in quick proto icmp from any to any icmp-type echorep code 0
1059 @5 pass in quick proto icmp from any to any icmp-type echo code 0
0 @6 pass in quick proto icmp from any to any icmp-type unreach code 3
0 @7 pass in quick proto icmp from any to any icmp-type timex code 0
205 @8 block in log all
205 @9 block in log quick on e1000g0 all head 101
0 @1 pass in quick proto tcp from any to any flags S/FSRPAU keep state keep frags group 101
79 @2 pass in quick proto udp from any to any keep state group 101
177 @3 pass in quick from any to any group 101
  • NOTE: The first column in ipfstat -hion output can differ on your machine, it is the rule-hit count.

Update DHCP server settings in the GZ

If you initially did just a host-only network and stopped at that, you might not have configured a router announcement in the macros, or the setting was/became invalid.

Make sure that this host (192.168.127.1) is now announced as the router for the host-only network clients, per DHCP macro examples above.

Spawn a local zone (typical procedure)

The local zone setup is explained in more detail in other documents, such as Building in zones.

For the purposes of this document, we only overview the creation of the zone and the tests that it has network access both ways.

Set up an LZ's VNIC

You might have set up some VNICs in the examples above. If you ran out of these addresses, follow the procedures above to add a VNIC and a DHCP reservation for its address.

In short:

Code Block
:; dladm create-vnic -l localstub127 -m 00:12:71:27:02:34 vnic127234

:; OWNERIP=$(getent hosts `hostname` | awk '{print $1}')
:; pntadm -A 192.168.127.234 -i 01001271270234 \
   -s ${OWNERIP} -m localstub127 -f 01 \
   -c "Build zone on localstub127" 192.168.127.0

Update /etc/ipf/ipnat.conf rules if you need to publish something (i.e. SSH access) as you see fit, and svcadm restart ipfilter in this case.

Configure the local zone

Code Block
:; zonecfg -z zone1
create -b
set zonepath=/zones/build/zone1
set brand=ipkg
set autoboot=true
set ip-type=exclusive
add net
set physical=vnic127101
end
verify
commit
exit

NOTE: Be certain to use the VNIC name, zone name and zone path which you want and not the ones you've copy-pasted from this example (wink)

Install and boot the local zone

Beside installing the zone, we will use a /etc/sysidcfg file to predefine the local zone's environmental configuration, so that the zone (and its possible clones) gets configured automatically.

  • Install the local zone:

    Code Block
    :; zoneadm -z zone1 install
    ...
      Next Steps: Boot the zone, then log into the zone console (zlogin -C)
                  to complete the configuration process.
    
    :; zfs snapshot -r rpool/zones/build/zone1@1
    Note
    titleNOTE about using a local package mirror

    If you are Making a LAN mirror of global package repositories on this machine's global zone, you might want to use that for LZ installation and subsequent upgrades, and bypass an HTTP proxy server that you might have configured in your environment. In that case you'd use the command like this for zone installation, instead of the one given above:

    Code Block
    :; http_proxy= https_proxy= zoneadm -z zone1 install -P openindiana.org=http://192.168.127.1:10002/
    :; zfs snapshot -r rpool/zones/build/zone1@1

    ... or likewise for the repo hosted on another box in your LAN – using the LAN's IP address (192.168.1.40 in our example). The remainder of the procedures below remains as described.

  • Prepare the /etc/sysidcfg file with environmental presets for the zone (for hands-free boots), boot it and monitor how it goes:

    Code Block
    :; zoneadm -z zone1 ready
    :; cat <<EOF > /zones/build/zone1/root/etc/sysidcfg
    terminal=vt100
    network_interface=PRIMARY {dhcp protocol_ipv6=no}
    security_policy=none
    nfs4_domain=dynamic
    timezone=Europe/Moscow
    root_password=fto/dU8MKwQRI
    name_service=NONE
    EOF
    ### the encrypted root password shown here is: abc123
    :; zoneadm -z zone1 halt
    :; zfs snapshot -r rpool/zones/build/zone1@2
    
    :; zoneadm -z zone1 boot
    :; zlogin -C zone1

    Unfortunately, I did not find a way for sysidcfg to pick up DNS settings from DHCP, so they are to be either pushed explicitly with name_service=DNS... configuration with a block like this:

    Code Block
    name_service=DNS { domain_name=myhome.com
         name_server=192.168.1.1,8.8.8.8
         search=myhome.com,mywork.com}

    ... or with name_service=NONE and a hack described below in order to allow local zones to use the DNS settings from DHCP properties.

    Note

    It seems that the ability for a local zone to get DNS settings from DHCP is crippled explicitly in /lib/svc/share/smf_include.sh routine smf_netstrategy:

    Code Block
    ...
    #   The network boot strategy for a zone is always "none".
    #
    smf_netstrategy () {
            if smf_is_nonglobalzone; then
                    _INIT_NET_STRATEGY="none" export _INIT_NET_STRATEGY
                    return 0
            fi
    ...

    Commenting away the block for smf_is_nonglobalzone (in the local zone's copy of the script) and doing svcadm restart svc://network/service:default does the trick for auto-generation of /etc/resolv.conf and fixup of /etc/nsswitch.conf, but further impact is unknown, so this is for now a "use at your own risk" kind of workaround. Comments requested on mailing list.

More information about /etc/sysidcfg options can be found here:

Troubleshooting

Note that with the /etc/sysidcfg file in place, the local zone proceeds to auto-configuration and reboot (which may take a minute of it seemingly hanging).

If you get errors like: "Could not contact a dhcp server on the network interface vnic127101" or "in.rdisc: No interfaces up", possibly after several minutes of waiting, double-check the DHCP server and firewall settings. In particular, check that the "owner" of the addresses is set correctly in the dhcptab (see details above).

Sniff whether the zone tries to use DHCP

If the local zone does not receive an IP address, you can try to see if it even tries:

Code Block
:; snoop -r -d vnic127001
Using device vnic127001 (promiscuous mode)

OLD-BROADCAST -> BROADCAST    DHCP/BOOTP DHCPREQUEST
OLD-BROADCAST -> (broadcast)  ARP C Who is 192.168.127.101, 192.168.127.101 ?
192.168.127.101 -> (broadcast)  ARP C Who is 192.168.127.101, 192.168.127.101 ?
OLD-BROADCAST -> (broadcast)  RARP C Who is 0:12:71:27:1:1 ?
192.168.127.101 -> 192.168.127.255 BPARAM C WHOAMI? 192.168.127.101
192.168.127.101 -> 192.168.127.255 BPARAM C WHOAMI? 192.168.127.101 (retransmit)

There may be several repetitions of these lines in varied order. If the server does not give it the address, there would only be repetitions of the DHCPREQUEST line every few seconds.

Access the LZ shell

You may also want to check the services and configuration from inside the local zone.

While the zone has not yet been configured, direct attempts to zlogin into it can fail, because the root user is not yet activated:

Code Block
:; zlogin zone1
[Connected to zone 'zone1' pts/3]
Login incorrect

[Connection to zone 'zone1' pts/3 closed]

In this case, use an administrative workaround:

Code Block
:; zlogin -S zone1 
[Connected to zone 'zone1' pts/3]
@zone1:~$ bash
bash-4.0# 

Now you can see what the zone has configured for itself by using ifconfig -a, what it is running with ps -ef, the state of services with svcs and so on.

Test internet access from the local zone

Now that your local zone seems to be configured and running, zlogin into it and see what networking settings it has grabbed:

Code Block
:; ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
vnic127101: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 9000 index 2
        inet 192.168.127.101 netmask ffffff00 broadcast 192.168.127.255
        ether 0:12:71:27:1:1 
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128 

:; netstat -rn
Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface 
-------------------- -------------------- ----- ----- ---------- --------- 
default              192.168.127.1        UG        1          0 vnic127101 
127.0.0.1            127.0.0.1            UH        6         56 lo0       
192.168.127.0        192.168.127.101      U         2          0 vnic127101 

Routing Table: IPv6
  Destination/Mask            Gateway                   Flags Ref   Use    If   
--------------------------- --------------------------- ----- --- ------- ----- 
::1                         ::1                         UH      2       0 lo0   




:; traceroute -nI www.ru 
traceroute: unknown host www.ru
### Oops... here DNS settings were not defined explicitly nor via DHCP with the workaround above.


### If DNS config is applied in any manner, name resolution and internet should work:
:; ping -ns pkg.openindiana.org
PING pkg.openindiana.org (91.194.74.133): 56 data bytes
64 bytes from 91.194.74.133: icmp_seq=0. time=53.891 ms
64 bytes from 91.194.74.133: icmp_seq=1. time=52.602 ms
64 bytes from 91.194.74.133: icmp_seq=2. time=53.448 ms
^C
----pkg.openindiana.org PING Statistics----
3 packets transmitted, 3 packets received, 0% packet loss
round-trip (ms)  min/avg/max/stddev = 52.602/53.314/53.891/0.655

:;  traceroute -nI pkg.openindiana.org
traceroute to pkg.openindiana.org (91.194.74.133), 30 hops max, 40 byte packets
 1  192.168.127.1  0.342 ms  0.086 ms  0.071 ms
 2  192.168.1.1  11.741 ms  2.199 ms  1.384 ms
...
12  87.245.245.10  61.736 ms  95.524 ms  96.185 ms
13  195.72.129.157  50.344 ms  54.766 ms  49.147 ms
14  91.194.74.133  53.484 ms  52.423 ms  51.936 ms

Further evolution

(optional) Publish a TCPv4 service (SSH, HTTP)

Rough outline of the routine, detailed steps are in the chapters above:

  • Start up an (SSH, HTTP, ...) server;
  • Update the ipnat.conf in the GZ;
  • Apply ipfilter settings;
  • Test accessibility of the service from your external LAN host (different from the routing GZ):

    Code Block
    ### Testing from another host
    :; (echo ""; sleep 1) | telnet 192.168.1.40 22101
    Trying 192.168.1.40...
    Connected to 192.168.1.40.
    Escape character is '^]'.
    SSH-2.0-Sun_SSH_1.5
    Protocol mismatch.
    Connection to 192.168.1.40 closed by foreign host.

Renaming the LZs/VNICs/IPs

If you rename the VNICs dedicated to a zone, such as when you clone a working zone and give it a new identity (in the example below I'll be changing 101 to 109), keep in mind the following configuration files:

  • /etc/hostname.vnic* – empty file, its existence notifies the OS that the named interface should get "plumbed" at startup.

    Code Block
    :; mv /etc/hostname.vnic127101 /etc/hostname.vnic127109

    If this file is not empty, it can contain additional parameters for ifconfig, such as static addressing or MTU size – in case of DHCP we likely want it empty.

  • /etc/dhcp.vnic* – empty file, notifies the OS that the named interface should get its settings via DHCP.

    Code Block
    :; mv /etc/dhcp.vnic127101 /etc/dhcp.vnic127109
  • /etc/hosts – matches IP addresses to textual names in absence of DNS or a different naming server; one of the lines should be the current host's name and address. The entry should be changed, i.e.

    Code Block
    -192.168.127.101  zone1
    +192.168.127.109  build-gcc444

    Without this things that reference the current host can fail, such as sendmail and some Solaris_audit tasks.

  • /etc/nodename – if you've also changed the local zone's name (i.e. while cloning it), you want to change its perception of its name.

    Code Block
    :; zonename > /etc/nodename
  • SSH keys – to be pedantic, you might want each cloned zone to use different SSH server keys, though this is not technically required.

    Code Block
    :; rm -f /etc/ssh/ssh_host_*_key* && \
       svcadm restart ssh

Now you can restart (init 6) the cloned/renamed zone to apply the new settings. Simple svcadm restart physical:default can also suffice for networking setup.