I came from a Linux background. For the past several years, I've used NX Free Edition from www.nomachine.com to connect to our servers remotely. However, they do not appear to offer a Solaris version for x86_64 -- only for SPARC. It is supposed to be possible to get FreeNX to work on OpenIndiana, but it appeared to be more complicated than I wanted to deal with.
So, that left me with needing to use VNC to connect to our OpenIndiana machines.
Years ago, VNC used to be a real pain in the neck to get working on Linux. You had to jump through hoops to get a graphical login screen with most distributions. I think SuSE was the only distribution I ever used that you could set it up without any grief.
So, when it came time for me to set up VNC on OpenIndiana, I just assumed that it was going to be a pain in the neck, so I started Googling. I found several sets of instructions for setting up VNC on Solaris. I think I ended up wasting about six hours getting it to work; and in the end, discovering that I didn't need to do most of what those web pages said.
If you would like a graphical connection with VNC, all you should need to do is the following:
1) Edit your /etc/gdm/custom.conf to look something like this:
# GDM configuration storage
2) Start the service:
# svcadm enable svc:/application/x11/xvnc-inetd
3) Restart GDM. You could do it with "svcadm restart gdm", but keep in mind that will kill your current desktop session, so you might want to just restart your machine at a convenient time after doing this, instead.
That's all there is to it.
4) If you are using a headless server or a serial console, you need to ensure that gdm will not start a session on the console. The way to do this is described in example 2 of the console-kit-daemon(1m) man page. Look in the /etc/ConsoleKit/seats.d directory for the file 00-primary.seat . In this file, change the line `Hidden=false' to `Hidden=true'. Make sure that you don't leave a backup copy of this file in the same directory, as both copies will be read, causing the console-kit-daemon to become very confused. Then, enable the svc:/system/consolekit:default service. Finally, if the svc:/application/graphical-login/gdm:default service is disabled, enable it. Running vncviewer on another host should show a normal gdm desktop from the headless server at this point. That's all there is to it.
Tunneling over SSH
Again, if you Google, you can find all sorts of complicated ways to use VNC by tunneling over SSH. However, all you really need to do is something like this:
1) In a terminal, connect to your machine via ssh:
$ ssh -X email@example.com
You should see something like this:
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Warning: No xauth data; using fake authentication data for X11 forwarding.
Last login: Fri Mar 8 22:29:36 2013 from ::ffff:127.0.0.
OpenIndiana (powered by illumos) SunOS 5.11 oi_151a7 October 2012
2) You have a couple of options here. I'll list what I think is the best option first.
a. Download RealVNC Viewer for Solaris
VNC Viewer for Solaris
5.0.5 x64 binary 8.3 MB
Copy the binary to your home directory.
Rename it something smaller, like, "realvnc"
Make it executable with something like "chmod +x realvnc"
Execute it with "./realvnc"
b. Alternatively, you could simply launch "vncviewer", which is part of the VNC install. It's not as nice as RealVNC's viewer, but it works. If you go that route, you may want to make a little script for it.
I made the following script, put it in my home directory, and named it "vnc".
vncviewer -geometry 1280x960 localhost
So, after logging in, all you would need to do is "./vnc"
Like I said, though, I think the RealVNC Viewer is a nicer option.
Notes about VNC Clients
My health hasn't been all that it could be for quite some time now, so I do most of my computer work on a MacBook Pro (laptop) while in bed. I thought while I was at it, maybe it might be helpful to some of you if I shared some of my experiences with the different clients I've used.
These comments are exclusively about my experience using these clients on my Mac connecting into OpenIndiana. Others should feel free to share your experiences, too, especially about clients on other platforms.
VNC Viewer for Solaris from RealVNC.com
VNC Viewer for Solaris
5.0.5 x64 binary 8.3 MB
For tunneling into a server via SSH, I think this is by far the best solution that I have found. It's fast, it looks nice, it doesn't seem to have any drawbacks that I can see, and it's free. It doesn't offer little perks like copy and pasting between the client and the server, however. If you want those, you either need to buy RealVNC's "VNC Viewer Plus", or find some other solution.
VNC Viewer that comes with OpenIndiana
TigerVNC Viewer for X version 1.0.1
Copyright © 2002-2005 RealVNC Ltd.
Copyright © 2000-2006 TightVNC Group
Copyright © 2004-2009 Peter Astrand for Cendio AB
See http://www.tigervnc.org for information on TigerVNC.
I used to use this viewer primarily when I needed a secure connection to a server, such as over the internet. It works well enough. It's a basic viewer, without many options. It doesn't look as "clean and crisp" as other viewers, however. I let it just auto adjust the encoding and pixel format, so perhaps this could have been improved if I had wanted to play around with that.
The biggest thing that I don't like about this viewer is that you cannot "CTRL Click" in Nautilus, or on the Desktop, etc. in the normal way.
To select multiple files in Nautilus, for instance, you have to open Nautilus, and click in it somewhere. Then you have to press the F8 Key to bring up the TigerVNC menu, select the CTRL menu item, so that it is checked. Then whenever you click on an item, it will be as if you CTRL-Clicked it. When you're done selecting your files, you have to press F8 again, and uncheck the CTRL button.
Very annoying. And I was never able to find any way to get around this. If anyone knows how, please feel free to post how to do that. Perhaps it's possible, but I haven't used this viewer since I tried RealVNC's viewer.
VNC Viewer for Mac OS X, downloaded from http://www.realvnc.com/
Built on Oct 2 2012 16:50:51
Copyright © 2002-2012 RealVNC Ltd.
VNC is a registered trademark of RealVNC Ltd. in the U.S. and in other countries.
This is for the free version - not the "VNC Viewer Plus", which offers other features, such as SSL and clipboard support. This is the Mac version, running on my laptop. When I used this, I did not tunnel into the servers through SSH.
This seems to work well enough.
It looks better than the TigerVNC viewer.
It does not have the CTRL Key problem that the TigerVNC viewer has.
The only real drawback I have found with this client is that you can only connect to one server at a time. It will not allow me to have multiple VNC windows open at the same time.
based on Chicken of the VNC by Jason Harris
based on VNCViewer by Helmut Maierhofer
released under the GNU Public License
source code and support available at http://chicken.sourceforge.net/
Copyright 1998-2000 by Helmut Maierhofer
Copyright 2002-2006 by Jason Harris
This is kind of a nice viewer.
You can configure the CTRL Key behavior by setting up your own profile.
Very minor issue: When I log in, I have to manually resize the window to the correct size.
It allows simultaneous connections to multiple servers.
It offers an option for tunneling through ssh, but the option does not work with OpenIndiana.
Bottom line - I recommend you use RealVNC's Solaris version. Tunnel in through ssh. Launch it connecting to localhost.
For insecure connections, I probably prefer RealVNC, as long as I don't need to connect to multiple servers at the same time. If you need to connect to more than one server simultaneously, then try Chicken.
I hope this helps!