Minimal OpenSolaris network install script

While preparing new version of zfsinstall, I've written sample version for minimal OpenSolaris installation over network. This script also includes disk partitioning. If complete installation is needed just replace pkg list with:

entire
SUNWcsd
SUNWcs

Try it if you've problems with graphics on old laptops and you can't run GUI installer.
You can find osinstaller script in MilaX mercurial repository on OpenSolaris.org.

Busybox on OpenSolaris

I hacked busybox and now it's working on my Solaris machine. This is not very fresh version (1.2.0) because all last releases become more Linux-specific. I've removed some applets and added "df","init","mount","umount","ps" from Solaris. Network applets like "ifconfig" are currently not implemented. For embedded Solaris we need native "ifconfig" with "plumb" function.

~$ uname -a
SunOS sysadm 5.11 snv_110 i86pc i386 i86pc
~$ ./busybox
BusyBox v1.2.0 (2009.04.20-10:06+0000) multi-call binary

Usage: busybox [function] [arguments]...
   or: [function] [arguments]...

 BusyBox is a multi-call binary that combines many common Unix
 utilities into a single executable.  Most people will create a
 link to busybox for each function they wish to use and BusyBox
 will act like whatever it was invoked as!

Currently defined functions:
 [, [[, awk, bunzip2, busybox, bzcat, cat, chgrp, chmod, chown, chroot, clear, cp, cpio, cut, date, dd, df, du, echo, ed, egrep, env, false,
 fgrep, find, grep, gunzip, gzip, halt, head, hostid, hostname, hush, id, init, kill, less, ls, md5sum, mkdir, mknod, mount, mv, nc, nslookup,
 poweroff, printf, ps, pwd, reboot, rm, sed, sh, sleep, sort, tar, tee, telnet, test, touch, tr, true, tty, umount, uname, unzip, vi, wc, whoami,
 xargs, yes, zcat

Drawing CPU part of boot chart

I'm using kstat for drawing CPU part of boot chart (see my previous posts). I need cpu values for "user","nice","system","idle","iowait" in such order (sample output):

cpu 8 0 141 313 230 0 0 

I've written the small kstat utility and added its call in a bootchartd script (utility writes /var/log/bootchart/proc_stat.log):

#include <kstat.h>
#include <sys/sysinfo.h>
#include <stdio.h>
#include <string.h>

static kstat_ctl_t *kc;
static kstat_t *ksp;

static unsigned int cpu_user, cpu_system, cpu_nice, cpu_idle, cpu_iowait;

int main(int argc, char **argv) {
cpu_stat_t *cpu_stat;

kc = kstat_open();

if (kc != NULL) {

ksp = kstat_lookup(kc, "cpu_stat", 0, "cpu_stat0");
if (ksp != NULL && ksp->ks_type == KSTAT_TYPE_RAW)
{
if (kstat_read(kc, ksp, NULL) != -1 &&  
ksp->ks_data_size == sizeof(cpu_stat_t))
{
  cpu_stat = (cpu_stat_t *)ksp->ks_data;
  cpu_user=cpu_stat->cpu_sysinfo.cpu[CPU_USER];
  cpu_nice=cpu_stat->cpu_sysinfo.cpu[CPU_WAIT];
  cpu_system = cpu_stat->cpu_sysinfo.cpu[CPU_KERNEL] ;
  cpu_iowait=cpu_stat->cpu_syswait.iowait;
  cpu_idle=cpu_stat->cpu_sysinfo.cpu[CPU_IDLE];

 printf("cpu  %d %d %d %d %d 0 0\n",
 cpu_user,cpu_nice,cpu_system,cpu_idle,cpu_iowait);
        }
    }
  }
}

Sample boot chart for MilaX:

Reducing Boot Time

In the last MilaX release (0.3.3) I've tried to reduce boot time from kernel startup to desktop environment up and running. I left only necessary SMF-services for LiveCD and changed startup sequence. The main idea - network services and compressed usr filesystem mounting can start after X server starting.
Here the services list after full booting:

STATE          STIME    FMRI
online         19:49:10 svc:/system/svc/restarter:default
online         19:49:13 svc:/network/datalink-management:default
online         19:49:14 svc:/system/filesystem/usr:default
online         19:49:14 svc:/system/filesystem/root:default
online         19:49:14 svc:/system/utmp:default
online         19:49:21 svc:/system/device/local:default
online         19:49:21 svc:/milestone/devices:default
online         19:49:22 svc:/system/filesystem/minimal:default
online         19:49:22 svc:/system/manifest-import:default
online         19:49:23 svc:/system/rmtmpfiles:default
online         19:49:23 svc:/system/filesystem/local:default
online         19:49:23 svc:/application/xserver:default
online         19:49:24 svc:/system/console-login:default
online         19:49:49 svc:/network/initial:default
online         19:49:57 svc:/network/loopback:default
online         19:49:58 svc:/system/system-log:default
online         19:50:01 svc:/system/cryptosvc:default
online         19:50:04 svc:/system/dbus:default
online         19:50:04 svc:/network/physical:default
online         19:50:06 svc:/system/identity:node
online         19:50:14 svc:/network/rpc/bind:default
online         19:50:18 svc:/system/hal:default
online         19:50:20 svc:/network/inetd:default
online         19:50:25 svc:/network/rpc/gss:default

Bootchart picture offers some details:

Certainly USB-booting yields the best results: for example on my DELL notebook I've booted MilaX from usb stick for 27 seconds. I've not included Moinak's code for compressed ramdisk support in this release, but I hope that I'll make it further.

Getting maximum info at booting

As I already written above, it's very useful to know all the current processes in boot time. You can get more interesting information. If we look at the contents of "live-devices-local" script in OpenSolaris LiveCD we'll see possibility to use iosnoop with output in logfile (iosnoop uses DTrace to monitor disk events in real time):

...
# Turn on I/O tracing if requested and possible
trace=`prtconf -v /devices|sed -n '/trace/{;n;p;}'|cut -f 2 -d\'`
if [ "$trace" = "on" ]; then
   if [ -n "$mntpt" ]; then
       outputfile="${mntpt}/traceout"
       echo "Enabling I/O Tracing ..." > /dev/console
       /opt/DTT/Bin/iosnoop -Deg > "$outputfile" 2> /dev/console &
       # Wait for iosnoop to actually initialize
       sleep 10
   else
       echo "Unable to enable I/O Tracing" > /dev/console
       echo "Must have a mountable Solaris root slice on harddisk" > /dev/console
       echo "to hold trace output" > /dev/console
   fi
fi
...

Here's another fast way without using "live-devices-local" script: boot with "-m milestone=none" and next (I'm using opensnoop):

mount -F tmpfs -o size=32m swap /var/log
/opt/DTT/Bin/opensnoop -eg > /var/log/opensnoop.log &
svcadm milestone all

Except iosnoop a lot of helpful information can be gathered using other DTrace scripts (opensnoop,errinfo) with various arguments. For example, the opensnoop script considerably simplifies search of all files necessary for booting without errors.

Boot chart with help of DTrace and Python

Anonymous tracing allows to receive more interesting information about Solaris booting process. I use a simple D-script (boot.d):

#!/usr/sbin/dtrace -Cs
#pragma D option quiet
/* I need process pid, his forks and life time */ 
proc:::create
{
 printf("<fork ppid=%d cpid=%d execname=%s time=%d />\n",
         pid,args[0]->pr_pid,execname,`lbolt*10/ `hz);
}

proc:::exec-success
{
 printf("<process pid=%d execname=%s time=%d />\n",
         pid,execname,`lbolt*10/ `hz);
}

proc:::exit
{

 printf("<end pid=%d execname=%s time=%d />\n", 
         pid, execname,`lbolt*10/ `hz);
}

Then I enable anonymous tracing:

dtrace -AFs /boot/boot.d
reboot

After reboot I write a logfile and disable anonymous tracing:

dtrace -ae -o bootlog
dtrace -A

Logfile looks like this:

CPU FUNCTION                                                                                                                                                 
  0 | exec_common:exec-success <process pid=1 execname=init time=63 />                                                                        
  0 | cfork:create             <fork ppid=1 cpid=4 execname=init time=66 />                                                                   
  0 | exec_common:exec-success <process pid=4 execname=ksh93 time=67 />                                                                       
  0 | exec_common:exec-success <process pid=4 execname=autopush time=87 />                                                                    
  0 | proc_exit:exit           <end pid=4 execname=autopush time=92 />                                                                        
  0 | cfork:create             <fork ppid=1 cpid=5 execname=init time=92 />                                                                   
  0 | exec_common:exec-success <process pid=5 execname=ksh93 time=92 />                                                                       
  0 | exec_common:exec-success <process pid=5 execname=soconfig time=94 /> 
...                   

Now by help of a python it's possible to parse logfile. I've written a small script bootchart.py for parsing and chart construction. It would be great to trace and draw the cpu part but this is not implemented yet.
To create boot chart you need run bootchart.py with logfile as an argument:

bootchart.py bootlog

This is a boot chart example for OpenSolaris 2008.11:

Boot charts for 2008.11 and snv101

Bootchart is a great tool for performance analysis and visualization of the Linux boot process. Eric Schrock and Dan Price worked on a Bootchart port using DTrace but I can't find any recipes on their blogs.
I've viewed bootchart source files for two days and by means of several hacks managed to receive boot charts for OpenSolaris. First of all I need to get uptime values in bootchart format (this is /proc/uptime content on linux). I've written the small utility (upt.c):

#include <sys/times.h>
#include <limits.h>
#include <dat/dat_platform_specific.h>

main {
struct tms  ts;
clock_t t = times(&ts);
                                                          
//we need jiffies from uptime value for bootchart
printf("%u\n",(unsigned long) ((DAT_UINT64) t*100 /CLK_TCK));                                                                    
}

Now the main script for logging (/boot/bootchartd):

#!/usr/bin/bash                                                                                         
                                                                                                  
PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/gnu/bin:$PATH"                                                 
                                                                                                  
# mounting dir for logging                                                                              
mount -F tmpfs -o size=32m swap /var/log/bootchart                                                                  
                                                                                                  
# start time hack for java parser                                                                       
echo "001" > /var/log/bootchart/ps.log                                                                  
echo "  PID  PPID S %CPU COMMAND" >> /var/log/bootchart/ps.log                                          
echo "    1     0 S  0.0 kernel" >> /var/log/bootchart/ps.log                                           
echo  >> /var/log/bootchart/ps.log                                                                      
                                                                                                  
while true; do                                                                                          
# uptime jiffies                                                                          
uptime=`/sbin/upt`                                                                                  
echo $uptime                                                                                        
# Log the command output and removing pid 0 for preventing "No processes found" error                                              
eval "ps -e -o pid,ppid,s,pcpu,comm"|grep -v sched                                                  
echo                                                                                                
/usr/gnu/bin/sleep 0.2                                                                              
done  >> /var/log/bootchart/ps.log      

Let’s create directory /var/log/bootchart:

mkdir /var/log/bootchart

Next I move /sbin/init to /bin:

mv /sbin/init /bin

and create a new init:

cat /sbin/init

#!/bin/sh

/boot/bootchartd &
exec /bin/init

Now it's possible to reboot. After booting I write a header file:

echo "title = Boot chart for $( hostname | sed q ) ($( date ))" > /var/log/bootchart/header             
echo "system.uname = $( uname -srvm | sed q )" >> /var/log/bootchart/header                             
if [ -f /etc/release ]; then                                                                            
echo "system.release = $( sed q /etc/release )" >> /var/log/bootchart/header                        
else                                                                                                    
echo "system.release = $( sed 's/\\.//g;q' /etc/issue )" >> /var/log/bootchart/header               
fi                                                                                                      

and create boot chart:

java -jar bootchart.jar

Parsing /var/log/bootchart
Wrote image: ./bootchart.png

This was the first boot chart created for OpenSolaris 2008.11 (Intel CPU 2.26GHz 512mb RAM). No disk and cpu utilization information yet:



Here the boot chart for snv101:

Installing pkg without pkg

Unfortunately I can't install some IPS packages without dependences. Christopher Bergström has written a small shell script for sunstudio installation. Some time I've tested it and have created the ruby version for installation of any package. This script does not process "driver" tag in manifest but in the rest is suitable for installation and a bit more exact than shell script. You can view the source code (getpkg.rb) here.
Sample usage (I want to install SUNWipkg to /opt/ipkg):

# getpkg inst_dir manifest_url 
getpkg.rb /opt/ipkg http://pkg.opensolaris.org/release/manifest/0/SUNWipkg%400.5.11%2C5.11-0.101%3A20081104T032047Z 

Preparing small miniroot with zfs and ssh

This is instructions for building smallest OpenSolaris miniroot (standalone RAM) with ssh and zfs.
I use the receptions familiar to funs of minimization according to Jim Connors's framework. But now we will use IPS for installation of packages and we should delete files of some packages manually ("--install" or "--uninstall --without-deps" not yet implemented in pkg).
Well, I'm running Solaris Express and will use "/opt/test" as working dir. Here are the packages for small miniroot (/opt/pkg.list):

SUNWcsd
SUNWcs
SUNWcakr
SUNWcar
SUNWckr
SUNWcnetr
SUNWcsl
SUNWesu
SUNWgss
SUNWgssc
SUNWkey
SUNWkvm
SUNWlibms
SUNWlibsasl
SUNWloc
SUNWlxml
SUNWnfsc
SUNWnfsckr
SUNWopenssl
SUNWos86r
SUNWpr
SUNWpsdcr
SUNWpsdir
SUNWrmodr
SUNWbip
SUNWsmapi
SUNWssh
SUNWsshcu
SUNWsshd
SUNWtecla
SUNWtls
SUNWtoo
SUNWusb
SUNWusbs
SUNWzfs
SUNWzfskr
SUNWzlib
SUNWzone
SUNWbash
SUNWrtls

As you can see, I have Realteck network adapter (SUNWrtls). Let's go:

# working dir
export ROOTDIR=/opt/test

# create the packaging db (pkg already installed on Nevada)
/usr/bin/pkg image-create -F -a opensolaris.org=http://pkg.opensolaris.org ${ROOTDIR}

cd ${ROOTDIR}
for pkg in `cat ../pkg.list`; do
/usr/bin/pkg install ${pkg}
done

But after that we have some additional packages because of dependences. Now we need to remove these unneeded packages (/opt/rm.txt) :

SUNWaudd
SUNWgccruntime
SUNWinstall-libs
SUNWmd
SUNWpicl
SUNWpool
SUNWwbsup

I cannot delete some packages and will use "pkg contents" command:

for pkg in `cat ../rm.txt`; do
pkg contents -m $pkg|grep file|sed -e 's/.*path=//; s/pkg.csize.*//'|\
awk '{print "./"$1}';
pkg contents -m $pkg|grep link|sed -e 's/.*path=//; s/target.*//'|\
awk '{print "./"$1}';
done > files.tmp

We need "libmeta.so" library for devfsadm:

cat files.tmp|sed 's/.*libmeta.*//g' > files.list

Now I delete packages files:

for file in `cat ../files.list `; do rm -rf $file; done

Basic miniroot configuration:

cd ${ROOTDIR}

# Copy usr/sfw/lib/libwrap.so.1 to usr/lib
cp ${ROOTDIR}/lib/svc/seed/global.db ${ROOTDIR}/etc/svc/repository.db

rm ${ROOTDIR}/usr/lib/libcrypto.so*
cp ${ROOTDIR}/usr/sfw/lib/libcrypto.so.0.9.8 ${ROOTDIR}/usr/lib


# Fix etc/vfstab
echo "/devices/ramdisk:a - / ufs - no nologging" >> ${ROOTDIR}/etc/vfstab

# Set the timezone
echo "TZ=US/Eastern" > ${ROOTDIR}/etc/TIMEZONE

# Set hostname
echo "small" > ${ROOTDIR}/etc/nodename

# Create /etc/inet/hosts and /etc/inet/ipnodes file
echo "127.0.0.1 localhost loghost" > ${ROOTDIR}/etc/inet/hosts
echo "192.168.1.1 small" >> ${ROOTDIR}/etc/inet/hosts
echo "::1 localhost loghost" > ${ROOTDIR}/etc/inet/ipnodes
echo "127.0.0.1 localhost loghost" >> ${ROOTDIR}/etc/inet/ipnodes
echo "192.168.1.1 small" >> ${ROOTDIR}/etc/inet/ipnodes

# Set nsswitch.conf
cp ${ROOTDIR}/etc/nsswitch.files ${ROOTDIR}/etc/nsswitch.conf

# Set IP Address
echo "small" > ${ROOTDIR}/etc/hostname.rtls0

# Set up default route
echo "192.168.1.200" > ${ROOTDIR}/etc/defaultrouter

# Set the environment variables for svccfg.
SVCCFG_DTD=${ROOTDIR}/usr/share/lib/xml/dtd/service_bundle.dtd.1
SVCCFG_REPOSITORY=${ROOTDIR}/etc/svc/repository.db
SVCCFG=/usr/sbin/svccfg

export SVCCFG_DTD SVCCFG_REPOSITORY SVCCFG

${SVCCFG} import ${ROOTDIR}/var/svc/manifest/milestone/sysconfig.xml

# turnoff boot-archive, manifest-import
${SVCCFG} -s system/boot-archive setprop start/exec=:true
${SVCCFG} -s system/manifest-import setprop start/exec=:true
${SVCCFG} -s network/rpc/bind delpg sysidtool
${SVCCFG} -s system/metainit:default setprop general/enabled=false

# Here's what needed to add ssh
${SVCCFG} import ${ROOTDIR}/var/svc/manifest/system/cryptosvc.xml
${SVCCFG} -s system/cryptosvc:default setprop general/enabled=true

${SVCCFG} import ${ROOTDIR}/var/svc/manifest/network/inetd.xml
${SVCCFG} -s network/inetd:default setprop general/enabled=true

${SVCCFG} import ${ROOTDIR}/var/svc/manifest/network/rpc/gss.xml
${SVCCFG} -s network/rpc/gss:default setprop general/enabled=true

${SVCCFG} import ${ROOTDIR}/var/svc/manifest/network/ssh.xml
${SVCCFG} -s network/ssh:default setprop general/enabled=true

# Creating RSA/DSA keys for ssh
/usr/bin/ssh-keygen -q -f ${ROOTDIR}/etc/ssh/ssh_host_rsa_key -t rsa -N ''
/usr/bin/ssh-keygen -q -f ${ROOTDIR}/etc/ssh/ssh_host_dsa_key -t dsa -N ''

# Edit etc/ssh/sshd_config to allow ssh to root account
ex -s ${ROOTDIR}/etc/ssh/sshd_config > /dev/null <<> /dev/null

# Remove packaging, xpg4, sfw - you can remove more but be carefully
rm -rf var/sadm/* var/pkg/*
rm -rf usr/xpg4 usr/sfw

Now I prepare a miniroot image:

# we need freespace ~10mb
size=`/usr/bin/du -ks ${ROOTDIR}|/usr/bin/nawk '{print $1+10240}'`

/usr/sbin/mkfile ${size}k /opt/miniroot
lofidev=`/usr/sbin/lofiadm -a /opt/miniroot`

# get the miniroot raw device and create a filesystem on it
rlofidev=`echo $lofidev |sed s/lofi/rlofi/`
/usr/sbin/newfs -m 0 $rlofidev

# mount ufs filesystem
/usr/sbin/mount -o nologging $lofidev /mnt

/usr/bin/find .  /usr/bin/cpio -pdum /mnt
/usr/sbin/umount /mnt

/usr/sbin/lofiadm -d /opt/miniroot
/usr/bin/7za a -tgzip -mx=9 /opt/miniroot.gz /opt/miniroot

Done. Now I have a small OpenSolaris 2008.11 miniroot with ZFS and SSH (32601K gzipped - I've also removed a number of unnecessary binaries from SUNWcs). It's possible to make an image even less.

Creating 90mb LiveCD using Distribution Constructor

Continuing a theme, it's possible to create a 90mb iso (minimal working LiveCD without Gnome and X) using the Distribution Constructor - command-line tool for building preconfigured, bootable OpenSolaris images. In the OpenSolaris 2008.11 release, this tool takes an XML manifest file as input, and builds an ISO image that is based on the parameters specified in the manifest file. I'm running Solaris Express and will use old-format configuration file.
I have added additional packages (SUNWgzip, SUNWgrub, SUNWslim-utils) in my short list (see post of December 08). Note: I need only "rtls" driver for this test so you should add packages for your network adapters. Besides, I use checkpoints as it will be necessary to change a script "/lib/svc/method/live-fs-root".
I have downloaded Distribution Constructor kit:

# hg clone ssh://anon@hg.opensolaris.org/hg/caiman/distro_constructor

I only have changed the list of packages and use my own checkpoint in "build_dist.bash":

# ./build_dist.bash small.conf

After installation of packages I have a pause (thanks to checkpoint). Now it's necessary to remove string "/usr/sbin/set_lang" in "/lib/svc/method/live-fs-root" and set "system/coreadm enabled=false" in generic_live.xml (otherwise there will be an errors after booting). It's only first test, you can change more. Then I finish creating the image. In DC output there will be some errors - the majority of them concerns Gnome which is not present.
I boot LiveCD in virtual machine - at first sight it works perfectly .
The test is fulfilled: for my list of packages the final iso has the size 90mb (but you can add any packages which are necessary for you).

OpenSolaris: life without Gnome

I have console OpenSolaris installed (see my post of December 08). But I have forgotten to install the most valuable package:

pkg install SUNWzfs-auto-snapshot

Now I can install X:

pkg install SUNWxorg-server
pkg install SUNWxorg-xsvc
pkg install SUNWxorg-agp
pkg install SUNWxorg-graphics-ddx
pkg install SUNWxorg-client-programs
pkg install SUNWxwfsw
pkg install SUNWxwopt
pkg install FSWxorg-fonts

After installation I run "xinit" and get xterm. However it would be better to install some minimalistic window manager, for example, dwm.

pkg install SUNWwget
wget http://code.suckless.org/dl/dwm/dwm-5.3.1.tar.gz

Gcc,binutils,gmake and headers are required:

pkg install SUNWxorg-headers
pkg install SUNWgmake
pkg install SUNWbinutils
pkg install SUNWgcc

I love vim. Also I need gzip:

pkg install SUNWvim
pkg install SUNWgzip

If you plan to compile some other apps, I advice to install substudio:

pkg install sunstudioexpress

Next I unpack archive, correct config.h and compile dwm. It's possible to start dwm xsession from smf. You can look how it's made in BeleniX or MilaX.


As a result I got system with short boot time and an easy and fast window manager.
At the end I need usual user:

mkdir /home
useradd -d /home/me -m -s /bin/ksh -c "System Developer" me
passwd me