You are on page 1of 171

SOLARIS 10 DEEP DIVE: SMF

SOLARIS 10
DEEP DIVE: SMF
Christine Tran
US Solaris Adoption Practice
Sun Microsystems

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Topics
• Part I: SMF Fundamentals
> Motivation for SMF
> Core concepts and terminology
> Command overview
> New boot process
• Part II: SMF in Action
> Quick tour of service administration
> Managing inetd services
> Tips for troubleshooting
> Recovering a repository

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Topics
• Part III: Advanced SMF
> Inside the manifest
> A little about XML
> Writing a manifest from scratch
> Exploring the repository with svccfg and svcprop
• Part IV: More Advanced SMF
> Security aspects
> Snaphots
> Handling signals
> Portable services

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Topics
• Part V: SMF Tools and SMF Community
> Tools: Webmin
> Tools: xmllint, jedit, Morphon
> SMF Community on OpenSolaris
> Resources
> Review
> Q&A
> Closing

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Part I:
SMF Fundamentals

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Motivation for SMF


• What is a service?
• Is running a software program the same as delivering a
service? What exactly is a service?
> Answer
> Answer
> Answer
• Does there currently exist an OS framework for service
support and management?
> Answer
> Answer
> Answer

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

How are services started today?


• init(1M) vi rc scripts in /etc/rc?.d/*
> Long-time running or one-time initializations
• inetd(1M) as defined by inetd.conf
> Short-lived to provide network functions
• init(1M) as defined by /etc/initab
> Restartable or one-time functions
• How are these services grouped? Could they be related?
How are the relationships described?

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Diagnostic ability
• An application fails to start
> Missing configuration file?
> Failed to mount a filesystem?
> Database is late in starting up?
> Missing another component?
• Lack of knowledge of service boundary and service
interdependencies limits error handling ability of system.
• Lacks common framework of error-handling
• Lacks restart capability

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Service Management Today


• Thousands of different text files, arbitrarily grouped and
managed with multiple administration techniques.
• Undeclared, often unknown dependencies; linear startup
is a by-product.
• Lacks common interface, each service is started a
different way.
• Does not address multiple instances of a service, for
example, web servers.
• Does not address services that may span multiple hosts,
for example, a grid.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Motivation for SMF

A Service Oriented Architecture


requires a definition of service
and a more robust framework to
deliver and manage it.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

SMF gives you


• A consistent service model in a common framework:
command interface, service description, property
specification, status view, etc.
• A meaningful system view.
• Ability to state dependencies (by-product is parallel boot).
• Restart capabilities
• All data stored in persistent, transaction-based repository.
• Snapshots allow undo and rollback to a working
configuration.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology


• A service is:
> An abstract description of a long-lived software object.
> May describe object on a collection of machines (grid).
> May have multiple instances (httpd).
> Each instance of a service has a well-defined state and a well-
defined error boundary [process contract].
> Each service defines methods: start, stop, refresh, etc.
> Each service defines dependencies: what it depends on, and
may define dependents: what depends on it.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology


• A service has a unique identifier called FMRI – Fault
Management Resource ID

svc://localhost/network/login:rlogin
Scheme
svc – SMF managed service
lrc – legacy RC script

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology


• A service has a unique identifier called FMRI – Fault
Management Resource ID

svc://localhost/network/login:rlogin
Location
localhost – current
hostname – future release

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology


• A service has a unique identifier called FMRI – Fault
Management Resource ID

svc://localhost/network/login:rlogin
Functional category
application
system
device
network
milestone
platform
site
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Functional Categories
• Application – traditional daemons
• Device – useful for dependencies
• Milestone – similar to run levels
• Network – converted services from inetd.conf
• Platform – platform-specific services
• System – platform-independent system services
• Site – reserved for local site use

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology


• A service has a unique identifier called FMRI – Fault
Management Resource ID

svc://localhost/network/login:rlogin
Service Description – related to method or RC script

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology


• A service has a unique identifier called FMRI – Fault
Management Resource ID

svc://localhost/network/login:rlogin
Instance – “default” is the default instance

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology


• FMRI can be addressed by the shortest unique match, for
example:
> svc://localhost/network/login:rlogin
> svc:/network/login:rlogin
> network/login:rlogin
> rlogin
• Some common names have changed to a different FMRI,
for example: syslog is now system-log

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Service states
• online – the service instance is enabled and has
successfully started.
• offline – the service instance is enabled, but the service is
not yet running or available to run, usually due to a
dependency that has not been satisfied, or an error in the
start method.
• disabled – the service instance is not enabled and is not
running.
• maintenance – the service instance has encountered an
error that must be resolved before it could be started
again. There are many reasons why a service could be in
this state.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Service states
• legacy_run – the legacy service is not managed by SMF,
but the service can be observed.
> Faults not handled by SMF, no automated restart
> Administrative error undetected
> Software or hardware error results in process death
• degraded – the service instance is enabled, but is running
at a limited capacity.
• uninitialized – this state is the initial state for all services
before their configuration has been read.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Service dependency
• A dependency is a formal description of other services
that are required to start a service.
• A service can be dependent on another service or files.
When a dependency is not met, the service stays offline.
Whether a dependency is satisfied depends on its type:
> require_all – all services are running or all files are present.
> require_any – at least one is running or at least one file is
present.
> optional_all – all are running, disabled, in maintenance, or not
present. For files, this type is the same as require_all.
> exclude_all – all are disabled, in maintenance, or when files
are not present.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Dependency actions
• Once running (online or degraded states are considered
running) if a service dependency is stopped or refreshed,
SMF determines whether to restart the service depending
on the restart_on attribute specified.

restart_on attribute
Reason for dependency stop None Error Restart Refresh
Error no yes yes yes
Non-error stop no no yes yes
Refresh no no no yes

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Service restarter
• /lib/svc/bin/svc.startd is the master process starter and
restarter. It
> reads the Service Configuration Repository and start services
on boot;
> restart services that have failed;
> shutdown services whose dependencies are no longer
satisfied;
> run legacy RC scripts at run-level transitions;
> provides system view of service status.
• There is a delegated restarter, inetd.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Delegated restarter
• Assumes responsibility for executing a set of services.
• Manages service faults; may gracefully handle conditions
such as signals or core dumps.
• May define/require additional service configuration.
• Often provides functionality common to a specific set of
services, for example: inetd

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

SMF Manifest
• Description of the initial configuration of a service in XML.
• Loaded into the SMF repository at boot time.
• Sun-delivered services live in /var/svc/manifest.
• For ISVs, manifests should be placed in the appropriate
subdirectory /var/svc/manifest.
• Manifests for anything that is specific to the customer's
site only can go in /var/svc/manifest/site. This is reserved
for local use.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

SMF Manifest
• A manifest identifies, at the very least:
> the service name
> what the service is dependent on
> ways to start and stop the service
• Can contain other attributes, such as the delegated
restarter, config files, log files and others.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Service Configuration Repository


• Located in /etc/svc.
• Distributed between local memory (volatile) and local file
(repository.db).
• repository.db holds the persistent service description read
from the manifests.
• volatile is a tmpfs allocated from swap containing
transient data (lock files, init state, log files).
• The underlying DB engine is sqlite 2.8
• Zones have their own repository.
• Changes to services should be made against repository.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Profile and Archive


• Profile – a set of service instances and their enabled or
disabled state. Useful for copying service states between
systems.
• Generated by svccfg extract
• Archive – a complete set of persistent data for all service
instances. Useful for copying service definitions between
systems.
• Generated by svccfg archive
• Output is an XML file similar to manifests.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

SMF Snapshots
• snapshot – a complete collection of properties for a
service instance.
• It provides a historical view of a service and simplifies
rollback of service configuration changes.
• No snapshot-at-will capability yet.
• Snapshot types:
> initial – taken on the first import of the manifest
> last_import – taken during last import
> running – properties of the currently running service instance
> previous – properties of the service instance just before running
> start – taken at the last successful service start
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Key files and directories


• /var/svc – manifests, logs and profiles
• /lib/svc – binaries, methods, seed, support files
• /etc/svc/repository.db – global configuration database
• /lib/svc/seed/global.db – minimal configuration database
• /lib/libscf.so.1 – repository APIs

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

New filesystems
• /etc/svc/volatile
> Directory where svc.startd stores log files in early stages of
boot, before /var is mounted read-write. Contains transient
data.
• /system/contracts
> Primary interface into contract subsystem, to allow service
instances to be restarted. Indirectly related to SMF.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Command overview
• General commands
> svcs(1) service status listings
> svcadm(1M) administrative actions
> svccfg(1M) general property manipulation
> svcprop(1) property reporting
• inetd management commands
> inetadm(1M) administrative actions/property modification
> inetconv(1M) conversion of legacy inetd.conf entries

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Command overview
• Daemons
> svc.startd(1M) dependency engine, master restarter
> svc.configd(1M) repository, authorization
> inetd(1M) delegated restarter
• Contracts subsystem
> ctrun(1M) execute with process contract
> ctstat(1M) display active contracts
> ctwatch(1M) monitor contract events
> libcontract(3LIB) contract APIs

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

New boot process


• Instead of booting to run level, SMF introduces the
concept of milestone.
• milestone – a service which specifies a collection of
services which declare a specific state of system-
readiness.
• You can boot to specific milestone, services not part of
that milestone are temporarily disabled.
• The default milestone is the milestone you will transition
to at every boot. It is usually milestone “all”.
• init S, boot -s and friends still work.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Milestones and run levels

SVR4 Run Level SMF Milestone


- none
s, S single-user
2 multi-user
3 multi-user-server
- all

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Legacy services
• Milestones are services, as such, they have manifests
which have defined dependencies.
• After all its dependencies have been satisfied, the start
method of the SMF major milestone executes each 'S'
script within /etc/rc?.d/ with the argument 'start'.
• /var/svc/manifest/milestone/single-user.xml specifies a
series of dependencies, and the start method is
/sbin/rcS start.
• SMF services first, legacy services last, run once only.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Part II:
SMF in Action

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Service administration
svcs – reports service status
• svcs -a show all services
• svcs -x show services not running, impact
• svcs -d show dependencies (what this depends on)
• svcs -D show dependents (what depends on this)
• svcs -p show member processes
• svcs -l/-v show additional details

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svcs
# svcs -a
STATE STIME FMRI
legacy_run Oct_12 lrc:/etc/rc2_d/S20sysetup
online Oct_12 svc:/network/ssh:default
disabled Oct_12
svc:/application/print/server:default
offline
Oct_12svc:/application/print/rfc1179:default
...

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svcs
# svcs -x
svc:/application/print/server:default (LP print
server)
State: disabled since Wed Oct 12 16:18:38 2005
Reason: Disabled by an administrator.
See: http://sun.com/msg/SMF-8000-05
See: lpsched(1M)
Impact: 2 dependent services are not running. (Use
-v for list.)

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svcs
# svcs -d http
STATE STIME FMRI
online Oct_12 svc:/network/loopback:default
online Oct_12 svc:/network/physical:default
# svcs -D loopback
STATE STIME FMRI
disabled Oct_12
svc:/network/dns/client:default
disabled Oct_12
svc:/network/dns/server:default
disabled Oct_12 svc:/network/rarp:default

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svcs
# svcs -p sendmail
STATE STIME FMRI
online 16:09:32 svc:/network/smtp:sendmail
16:10:32 23724 sendmail
16:10:32 23725 sendmail

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svcs
# svcs -l rarp
fmri svc:/network/rarp:default
name Reverse Address Resolution Protocol
(RARP) server
enabled false
state disabled
next_state none
state_time Wed Oct 12 16:18:41 2005
restarter svc:/system/svc/restarter:default
dependency require_all/error
file://localhost/etc/nsswitch.conf (online)
dependency require_any/error

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Service Administration
svcadm – administer services
-t temporary
-r recursive
-s synchronously waits for completion
• svcadm enable/disable enable/disable services
• svcadm enable/disable -t temporarily, until rebooted
• svcadm restart restart specified service
• svcadm refresh re-read service configuration
• svcadm clear/mark clear/mark maintenance state
• svcadm milestone transition to a milestone
• svcadm -d milestone transition and set as default
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

svcadm
# svcs sendmail
STATE STIME FMRI
online 16:49:59 svc:/network/smtp:sendmail
# svcadm disable -t sendmail
# svcs sendmail
STATE STIME FMRI
disabled 16:51:27 svc:/network/smtp:sendmail
• Does not show service in temporary state. The presence
of this property tells us that this setting is temporary.
# svcprop sendmail | grep general_ovr/enabled
general_ovr/enabled boolean false

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svcadm keyserv
# svcs keyserv
STATE STIME FMRI
disabled Oct_12
svc:/network/rpc/keyserv:default
# svcadm enable keyserv
# svcs keyserv
STATE STIME FMRI
maintenance 16:58:57
svc:/network/rpc/keyserv:default
• Keyserv is in maintenance mode, why?

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svcadm keyserv
# svcs -x
svc:/network/rpc/keyserv:default (RPC encryption key
storage)
State: maintenance since Wed Oct 19 16:58:57 2005
Reason: Start method exited with
$SMF_EXIT_ERR_CONFIG.
See: http://sun.com/msg/SMF-8000-KS
See: keyserv(1M)
See: /var/svc/log/network-rpc-keyserv:default.log
Impact: This service is not running.
• Actually, it is because the domain name is not set on this
host.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svcadm keyserv
# domainname test.com
# domainname
test.com
• There is a service called domain, which takes care of
system identifying info; it has to be restarted.
# svcs -o fmri,desc domain
FMRI DESC
svc:/system/identity:domain system identity(domainname)
# svcs domain
STATE STIME FMRI
online Oct_12 svc:/system/identity:domain
# svcadm restart domain

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svcadm keyserv
# svcs keyserv
STATE STIME FMRI
maintenance 16:58:57
svc:/network/rpc/keyserv:default
• Keyserv is still in maintenance, why?
# svcadm clear keyserv
# svcs keyserv
STATE STIME FMRI
online 17:02:48
svc:/network/rpc/keyserv:default
• Will this survive a reboot?

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svcadm milestone
• What milestone am I in?
# svcprop -p options_ovr/milestone
system/svc/restarter:default
svcprop: Couldn't find property
`options_ovr/milestone' for instance
`svc:/system/svc/restarter:default'.
• If you see this, you are in milestone “all”.
# svcadm milestone single-user
# Requesting System Maintenance Mode
Console login service(s) cannot run
Oct 19 17:19:00 zone1 syslogd:going down on signal 15
svc.startd: The system is coming down for
administration. Please wait.
svc.startd: Killing user processes: done.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

svcadm milestone
svc.startd: Killing user processes: done.
Root password for system maintenance (control-d to
bypass):
single-user privilege assigned to /dev/console.
Entering System Maintenance Mode
# svcprop -p options_ovr/milestone
system/svc/restarter:default
svc:/milestone/single-user:default
# reboot
# svcprop -p options_ovr/milestone
system/svc/restarter:default
svcprop: Couldn't find property
`options_ovr/milestone' for instance
`svc:/system/svc/restarter:default'.
• I'm back in milestone “all”.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

svcadm milestone
# svcadm milestone -d single-user
# Requesting System Maintenance Mode
# reboot
# svcprop -p options_ovr/milestone
system/svc/restarter:default
svc:/milestone/single-user:default
• Every subsequent reboot will get me back here. To get
back to full service, set default milestone to be “all”.
# svcadm milestone -d all

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Managing inetd services


• Definition for inetd-started services were in inetd.conf but
inetd.conf is now deprecated. Look how short it is!
• inetd.conf entries are converted to SMF manifests and
imported into the repository on initial boot.
• The old way of adding a service by modifying inetd.conf
and HUPing inetd no longer works.
• If you have an application that depends on what's in
inetd.conf, there is a tool called inetconv that will convert
the inetd.conf entry to an SMF manifest.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

inetd
• inetd is now a delegated restarter for inet services.
• It manages inet-specific properties such as
> tcp_wrappers
> max connection rate
• It manages state for inet services.
• It listens for connections, etc.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

inetconv
# grep tftp inetd.conf
# TFTPD - tftp server (primarily used for booting)
# tftp dgram udp6 wait root /usr/sbin/in.tftpd
in.tftpd -s /tftpboot
• It's commented out. Uncomment the entry. You can
HUP inetd but take my word that that won't do anything.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

inetconv
# inetconv
inetconv: Notice: Service manifest for 100235/1
already generated as /
var/svc/manifest/network/rpc/100235_1-
rpc_ticotsord.xml, skipped
tftp -> /var/svc/manifest/network/tftp-udp6.xml
Importing tftp-udp6.xml ...Done
inetconv: Notice: Service manifest for 100083/1
already generated as /
var/svc/manifest/network/rpc/100083_1-rpc_tcp.xml,
skipped
inetconv: Notice: Service manifest for 100068/2-5
already generated as /
var/svc/manifest/network/rpc/100068_2-5-rpc_udp.xml,
skipped

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

inetconv
# svcs tftp
svcs: Pattern 'tftp' doesn't match any instances
# svcs -l udp6
fmri svc:/network/tftp/udp6:default
name tftp
enabled true
state online
next_state none
state_time Wed Oct 19 17:40:16 2005
restarter svc:/network/inetd:default

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Service Administration
inetadm – administer inetd-started services
• inetadm list all services managed by inetd
• inetadm -p show default inetd service property values
• inetadm -l show all properties for a service
• inetadm -e/-d enable/disable a service
• inetadm -m modify property values
• inetadm -M modify inetd default property values

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

inetadm
# inetadm -l udp6
SCOPE NAME=VALUE
name="tftp"
endpoint_type="dgram"
proto="udp6"
isrpc=FALSE
wait=TRUE
exec="/usr/sbin/in.tftpd -s /tftpboot"
user="root"
...
• Let's say you want to run with some other option, -X
instead of -s.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

inetadm
# inetadm -m udp6 exec="/usr/sbin/in.tftpd -X /
tftpboot"
# inetadm -l udp6
SCOPE NAME=VALUE
name="tftp"
endpoint_type="dgram"
proto="udp6"
isrpc=FALSE
wait=TRUE
exec="/usr/sbin/in.tftpd -X /tftpboot"
user="root"

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

inetadm
• Can also use inetadm to manage inetd-services as well
as svcadm.
# svcs udp6
STATE STIME FMRI
online 17:49:44
svc:/network/tftp/udp6:default
# inetadm -d udp6
# svcs udp6
STATE STIME FMRI
disabled 18:01:53
svc:/network/tftp/udp6:default

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

inetadm
• Usually if you don't use a service, it's enough to disable it
using svcadm. But you want it really gone then delete it
from your repository using svccfg delete FMRI.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Service Administration
svcprop – get service configuration properties
• -p property FMRI fetch a property
• -s snapshot FMRI fetch property from the snapshot

# svcprop -p inetd_start/exec tftp/udp6


/usr/sbin/in.tftpd\ -s\ /tftpboot
# svcprop -p restarter/state tftp/udp6
online

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Service Administration
svccfg – modify service configurations
• select FMRI select a service
• listprop show properties for a service
• setprop change a property value for a service
• delprop delete a property value for a service
• delete delete a service
• export|import export|import a service description in XML
• extract|apply extract|apply a profile
• archive dump a full XML service description for
all services
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

svccfg
# svccfg -s print/server listprop 'lpsched/fd_limit'
lpsched/fd_limit count 0
# svccfg
svc:> select print/server
svc:/application/print/server> listprop
lpsched/fd_limit
lpsched/fd_limit count 0
svc:/application/print/server> setprop
lpsched/fd_limit = 8192
svc:/application/print/server> exit
# svcadm refresh print/server
# svccfg -s print/server listprop 'lpsched/fd_limit'
lpsched/fd_limit count 8192

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svccfg
• Can be used in batch or interactive mode.
• Understands globbing to form regex
• We'll work through a more pertinent example later.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Tips for troubleshooting


• If a single service is broken, -x is your friend. svcs -x will
display the service logfile, if it exists.
• See service start message:
> boot -m verbose.
• For a system that hangs during boot:
> boot -m verbose,milestone=none
> log in at prompt
> svcadm milestone all
• Watch system progress with svcs.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Tips for troubleshooting


• truss the service starting up by trussing the restarter
• Or modify the start method, wrapping it in truss
# svcprop -p start/exec apache2
/lib/svc/method/http-apache2 start
# svccfg -s apache2
svc:/network/http:apache2> setprop start/exec =
“/usr/bin/truss -ealfo /tmp/apache2.truss
/lib/svc/method/http-apache2 start”
• Don't forget svcadm clear once fault is rectified.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Last resorts
• Have changes been made to repository?
> How many services are affected?
> What snapshots exist?
> Can you revert back to a previous snapshot?
• Read /lib/svc/share/README.
• Restore repository from backup.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Revert a snapshot
# svccfg -s svc:/network/dns/server:default
svc:/network/dns/server:default> listsnap
initial
last-import
previous
running
start
svc:/network/dns/server:default> revert start
svc:/network/dns/server:default> exit
# svcadm refresh dns/server
# svcadm restart dns/server

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Recovering a repository
• The SMF repository can be recovered in case of
corruption.
• You may want to save the current repository in
/etc/svc/repository.db for post mortem.
• Restore script is /lib/svc/bin/restore_repository.
• If you restore from seed, next boot would be the initial
boot, you will see SMF reading and importing manifests.
# zlogin -C zone1
[Connected to zone 'zone1' console]
40/104

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

restore_repository
# /lib/svc/bin/restore_repository
The following backups of /etc/svc/repository.db
exist, from oldest to newest:
boot-20051012_125317
manifest_import-20051012_130044
boot-20051012_161835
boot-20051019_172301
boot-20051019_180815
# Please enter one of:
1) boot, for the most recent post-boot backup
2) manifest_import, for the most recent
manifest_import backup.
3) a specific backup repository from the above list

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

restore_repository
4) -seed-, the initial starting repository. (All
customizations will be lost.)
5) -quit-, to cancel.
Enter response [boot]: -seed-
After confirmation, the following steps will be
taken:
svc.startd(1M) and svc.configd(1M) will be quiesced,
if running.
/etc/svc/repository.db -- renamed -->
/etc/svc/repository.db_old_20051019_182048
/lib/svc/seed/nonglobal.db -- copied -->
/etc/svc/repository.db
and the system will be rebooted with reboot(1M).

Proceed [yes/no]? yes


Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Part III:
Advanced SMF

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Manifest revisited
• Manifest is the description of the initial configuration of a
service, delivered in XML.
• Automatically imported into repository on boot, install,
upgrade, and pkgadd by svc:/system/manifest-import.
• Ignored once imported unless modified.
• Do not modify Sun-delivered manifests! If you need to
change service properties, do it in the repository. We'll
talk more about this.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Order is important
• Service bundle
• Service name
• Create default instance
• Single instance
• Dependencies
• Exec methods
• Property groups
• Stability level
• Template

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

A little about XML


• XML markups contain data about data, metadata, on the
information they bracket.
> <time>1200</time>
• Tells you at 1200 is a time unit, instead of weight or
distance. Whatever processes XML data can recognize
that data for what it is.
• Can contain any kind of data. How do we differentiate?
An XML file of train schedule will have different formatting
than an XML file of TV schedule, not just formatting, but
different processing and different usage. Yet,
<time>1200</time> can appear in both XML files.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Introducing DOCTYPE
• Look at the top of each manifest, they all have this line:
> <!DOCTYPE service_bundle SYSTEM
'/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
• DTD – Document Type Definition, contrast with
DOCTYPE for HTML.
• Tells you whether you (especially if you are an XML-
processing program) are reading a train schedule or a TV
Guide listing.
• DTD is one example of a type of markup called
declaration. Specifically, the DTD is used to constrain the
grammar of an XML file.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Grammar?
• Grammar governs the “correctness” of an XML document.
A train schedule must have: source, terminus, track
number, departure and arrival time. It won't have
channel, show name, show time, duration, starring, rating.
• Looking at just data, they look very similar. The DTD
defines and enforces how they must be marked up in
XML, one way and one way only.
• DTD is a collection of rules, or declarations, describing
markup objects.
• An element declaration defines what it can contain and in
which order, also known as a content model.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Define your own language


• Strictly speaking, XML is not a markup language. It has
no tag with absolute meaning. <time> can mean
absolute time or elapsed time. Contrast this with
<b></b>, this always means bold.
• XML lays down framework for you to construct your own
markup language.
• Adding elements in the DTD adds objects to your
language. An element not declared in the DTD is illegal.
An element containing anything not declared in the DTD
is illegal.
• We'll see why this is important later on.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

What's in the DTD?


• A bunch of things! None resembling what you might find
in a manifest.
> <!-- comments --> are comments
> <!ELEMENT> or <!ATTLIST> are declarations
> <blah /> is same as <blah></blah>
• Ordering in DTD is not same as the required ordering in
manifest. For example, declaration for property group is
before declaration of service FMRI, and service bundle is
declared last.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Here's an snippet
<!ELEMENT dependency
( service_fmri*, stability?, ( propval |
property )* ) >

<!ATTLIST dependency
name CDATA #REQUIRED
grouping ( require_all | require_any |
exclude_all |
optional_all ) #REQUIRED
restart_on ( error | restart | refresh |
none ) #REQUIRED
type CDATA #REQUIRED
delete ( true | false ) "false" >

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Why XML?
• Think about what's in /etc/rc?.d today. Anyone can write
anything in any format and drop it in /etc/rc?.d. In fact,
what's in /etc/rc?.d is closer to methods than manifests.
• A unified service management tool depends on having
uniform data to process. The repository is a database
which relies on having correct data in the correct “cell”.
• With XML you get a parser and a validator for free.
Anything that XML can't validate won't get imported.
• The parser is in a library called libxml2. It comes with
xmllint, which is a generic XML validator.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

A quick tour inside the manifest


• We said a manifest must have at least
> the service name
> what the service is dependent on
> ways to start and stop the service
> in theory you need a restarter too but there is the default
svc.startd.
• House-keeping robot analogy
> Lights
> Depends on Power
> On/Off switch on the wall, flip up or down
> Robot is the default restarter

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

tftp-udp6.xml
service
name='network/tftp/udp6'
type='service'
<exec_method
type='method'
name='inetd_start'
exec='/usr/sbin/in.tftpd -s /tftpboot'
<exec_method
type='method'
name='inetd_offline'
exec=':kill_process'

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

tftp-udp6.xml
<restarter>
<service_fmri
value='svc:/network/inetd:default' />
</restarter>

• Note that the method isn't “start”, but “inetd_start”.


• Wait! No dependencies? Well, having inetd as your
delegated restarter implicitly marks you as dependent on
inetd. Who starts inetd, then? It's started in
/var/svc/profile/generic_open.xml
• If you look at /var/svc/manifest/network/nfs/server.xml,
you will see that there is no explicitly defined restarter.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Writing a manifest from scratch


• Not all Sun products are delivered in a manifest, the
reason seems to be time and effort. The README file for
MySQL still says to link a startup script in /etc/rc3.d to the
mysql.server script.
• If no manifest is delivered, the service can still run as a
legacy service.
• We're going to convert MySQL into an SMF service.
• You may want to do this in a zone. Remember that each
zone has their own manifests and SMF repository.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Setting up MySQL database


• First, we need to set up the database. Instructions are
in /etc/sfw/mysql/README.solaris.mysql. The startup
script is /etc/sfw/mysql/mysql.server.
root@mysql~> /usr/sfw/bin/mysql_install_db
Preparing db table
Preparing host table
Preparing user table
Preparing func table
Preparing tables_priv table
Preparing columns_priv table
Installing all prepared tables
13:38:46 /usr/sfw/sbin/mysqld: Shutdown Complete

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

MySQL
• Add users and groups
root@mysql~> groupadd mysql
root@mysql~> useradd -g mysql mysql
root@mysql~> chgrp -R mysql /var/mysql
root@mysql~> chmod -R 770 /var/mysql
• Add configuration files
root@mysql~> installf SUNWmysqlr /var/mysql d 770
root mysql
root@mysql~> cp /usr/sfw/share/mysql/my-medium.cnf
/var/mysql/my.cnf

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

MySQL
• Start MySQL manually to see that everything works
root@mysql~> /usr/sfw/sbin/mysqld_safe --user=mysql &
Starting mysqld daemon with databases from /var/mysql
• Set the passwords
root@mysql~> /usr/sfw/bin/mysqladmin -u root password
'mysql'
root@mysql~> /usr/sfw/bin/mysqladmin -u root -h mysql
password 'mysql'
• Check the status of MySQL
root@mysql~> /usr/sfw/bin/mysqladmin -u root -pmysql
status
Uptime: 37 Threads: 1 Questions: 1 Slow queries: 0
Opens: 6 Flush tables: 1 Open tables: 0 Queries
per second avg: 0.02
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Copy and modify


• Great! Everything works. Now you are ready to write
your manifest!
• Find a manifest similar to your application and tweak it.
Since MySQL is an application, I'll look in
/var/svc/manifest/application. The print/server.xml seems
closest to what I want, so I'll start from that. Remember
that site-specific manifests go into “site”.
• Most everything is already filled out, I just change a few
relevant properties.
root@mysql~> cp
/var/svc/manifest/application/print/server.xml
/var/svc/manifest/site/mysql.xml

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Name
• Change the name
<service_bundle type='manifest' name='SUNWsqlr:mysql'>
<service
name='application/mysql'
type='service'
version='1'>

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Dependency
• I leave the existing dependencies alone. I may want to
add a dependency on the config file, so MySQL doesn't
start if it's missing.
<dependency
name='database'
grouping='require_all'
restart_on='none'
type='path'>
<service_fmri
value='file://localhost/var/mysql/my.cnf'/>
</dependency>

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

This just in
• File dependencies going may be going away.
• CR 6240573
> Synopsis: File dependencies are currently overused by SMF
consumers. The problem is that a file dependency means "wait
for this file to become available before starting". That's not the
usual case. Most users of file dependencies are trying to
express the concept that "if this file is missing, there is a
problem."
• No file event to signal SMF if a file has changed.
• Actions in a file dependency doesn't do anything, yet.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Methods
• Change the start/stop methods
<exec_method
type='method'
name='start'
exec='/etc/sfw/mysql/mysql.server start'
timeout_seconds='60' />
<exec_method
type='method'
name='stop'
exec='/etc/sfw/mysql/mysql.server stop'
timeout_seconds='60' />

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Description and documentation


• For now, delete the property groups as they don't apply to
mysql
• Change the description and documentation location
<loctext xml:lang='C'>
MySQL server
</loctext>
<documentation>
<manpage title='mysql' section='1M'
manpath='/usr/sfw/man' />
</documentation>

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Importing your manifest


• You're done! Now import it.
root@mysql~> svccfg import mysql.xml
root@mysql~> svcs -l mysql
fmri svc:/application/mysql:default
name MySQL server
enabled false
state disabled
next_state none
state_time Tue Jun 07 19:20:51 2005
restarter svc:/system/svc/restarter:default

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Importing your manifest


dependency require_all/none
svc:/system/filesystem/local (online)
dependency require_all/none
svc:/system/filesystem/usr (online)
dependency require_all/refresh
svc:/system/identity:domain (online)
dependency optional_all/none svc:/system/system-log
(online)
dependency require_all/none
file://localhost/var/mysql/my.cnf (online
root@mysql~> svcadm enable mysql
root@mysql~> svcs mysql
STATE STIME FMRI
online 19:21:19
svc:/application/mysql:default
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Done!
• Congratulations! You have successfully written a service
manifest. Now MySQL is an SMF-managed service.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

SMF on OpenSolaris
• Collection of manifests and methods are at
http://www.opensolaris.org/os/community/smf/manifests/
• There is a different manifest for MySQL submitted there.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Exploring the repository


• We said the repository is a database containing all
descriptions of all services SMF knows about, and their
states.
• Make changes to service against its content in the
repository, not in the manifest. The days of looking in a
text file to find what's running is over. In fact, what's in a
manifest can differ vastly with the running configuration.
• We'll fix Bug ID 6270245 as an example.
• Bug ID: 6270245
> Synopsis: apache2 needs svc:/system/filesystem/local
dependency when Sun Volume Manager metadevices are in
use
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Two ways to fix


• Two ways to fix:
• Delete the service from the repository, fix the manifest
and re-import. This is not the recommended way.
• Fix the currently running service configuration. This is the
recommended way, and we'll talk about why after we've
finished the exercise.
• We're going to do this both ways, in order to better
understand the repository and the manifest.
• In this case the service can be addressed as http or
apache2, http is the service, apache2 is the instance.
• Disable the service, then delete it from the repository.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Fix 1: deleting http from repository


# svccfg
svc:> list network/http
network/http
svc:> delete network/http
svc:> exit
• Or
# svccfg delete apache2
# svcs apache2
svcs: Pattern 'apache2' doesn't match any instances
STATE STIME FMRI

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Modifying manifest
• Once again, look around for something to copy from, I
find my filesystem/local dependency in inetd-upgrade.xml
and ssh.xml, respectively:
inetd-upgrade.xml: <dependency name='filesystem'
ssh.xml: <dependency name='fs-local'
• Note the name, filesystem and fs-local. These are just
common names, they are not FMRI. In fact, these can be
anything descriptive.
• Add this to my
/var/svc/manifest/network/http-apache2.xml
• Import your fixed manifest and check.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

filesystem/local dependency
dependency name='myfs-local'
grouping='require_all'
restart_on='none'
type='service'>
<service_fmri
value='svc:/system/filesystem/local'/>
</dependency>

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Compare with DTD


<!ELEMENT dependency
( service_fmri*, stability?, ( propval | property
)* ) >

<!ATTLIST dependency
name CDATA #REQUIRED
grouping ( require_all | require_any | exclude_all
|
optional_all ) #REQUIRED
restart_on ( error | restart | refresh | none )
#REQUIRED
type CDATA #REQUIRED
delete ( true | false ) "false" >

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

In depth
• name is myfs-local, a descriptive name.
• grouping is require_all, self-explanatory, but when is
grouping not require_all? When the dependency is a
milestone, for example.
• restart_on is none. If you depend on a service, and its
state changes, you may want to re-examine your own
service.
• Refer back to slide “reason for dependency stop”
• type is service; can be service or file.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

restart_on
• restart_on none: do not restart, you require this
dependency to be satisfied once, on startup, that's all.
• restart_on error: restart when this service faults. Why?
Opposite of previous case, you depend on this service all
the time. If it faults (restart is implicit) and does not come
back up, you need to restart and check your dependency.
What state would you be in?
• restart_on restart: restart when this service has an error
stop, or a non-error stop.
• restart_on refresh: restart on all the above conditions,
and when this service re-read its configuration file.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

svccfg import
# svccfg import http-apache2.xml
# svcs -l apache2
fmri svc:/network/http:apache2
name Apache 2 HTTP server
enabled false
state disabled
next_state none
state_time Mon Jun 06 18:26:41 2005
restarter svc:/system/svc/restarter:default
dependency require_all/error
svc:/network/loopback:default (online)
dependency optional_all/error
svc:/network/physical:default (online)
dependency require_all/none svc:/system/filesystem/local
(online)
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Fix 2: modify properties in repository


• Fix by modifying current service properties in the
repository.
• Now that I've written the manifest, I know better what to
do. Basically, take that block of dependency and some
how get it into the running service configuration of
apache2.
• Disable and delete apache2 from the repository.
• Re-import the original, unrepaired manifest; it should only
have two dependencies.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Put back original manifest


# svccfg -s apache2 delete
# svccfg import http-apache2.xml
# svccfg -s apache2 listpg
loopback dependency
physical dependency
start method
stop method
refresh method
httpd application
startd framework
general framework
restarter framework NONPERSISTENT
restarter_actions framework NONPERSISTENT
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

What I'm interested in


# svccfg
svc:> help
General commands: help set repository end
Manifest commands: inventory validate import export
archive
Profile commands: apply extract
Entity commands: list select unselect add delete
Snapshot commands: listsnap selectsnap revert
Property group commands: listpg addpg delpg
Property commands: listprop setprop delprop editprop
Property value commands: addpropvalue delpropvalue
setenv unsetenv
svc:>

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Adding properties
• Add a new dependency to my property group
svc:/network/http:apache2> addpg myfs-local
dependency
• In your manifest, this conforms to
<dependency name='myfs-local'
• Add other properties
svc:/network/http:apache2> setprop myfs-
local/grouping = astring: ”require_all”
svc:/network/http:apache2> setprop myfs-
local/restart_on = astring: ”none”
svc:/network/http:apache2> setprop myfs-local/type =
astring: ”servicr”

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Oops! A typo.
svc:/network/http:apache2> listprop
myfs-local dependency
myfs-local/group astring require_all
myfs-local/restart_on astring none
myfs-local/type astring servicr
svc:/network/http:apache2> editprop (invokes vi)
# Property group "myfs-local"
# delprop myfs-local
# addpg myfs-local dependency
# setprop myfs-local/group = astring: (require_all)
# setprop myfs-local/restart_on = astring: (none)
# setprop myfs-local/type = astring: (servicr)

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Add the FMRI


• Change this line to
setprop myfs-local/type = astring: “service”
• Upon quitting vi, the temp file is executed. This
corresponds to
grouping='require_all'
restart_on='none'
type='service'>
• Now add the FMRI of the dependency
svc:/network/http:apache2> setprop myfs-local/
entities = fmri: “svc:/system/filesystem/local”

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Check your configuration


svc:/network/http:apache2> listprop
...
filesystem-local dependency
filesystem-local/group astring require_all
filesystem-local/restart_on astring none
filesystem-local/type astring service
filesystem-local/entities fmri
svc:/system/filesystem/local
...
svc:/network/http:apache2> exit

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Refresh the service


# svcadm refresh apache2
# svcs -l apache2
fmri svc:/network/http:apache2
name Apache 2 HTTP server
...
restarter svc:/system/svc/restarter:default
dependency require_all/error
svc:/network/loopback:default (online)
dependency optional_all/error
svc:/network/physical:default (online)
dependency require_all/none
svc:/system/filesystem/local (online)

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

New property added


# svccfg -s apache2 listpg
loopback dependency
physical dependency
start method
stop method
refresh method
httpd application
startd framework
general framework
restarter framework NONPERSISTENT
restarter_actions framework NONPERSISTENT
myfs-local dependency

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

The reason why ...


• You've modified the service properties for apache2 in the
repository. Don't be confused the next time you look at
http-apache2.xml and wonder what happened. Manifests
are read and imported only once (unless they are
modified.)
• Don't modify manifests in place. Upon boot
svc:/system/manifest-import searches
/var/svc/manifest for new XML files and ones which have
changed and import them.
• This allows us to upgrade service properties (via a patch,
for example) by dropping new manifest in, instead of
manipulating the repository using svccfg.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

The reason why ...


• When svccfg import is run on a new manifest for services
which already exist, it first compares the current values to
those in the last-import snapshot.
• If values don't differ, it assumes that there's been no
customization, and upgrades the properties to what's in
the manifest.
• Otherwise it issues warnings to the user, but leaves the
customizations in place.
• So any patch which alters a manifest will cause manifest-
import to try to upgrade the corresponding service, no
post-install action necessary.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

The reason why ...


• If you had modified that manifest in place and re-imported
it, or even if you copied it somewhere else, modified it,
and re-imported it, svccfg will think that the modified
version is the original, official version (via the last-import
snapshot comparison), will will override all your
customization after the patch.
• The moral of the story: leave those manifests alone.
Make changes against the repository with svccfg.
• man smf_bootstrap(5) for the gory details.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Part IV:
More Advanced SMF

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

SMF security
• SMF uses RBAC to delegate service management
responsibility. We'll talk about managing specific services
first.
• SMF provides these propval to be put into individual
manifest to authorize specific services:
> action_authorization – can execute method
> value_authorization – can change value of existing properties
> modify_authorization – change, add, delete properties
• These are fixed keywords.
• Let's take a look smtp-sendmail.xml and ssh.xml as
examples.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

action_authorization
smtp-sendmail.xml:
<property_group name='general' type='framework'>
<propval name='action_authorization' type='astring'
value='solaris.smf.manage.sendmail' />
</property_group>

ssh.xml:
<property_group name='general' type='framework'>
<!-- to start stop sshd -->
<propval name='action_authorization'
type='astring' value='solaris.smf.manage.ssh' />
</property_group>

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

prof_auth
• Where will you find this? /etc/security/prof_auth
Mail Management:::Manage sendmail &
queues:auths=solaris.smf.manage.sendmail;help=RtMail
Mngmnt.html
Network Security:::Manage network and host
security:auths=solaris.smf.manage.ssh,solaris.network.*;
help=RtNetSecure.html
• These RBAC roles have this privilege, which privilege is
this? The one defined by the keyword
action_authorization, which allows you to execute
methods.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Delegating to a user
• But you haven't set up roles properly, you want to
delegate service administration to a user.
# echo christine::::auths=solaris.smf.manage.ssh >>
/etc/user_attr
• You have to re-login for authorization to take effect.
# su – christine
$ id
uid=85228(christine) gid=10(staff)

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Delegating to a user
$ svcs ssh
STATE STIME FMRI
online Oct_19 svc:/network/ssh:default
$ date
Fri Oct 21 15:47:40 EDT 2005
$ svcadm restart ssh
$ tail /var/svc/log/network-ssh:default.log
[ Oct 21 15:47:42 Stopping because service
restarting. ]
[ Oct 21 15:47:42 Executing stop method (:kill) ]
[ Oct 21 15:47:42 Executing start method
("/lib/svc/method/sshd start") ]
[ Oct 21 15:47:42 Method "start" exited with status 0
]
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

A little problem
$ svcadm disable ssh
svcadm: svc:/network/ssh:default: Permission denied.
$ svcadm disable -t ssh
$ svcs ssh
STATE STIME FMRI
disabled 15:55:43 svc:/network/ssh:default
$ svcprop ssh |grep general_ovr
general_ovr/enabled boolean false
• Service can only be temporarily disabled, why?
See BugID 6303496.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Needs value_authorization
• Bug ID 6303496
> Synopsis cron cannot be smf managed with
solaris.smf.manage.cron
• It turns out that you also need value_authorization, the
keyword to change existing property values.
• Why? Because you're actually changing the value of
some property. Which property?
• And exactly where is that general_ovr property listed?
• general and general_ovr appear to be property groups of
type framework. Check the DTD.
• We'll talk more about instances later.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Looking for general_ovr


# svccfg -s ssh
svc:/network/ssh> listprop general*
general framework
general/action_authorization astring
solaris.smf.manage.ssh
general/entity_stability astring Unstable
general/single_instance boolean true
# svccfg -s ssh:default
svc:/network/ssh:default> listprop general*
general framework
general/enabled boolean true
general_ovr framework
NONPERSISTENT
general_ovr/enabled boolean false
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

value_authorization
# svccfg -s ssh
svc:/network/ssh> listprop
general/action_authorization astring
solaris.smf.manage.ssh
• Do you need to addpg?
svc:/network/ssh> setprop general/value_authorization
= astring: "fooblat"
svc:/network/ssh> exit
• You still need to make one change for this to work.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

value_authorization
# tail /etc/user_attr
christine::::auths=solaris.smf.manage.ssh,fooblat
# svcadm refresh ssh
# su - christine
$ svcadm disable ssh
$ svcs ssh
STATE STIME FMRI
disabled 16:42:47 svc:/network/ssh:default
$ svcprop ssh|grep general_ovr
$
• Yes, it's a permanent change.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Value can be anything


• The value of the auth used has no specific meaning
except that it must exist in /etc/user_attr.
• In the manifest
<propval name='value_authorization' type='astring'
value='fooblat' />
• ... is equal to
<propval name='value_authorization' type='astring'
value='solaris.smf.manage.ssh' />
• ... but the later is more descriptive. In fact:
# auths christine
solaris.smf.manage.ssh,fooblat,solaris.device.cdrw,s

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

SMF Security
• If you look at /etc/security/prof_attr, you will see
Service Management:::Manage
services:auths=solaris.smf.manage, solaris.smf.modify
• These are two main authorizations in SMF:
> solaris.smf.modify – add, delete, modify services
> solaris.smf.manage – restart, refresh, or other state
modification
• They don't appear in any manifest because they apply all
services.
• There are others: method, dependency, application, and
framework. man smf_security(5) for the gory details.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Blueprint
Restricting Service Administration in the Solaris 10
Operating System by Glenn Brunette (June 2005)
This Sun BluePrints Cookbook describes how to use the Solaris 10
Service Management Facility (SMF) to require specific authorizations for
certain types of operations. Using this capability, it is possible to
delegate access to core service management functions based on the
concept of least privilege--if a user or service does not strictly need to
have some degree of privilege, then that privilege should not be granted.
SMF allows organizations to have much finer grained access control
policies than was possible before the Solaris 10 Operating System.
http://www.sun.com/blueprints/0605819-2887.pdf

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Snapshots
# svccfg -s ssh
svc:/network/ssh> listsnap
Instance not selected.
svc:/network/ssh> select default
svc:/network/ssh:default> listsnap
last-import
initial
running
• Where are snapshot start and previous?
# svcs ssh
STATE STIME FMRI
disabled Oct_21 svc:/network/ssh:default

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

What's the difference?


# svccfg -s ssh listprop '*general*'
general framework
general/entity_stability astring Unstable
general/single_instance boolean true
general/action_authorization astring
solaris.smf.manage.ssh
general/value_authorization astring yadayada
# svccfg -s ssh:default listprop '*general*'
general framework
general/enabled boolean false
general_ovr framework NONPERSISTENT

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

instance
• Instances are units of administrations.
• Services are mechanism for sharing properties.
• Instance can have properties, and its parent service can
have properties.
• Instances have snapshots, services do not.
• If unnamed, that instance is called “default”.
• Snapshots are composite view of the instance and
service properties at one point in time.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

revert
svc:/network/ssh:default> listsnap
last-import initial
running start
svc:/network/ssh:default> selectsnap initial
[initial]svc:/network/ssh:default> revert
svc:/network/ssh:default> unselect
svc:/network/ssh> listprop general*
general framework
general/action_authorization astring
solaris.smf.manage.ssh
general/entity_stability astring Unstable
general/single_instance boolean true
• value_authorization has disappeared.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Snapshots
• Snapshots are not good for rolling back a single change
because
> No control over when snapshots are taken
> Alters the service properties as well
• To undo a property change, it's simpler to use svccfg
setprop commands
• Can use svcprop or svccfg export to take textual
snapshots.
• However, when you use svccfg export, you use the
service name, even though the output file contains the
description of the instance(s).

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

ignore_error property
• From http-apache2.xml:
<property_group name='startd' type='framework'>
<!-- sub-process core dumps shouldn't
restart session -->
<propval name='ignore_error' type='astring'
value='core,signal' />
</property_group>

• SMF is instructed to ignore signals sent to a service or


core dumps by a service; otherwise SMF will try to restart
the service.
• man svc.startd(1M) for the gory details; description at the
startd/ignore_error paragraph.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Signaling httpd
# ps -ef|grep http
webservd 29855 29853 0 11:46:46 ? 0:00
/usr/apache2/bin/httpd -k start
root 29853 27175 0 11:46:45 ? 0:00
/usr/apache2/bin/httpd -k start
webservd 29857 29853 0 11:46:46 ? 0:00
/usr/apache2/bin/httpd -k start
webservd 29856 29853 0 11:46:46 ? 0:00
/usr/apache2/bin/httpd -k start
webservd 29858 29853 0 11:46:46 ? 0:00
/usr/apache2/bin/httpd -k start
webservd 29854 29853 0 11:46:46 ? 0:00
/usr/apache2/bin/httpd -k start
# kill -HUP 29853 <-- started by root

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Logs
# tail -f /var/apache2/logs/error_log
(nothing ...)
# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 11:46:44 Executing start method
("/lib/svc/method/http-apache2 start") ]
[ Oct 26 11:46:45 Method "start" exited with status 0
]
(nothing ... )
• What would happen, by the way, if you kill just httpd
processes started by webservd?

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Keep on killing httpd


# ps -ef|grep http
webservd 29854 27175 0 11:46:46 ? 0:00
/usr/apache2/bin/httpd -k start
# kill -9 29854 <-- last one
# tail -f /var/apache2/logs/error_log
[Wed Oct 26 12:46:53 2005] [notice] Digest:
generating secret for digest authentication ...
[Wed Oct 26 12:46:53 2005] [notice] Digest: done
[Wed Oct 26 12:46:54 2005] [notice] Apache/2.0.52
(Unix) DAV/2 configured -- resuming normal
operations

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

httpd restarted by SMF


# tail -f /var/svc/log/network-http:apache2.log
Stopping because all processes in service exited. ]
[ Oct 26 11:55:05 Executing stop method
("/lib/svc/method/http-apache2 stop") ]
httpd (pid 29853?) not running
[ Oct 26 11:55:05 Method "stop" exited with status
0 ]
[ Oct 26 11:55:05 Executing start method
("/lib/svc/method/http-apache2 start") ]
[ Oct 26 11:55:06 Method "start" exited with status 0
]
# svcs apache2
STATE STIME FMRI
online 12:46:53 svc:/network/http:apache2
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Removing ignore_error property


# svccfg -s apache2
svc:/network/http:apache2> listprop startd*
startd framework
startd/ignore_error astring core,signal
svc:/network/http:apache2> delprop
startd/ignore_error
svc:/network/http:apache2> listprop startd*
startd framework
svc:/network/http:apache2> exit
# svcadm refresh apache2

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Log of httpd refreshing


# tail -f /var/apache2/logs/error_log
[Wed Oct 26 12:55:30 2005] [notice] Graceful restart
requested, doing restart
[Wed Oct 26 12:55:30 2005] [warn] (128)Network is
unreachable: connect to listener
[Wed Oct 26 12:55:30 2005] [notice] Digest: generating
secret for digest authentication ...
[Wed Oct 26 12:55:30 2005] [notice] Digest: done
[Wed Oct 26 12:55:31 2005] [notice] Apache/2.0.52 (Unix)
DAV/2 configured -- resuming normal operations
# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 12:55:30 Rereading configuration. ]
[ Oct 26 12:55:30 Executing refresh method
("/lib/svc/method/http-apache2 refresh") ]
[ Oct 26 12:55:30 Method "refresh" exited with status 0 ]

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

httpd without ignore_error


# kill -9 29999 <-- started by root
# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 13:13:02 Stopping because process received
fatal signal from outside the service. ]
[ Oct 26 13:13:02 Executing stop method
("/lib/svc/method/http-apache2 stop") ]
httpd (pid 29999?) not running
[ Oct 26 13:13:02 Method "stop" exited with status
0 ]
[ Oct 26 13:14:02 Method or service exit timed out.
Killing contract 720 ]
# svcs apache2
STATE STIME FMRI
maintenance 13:14:02 svc:/network/http:apache2

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Clearing and restarting


# svcadm clear apache2
# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 13:16:04 Leaving maintenance because clear
requested. ]
[ Oct 26 13:16:04 Enabled. ]
[ Oct 26 13:16:04 Executing start method
("/lib/svc/method/http-apache2 start") ]
[ Oct 26 13:16:05 Method "start" exited with status 0
]
• What would happen if you kill any httpd processes started
by webservd, now that you no longer have the
ignore_error property?

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Logs
# tail -f /var/apache2/logs/error_log
[Wed Oct 26 13:24:23 2005] [notice] caught SIGTERM,
shutting down
[Wed Oct 26 13:24:25 2005] [notice] Digest:
generating secret for digest authentication ...
[Wed Oct 26 13:24:25 2005] [notice] Digest: done
[Wed Oct 26 13:24:26 2005] [notice] Apache/2.0.52
(Unix) DAV/2 configured -- resuming normal
operations
• SMF stops and restarts httpd:apache because svc.startd
will interpret any externally-generated signal as a failure
of the service, and will execute the stop and start method
to try and recover.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Logs
# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 13:24:23 Stopping because process received
fatal signal from outside the service. ]
[ Oct 26 13:24:23 Executing stop method
("/lib/svc/method/http-apache2 stop") ]
[ Oct 26 13:24:23 Method "stop" exited with status
0 ]
[ Oct 26 13:24:24 Executing start method
("/lib/svc/method/http-apache2 start") ]
[ Oct 26 13:24:25 Method "start" exited with status 0
]
• What's different though? Why isn't it in maintenance
mode?

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Portable SMF
• Perhaps you have a large number of servers, or even
zones, on which you'd like to imprint an identical set of
services.
• Well, this is not easy. This is “provisioning”, and SMF is a
service management tool.
• You can't just “pack up” the database, or the XML dump
generated by svccfg archive, or all the manifests ... Why
not?
• What about the methods, the config files, the binaries,
everything else that makes up the apps?

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Duplicating service properties


• If, however, you have an application already installed,
and you would like to transfer service properties (such as
dependencies, the exec method, anything that might be
defined by the manifest), you can do that by exporting
and then importing a manifest.
# svccfg export svc:/network/http > apache2.export
• Move apache2.export to another system.
# svccfg import apache2.export
• Note: the argument to svccfg export is a service, not an
instance, although the XML file will contain definition of
the instance(s).

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Duplicating service states


• You can get a profile, or an XML file that describes the
state of all services by using svccfg extract.
• This XML file can be used to duplicate the on|off state of
services from system to system. Use svccfg apply.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Profiles
• A profile is an XML description of all service instance
states on a system.
• Profiles are in /var/svc/profile.
• Profiles are applied in order: generic, platform, site
• Never modify existing profiles, use site.xml for local
customization.
• An example of something that may create a use a profile
is the Solaris Security Toolkit 4.2 (aka JASS).

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Part V:
SMF Tools and Community

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Webmin

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

About Webmin
• A web-based interface for UNIX system administration
• It comes with Solaris 10, or get it at
http://www.webmin.com
• Version 1.190 an
later supports SMF,
even has a feature to
create a new manifest.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

SMF in Webmin

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Create new service


• This button will walk you through creating a new service
manifest.
• You don't need to know XML, but you do need to
understand what goes into a manifest.
• Will ask you a series of questions such as:
> Name of service
> What are the dependencies,
> Which is the delegated restarter
> Service properties
> Man pages

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

GUI Tools
• With Webmin you can monitor and manage SMF services
through a browser.
• A GUI interface to SMF is being developed.
• Watch for a SunMC SMF module due near the end of this
year.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

More tools
• Use xmllint to check for errors in the manifest file without
importing the manifest.
• svccfg validate does something similar, without changing
the repository.
• Jedit is a free text editor with XML with syntax checker
and validator. Comes with Solaris 10
(/usr/demo/jds/bin/jedit) or get it from http://www.jedit.org.
• Morphon Technologies has a good and free XML editor,
but sadly no longer in development, get it from
http://www.morphon.com.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

SFM community on OpenSolaris


• http://www.opensolaris.org/os/community/smf/
• Submit manifests or methods you've written here:
http://www.opensolaris.org/os/community/smf/manifests/
• greenline-interest@sun.com migrating to
smf-discuss@opensolaris.org
• Subscribe here:
http://www.opensolaris.org/os/community/smf/discussions
• Read it here:
http://www.opensolaris.org/jive/forum.jspa?forumID=24
• File bugs here:
• http://bugs.opensolaris.org/
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF

Resources
• Internal
> Greenline project page: http://greenline.eng
> greenline-interest alias.
• External
> BigAdmin: http://www.sun.com/bigadmin
> Especially the Service Developer Intro at
http://www.sun.com/bigadmin/content/selfheal/sdev_intro.html
> SMF community: http://opensolaris.org/os/community/smf/
> SMF mailing list: smf-discuss@opensolaris.org
• Blogs
> Liane Praza's blog: http://blogs.sun.com/roller/page/lianep
> Stephen Hahn's blog: http://blogs.sun.com/sch

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Review


Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Questions?


Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

And lastly ...

Thank you!

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

References
• Presentations
L. Praza. smf(5): Service Management Facility.
R. Manus, G. Hiregoudar. Solaris Management Framework – Workshop.

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

Changes
• Last modified: 03/10/2005

Sun Proprietary/Use by Permission Only


SOLARIS 10 DEEP DIVE: SMF

SOLARIS 10
DEEP DIVE: SMF
christine.tran@sun.com
US Solaris Adoption Practice
Sun Microsystems

Sun Proprietary/Use by Permission Only

You might also like