Advanced Linux Installations and Upgrades with Kickstart

by Ethan McCallum 11/04/2004 Editor's note: Ethan has collected this series and other information into Managing RPM-Based Systems with Kickstart and Yum. This series concludes in Pre-patched Kickstart Installs.

In Hands-Off Fedora Installs with Kickstart, I provided an overview of the Kickstart process. This article is a collection of techniques that may interest people who want to do more with Kickstart. It covers Kickstart customization, scalability, and security, including:

dynamic ks.cfg storing ks.cfg on removable media default booting to kickstart pre- and postinstall scripts custom RPMs and package groups using Kickstart to upgrade

Dynamic ks.cfg
Most Kickstart experiments begin with a single ks.cfg file, though this approach is less suitable for large deployments. Even a farm of cloned hardware will require some settings unique to each host. That means you have either several one-use ks.cfgs, or one file to tweak for each Kickstart target. These methods are brittle because they bind two elements that may vary independently of one another: host-specific data (the IP address) and build-specific data (packages to install). When either one changes, the ad hoc edits to resync the two may introduce errors. You can avoid this hassle if you're programmatically inclined: pre-generate a series of ks.cfg, or generate the data on the fly. The latter requires that your Kickstart clients fetch ks.cfg via HTTP. Both approaches use the mail-merge concept: create a template ks.cfg with placeholders for the dynamic info, then merge that with a data source that contains one set of values for each host. A Perl or PHP template, for example, may include the following:

## ... other settings ... network --device ${network_device} --bootproto static --ip ${network_address} --netmask ${network_mask} --gateway ${network_gateway} --nameserver ${nameserver_1} --nameserver ${nameserver_2} --hostname ${hostname}

(We've split the above for readability; it is really one line.) Pre-generating files requires that you create one file per target host (the Kickstart client). Specify that host's file at the CD's boot: prompt. For example:

boot: linux ks=http://server/ks_configs/host5.cfg

While this method certainly works, it requires you to remove the generated files later (or leave them around forever). By comparison, the web-based method requires no such cleanup, since the data lasts only as long as the HTTP request. Clients specify a unique identifier in the URL:

boot: linux ks=http://server/config.pl?target=host5

This identifier is a key to the rest of that host's data, which the script uses to populate the template. A completely automated solution would detect the target host's identity rather than rely on URL parameters. For example, it could mine DHCP data to match the incoming IP to a MAC address. This sort of solution would require a hardware inventory system that I suspect is quite rare, though. Neither method relies on a particular implementation: use PHP, Java, Perl, or whatever you fancy. Perl fans have the Template Toolkit. A Java solution could use Apache's Velocity. These last two have the added bonus of being usable in both web (on the fly) and command-line (pre-generated) scenarios. Most programming languages provide built-in support for a variety of data formats, so you have your choice of storing the machine data in flat files, XML, an RDBMS, or whatever else comes to mind.

Storing ks.cfg on Removable Media

ks.cfg contains the root password, and as such, sensitive shops may not want it to cross the wire via HTTP or FTP. You can trade the convenience of clients downloading ks.cfg for the added security of storing it on removable media, such as a disk, a CD, or an external drive. The disk route is the easiest: copy the file to the root directory of a formatted disk. When you boot the target host from the CD, enter:

boot: linux ks=floppy:/ks.cfg floppy here is a device alias for fd0. Specify another partition's name to load ks.cfg from that device: boot: linux ks=hd:{partition}:/ks.cfg {partition} is an identifier from the /dev directory, such as sda1. (Remember: USB, FireWire, and other such
devices appear as SCSI devices.) The CD method requires slightly more work, because it creates a bootable CD. In the distribution, the isolinux directory represents the root directory of a boot CD. Copy it to a scratch space, and then copy your custom ks.cfg to {scratch space}/isolinux. Change to the directory above isolinux and call mkisofs to generate the ISO image:

$ mkisofs -o boot-ks.iso -r -b isolinux.bin \ -c boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -R -J -v -T isolinux/

(Note that the boot image file, isolinux/isolinux.bin, must be writable or mkisofs will fail.) Follow your standard procedure for burning this ISO to media. If you use VMWare as a Kickstart test platform, you can save a disc by specifying the ISO as the guest's CD-ROM device. Boot a test machine from the CD and fire off Kickstart using the local ks.cfg:

boot: linux ks=cdrom:/ks.cfg

There are, of course, trade-offs here. Disks tend to have shorter and less predictable life spans than CDs and require you to have a boot CD handy. Then again, CDs and CD-RWs are good for a limited number of writes. Furthermore, because of the strong ties between the boot media and the target OS revision, your boot CDs become coasters at the next upgrade rollout. Using other removable media requires that your boot medium's kernel support your SCSI (or USB, or FireWire) adapter. Whatever the case, you needn't limit yourself to a single config file. You could copy a series of pre-generated config files to the disk or CD, then specify the target host's file at the boot: prompt:

linux ks=hd:sdb5:/ks-host5.cfg linux ks=floppy:/ks-host5.cfg

As ks.cfg is just a few K in size, space shouldn't be a problem. There's no right answer here. Arguably, storing ks.cfg on removable media works best for small or medium shops or for large one-offs such as clone farms. If your goal is to make sure sensitive Kickstart data doesn't cross the normal network, set up a private build network using a portable switch and serve Kickstart data from a laptop.

Default Booting to Kickstart

Whether you're building a clone farm or repeatedly testing a Kickstart setup, you'll tire of entering linux ks=... each time you start a build. (Maybe that's just me; people have accused me of selective laziness.) You can create a boot CD that automatically loads your ks.cfg, making Kickstart a true fire-and-forget process. The key to this magic is isolinux/isolinux.cfg. To start, define a new boot selection by adding a new stanza:

label custom_kickstart kernel vmlinuz append initrd=initrd.img ramdisk_size=8192 ks=cdrom:/ks.cfg

This lets you enter:

linux custom_kickstart
at the CD's boot: prompt to fire off a Kickstart. The operation will use the ks.cfg in the CD's root directory. Change the default directive to make this the default:

default kickstart
To be fancy, associate an explanatory message with an F-key:

F8 our_site.msg
(There are other directives in isolinux.cfg worthy of experiment, but they're beyond the scope of this article.) The specified ks.cfg needn't exist on the boot CD; you can specify other valid sources, such as a disk or a URL. If your dynamically generated ks.cfg doesn't require parameters to identify the target host, you can have a single boot CD for the entire shop, or at the least, one CD per OS version.

Pre- and Postinstall Scripts

Kickstart does a lot of host customization for you--setting the root password, the time zone, and so on--but it won't do certain other tweaks, such as disabling services via chkconfig or setting up site-specific directory structures. You can automate these tasks using the pre- and postinstall scripts. (There is one of each.) Per their names, these scripts run before and after the installation, respectively.

The preinstall script starts toward the end of ks.cfg under the %pre directive. The syntax is similar to that of an RPM spec file. At this early stage there is a bare-bones assortment of tools at your disposal, just enough to shuffle files around or mount NFS shares. The postinstall script (%post) has more potential. By this point, with the OS installed, the script defaults to running chrooted in the newly installed OS under /mnt/sysimage. You can disable this by using %post's --nochroot flag, by the way. Use this script to alter runtime services with chkconfig, install a rudimentary firewall rule set, or do anything else you'd rather take place before the machine reboots into full-service mode. Both the %pre and %post directives accept the --interpreter flag, which specifies an interpreter other than the default Bourne shell. You can choose only from the interpreters available at runtime, obviously; but likely the chrooted environment of the postinstall script will have a wider selection. Pre- and postinstall scripts certainly have their place, but they're easy to misuse, difficult to test, and require updates as your build environment evolves. Simplify your build by extracting %pre and %post script functionality into custom RPMs or formal update processes (such as cfengine and rsync wrappers). These have the added benefit of being usable (and testable) outside of the install process.

Custom RPMs and Package Groups

It's not uncommon for a Linux-savvy shop to create custom RPMs, such as homegrown software or site-specific adjustments of third-party packages. Kickstart's ties to RPM (via Anaconda) permit you to transparently add these RPMs to the installation process. This requires that you copy your RPMs to the tree, create a new group in the comps file, and update package headers. The first step is the easiest: copy your custom RPMs to {install tree}/Fedora/RPMS. Make sure the files have appropriate read-access permissions. The comps file is Fedora/base/comps.xml in the install tree. It contains definitions of package groups, which are collections of RPMs that you can select and install en masse. (A detailed tour of the comps file is beyond the scope of this article, but you need to know very little to add a group.) It's cleaner and more future-proof to put your custom RPMs in a new, separate group. For example, create a <group> tag in comps.xml:

<group> <id>local_packages</id> <name>Local RPMs</name> <description>home-built RPMs</description> <uservisible>false</uservisible> <packagelist> <packagereq type="default">package1</packagereq> <packagereq type="default">package2</packagereq> <packagereq type="optional">package3</packagereq> </packagelist> </group>
I find it easiest to define new groups at the top of comps.xml, or better still to store them in a separate file and insert them into comps.xml as needed. The <id> tag is a unique identifier for the group. Specify this name in ks.cfg preceded by an @ symbol to install all of the group's packages:

%packages ... @ local_packages ...

The body of the <name> tag will appear on the Package Group Selection screen during a manual install, provided <uservisible> is true. The aptly named <description> tag provides a brief description of the package group. A global shop could take advantage of localization. Add extra <name> and <description> tags, each with the xml:lang attribute, to set that language's code:

<name xml:lang="fr">Trucs</name> ... <description xml:lang="fr">Les Trucs...</description> <packagelist> holds the list of this group's packages in <packagereq> tags. The type attribute reflects a package's status within the group: optional (addable), default (included by default), or mandatory
(unremovable). Don't rely on the filename for <packagereq>. If you're unsure, query the RPM itself for its internal name:

$ rpm -q --qf '%{NAME}\n' -p some_file.rpm

Package headers contain information about each RPM in the install tree, such as checksums and dependencies. The query process is rather I/O-intensive, so pre-generating the data speeds up the install process. To (re-)generate the package headers, run genhdlist (from the package anaconda-runtime):

$ /usr/lib/anaconda-runtime/genhdlist \ --productpath Fedora \ {path} {path} is the fully-qualified path to the parent of the Fedora directory. For the curious, the header data goes in
Fedora/base/hdlist and hdlist2.

Using Kickstart to Upgrade

Vendors release new versions more frequently these days, which means OS upgrades are fast becoming a regular occurrence. Why click through the upgrades by hand when Kickstart can do them for you? Upgrades are similar to installs, though ks.cfg requires fewer directives:

upgrade (as opposed to install) installation method (for example, url or nfs) installation language (lang) installed language support (langsupport)
device spec, if you need the device for the install ( device) installation process keyboard setup (keyboard) boot loader configuration (bootloader)

(Refer to the previous article in this series or the Kickstart documentation for detailed explanations of these directives.) The upgrade process ignores most other options, because it shouldn't change settings such as the time zone, the root password, or the network setup.

I've provided a sample upgrade, ks.cfg. It is very generic, and accordingly, you could have a single such file for an entire site. The upgrade operation otherwise functions similarly to the installation: boot from your preferred media, direct Anaconda to ks.cfg, and walk away. Please note that boot media has very strong ties to the OS revision; you must boot from the newer CDs to upgrade. However, the machine hosting the installation media needn't run the same OS version, nor even the same OS. My experience is that Kickstart's upgrade is nondestructive: while it doesn't delete any files or alter disk layouts, it installs new packages (including new kernels, though, so watch out if you rely on a custom kernel). That said, test the process before turning it loose on your environment.

Conclusion
There are several ways to customize the Kickstart process. Applying these techniques to your Kickstart infrastructure should take you closer to a completely hands-off process for installing and retasking machines.

Resources

Download the sample ks.cfg used for upgrades. Hands-Off Fedora Installs with Kickstart provides a general introduction to Kickstart. Chapter 7 of the Red Hat 9 Customization Guide gives Kickstart documentation. The Fedora team has information on tweaking comps.xml available on its web site. Based on my experience, adding the <package> tags to comps.xml is unnecessary. It would appear Anaconda pulls this information from the generated headers list. (Perhaps the documentation is out of date?) "Software Packaging with RPM" explains how to create custom RPMs.

Ethan McCallum grew from curious child to curious adult, turning his passion for technology into a career