wiki:PXEInstall

PXE Installation

This page covers PXE installation. The actual Debathena Installer is covered in a separate page.

tl;dr

PXE serves up an initial installation script that asks the user some questions, kexec's into a new network install using the answers to those questions, and then runs install-debathena.sh after the base system has been installed.

Infrastructure

  • The PXE server
  • demeter.mit.edu (aka athena10.mit.edu): 18.9.60.73. This IP is hardcoded in the following places:
    • the preseed files
    • the stage1 installer (for doing low-rent DNS queries)
    • the actual installer for each suite (for fetching install-debathena.sh)
    • TODO: probably other places, but in theory nothing outside of trunk/athena/debathena/scripts/installer/pxe

The PXE Server

The PXE server is maintained by the Network team, running PXELINUX. There is an entry called "Debathena". Currently, it contains a Natty kernel and initrd from the i386 netboot images. We should try and keep this up to date periodically, and will definitely need to do so for hardware support (the kernel needs to support the NICs in all supported machines).

The following command line is passed to the kernel: netcfg/get_hostname=athena-pxe-install locale=en_US keyboard-configuration/layoutcode=us interface=auto url=http://18.9.60.73/installer/current/debathena.preseed

Notes:

  • We have to specify a hostname. It will get overwritten by the DHCP config, but if DHCP barfs (which it will) or if it gets given an address with no PTR record (which it will), it needs a value there or the Debian installer fails miserably.
  • This should continue to be i386 for the foreseeable future.
  • The current directory in the url is a symlink, which is pointed at stage1. You can repoint the symlink for testing, since we can't get a separate "Testing" entry on the PXE server.

Server Details

The server is currently installer.mit.edu (18.7.21.127 as of Aug 2013). You can connect to it via tftp. The pxelinux config file can be retrieved via tftp at /debathena/pxelinux.cfg/default. Pathnames referenced in the config file are relative to /debathena (that is, kernel vmlinuz refers to /debathena/vmlinuz).

Preseeding and include_command

The preseed file generally sets a single include_command which runs as early in the install as possible. The preseed command wget's the "debathena-loader.sh" file and runs it. debathena-loader's job is to wget the installer tarball, extract it, and run the "installer.sh" file in there. Note that in include_command, the shell is not connected to a terminal, which is why we explicitly chvt and run the script with I/O redirection. The contents of debathena-loader.sh could all go in the single line in the initial preseed file, but it'll be ugly and we're more likely to screw up. Also, we have no idea what the max size of an include_command is.

Stage 1

The purpose of the Stage 1 installer is to interact with the user -- the second stage should be unattended. At this time, we do things like ask the user what suite and metapackage to install, talk about partitioning, get network configuration, etc.

First, we set some initial values: mirrors.mit.edu as the mirror, the newest cluster release as the distro, auto partitioning, and arch of i386, unless we find the lm flag in /proc/cpuinfo, in which case we use amd64.

The user gets a menu, and they can choose between debathena-cluster, "normal Debathena" (i.e. not -cluster), or plain vanilla Ubuntu. (The latter is provided for convenience only). Also, they can get plain old /bin/sh and BusyBox? (and maybe anna or maybe not). They can also select a different mirror, use the beta installer, change the distro/version/suite, change the architecture, and toggle between manual and automatic partitioning for the cluster install.

Once the user has decided what they want, we move on to network configuration. If the user has a "real" DHCP address (i.e. not 18.2 or 10.whatever), they can elect to use that and not configure network, or they can move on to static IP configuration. (People selecting -cluster are forced to use a static IP). Network configuration:

  1. Check if we can get to our low-rent DNS resolver. If so, we can accept a hostname instead of an IP.
  2. Prompt the user for an IP (or hostname if DNS is available).
  3. If we get a hostname, look up the A record. If there isn't one, demand an IP.
  4. If we get an IP, look up the PTR record, and set the target system's hostname to the value. Otherwise, set it to install-target-host.
  5. Given the IP, use magic the netparams script and masks file to get the netmask, broadcast, gateway, and CIDR notation. If we're not on cluster, give the user a chance to edit them by hand.
  6. Kill dhclient if it's running, and flush eth0. Set the IP address, route, and other info. Set the MIT nameservers and clobber any existing values in resolv.conf

Not that we're all set, we retrieve the kexec binary, and the arch-specific net-install initrd and kernel from debathena.mit.edu/net-install. And we set up the kernel args to pass:

  • networking information (netcfg/*)
  • keyboard layout, otherwise the user will get prompted in stage 2 (keyboard-configuration/layoutcode for Natty and later, console-setup/layoutcode for earlier releases)
  • Any hardware-specific args, such as reboot=pci
  • locale (currently forced to en_US)
  • url=http://18.9.60.73/installer/$distro/debathena.preseed
  • The user's selections for installation type, mirror, beta vs production, and partitioning type (auto vs manual). These are prefixed with (da/*).

We then sanity-check things (we're going to exceed 512 bytes eventually, I know it) and then kexec out of there.

If we're still around after the kexec, horrible things happened, and we tell the user this.

Notes:

  • The initrd doesn't have date(1) (yes, really), so we use netcat to talk to kerberos.mit.edu (with a hardcoded IP) on the daytime port for datestamps. This is a awesome terrible idea.
  • A lot of assumptions are made about the way MIT's network is laid out. If you manage to get to Stage 1 through a method other than PXE on a wired interface, terrible things will happen.
  • You should be able to do useful things running the stage1 installer with --test.
  • If your preferred network device is something other than eth0, you lose.
  • If you want to install to something other than the first IDE/SATA disk, you lose.
  • The stage1 installer.sh script must never exit (it kexec's out, or loops forever with a shell). If it does exit, the install will fall through to whatever the PXE server initially gave us, which is undoubtedly wrong.

The Stage 2 Install

The stage 2 install is designed to be invoked from 2 places: The stage 1 install, and the hopefully-obsolete-by-the-time-you-read-this auto-upgrade code.

First, we do some hardware-specific hacks (I'm looking at you, Realtek).

Then we parse the kernel's command-line, looking for options we care about. In particular, we support:

  • debathena/clusteraddr: DEPRECATED, an IP address, which also serves as a flag indicating that this is a cluster machine. (Note the different prefix, this is not a typo).
  • da/pxetype: cluster, workstation, vanilla, etc. We fallback to debathena/pxetype but will stop doing that.
  • da/i: beta or production
  • da/m: mirror site
  • da/part: auto or manual partitioning

For the fallback values (clusteraddr or debathena/pxetype), we set some defaults of installertype=production and mirrorsite=mirrors.mit.edu. Eventually, we should stop doing this, and just error out if the necessary arguments aren't there. We already barf if pxetype isn't present.

We re-run net configuration in case we get a debathena/clusteraddr, but we do everything automatically. We will stop doing this eventually.

We now build-up a pre-seed file by copying the netcfg arguments, and setting the mirror hostname. We also copy in the contents of preseed.common. We add preseed.autoinstall (possibly skipping the partman section) for cluster machines. We then fetch the actual Debathena Installer (the beta version, if necessary), set pxe-install-flag, and exit out into the main installer, which will use these preseed values.

Other files in the stage2 installer

  • athena/{netparams,masks} -- these will go away when we stop re-doing netcfg in stage 2
  • check-disks.sh -- Run by the partman component. Find the first IDE/SATA disk, and ensure that the disk itself is big enough for the install. This is defined as "20GB" by a variable at the top of this script.
  • lvm-cluster-machine.partman -- a partman "recipe" for the layout of a typical cluster machine. See /usr/share/doc/debian-installer/devel/partman-auto-recipe.txt.gz
  • postinstall.sh - Run for debathena installation to do some sanity checking and run the debathena installer.
  • preseed.autoinstall - For things that should be set for cluster machines
  • preseed.common - Generally useful preseed values

Stuff on debathena.mit.edu

The necessary kernel and initrd for stage 2 are fetched from http://debathena.mit.edu/net-install/$DISTRO/$ARCH. These are bootstrapped from the upstream netboot installer. For example, for precise, i386:

 http://mirrors.mit.edu/ubuntu/dists/precise/main/installer-i386/current/images/netboot/netboot.tar.gz  http://mirrors.mit.edu/ubuntu/dists/precise/main/installer-i386/current/images/netboot/ubuntu-installer/i386/linux  http://mirrors.mit.edu/ubuntu/dists/precise/main/installer-i386/current/images/netboot/ubuntu-installer/i386/initrd.gz

Repeat, substituting "amd64" for "i386" in the URLs above.