Ticket #960 (new enhancement)

Opened 13 years ago

Last modified 10 years ago

Make the stage1 iPXE instead of Linux

Reported by: geofft Owned by:
Priority: normal Milestone: The Distant Future
Component: -- Keywords:
Cc: oremanj Fixed in version:
Upstream bug:


So currently the way that Debathena PXE boot works is that we load this one kernel (currently Jaunty), prompt for what you want to do, including all the config options if it's cluster, and kexec the distro you actually want to install. We're running into the newest cluster hardware not actually supported by Jaunty's kernel, and it's both annoying to have to get netops to change the kernel instead of just having a stage1 that always works and stage2s we can customize, and worrisome that there might be something we _de_support with Jaunty.

You might think, though, that since the PXE option ROM can download the stage1, surely there's a way we can use it to download the stage2. There is, it's called UNDI, and it's a crappy API exposed by network cards that's only really usable in real mode. So we have no hope of a Linux UNDI driver existing. But, iPXE supports speaking UNDI (as well as native network drivers when they exist for iPXE, and when they exist they generally work better). We can make an iPXE stage 1 that does the prompting, downloads the appropriate Linux kernel, and boots that.

Then we can just leave an appropriately-configured iPXE (or even one that downloads its configuration from the network) as the "Debathena" PXE menu option and not worry about the stage1 any more.

Josh points out that the iPXE command language now has  support for user input.

Change History

comment:1 Changed 13 years ago by geofft

Looked a bit more at this.

Josh says that we can keep the same DHCP IP address/lease if we tell iPXE set use-cached 1 before running dhcp, and if we use the undionly.kkpxe (not .kpxe) build of iPXE (see also his  comment to the same effect on ipxe-devel). Since DHCP is generally slow, this would be a win.

I've built an undionly.kkpxe with the EMBED parameter pointing to a script that does:

set use-cached 1
set dns
chain http://debathena.mit.edu/ipxe/boot

This works, as evidenced by "kvm -boot n -net nic -net user,tftp=/mit/debathena/web_scripts/website/ipxe,bootfile=/undionly.kkpxe". So if we give this undionly.kkpxe to Network and tell them to call it the Debathena option, we can chainload our own stuff on the other end.

Note that we don't actually want to give this particular copy. Apart from wanting to build it on the build server in some reasonable way, current iPXE doesn't support multiple DNS servers. I'd like to add that support before we deploy this.

Also, as the name would imply, undionly.kkpxe includes only the UNDI drivers, and it might be nice to build in some native drivers for common cluster hardware. But Josh tells me that use-cached won't work with native drivers, so this requires a second DHCP request. This doesn't strike me as a fundamental limitation, though... still, we should see if UNDI works well enough on the hardware we care about or not, before caring.

Obviously we also need to write a better iPXE script that does what we want, too...

comment:2 Changed 13 years ago by geofft

Note that PXELINUX as booted out of Network's PXE stack to chainload our kernel/initrd already uses UNDI and the existing IP lease when downloading our stage1 kernel and initrd, so we know that UNDI definitely works on all cluster hardware.

comment:3 Changed 11 years ago by jdreed

I'd like to test chainloading, but my reading of the "documentation" suggested that you needed to use ipxe.lkrn if you were coming from PXELINUX. I have asked network for a test option chainloaded from PXELINUX for that for testing. I guess we can try the undionly one, but I don't love the idea of building our own PXE.

comment:4 Changed 11 years ago by jdreed

Anyone who wants to play with iPXE can now boot the Debathena PXE entry, hold down shift, and at the boot: prompt, type ipxe, and then press C-b to get to the iPXE command line.

comment:5 Changed 10 years ago by jdreed

OK, I played with this a bit on my local config. This looks sort of promising, but we still don't have support for multiple DNS servers. I'm not convinced this is a real problem: if one DNS server is down, it's likely to be dealt with quickly. Also, if we're rolling our own, we can include nslookup support, and try the next DNS server if it fails. I'm torn, because I like the idea of "never" having to worry about what's on the PXE server. OTOH, we want to pretend to support UEFI soon, and iPXE definitely doesn't. I'd really like us to have as little skew as possible between UEFI and legacy boot code paths.

Note: See TracTickets for help on using tickets.