Using FOG in an educational setting

If you teach IT in a vocational training (VET) program, sooner or later you hit the same question: how do I get thirty identical computers without running the installer thirty times?

The classic answer is cloning. And for cloning over the network, with no USB sticks and no going machine by machine, my tool of choice is FOG Project: free software with a web interface that captures the disk of a “template” machine and pushes it over the network to the rest of the lab.

In this article I'll walk you through how FOG fits into a vocational classroom, what makes the educational setting special, and the traps I've fallen into so you don't have to. I won't talk about my actual network (addresses, machine names, switch models…), but I will talk about the ideas, which travel to any school.

What FOG is, in a sentence and a half

FOG is a server that handles three things:

1. Booting machines over the network (PXE) and offering them a menu.
2. Capturing a machine's disk and storing it on the server as an image.
3. Deploying that image to one or many machines at once.

Under the hood it's a very instructive stack: Linux, Apache/PHP for the web interface, MariaDB for configuration, TFTP for booting, NFS for moving images around, and a small Linux of its own (FOS) that each machine downloads to do the heavy lifting of copying partitions. Everything a systems and networking syllabus covers is in there, running for real.

Why a vocational lab is a special case

A computer lab isn't an office full of identical machines. It has its own traits, and they shape how you use FOG:

• Lots of machines, all the same… in theory. Thirty seats per room, but bought in different batches, with motherboards from several vendors and disks from different tiers. “Identical” lasts until the first clone boots.
• The software changes every year. What you install in September isn't what you need in February. The image isn't something you build once: it's a cycle (prepare, capture, deploy, fix, recapture).
• Students break things, sometimes on purpose. It's part of learning. When restoring a whole lab takes minutes instead of an afternoon, it changes what you can afford to let them do.
• Little time and few people. Maintenance usually falls on the school's IT department. What isn't automated doesn't get done.
• The network isn't yours. There's usually a school-wide DHCP server, VLANs, a switch managed by someone else. FOG has to coexist with all that.

The workflow

My FOG workflow is always the same, and I think it's a good skeleton for teaching it in class too.

1. Prepare the template (golden image)

You install and configure one single machine the way you want the whole lab to end up. Before capturing it, you have to generalize it: strip out everything that makes it unique. If you don't, every clone inherits the template's identity.

The bare minimum I wipe before capturing:

# Blank machine identity: each clone will generate its own
truncate -s 0 /etc/machine-id
rm -f /var/lib/dbus/machine-id
ln -sf /etc/machine-id /var/lib/dbus/machine-id

# Machine's SSH host keys: if they travel in the image, all clones
# “look alike” cryptographically
rm -f /etc/ssh/ssh_host_*

On top of that come authentication caches, the home directories of test accounts, any snapshots lying around (backup ones are huge and pointless on a clone) and the logs.

2. Capture

You boot the template over PXE, FOG captures it to an image on the server, and the machine shuts down. From the web interface it's a few clicks; you can also create the task through the API or straight in the database if you want to script it.

3. Deploy

You pick a machine, a group (I use one per lab) or several, create a Deploy task and boot them over the network. FOS downloads the image, writes it to disk, and the machine reboots already cloned.

4. Individualize on first boot

This is the most overlooked part. The clone arrives with a blank identity and has to give itself a name, regenerate keys and so on at first boot. I do it with a systemd oneshot service that runs only once. If it doesn't run, you end up with thirty machines sharing the same machine-id and the same SSH keys, a problem that is as silent as it is annoying (things like DHCP or your inventory start mixing machines up).

For later configuration (users, profiles, per-lab software) I rely on Ansible: FOG leaves the system nailed down and Ansible fine-tunes it.

The traps (the useful part)

This is what I really wish I'd read before I started.

PXE with a DHCP server that isn't yours: proxyDHCP

Normally a school already has a DHCP server that you can't (and shouldn't) touch. If you bring up FOG's DHCP, you'll have two servers answering and a whole lab without a network.

The fix is proxyDHCP mode: a dnsmasq that doesn't hand out addresses, it only answers PXE requests, telling the client where the boot server is and which file to download. Addressing keeps coming from the usual DHCP server.

Two lessons from this:

• proxyDHCP only works on the same layer-2 network as the clients. PXE requests are broadcast and don't cross routers. If the server sits on another VLAN, the machines won't see anything.
• BIOS and UEFI don't behave the same. A configuration that boots BIOS machines just fine can leave UEFI ones downloading nothing, because the boot file field ends up empty. You have to declare the boot file per architecture. It's one of those things you only find out with a real machine in front of you.

And a detail that seems minor and isn't: the boot service must be enabled, not just running. A dnsmasq stopped “just for a moment” and left disabled means that after the server's next reboot, PXE doesn't come back and nobody knows why.

The server's IP lives in many places

FOG stores the server's address in the database, in the installer's answer file, in the iPXE menu, in the dnsmasq configuration and even in the certificate the client uses. If you change the server's IP only at the operating system level, FOG will keep serving everything with the old one.

Tips:

• Give the server a static IP (or a DHCP reservation). A “dynamic but stable” IP works right up until the day it stops being stable.
• After any network change, check what FOG actually hands out to clients, not just what it has configured. And the ultimate test is always booting a real machine.
• Parameterize your monitoring scripts with an environment variable instead of typing the IP by hand.

Secure Boot and signed drivers

This one cost me an afternoon. If you install kernel modules signed with your own key (in my case, VirtualBox's, with a MOK key), that key lives in the motherboard's NVRAM, not on the disk. FOG copies the disk, so it never replicates the key enrollment. The result: on a clone with Secure Boot enabled, the module is rejected, the virtual machine won't start… and everything else works perfectly, so it goes unnoticed until a student needs it.

My decision was to run the whole fleet with Secure Boot disabled and to check it on every new machine:

mokutil --sb-state        # should say: SecureBoot disabled

Cloning across different hardware

The image is captured on one motherboard and deployed to others from different vendors. The kernel boots with an initramfs that, by default, only includes the drivers for the hardware where it was generated. On a different motherboard (another disk controller, another SATA mode…) the result is the dreaded “Unable to mount root fs” and a kernel panic.

What worked for me:

• Generate the initramfs with all the storage drivers (MODULES=most in initramfs.conf) right before capturing.
• With low-end NVMe SSDs, add boot parameters that make the kernel wait for the disk (rootwait) and disable problematic power-saving states. They're harmless on SATA, so I always leave them in.

It's a great “debug a boot” exercise for students: a black screen, a cryptic message and a chain of causes you have to rule out one by one.

A deadlock nobody sees coming

A case for the “systems waiting on each other” notebook. My first-boot service had to run before the SSH server and the session manager, and among its tasks was generating the SSH keys and restarting the service. But the SSH restart waited for the first-boot service to finish… and the first-boot service waited for the restart. Deadlock. Since it was a oneshot service with no timeout, it stayed that way forever: the port accepted connections but never got as far as a banner, and there was no login screen.

The fix was to replace restart with --no-block try-restart. The valuable part is the lesson: test the first boot of a real clone, not just the template. The template never goes down that path.

Multicast: lovely in theory, depends on the network

With multicast, the server sends the image just once and every machine receives it at the same time. For deploying a whole lab it's a dream… if your network allows it.

In my tests the session started, but the stream died around 4% and the clients threw read errors. The cause is usually the switch: IGMP snooping without a querier, or with a badly configured one, keeps multicast traffic from being sustained. You'll have to talk to whoever runs the network (or enable an IGMP querier / disable snooping on that VLAN). In the meantime, unicast always works, just a bit slower.

Waking the machines up: Wake-on-LAN

Deploying overnight or before the first class is far more comfortable if the machines power on by themselves. WoL has two halves:

• On the machine: the BIOS must have Wake on LAN / Power On by LAN enabled and the deep power-saving mode (ErP / Deep Sleep) disabled, since it cuts power to the network card. And the operating system must “arm” the card on every boot.
• On the network: the magic packet is a broadcast. It has to reach the segment where the machines are. At first I ruled out WoL as unworkable… and the problem wasn't the card, it was the packet's delivery. When I sent it from a machine with an interface on that same network, the vast majority of the fleet woke up.

The API: “403” doesn't mean “off”

FOG has a REST API, and it's enabled out of the box. If you get a 403, don't go into the database to turn it on: you're missing headers. It needs two tokens:

curl -s \
  -H "fog-api-token: $SYSTEM_TOKEN" \
  -H "fog-user-token: $USER_TOKEN" \
  http://<server>/fog/task/active

No headers, 403; only the system token, 401; both, 200. And the system token is sent exactly as the interface stores it (in base64), without decoding it. With the API you can register machines, create tasks and build your own deployment monitors.

What you can teach with this

Beyond “getting the lab ready”, FOG is an excellent testbed for the curriculum:

• Networking: DHCP, proxyDHCP, TFTP, NFS, broadcast versus multicast, IGMP, VLANs, and why PXE boot doesn't cross a router.
• Operating systems: the BIOS/UEFI boot process, GRUB, initramfs, systemd (oneshot units, dependencies and deadlocks), machine identity, Secure Boot.
• Systems administration: the lifecycle of an image, generalization, automation with scripts and Ansible, backups and rollback.
• Databases and APIs: MariaDB as the tool's “brain” and a REST API with header-based authentication.
• Security: default credentials, who can boot over the network, DHCP/PXE spoofing attacks, key management.
• Methodology: “document what you do”, “test the real path, not the happy one” and “one change, one check”.

One proposal I find very worthwhile: set up a practice FOG on virtual machines (a server and a couple of clients on an isolated network). Students can break PXE, pick the wrong architecture or cause a deadlock on first boot with no risk to the real lab.

Things I'd do from day one

1. Change the default password of the web interface. It's the first one anybody tries.
2. Control who can boot over PXE. A curious student and a network boot menu make for an interesting combination; a BIOS password and a sensible boot order help.
3. Back up what FOG holds in the database, not just the images: groups, hosts and tasks are hours of work.
4. Keep an image of the original system (the factory Windows, for example) before wiping it. When someone asks whether you can go back, you'll be glad you have it.
5. Document every network or IP change along with its rollback procedure.
6. Test the first boot of a real clone after every recapture, before deploying to the lab.

Wrapping up

FOG isn't magic: it's a collection of classic pieces (PXE, TFTP, NFS, DHCP, a database and a web interface) glued together well. And that's precisely why it works so well in a vocational lab: it gives you a useful result from day one and, when something breaks, it forces you to understand the piece that broke.

If I had to keep just one idea, it would be this: the image isn't the product, the process is. A documented, repeatable process, tested on a real clone, is worth far more than the best image in the world.

In upcoming articles: automating lab configuration with Ansible, and how to integrate centralized authentication on the machines of a cloned fleet.