Compiling a Custom FreeBSD Kernel¶

A complete walkthrough for building and installing a custom kernel on FreeBSD, using the native buildworld/buildkernel toolchain and, where available, ZFS boot environments for safe rollback.

1. Overview and prerequisites¶

FreeBSD builds the kernel and the base userland ("world") from one unified source tree rather than treating the kernel as a standalone project. The standard flow is:

buildworld -> buildkernel -> installkernel -> reboot -> installworld -> etcupdate -> reboot

buildworld compiles the entire base system (libc, compiler toolchain, core utilities) using itself, which is also what makes cross-version upgrades safe - you always build with a toolchain compatible with the source you're building. buildkernel then compiles the kernel using the freshly built tools.

Budget: 30 GB+ of free space for /usr/src and /usr/obj, and 30 minutes to a few hours of CPU time depending on hardware and whether you run a full buildworld or just rebuild the kernel.

You need:

• The base system's own toolchain (clang/lld, included by default) - no extra compiler packages needed for a native build.
• Source tree checked out under /usr/src (Section 2).
• Root access for installkernel/installworld. The build steps themselves (buildworld, buildkernel) can run as an unprivileged user if that user can write to /usr/obj (or you point MAKEOBJDIRPREFIX somewhere it can); the Handbook simply runs everything as root.
• If you're only rebuilding the kernel (not the whole world) after a small config tweak, you can often skip buildworld - see Section 9 for when that's safe.

Do you actually need a custom kernel? Most drivers and features ship as loadable modules (kldload), including pf, pflog, pfsync, carp, if_wg and vmm. Load them at boot with sysrc kld_list+="pf carp" or the matching _load="YES" line in /boot/loader.conf. A custom kernel is needed for options that can't be modules (ALTQ is the classic example), for stripping debugging overhead, or for kernel development. A custom kernel also means freebsd-update will no longer patch your kernel - you own its security updates from source.

2. Obtaining source¶

FreeBSD source lives in Git. The Subversion repository was frozen after the 2020-21 migration, and the even older CVS/cvsup/csup tooling is long gone, so any old notes that edit a supfile no longer apply.

Install Git and clone the source tree:

pkg install git            # or git-lite
git clone -b releng/15.0 https://git.FreeBSD.org/src.git /usr/src

Pick the branch with -b, or switch later:

cd /usr/src
git checkout releng/15.0     # a release branch
git checkout stable/15       # the stable branch for that major version
git checkout main            # CURRENT (development head)

Update an existing checkout:

cd /usr/src
git pull --ff-only

Which branch to track:

• releng/X.Y - a specific release branch, receiving only security and errata fixes. Matches what freebsd-update tracks for that release; the conservative choice for a production machine.
• stable/X - the stable branch for a major version; gets features merged back from main but stays ABI-stable within that branch.
• main - CURRENT, the development head that becomes the next major release. Expect occasional breakage; useful for testing new features or hardware support.

Check freebsd-version -ku against the branch you intend to track so the build starts from a consistent base rather than jumping major versions by accident (see the "unexpected -CURRENT" row in Section 8 - checking out the wrong branch is an easy mistake to make).

3. Kernel configuration file¶

Unlike Linux's .config, FreeBSD kernel options live in a plain-text config file. Every install ships a GENERIC config under /usr/src/sys/<arch>/conf/ (amd64, arm64, and so on).

3.1 Keep your config outside the source tree¶

Store your configs somewhere that survives a fresh clone or git clean of /usr/src, such as /root/kernels, and either symlink them into the conf directory or point the build at them with KERNCONFDIR:

mkdir -p /root/kernels
vi /root/kernels/MYKERNEL

# Option 1: symlink into the source tree
ln -s /root/kernels/MYKERNEL /usr/src/sys/amd64/conf/MYKERNEL

# Option 2: no symlink - tell make where configs live
make buildkernel KERNCONFDIR=/root/kernels KERNCONF=MYKERNEL

Name the config in caps by convention; this name becomes KERNCONF in later build steps. Version /root/kernels in Git alongside your other host config so you can see what changed between builds.

3.2 Write it as deltas against GENERIC¶

# /root/kernels/MYKERNEL
# Pull in everything GENERIC has, then layer changes on top
include GENERIC
ident       MYKERNEL

# Remove something you don't need
nooptions   COMPAT_FREEBSD32

# Add a driver or option not in GENERIC
device      wg             # WireGuard (also available as the if_wg module)
options     KDTRACE_HOOKS  # DTrace support

# Disable debugging aids for a leaner/faster production kernel
nooptions   INVARIANTS
nooptions   INVARIANT_SUPPORT
nooptions   WITNESS
nooptions   WITNESS_SKIPSPIN

If config can't find GENERIC when the file lives outside the conf directory (the KERNCONFDIR route), give it the full path: include "/usr/src/sys/amd64/conf/GENERIC".

The include GENERIC pattern is strongly preferred over copying and editing the whole file - your custom config then only has to state deltas, which survive source updates far more cleanly than a full copy that silently drifts from GENERIC over time.

Common things people add/remove:

• Add: device wg, options KDTRACE_HOOKS (DTrace), device vmm (bhyve virtualization host), ZFS-related options if not already present.
• Remove for a smaller/faster kernel: INVARIANTS/WITNESS/DDB debugging aids (leave these in if you're troubleshooting a driver or planning to file a bug - they make crash dumps far more useful). Note that main/CURRENT's GENERIC has these debugging options on by default, which is why CURRENT feels slower than a release.
• makeoptions DEBUG=-g plus options KDB and options DDB together give you an interactive kernel debugger on panic - worth keeping on a build you're actively developing against.

The full catalogue of options lives in /usr/src/sys/conf/NOTES (machine-independent) and /usr/src/sys/<arch>/conf/NOTES (architecture-specific).

4. Firewall and failover options (pf, ALTQ, CARP)¶

A firewall or router box is the most common reason to build a custom kernel, because pf's ALTQ traffic shaping can't be loaded as a module.

# pf and friends - compiled in rather than loaded as modules
device      pf
device      pflog          # pflog0 interface for logging
device      pfsync         # state table sync between failover firewalls

# CARP - shared virtual IPs for redundant firewalls/gateways
device      carp

# ALTQ - queueing for pf; must be compiled in
options     ALTQ
options     ALTQ_CBQ       # Class Based Queuing
options     ALTQ_RED       # Random Early Detection
options     ALTQ_RIO       # RED In/Out
options     ALTQ_HFSC      # Hierarchical Packet Scheduler
options     ALTQ_PRIQ      # Priority Queuing
options     ALTQ_FAIRQ     # Fair Queuing
options     ALTQ_CODEL     # Controlled Delay

• pf, pflog, pfsync and carp all work as modules too (sysrc pf_enable=YES pflog_enable=YES loads pf automatically; sysrc kld_list+=carp for CARP), so compile them in only if you want a monolithic kernel.
• ALTQ_NOPCC: older Handbook editions listed this as "required for SMP builds." Current altq(4) documents it only as "required if the TSC is unusable," so leave it out unless you hit that case.
• ALTQ only works with drivers that support it; altq(4) lists them. Check your NIC is on the list before building a shaper around it.

5. Building¶

cd /usr/src
make -j$(sysctl -n hw.ncpu) buildworld
make -j$(sysctl -n hw.ncpu) buildkernel KERNCONF=MYKERNEL

buildworld must complete before buildkernel when you're changing branches or doing a significant version jump, since buildkernel relies on the toolchain buildworld just produced. For a same-version kernel-only tweak (just editing MYKERNEL, not moving to new source), you can often run buildkernel alone - see Section 9 for the exact conditions where that's safe.

Tuning the build with /etc/src.conf and /etc/make.conf:

# /etc/src.conf - controls what gets built
WITHOUT_TESTS=yes
WITHOUT_DEBUG_FILES=yes
WITH_CCACHE_BUILD=yes       # needs devel/ccache installed
WITH_META_MODE=yes          # incremental rebuilds; needs filemon (kldload filemon)

# /etc/make.conf - compiler/linker flags, global build options
KERNCONF=MYKERNEL           # default KERNCONF so you can omit it on the command line

src.conf toggles which subsystems/components are built at all (tests, docs, debug symbols); make.conf carries make-level settings that apply more broadly, including to ports if you build those too. Avoid global CFLAGS overrides in make.conf - the base system already chooses sensible flags, and custom ones are a common source of hard-to-reproduce build failures.

• -j$(sysctl -n hw.ncpu) parallelizes the same way nproc does on Linux.
• A full buildworld typically takes 30 minutes to a couple of hours depending on hardware; buildkernel alone is much faster, usually a few minutes.
• Logs go to your terminal by default; redirect to a file (... 2>&1 | tee /var/tmp/build.log) for a long unattended build, ideally inside tmux so an SSH drop doesn't kill it.
• The build ends with a banner like >>> Kernel build for MYKERNEL completed on .... Anything else at the end of the output means it failed - scroll up to the first Error code line.

6. Installing¶

Take a boot environment first if you're on ZFS (Section 7), then:

cd /usr/src
make installkernel KERNCONF=MYKERNEL
shutdown -r now

# after the reboot, on the new kernel (single-user mode is the most conservative):
cd /usr/src
etcupdate -p                 # pre-world: merge files installworld needs (e.g. new users/groups)
make installworld
etcupdate -B                 # merge the rest of /etc, skipping the build step
make delete-old              # remove files that no longer exist in this version
shutdown -r now

# once everything (including ports/packages) works against the new world:
make delete-old-libs

Order matters: install the kernel first, reboot into it, then install world - this is FreeBSD's documented upgrade order because a newer kernel is generally compatible with an older userland (for one boot), but not reliably the reverse. For a same-version kernel-only change, you can skip installworld/etcupdate entirely and just install and boot the new kernel.

etcupdate does a three-way merge between the old stock /etc, the new stock /etc, and your local changes, so it only stops to ask when your edits genuinely conflict with upstream changes. Resolve those with etcupdate resolve and review what changed with etcupdate status and etcupdate diff.

• First run on a system that's never used etcupdate: bootstrap its baseline with etcupdate extract (ideally from the source of the version currently installed) before your first upgrade, or it has no "old stock" copy to merge against.
• mergemaster was the older interactive tool for the same job. It was deprecated for years and removed from the base system in FreeBSD 15.0, so older notes that say mergemaster -Ui should be read as etcupdate.

Skipping the /etc merge after a version bump is a common source of a system that boots but has broken /etc/rc.d scripts or mismatched config file formats - don't skip it on anything but a pure kernel-config tweak with no world rebuild.

7. Rollback: boot environments, kernel.old and test boots¶

7.1 ZFS boot environments¶

If your root filesystem is ZFS (the default on most modern installs), create a boot environment before installing anything new. It's FreeBSD's equivalent of a Btrfs/LVM snapshot rollback, and the single best safety net for kernel and world upgrades:

bectl create pre-MYKERNEL        # snapshot the current, working BE
make installkernel KERNCONF=MYKERNEL
shutdown -r now
# ...continue with installworld/etcupdate from Section 6

If the new kernel or world is broken, select the old boot environment from the loader menu (it lists BEs automatically on ZFS-root systems), or roll back explicitly:

bectl list                       # see available boot environments
bectl activate pre-MYKERNEL      # make it the default for next boot
shutdown -r now

7.2 kernel.old (any filesystem)¶

On UFS there's no built-in snapshot, but installkernel always moves the previous /boot/kernel to /boot/kernel.old before installing the new one. To boot it, escape to the loader prompt from the boot menu and run:

unload
boot kernel.old

Then fix or reinstall the kernel from there. Note that running installkernel twice in a row replaces kernel.old with your first (possibly broken) custom kernel, so copy a known-good kernel aside (cp -Rp /boot/kernel /boot/kernel.good) before iterating.

7.3 One-time test boot with nextboot¶

You can install a new kernel under a different name and boot it exactly once, leaving the current kernel as the default if the test kernel hangs or panics:

make installkernel KERNCONF=MYKERNEL INSTKERNNAME=testkernel   # installs to /boot/testkernel
nextboot -k testkernel
shutdown -r now

If it works, install it normally; if it doesn't, a power cycle brings you back to /boot/kernel.

8. Verifying and troubleshooting¶

uname -v                    # build ident (MYKERNEL) and timestamp
freebsd-version -kru        # installed kernel, running kernel, userland versions
dmesg | grep -i error       # scan for driver/hardware issues
kldstat                     # loaded kernel modules
pfctl -s queue              # if you built ALTQ: confirm queues load
zpool status                # if using ZFS, confirm pools are healthy
service -e                  # services enabled to start (sanity check after etcupdate)

If you built with options KDB and options DDB, a panic drops you into an interactive debugger (db> prompt) instead of just rebooting - bt for a backtrace is the first thing to run there before you reboot and lose the state.

9. Tips for repeat builds¶

When you can skip buildworld: if you're only editing MYKERNEL (adding/removing a driver or option) and haven't updated the source tree or changed branches, make buildkernel KERNCONF=MYKERNEL alone against your existing /usr/obj toolchain is generally safe and much faster. Any time you git pull/git checkout a different point in source history, do a full buildworld first.

Speeding up iteration:

# Enable ccache and meta mode once, in /etc/src.conf
WITH_CCACHE_BUILD=yes
WITH_META_MODE=yes

# Skip rebuilding modules you don't use (in the kernel config or on the command line)
make buildkernel KERNCONF=MYKERNEL -DKERNFAST          # reuse objects, skip cleaning
make buildkernel KERNCONF=MYKERNEL MODULES_OVERRIDE="zfs if_wg"

# Rebuild and install just the kernel after a config-only change
make buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL

-DKERNFAST only helps when the previous kernel build of the same config is still in /usr/obj; MODULES_OVERRIDE limits which modules get built and installed, so don't use it on a machine that loads modules you've left out.

Keeping source current:

cd /usr/src
git pull --ff-only
less UPDATING      # read the entries since your last build before you start
# then a full buildworld/buildkernel/installkernel/installworld/etcupdate cycle

Diffing your config against GENERIC to see your full set of deltas at a glance:

diff /usr/src/sys/amd64/conf/GENERIC /root/kernels/MYKERNEL

(With include GENERIC, your config file is the list of intentional changes, so this mostly matters for older full-copy configs.)

Cross-building for another architecture (e.g. building an arm64 kernel from an amd64 host) uses the same buildworld/buildkernel targets with TARGET/TARGET_ARCH set:

make TARGET=arm64 TARGET_ARCH=aarch64 buildworld
make TARGET=arm64 TARGET_ARCH=aarch64 buildkernel KERNCONF=GENERIC

Useful for building images for Raspberry Pi or other arm64 boards from a faster amd64 build machine, then transferring the result rather than compiling natively on the slower target.

References¶

• FreeBSD Handbook: Configuring the FreeBSD Kernel
• FreeBSD Handbook: Updating FreeBSD from Source
• FreeBSD Handbook: PF and ALTQ
• man 5 src.conf, man 7 build, man 8 config, man 8 etcupdate, man 8 nextboot, man 8 bectl, man 4 altq, man 4 carp
• /usr/src/UPDATING and /usr/src/sys/conf/NOTES