The Linux Kernel¶

The kernel is the one program that is always running. It owns the hardware, decides which process runs on which CPU, hands out memory, mediates every file and network operation, and enforces the security boundaries everything else relies on. Almost every hard production problem — latency spikes, OOM kills, container escapes, packet drops, "disk is full but du says otherwise" — eventually ends up here.

This page is the hub for kernel documentation. Related pages:

• Compiling a Custom Linux Kernel

Table of Contents¶

1. What the Kernel Is
2. Versions and the Release Process
3. The Boot Process
4. The Kernel Command Line
5. System Calls
6. Process Management and Scheduling
7. Memory Management
8. VFS, Filesystems, and the Block Layer
9. Networking Stack
10. Device Drivers and the Device Model
11. Kernel Modules
12. Kernel Interfaces: /proc, /sys, and sysctl
13. Namespaces and cgroups (Container Primitives)
14. eBPF
15. Security
16. Logging, Observability, and Tracing
17. Crashes, Oopses, Panics, and kdump
18. Live Patching and kexec
19. Tuning Reference
20. Rust in the Kernel
21. The Source Tree
22. Contributing Upstream
23. Cheat Sheet
24. Further Reading

What the Kernel Is¶

Linux is a monolithic, modular kernel:

• Monolithic — the scheduler, memory manager, filesystems, network stack, and drivers all run in one privileged address space. A call between subsystems is a function call, not a message. Fast, but a bug in any driver can take down the whole system.
• Modular — most of that code can be built as loadable kernel modules (.ko) and inserted or removed at runtime without rebooting.

Contrast with microkernels (seL4, QNX, MINIX 3), which push drivers and filesystems into user-space servers for isolation at the cost of IPC overhead.

Kernel space vs user space¶

+----------------------------------------------------------+
|  User space (ring 3)                                      |
|   bash, systemd, nginx, containerd, kubelet, glibc/musl   |
+---------------------- system call interface -------------+
|  Kernel space (ring 0)                                    |
|   syscalls | VFS | scheduler | MM | net | security (LSM)  |
|   block layer | device model | IPC | eBPF | tracing       |
|   arch-specific code | device drivers                     |
+----------------------------------------------------------+
|  Hardware: CPU, RAM, disks, NICs, GPUs, firmware          |
+----------------------------------------------------------+

The CPU enforces the split with privilege levels (rings on x86, exception levels on ARM). User processes can't touch hardware or other processes' memory directly — they ask the kernel through system calls.

What the kernel is not¶

"Linux" in casual use means a whole OS. Strictly, Linux is only the kernel. The C library (glibc, musl), init system (systemd, OpenRC), shell, coreutils, and package manager are separate projects that a distribution bundles together.

Versions and the Release Process¶

uname -r          # 6.12.0-55.el10.x86_64
uname -a          # everything, incl. build date and arch
cat /proc/version # compiler and build info

Version string anatomy¶

6.12.0-55.el10.x86_64

Release cycle¶

1. Merge window (~2 weeks) — Linus pulls new features from subsystem maintainers.
2. -rc1 … -rc7/-rc8 — one release candidate a week; fixes only.
3. Final release — roughly every 9–10 weeks.

Kernel lines¶

Check what's current at https://www.kernel.org/.

The Boot Process¶

Firmware (UEFI / BIOS)
   └─> Bootloader (GRUB2, systemd-boot, U-Boot)
         └─> Kernel image (vmlinuz) + initramfs loaded into RAM
               └─> Kernel decompresses, initializes CPU, memory, drivers
                     └─> Unpacks initramfs as a tmpfs root, runs /init
                           └─> initramfs finds the real root (LVM, LUKS, RAID, NFS...)
                                 └─> switch_root to real /
                                       └─> /sbin/init (systemd) becomes PID 1

Files in /boot¶

Initramfs tools by distro¶

Rebuild the initramfs whenever you change storage drivers, LUKS/LVM setup, or module options that must apply in early boot.

Boot timing¶

systemd-analyze                 # firmware / loader / kernel / initrd / userspace
systemd-analyze blame
systemd-analyze critical-chain
dmesg | head -50                # earliest kernel messages

The Kernel Command Line¶

Parameters passed by the bootloader configure the kernel (and initramfs, and systemd) before anything else runs.

cat /proc/cmdline

Common parameters¶

Making changes persistent¶

# RHEL / Fedora (BLS)
sudo grubby --update-kernel=ALL --args="console=ttyS0,115200"
sudo grubby --update-kernel=ALL --remove-args="quiet"
sudo grubby --info=DEFAULT

# Debian / Ubuntu
sudoedit /etc/default/grub         # GRUB_CMDLINE_LINUX_DEFAULT="..."
sudo update-grub

# Generic GRUB2
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# systemd-boot / UKI
sudoedit /etc/kernel/cmdline

For a one-off, press e at the GRUB menu, edit the linux line, and boot with Ctrl-x.

System Calls¶

A system call is the controlled doorway from user space into the kernel. glibc wraps most of them (open() → openat syscall), and the CPU switches privilege level via syscall (x86_64) or svc (arm64).

Core families:

strace -f -e trace=openat,execve ./program   # what is it doing?
strace -c ./program                           # syscall counts and time
ausyscall --dump | less                       # syscall number table (audit package)

vDSO — a small kernel-provided shared library mapped into every process so hot calls like clock_gettime avoid a real mode switch.

io_uring (5.1+) — shared ring buffers between user space and kernel for batched async I/O with very few syscalls. Powerful, but has been a major source of kernel CVEs; several distros and Google disable it by default (kernel.io_uring_disabled sysctl).

Process Management and Scheduling¶

Tasks¶

To the kernel, processes and threads are both tasks (struct task_struct). Threads are just tasks that share an address space, file table, and so on — the difference is which CLONE_* flags were passed.

Process states as shown by ps/top:

Kernel threads appear in ps with names in brackets: [kworker/0:1], [ksoftirqd/0], [kswapd0], [rcu_sched].

ps -eo pid,ppid,stat,ni,pri,cls,comm --sort=-pcpu | head
cat /proc/<pid>/status
cat /proc/<pid>/sched

Scheduler classes¶

Checked in priority order — a runnable task in a higher class always wins.

The fair class was CFS (Completely Fair Scheduler) from 2.6.23 until 6.6, when it was replaced by EEVDF (Earliest Eligible Virtual Deadline First), which handles latency more predictably. sched_ext (6.12) lets you load an entire scheduling policy as a BPF program.

PREEMPT_RT — the real-time patchset was fully merged in 6.12, so mainline kernels can be built fully preemptible.

Controlling priority and placement¶

nice -n 10 cmd                    # start with lower priority (-20 highest, 19 lowest)
renice -n 5 -p <pid>
chrt -f 50 cmd                    # SCHED_FIFO priority 50
chrt -p <pid>                     # show policy
taskset -c 2,3 cmd                # pin to CPUs 2-3
taskset -cp <pid>
ionice -c3 cmd                    # idle I/O class

Load average¶

uptime shows 1/5/15-minute averages of tasks that are runnable or in D state — on Linux, heavy I/O wait raises load even with idle CPUs. Compare against nproc, and look at vmstat 1 (r and b columns) and PSI (below) for the real story.

Pressure Stall Information (PSI)¶

cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/io

some avg10=4.20 means at least one task was stalled 4.2 % of the last 10 s. full means all non-idle tasks were stalled. Per-cgroup versions exist in cgroup v2 (cpu.pressure, etc.). systemd-oomd uses memory PSI.

Interrupts¶

cat /proc/interrupts       # hardware IRQ counts per CPU
cat /proc/softirqs         # NET_RX, TIMER, BLOCK, RCU...
cat /proc/irq/<n>/smp_affinity_list

Hardware interrupts do minimal work and defer the rest to softirqs, tasklets, or workqueues (kworker threads). High si in top usually means network receive processing.

Memory Management¶

Virtual memory¶

Every process gets its own virtual address space. The kernel and MMU map virtual pages (usually 4 KiB) to physical frames via page tables; the TLB caches translations. Pages are only backed by physical memory when first touched (demand paging).

cat /proc/<pid>/maps        # memory regions
cat /proc/<pid>/smaps_rollup
pmap -x <pid>

Reading memory usage¶

free -h
cat /proc/meminfo
vmstat 1

"Linux ate my RAM" — the page cache uses idle memory and gives it back under pressure. See https://www.linuxatemyram.com/.

Reclaim and swap¶

When free memory drops below watermarks, kswapd reclaims pages in the background; if that's not enough, allocating tasks do direct reclaim (latency!). Clean file pages are dropped, dirty ones written back, anonymous pages swapped out.

• vm.swappiness (0–200, default 60) — bias between reclaiming anonymous memory (swap) vs file cache.
• MGLRU (multi-gen LRU, 6.1+) — improved page aging; check /sys/kernel/mm/lru_gen/enabled.
• zswap — compressed cache in front of a swap device. zram — compressed RAM block device used as swap (Fedora default).

swapon --show
cat /proc/swaps

Overcommit¶

The OOM killer¶

When memory is exhausted and reclaim fails, the OOM killer picks a victim by oom_score (mostly memory footprint) and kills it.

cat /proc/<pid>/oom_score
echo -500 | sudo tee /proc/<pid>/oom_score_adj    # -1000 = never kill, 1000 = kill first
journalctl -k | grep -i -A20 'out of memory'
dmesg -T | grep -i 'killed process'

In systemd units: OOMScoreAdjust=-500. In containers, a cgroup hitting memory.max triggers an OOM kill inside that cgroup only — this is Kubernetes' OOMKilled (exit code 137).

User-space OOM daemons (systemd-oomd, earlyoom) act earlier, based on PSI, to avoid the long thrash before the kernel OOM killer fires.

Huge pages¶

cat /sys/kernel/mm/transparent_hugepage/enabled   # [always] madvise never
grep -i huge /proc/meminfo

Many databases (MongoDB, older Redis, Oracle) recommend madvise or never for THP due to latency spikes from compaction.

NUMA¶

On multi-socket systems memory is local to a CPU node; remote access is slower.

numactl --hardware
numastat -p <pid>
numactl --cpunodebind=0 --membind=0 cmd

VFS, Filesystems, and the Block Layer¶

The Virtual File System¶

The VFS is a common abstraction over every filesystem — ext4, XFS, Btrfs, NFS, procfs, tmpfs, FUSE all present the same open/read/write/stat interface. Key objects:

"Everything is a file" works because devices, pipes, sockets, and pseudo-files all implement the VFS operations.

cat /proc/filesystems         # filesystems the kernel supports right now
findmnt                       # mount tree
stat file
ls -l /proc/<pid>/fd
lsof +L1                      # deleted-but-open files (the "df full, du not" mystery)

Page cache and writeback¶

Reads and writes go through the page cache. Writes dirty pages in memory; flusher threads write them back based on:

On machines with lots of RAM, use the _bytes variants so a percentage doesn't mean tens of GiB of dirty data.

sync
echo 3 | sudo tee /proc/sys/vm/drop_caches   # drop clean caches — benchmarking only

Block layer¶

filesystem -> bio -> blk-mq (per-CPU software queues -> hardware queues) -> driver -> device

blk-mq (multi-queue) is the only block path in modern kernels. I/O schedulers:

cat /sys/block/nvme0n1/queue/scheduler
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler
lsblk -o NAME,ROTA,SCHED,SIZE,TYPE,MOUNTPOINTS
iostat -xz 1

Persist with a udev rule:

# /etc/udev/rules.d/60-iosched.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"

Stacked block devices — device-mapper (LVM, LUKS/dm-crypt, dm-multipath, dm-verity) and md (software RAID) — sit between filesystem and hardware.

Networking Stack¶

NIC -> driver (NAPI polling) -> [XDP] -> sk_buff -> tc ingress -> netfilter PREROUTING
    -> routing -> INPUT -> socket -> application
                -> FORWARD -> POSTROUTING -> tc egress -> driver -> NIC

Concepts:

• sk_buff — the kernel's packet structure.
• NAPI — switches from interrupt-per-packet to polling under load.
• GRO/GSO/TSO — batch packets to cut per-packet overhead (ethtool -k eth0).
• RSS/RPS/RFS — spread receive processing across CPUs.
• Netfilter — hooks for filtering/NAT. nftables is the modern front end; iptables on current distros is usually iptables-nft, a compatibility layer. Kubernetes kube-proxy uses iptables, IPVS, or (newer) nftables mode; Cilium replaces it with eBPF.
• conntrack — stateful connection tracking (conntrack -L, nf_conntrack_max). A full conntrack table drops new connections with nf_conntrack: table full in dmesg.
• tc — traffic control: qdiscs (fq_codel default, fq for BBR), shaping, eBPF hooks.
• XDP — eBPF at the driver, before an sk_buff exists. Line-rate filtering/load balancing.
• Network namespaces, veth, bridges, VXLAN — the building blocks of container networking.

ip -s link
ip route
ss -tulpn                # listening sockets
ss -s                    # socket summary
nstat -az | grep -i drop
cat /proc/net/softnet_stat   # 2nd column = drops due to backlog
ethtool -S eth0 | grep -iE 'drop|err|miss'
sysctl net.ipv4.tcp_congestion_control   # cubic default; bbr available

Device Drivers and the Device Model¶

Drivers bind to devices on a bus (PCI, USB, platform, I²C, virtio). The device model exposes this tree in /sys, and udev (systemd-udevd) reacts to kernel uevents to create /dev nodes, load modules, and apply rules.

lspci -nnk               # PCI devices + driver in use + available modules
lsusb -t
ls -l /sys/class/net/eth0/device/driver
udevadm info /dev/sda
udevadm monitor --kernel --udev
dmesg | grep -i firmware

Device files:

ls -l /dev/sda /dev/null
# brw-rw---- 1 root disk 8, 0 ...   b = block, major 8, minor 0
# crw-rw-rw- 1 root root 1, 3 ...   c = character
cat /proc/devices

Firmware blobs (linux-firmware) live in /lib/firmware and are loaded by drivers on demand.

Kernel Modules¶

lsmod                         # loaded modules
modinfo br_netfilter          # description, params, deps, signature
sudo modprobe br_netfilter    # load with dependencies
sudo modprobe -r br_netfilter # unload
systool -vm <module>          # current parameter values (sysfsutils)
cat /sys/module/<mod>/parameters/<param>
ls /lib/modules/$(uname -r)/
sudo depmod -a                # rebuild dependency map

Prefer modprobe over insmod/rmmod — it resolves dependencies and reads config.

Persistent configuration¶

# Load at boot
echo -e "overlay\nbr_netfilter" | sudo tee /etc/modules-load.d/k8s.conf

# Module options
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm.conf

# Blacklist (prevents autoload, not explicit modprobe)
echo "blacklist nouveau" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf

# Truly prevent loading
echo "install usb_storage /bin/false" | sudo tee /etc/modprobe.d/no-usb-storage.conf

Rebuild the initramfs if the module is loaded during early boot.

Out-of-tree modules and DKMS¶

DKMS rebuilds third-party modules (NVIDIA, ZFS, VirtualBox, some Wi-Fi) automatically on every kernel update.

dkms status
sudo dkms autoinstall

Module signing and Secure Boot¶

With Secure Boot, the kernel enters lockdown and refuses unsigned modules. Sign out-of-tree modules with a Machine Owner Key:

sudo mokutil --import /var/lib/dkms/mok.pub   # enrol on next boot
mokutil --sb-state
cat /sys/kernel/security/lockdown

module.sig_enforce=1 requires signatures even without Secure Boot.

Kernel Interfaces: /proc, /sys, and sysctl¶

Useful /proc/<pid> entries: cmdline, environ, status, limits, fd/, maps, smaps_rollup, stack (kernel stack, root), wchan, cgroup, ns/, mountinfo, io.

sysctl¶

sysctl -a | grep -i forward
sysctl net.ipv4.ip_forward
sudo sysctl -w net.ipv4.ip_forward=1         # runtime only

# Persistent: drop-in files, applied at boot by systemd-sysctl
echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/90-forward.conf
sudo sysctl --system                          # reload all

Load order: /usr/lib/sysctl.d/, /run/sysctl.d/, /etc/sysctl.d/, then /etc/sysctl.conf — later files win for the same key. Name files with numeric prefixes.

Note that some net.* sysctls are per network namespace — setting them on the host doesn't change them inside a container. Kubernetes exposes "safe" namespaced sysctls via securityContext.sysctls.

Namespaces and cgroups (Container Primitives)¶

A container is just a process with its own namespaces (what it can see), cgroups (what it can use), plus capabilities, seccomp, and an LSM profile (what it may do).

Namespaces¶

lsns
ls -l /proc/$$/ns
sudo unshare --pid --fork --mount-proc --net --uts bash   # mini container
sudo nsenter -t <pid> -n ip addr                           # enter a container's netns
sudo nsenter -t <pid> -a                                   # all namespaces
ip netns add test && ip netns exec test ip link

Debug a Kubernetes pod's network from the node: find the container PID (crictl inspect <id> | jq .info.pid) then nsenter -t <pid> -n tcpdump -i eth0.

cgroups v2¶

A single unified hierarchy at /sys/fs/cgroup (the default on all current major distros; v1 is deprecated). systemd owns the tree: slices → services/scopes → processes. Kubernetes uses kubepods.slice.

stat -fc %T /sys/fs/cgroup        # cgroup2fs = v2
systemd-cgls
systemd-cgtop
cat /proc/<pid>/cgroup
cat /sys/fs/cgroup/cgroup.controllers

Key controller files:

# Limit a one-off command
systemd-run --scope -p MemoryMax=512M -p CPUQuota=50% stress-ng --vm 1
# Set on a running service
sudo systemctl set-property nginx.service MemoryMax=1G

eBPF¶

eBPF lets you load small, verified programs into the kernel and attach them to hooks — without writing a module or risking a crash. The verifier proves the program terminates and only accesses memory safely; a JIT compiles it to native code.

Attach points: kprobes/kretprobes, tracepoints, uprobes, perf events, XDP, tc, cgroup hooks, sockets, LSM hooks, sched_ext. Data moves through maps (hash, array, ring buffer, per-CPU).

Uses:

• Observability — bpftrace, BCC tools, Pixie, Parca, Pyroscope
• Networking — Cilium, Calico eBPF data plane, Katran load balancer
• Security — Falco, Tetragon, Tracee, BPF LSM

sudo bpftool prog list
sudo bpftool map list
sudo bpftool feature probe | less

# Who is opening which files?
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args.filename)); }'
# (older bpftrace versions use args->filename)

# Syscall counts by process
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# BCC tools (package: bcc-tools / bpfcc-tools)
sudo execsnoop    # new processes
sudo opensnoop
sudo biolatency   # block I/O latency histogram
sudo tcpconnect
sudo runqlat      # scheduler run-queue latency

BTF + CO-RE (Compile Once, Run Everywhere) let one compiled BPF program run across kernel versions; check /sys/kernel/btf/vmlinux exists.

Security¶

Privilege model¶

• Capabilities split root into ~40 privileges (CAP_NET_ADMIN, CAP_SYS_ADMIN, CAP_NET_BIND_SERVICE...). CAP_SYS_ADMIN is effectively "root lite" — avoid granting it.

getpcaps <pid>
grep Cap /proc/<pid>/status && capsh --decode=<hex>
getcap /usr/bin/ping
sudo setcap cap_net_bind_service=+ep ./server

• seccomp — filters which syscalls a process may make. Docker/containerd apply a default profile; Kubernetes seccompProfile: RuntimeDefault.
• no_new_privs — prevents gaining privileges via setuid exec.

Linux Security Modules (LSMs)¶

cat /sys/kernel/security/lsm
getenforce && sestatus
aa-status

CPU vulnerabilities¶

grep -r . /sys/devices/system/cpu/vulnerabilities/

Shows the mitigation status for Spectre, Meltdown, MDS, Retbleed, etc.

Hardening sysctls¶

# /etc/sysctl.d/80-hardening.conf
kernel.kptr_restrict = 2              # hide kernel pointers
kernel.dmesg_restrict = 1             # dmesg for root only
kernel.yama.ptrace_scope = 1          # ptrace only descendants
kernel.unprivileged_bpf_disabled = 1
net.core.bpf_jit_harden = 2
kernel.kexec_load_disabled = 1        # one-way switch
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_fifos = 2
fs.protected_regular = 2
fs.suid_dumpable = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.tcp_syncookies = 1

Test before deploying — unprivileged_userns_clone-style restrictions or BPF restrictions can break rootless containers, browsers' sandboxes, and some monitoring agents. kernel-hardening-checker (a Python tool) audits a config, cmdline, and sysctls against recommendations.

Logging, Observability, and Tracing¶

Kernel log¶

dmesg -T                 # human timestamps
dmesg -l err,warn
dmesg -w                 # follow
journalctl -k            # kernel messages, this boot
journalctl -k -b -1      # previous boot (needs persistent journal)
journalctl -k -p err

Enable persistent journal so you can read logs from before a crash: sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald (or Storage=persistent in journald.conf).

printk log levels¶

cat /proc/sys/kernel/printk   # console / default / minimum / boot-default
sudo dmesg -n 4               # only <4 to console

Dynamic debug turns on pr_debug() messages for specific code:

echo 'module nfs +p' | sudo tee /sys/kernel/debug/dynamic_debug/control

The toolbox¶

ftrace quick start¶

cd /sys/kernel/tracing
cat available_tracers
echo function_graph > current_tracer
echo vfs_read > set_graph_function
echo 1 > tracing_on; sleep 1; echo 0 > tracing_on
head -50 trace
echo nop > current_tracer

# Friendlier front end
sudo trace-cmd record -p function_graph -g do_sys_openat2 -- ls
sudo trace-cmd report | less

perf quick start¶

sudo perf top -g
sudo perf record -F 99 -a -g -- sleep 30
sudo perf report --stdio | head -100
sudo perf stat -e cycles,instructions,cache-misses -a sleep 5
sudo perf sched latency

kernel.perf_event_paranoid controls what unprivileged users can profile.

Crashes, Oopses, Panics, and kdump¶

Reading an oops¶

Key lines: the reason (BUG: kernel NULL pointer dereference), RIP: (faulting function + offset, e.g. ext4_foo+0x1a/0x80 [ext4]), the Tainted: flags, and the Call Trace:. Decode offsets to source lines with scripts/decode_stacktrace.sh and a matching vmlinux with debuginfo.

Taint flags¶

cat /proc/sys/kernel/tainted   # 0 = clean

Upstream developers generally won't debug reports from kernels tainted with proprietary modules. tools/debugging/kernel-chktaint in the source tree decodes the number.

Panic behaviour¶

kernel.panic = 10             # reboot 10 s after panic (0 = hang forever)
kernel.panic_on_oops = 1      # treat oops as panic (good with kdump)
kernel.softlockup_panic = 0
kernel.hung_task_panic = 0

kdump¶

kdump reserves memory for a second "capture" kernel. On a panic, the system kexecs into it and saves the crashed memory image to /var/crash.

# RHEL/Fedora
sudo dnf install kexec-tools crash kernel-debuginfo
sudo grubby --update-kernel=ALL --args="crashkernel=auto"   # or a size; RHEL 9+ uses kdumpctl reset-crashkernel
sudo systemctl enable --now kdump
kdumpctl status

# Debian/Ubuntu
sudo apt install kdump-tools crash linux-image-$(uname -r)-dbgsym

# Test (this WILL crash the box)
echo 1 | sudo tee /proc/sys/kernel/sysrq
echo c | sudo tee /proc/sysrq-trigger

# Analyse
sudo crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/<dir>/vmcore
#   crash> bt      log     ps      kmem -i     files <pid>

Magic SysRq¶

cat /proc/sys/kernel/sysrq      # bitmask; 1 = all, 176 = sync+remount+reboot
echo 1 | sudo tee /proc/sys/kernel/sysrq
echo w | sudo tee /proc/sysrq-trigger   # dump blocked (D-state) tasks
echo t | sudo tee /proc/sysrq-trigger   # dump all tasks
echo m | sudo tee /proc/sysrq-trigger   # memory info
echo l | sudo tee /proc/sysrq-trigger   # backtrace all CPUs

On a frozen machine with a keyboard: Alt+SysRq+ R E I S U B — unRaw keyboard, tErminate, kIll, Sync, Unmount (remount ro), reBoot.

Live Patching and kexec¶

Live patching¶

Apply security fixes to the running kernel without a reboot by redirecting functions (via ftrace) to patched versions. Taints the kernel with K.

cat /sys/kernel/livepatch/*/enabled

Live patches buy time between reboots; they don't replace them — not every fix can be live-patched.

kexec¶

Boot a new kernel directly from the running one, skipping firmware and bootloader — much faster reboots on servers with long POST.

sudo kexec -l /boot/vmlinuz-<ver> --initrd=/boot/initramfs-<ver>.img --reuse-cmdline
sudo systemctl kexec

Is a reboot pending?¶

# Running vs newest installed
uname -r; ls -t /boot/vmlinuz-* | head -1
dnf needs-restarting -r          # RHEL/Fedora
cat /var/run/reboot-required     # Debian/Ubuntu

Tuning Reference¶

Always measure before and after; most defaults are good. These are the ones that come up most in real infrastructure.

Kubernetes / container nodes¶

# /etc/modules-load.d/k8s.conf
overlay
br_netfilter

# /etc/sysctl.d/90-k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 8192
net.netfilter.nf_conntrack_max = 1048576

too many open files from kubelet/log shippers is often inotify instances, not file descriptors.

Elasticsearch / OpenSearch¶

vm.max_map_count = 262144     # required minimum
vm.swappiness = 1             # or disable swap

High-connection servers¶

net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.core.netdev_max_backlog = 16384
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Files and processes¶

fs.file-max = 2097152         # system-wide (usually already huge on 64-bit)
kernel.pid_max = 4194304

Per-process limits are separate: ulimit -n, /etc/security/limits.d/, or LimitNOFILE= in systemd units.

Memory¶

vm.swappiness = 10
vm.dirty_background_bytes = 268435456   # 256 MiB
vm.dirty_bytes = 1073741824             # 1 GiB
vm.min_free_kbytes = 131072             # bigger cushion for bursty allocs

Profiles¶

tuned (RHEL/Fedora) bundles sysctls, I/O schedulers, CPU governors and more:

tuned-adm list
tuned-adm active
sudo tuned-adm profile throughput-performance   # or latency-performance, virtual-guest

Rust in the Kernel¶

Initial Rust support landed in 6.1 (December 2022) — the first new language accepted alongside C. It provides safe abstractions over kernel APIs so drivers can be written with Rust's memory-safety guarantees, eliminating whole classes of bugs (use-after-free, data races) that dominate kernel CVEs.

Status and landmarks:

• Lives in rust/ (bindings and the kernel crate) with drivers under drivers/.
• Real drivers have shipped: Android binder, the QR-code panic screen for DRM, the ASIX PHY driver, NVMe and null block reference drivers, and the Nova NVIDIA GPU driver (in progress), plus the Asahi Apple GPU driver out of tree.
• Built with a specific minimum rustc plus bindgen; check with make LLVM=1 rustavailable, enable with CONFIG_RUST=y.
• At the 2025 Maintainers Summit, Rust support was declared no longer experimental.

The model is C and Rust coexisting long-term — subsystems opt in; nobody is rewriting the core in Rust.

The Source Tree¶

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git — or browse with cross-references at https://elixir.bootlin.com/.

The user-space ABI is sacred — "we do not break user space." Internal kernel APIs, by contrast, change freely, which is why out-of-tree modules break between versions.

For building from source, see Compiling a Custom Linux Kernel.

Checking a running kernel's config¶

grep CONFIG_BPF_SYSCALL /boot/config-$(uname -r)
zcat /proc/config.gz | grep CONFIG_IKCONFIG   # if built with CONFIG_IKCONFIG_PROC

Contributing Upstream¶

Development happens over email, not merge requests.

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
# Work on the relevant subsystem tree (see MAINTAINERS "T:" lines)
git commit -s                                  # Signed-off-by (DCO) is mandatory
git format-patch -1 --base=auto
./scripts/checkpatch.pl 0001-*.patch
./scripts/get_maintainer.pl 0001-*.patch       # who to send it to
git send-email --to=... --cc=... 0001-*.patch

b4 (pip install b4) simplifies preparing series and pulling patches from https://lore.kernel.org/.

Good starting points: the KernelNewbies site, fixing checkpatch or compiler warnings in drivers/staging/, running and fixing kselftests (make -C tools/testing/selftests run_tests), testing release candidates and reporting regressions, and the Linux Foundation Mentorship Program.

Read first: Documentation/process/submitting-patches.rst and Documentation/process/coding-style.rst.

Cheat Sheet¶

# Identity
uname -r                              # running version
cat /proc/cmdline                     # boot parameters
cat /proc/sys/kernel/tainted          # taint state

# Logs
dmesg -T -l err,warn
journalctl -k -b -1

# Modules
lsmod; modinfo <m>; modprobe <m>; modprobe -r <m>

# Tunables
sysctl <key>; sysctl -w <key>=<v>; sysctl --system

# Resources and pressure
free -h; vmstat 1; iostat -xz 1; cat /proc/pressure/*

# Processes
ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /D/'   # stuck in D state
cat /proc/<pid>/stack

# Containers
lsns; nsenter -t <pid> -n <cmd>; systemd-cgls; cat /proc/<pid>/cgroup

# Tracing
perf top -g; bpftrace -l 'tracepoint:syscalls:*'; trace-cmd record ...

# Hardware/drivers
lspci -nnk; lsblk; udevadm info /dev/<dev>

# Security
cat /sys/kernel/security/lsm
grep -r . /sys/devices/system/cpu/vulnerabilities/

Further Reading¶

• docs.kernel.org — official documentation, including the admin guide and kernel-parameters
• kernelnewbies.org — per-release changelogs in plain English (LinuxChanges)
• LWN.net — the best ongoing coverage of kernel development
• elixir.bootlin.com — cross-referenced source browser
• brendangregg.com — performance analysis, flame graphs, eBPF tools
• Linux Kernel Development — Robert Love (dated but still the clearest overview)
• Understanding the Linux Kernel — Bovet & Cesati
• Systems Performance, 2nd ed. and BPF Performance Tools — Brendan Gregg
• Linux Device Drivers, 3rd ed. — free online, old APIs but sound concepts
• man 7 pages: namespaces, cgroups, capabilities, bpf-helpers, sched, proc (man 5 proc)