MIPS XIP Docs
Linux 6.12.34 · MIPS32r2 · GPL-2.0

MIPS XIP Kernel

Execute-In-Place Linux kernel for memory-constrained embedded devices. Build firmware, IoT nodes, and router systems that run directly from SPI-NOR flash — no RAM copy required.

2304 KiB RAM Saved
5 Kernel Patches
4 KiB Boot Shim
<1 KiB PID 1 Binary

Project Overview

This project implements CONFIG_XIP_KERNEL for the MIPS architecture on Linux 6.12.34. It enables the kernel to execute its .text, .rodata, and .init.text sections directly from SPI-NOR flash without copying to RAM or decompressing.

Why XIP Matters: On routers with 8–16 MiB of total RAM, a stock compressed kernel consumes ~2.3 MiB just for the decompressed copy. XIP eliminates this overhead entirely.

Key Capabilities

💾

Flash Execution

Kernel code runs directly from SPI-NOR flash at physical address 0x9FC01000. No decompression, no RAM copy.

🧠

RAM Optimization

Frees ~2304 KiB of宝贵 RAM on memory-constrained devices. Critical for IoT nodes and budget routers.

🔧

Minimal Patches

Only 5 kernel patches required. Uses MIPS KSEG0/KSEG1 direct-mapped segments — no page-table fixups needed.

🧪

Verified Boot

Static layout assertions + QEMU smoke tests ensure correctness. CI runs on every push.

Supported Configurations

DefconfigPurposeNetworkingUse Case
xip_qemu_malta_defconfigMinimal XIP testNoDevelopment & verification
lwrt_qemu_malta_defconfigRouter userspaceFull IPv4 + nftablesLWRT router firmware

Quick Start Guide

Build a bootable XIP kernel image in under 5 minutes.

Prerequisites

bash
# Ubuntu/Debian
sudo apt update
sudo apt install -y clang lld llvm binutils-mipsel-linux-gnu \
  qemu-system-mips flex bison bc libelf-dev libssl-dev \
  python3 python3-pip texinfo

# Verify toolchain
clang --version
mipsel-linux-gnu-ld --version
qemu-system-mips --version

Build Everything

bash
# Clone the repository
git clone https://github.com/user/mips-xip-kernel.git
cd mips-xip-kernel

# Full build: download + patch + compile + assemble
make

# Static layout verification
make verify

# Boot test in QEMU
make test

# Interactive QEMU session
make run

Build Outputs

text
build/
├── linux-6.12.34/          # Patched kernel source tree
│   ├── vmlinux             # ELF kernel image
│   └── System.map          # Symbol map
├── out/
│   ├── xip-bios.bin        # Final flash image (shim + kernel ROM)
│   ├── shim.elf            # Boot shim ELF
│   ├── shim.bin            # Boot shim raw binary
│   └── kernel.bin          # Kernel ROM blob (objcopy -O binary)
└── boot.log                # QEMU serial output (if make test/run)

System Architecture

Application Layer
Freestanding PID 1
LWRT Router Userspace
IoT Application
↓ syscalls (raw MIPS o32)
Kernel Layer
XIP Kernel (ROM)
RAM Data
TLB Handlers (RAM)
↓ execute from flash
Hardware Layer
SPI-NOR Flash
RAM (8–16 MiB)
MIPS32r2 CPU

Boot Sequence

  1. CPU Reset: Jumps to reset vector at 0xBFC00000 (virtual) / 0x1FC00000 (physical)
  2. Boot Shim (4 KiB): Initializes GT-64120 system controller, fakes YAMON protocol, jumps to kernel_entry
  3. head.S XIP Data Copy: Copies writable data from flash LMA to RAM VMA (__data_loc → _sdata)
  4. BSS Clear: Zeros the .bss section in RAM
  5. setup.c: Memblock accounting reserves only RAM-resident sections ([_sdata, _end))
  6. Kernel Init: Standard Linux initialization with XIP-aware memory layout
  7. PID 1: Freestanding init binary executes, outputs markers, powers off

Firmware Fundamentals

Firmware is the software that bridges hardware and operating systems. In embedded Linux systems, firmware typically consists of a bootloader, kernel, device tree, and root filesystem — all packed into a flash image.

Firmware Image Components

Boot Shim
4 KiB — Hardware init + jump to kernel
Kernel (ROM)
XIP — .text + .rodata in flash
Kernel (RAM)
.data + .bss copied at boot
Initramfs
Root filesystem in RAM

Flash Memory Types

Flash TypeTypical SizeSpeedXIP SupportCommon Devices
SPI-NOR4–16 MiB~50 MB/sYes (this project)Routers, IoT nodes
SPI-NAND128 MiB–2 GiB~200 MB/sLimitedSet-top boxes, NAS
eMMC4–64 GiB~400 MB/sNoPhones, tablets
NOR (parallel)1–32 MiB~100 MB/sYesLegacy embedded

OpenWrt Firmware Image Structure

text
┌─────────────────────────────────────────────────┐
│  U-Boot Header (64 bytes)                       │
├─────────────────────────────────────────────────┤
│  Kernel (compressed or XIP)                     │
│  ├── Entry point                                │
│  ├── Device Tree Blob (DTB)                     │
│  └── Initramfs                                  │
├─────────────────────────────────────────────────┤
│  Root Filesystem (SquashFS + JFFS2 overlay)     │
├─────────────────────────────────────────────────┤
│  Bootloader (U-Boot / Breed)                    │
└─────────────────────────────────────────────────┘

Build System

The build system is a Makefile-driven pipeline that orchestrates kernel download, patching, compilation, and flash image assembly.

Makefile Targets

TargetCommandDescriptionTime
makeall: imageFull build pipeline~5 min
make kernelbuild-kernel.shDownload, patch, compile kernel~3 min
make imagebuild-image.shAssemble flash image~10 sec
make verifyverify-layout.pyStatic ELF assertions~1 sec
make testsmoke-test.pyQEMU boot test~30 sec
make runrun-qemu.shInteractive QEMU sessionmanual
make cleanrm -rf $(OUT)Remove build outputsinstant
make distcleanrm -rf $(WORK)Remove entire work directoryinstant

Build Pipeline Flow

1
Fetch

Download linux-6.12.34.tar.xz from kernel.org

2
Extract

Untar to build/linux-6.12.34/

3
Patch

Apply 5 XIP patches via patch -p1

4
Userspace

Build initramfs (LWRT rootfs or demo init)

5
Configure

Copy defconfig, run olddefconfig

6
Compile

Build vmlinux with clang + mipsel binutils

7
Assemble

Concatenate shim + kernel ROM → xip-bios.bin

build-kernel.sh Details

bash
#!/bin/bash
set -euo pipefail

KVER=6.12.34
WORK=build
KDIR=$WORK/linux-$KVER
OUT=$WORK/out

# Which defconfig to build. Defaults to the bare XIP boot-test config; the
# LWRT image build overrides it with DEFCONFIG=lwrt_qemu_malta_defconfig.
DEFCONFIG="${DEFCONFIG:-xip_qemu_malta_defconfig}"

# 1. Fetch kernel tarball
curl -fSL "https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-${KVER}.tar.xz" \
  -o "$WORK/linux-${KVER}.tar.xz"

# 2. Extract
tar -xf "$WORK/linux-${KVER}.tar.xz" -C "$WORK"

# 3. Apply XIP patches
for p in patches/*.patch; do
  patch -p1 -d "$KDIR" < "$p"
done

# 4. Build the initramfs userspace. With no arguments this builds the tiny
#    freestanding demo PID 1; if LWRT_ROOTFS points at a staged rootfs, the
#    script instead emits a gen_init_cpio list describing it (this is how the
#    LWRT binary is baked in). Either way it writes $OUT/initramfs.list.
bash scripts/build-userspace.sh "$OUT"

# 5. Configure kernel from the selected defconfig, then point the embedded
#    initramfs at the generated list.
cp "configs/$DEFCONFIG" "$KDIR/arch/mips/configs/"
make -C "$KDIR" CC=clang ARCH=mips CROSS_COMPILE=mipsel-linux-gnu- "$DEFCONFIG"
"$KDIR/scripts/config" --file "$KDIR/.config" \
  --set-str INITRAMFS_SOURCE "$OUT/initramfs.list"
make -C "$KDIR" CC=clang ARCH=mips olddefconfig

# 6. Compile
make -C "$KDIR" CC=clang ARCH=mips \
  CROSS_COMPILE=mipsel-linux-gnu- \
  -j$(nproc) vmlinux

Kernel Configuration

The kernel configuration is carefully tuned for minimal footprint while retaining essential embedded functionality.

Minimal XIP Config

kconfig
# Architecture
CONFIG_MIPS=y
CONFIG_32BIT=y
CONFIG_CPU_MIPS32_R2=y
CONFIG_CPU_LITTLE_ENDIAN=y

# XIP (the core feature)
CONFIG_XIP_KERNEL=y
CONFIG_XIP_PHYS_ADDR=0x1fc01000

# Optimization
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_EXPERT=y
CONFIG_SLUB_TINY=y
CONFIG_KALLSYMS=n

# Boot
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""

# Hardware
CONFIG_MIPS_MALTA=y
CONFIG_PCI=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y

# Power management
CONFIG_POWER_RESET_PIIX4_POWEROFF=y

# Disabled subsystems
CONFIG_SMP=n
CONFIG_FPU=n
CONFIG_BLOCK=n
CONFIG_PROC_FS=n
CONFIG_SYSFS=n
CONFIG_NET=n
CONFIG_USB=n
CONFIG_DEBUG_INFO=n

LWRT Router Config

Extends the minimal config with just enough for the LWRT userspace to come up as PID 1: the virtual filesystems, sysctl, IPv4 networking and the QEMU malta NIC. The embedded LWRT initramfs is gzip-compressed.

kconfig
# Networking core (lean: dhcp client / dns / httpd)
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
# CONFIG_IPV6 is not set

# Network driver (QEMU malta NIC)
CONFIG_NETDEVICES=y
CONFIG_ETHERNET=y
CONFIG_NET_VENDOR_AMD=y
CONFIG_PCNET32=y

# VFS the LWRT init needs
CONFIG_PROC_FS=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y

# IPC / syscalls the Rust std runtime expects
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_SYSCTL=y

# Compress the embedded initramfs to fit the malta byteswap window
CONFIG_RD_GZIP=y
CONFIG_INITRAMFS_COMPRESSION_GZIP=y
Why no netfilter/bridge on malta?

The full firewall (nftables / conntrack / NAT) and the bridge/802.1q datapath are deliberately absent from this config. Two QEMU-malta-only ceilings — neither of which exists on real hardware — make them impossible to fit here:

  1. R_MIPS_26 window. XIP runs from the malta -bios window at phys 0x1fc01000 → KSEG0 0x9fc01000, leaving only ~4 MiB before the 0xa0000000 segment boundary. MIPS jal cannot jump across it, so the flash-resident text must stay under ~4 MiB.
  2. Byteswap window. QEMU only byte-swaps the first 0x3e0000 (~3.9 MiB) of a -bios image on mipsel, so the entire flash image (shim + kernel ROM + embedded initramfs) must fit under that.

The complete feature set is deferred to the MT7628 defconfig: that SoC maps its SPI flash XIP window at 0x1c000000 (KSEG0 0x9c000000) with 64 MiB of headroom and has no byteswap quirk, so the full firewall links and fits there. The malta config exists to validate that LWRT boots and runs on an XIP kernel.

Image Assembly

The final flash image is assembled by concatenating the boot shim and the kernel ROM blob.

Image Layout

Boot Shim
4 KiB
0x1FC00000
Kernel ROM (.text + .rodata + .init.text)
~900 KiB
0x1FC01000
Data (LMA in flash)
~200 KiB
after .init

Assembly Script

bash
#!/bin/bash
set -euo pipefail

WORK=build
KDIR=$WORK/linux-6.12.34
OUT=$WORK/out
mkdir -p "$OUT"

# 1. Extract kernel_entry address from System.map
ENTRY=$(awk '$3 == "kernel_entry" {print "0x"$1}' "$KDIR/System.map")

# 2. Build boot shim (4 KiB at reset vector)
mipsel-linux-gnu-gcc -c -o shim.o shim/shim.S \
  -mips32r2 -EL -mno-abicalls -fno-pic -nostdlib
mipsel-linux-gnu-ld -o shim.elf shim.o \
  -T shim/shim.ld --defsym kernel_entry="$ENTRY" \
  -nostdlib --no-dynamic-linker
mipsel-linux-gnu-objcopy -O binary shim.elf shim.bin

# 3. Extract kernel ROM blob
mipsel-linux-gnu-objcopy -O binary -j .text -j .rodata \
  -j .init.text -j .data -j .init.data \
  "$KDIR/vmlinux" kernel.bin

# 4. Concatenate: shim + kernel → flash image
cat shim.bin kernel.bin > xip-flash.bin

# 5. Word-swap for QEMU malta (big-endian BIOS on mipsel)
python3 -c "
import sys
data = open('xip-flash.bin','rb').read()
swapped = b''
for i in range(0, len(data), 4):
    word = data[i:i+4]
    if len(word) == 4:
        swapped += bytes([word[3],word[2],word[1],word[0]])
    else:
        swapped += word
open('$OUT/xip-bios.bin','wb').write(swapped)
"

echo "Flash image: $OUT/xip-bios.bin ($(stat -c%s $OUT/xip-bios.bin) bytes)"

Boot Shim

The boot shim is a 4 KiB MIPS assembly program that initializes hardware and bridges the gap between the reset vector and the Linux kernel entry point.

What the Shim Does

  1. Skip QEMU board ID: Word at offset 0x10 contains a board identifier — skip over it
  2. Disable interrupts: di instruction prevents any interrupt during init
  3. GT-64120 init: Programs the system controller registers exactly as YAMON does:
    • Moves GT registers to 0x1BE00000
    • Sets PCI I/O decode at 0x18000000 (needed for serial at 0x180003F8)
    • Configures PCI memory windows
  4. Fake YAMON protocol: Sets registers a0=argc, a1=argv, a2=envp, a3=memsize
  5. Jump to kernel: Loads kernel_entry address (injected via linker --defsym) and jumps
Size Constraint: The shim must fit in exactly 4 KiB. The linker script enforces this with an assertion: ASSERT(SIZEOF(.text) <= 4096, "Shim too large")

Embedded Systems Fundamentals

Embedded systems are specialized computing devices designed for specific tasks, often with strict constraints on power, memory, cost, and real-time performance.

Embedded Linux Architecture

LayerComponentDescription
ApplicationUser programsDomain-specific logic (routing, sensor reading, etc.)
LibrariesuClibc, muslMinimal C library for embedded systems
MiddlewareBusyBox, procdInit system, shell, core utilities
KernelLinuxProcess management, drivers, networking
BootloaderU-Boot, BreedHardware init, kernel loading
Board SupportDevice Tree, patchesHardware-specific configuration

Resource Constraints

RAM

8 MiB

Typical for budget routers. XIP saves 2.3 MiB — nearly 30% of total RAM.

Flash

16 MiB

SPI-NOR flash stores kernel, rootfs, and configuration.

CPU

MIPS32r2 @ 580MHz

No FPU, no SMP, single-core. Optimization is critical.

Power

~5W

Wall-powered but thermal constraints in compact enclosures.

Memory Management

Memory management in XIP kernels requires careful attention to the distinction between flash-resident and RAM-resident sections.

MIPS Memory Map

text
MIPS Virtual Address Space (32-bit)
┌────────────────────────────────────────────────┐ 0xFFFFFFFF
│  KSEG3 (0xBFC00000) — Cached, mapped           │
│  └── Reset vector: 0xBFC00000                  │
├────────────────────────────────────────────────┤ 0xA0000000
│  KSEG1 (0xA0000000) — Uncached, unmapped       │
│  └── I/O registers, flash if > 512 MiB         │
├────────────────────────────────────────────────┤ 0x80000000
│  KSEG0 (0x80000000) — Cached, unmapped         │
│  └── RAM: 0x80000000 – 0x8FFFFFFF              │
│  └── XIP virt: 0x80000000 + phys_addr         │
├────────────────────────────────────────────────┤ 0x00000000
│  USEG (0x00000000) — User space (mapped)       │
└────────────────────────────────────────────────┘

XIP Memory Sections

SectionLocationAddressWritableNotes
.textFlash (ROM)0x9FC01000NoKernel code — executes in place
.rodataFlash (ROM)after .textNoRead-only data (strings, constants)
.init.textFlash (ROM)after .rodataNoInit functions (freed after boot)
.dataRAM0x8xxxxxxxYesGlobal/static variables
.bssRAMafter .dataYesZero-initialized data
.init.dataRAMafter .dataYesInit-only data (freed after boot)
.data..xip_patchable_textRAMafter .dataYesTLB handlers (uasm-generated)

Data Copy at Boot

head.S Copy Loop: Before BSS clearing, the kernel copies writable data from flash to RAM:
__data_loc (flash LMA) → _sdata (RAM VMA) through __init_end
This mirrors what ARM and RISC-V XIP heads do.

Cross-Compilation

Cross-compilation builds software on one architecture (x86_64) targeting another (MIPS). This project uses clang as the compiler with mipsel-linux-gnu- binutils for linking.

Toolchain Components

ToolPurposePackage
clangC compiler (cross-compilation via --target)clang
mipsel-linux-gnu-ldLinker for MIPS little-endianbinutils-mipsel-linux-gnu
mipsel-linux-gnu-objcopyBinary format conversionbinutils-mipsel-linux-gnu
mipsel-linux-gnu-gccAssembly (boot shim)gcc-mipsel-linux-gnu
qemu-system-mipsEmulation for testingqemu-system-mips

Why Clang?

Toolchain Setup

Ubuntu/Debian

bash
# Install cross-compilation toolchain
sudo apt install -y \
  clang lld llvm \
  binutils-mipsel-linux-gnu \
  gcc-mipsel-linux-gnu \
  qemu-system-mips

# Verify
clang --version
mipsel-linux-gnu-ld --version
qemu-system-mips --version

Build Dependencies

bash
# Kernel build dependencies
sudo apt install -y \
  flex bison bc libelf-dev libssl-dev \
  libncurses-dev cpio wget xz-utils \
  python3 python3-pip

Freestanding Programs

Freestanding programs run without any C library or OS support. This project includes a minimal PID 1 init binary that uses raw MIPS syscalls.

userspace/init.c

c
// Freestanding PID 1 — no libc, raw MIPS o32 syscalls
#include <stddef.h>

#define SYS_exit    4001
#define SYS_write   4004
#define SYS_pause   4029
#define SYS_sync    4036
#define SYS_reboot  4088

static void syscall1(int n, int a0) {
    register int v0 __asm__("v0") = n;
    register int a0r __asm__("a0") = a0;
    __asm__ volatile("syscall" : "+r"(v0) : "r"(a0r) : "memory");
}

static void write_str(const char *s, int len) {
    register int v0 __asm__("v0") = SYS_write;
    register int fd __asm__("a0") = 1;  // stdout
    register const char *buf __asm__("a1") = s;
    register int sz __asm__("a2") = len;
    __asm__ volatile("syscall" : "+r"(v0) : "r"(fd), "r"(buf), "r"(sz) : "memory");
}

void _start(void) {
    write_str("XIP-USERSPACE-OK\n", 17);
    write_str("XIP-POWEROFF: requesting power off\n", 35);
    syscall1(SYS_sync, 0);
    // reboot(LINUX_REBOOT_CMD_POWER_OFF = 0x4321fedc)
    register int v0 __asm__("v0") = SYS_reboot;
    register int a0 __asm__("a0") = 0xfee1dead;
    register int a1 __asm__("a1") = 672274793;
    register int a2 __asm__("a2") = 0x4321fedc;
    __asm__ volatile("syscall" : "+r"(v0) : "r"(a0), "r"(a1), "r"(a2) : "memory");
    while(1); // should never reach here
}
Why Freestanding? On a device with 8 MiB RAM, even musl libc adds overhead. Raw syscalls keep the init binary under 1 KiB.

Execute-In-Place (XIP)

XIP allows code to execute directly from non-volatile storage (flash) without first copying it to RAM. This is fundamentally different from traditional boot where the kernel is decompressed into RAM before execution.

Traditional vs XIP Boot

Traditional Boot

1. Bootloader loads compressed kernel to RAM
2. Kernel decompresses itself (~900 KiB → ~2.3 MiB)
3. Decompressed kernel copied to final RAM location
4. Kernel starts executing from RAM
5. RAM used: ~2.3 MiB for kernel alone

XIP Boot

1. Boot shim initializes hardware
2. Kernel executes directly from flash
3. Only writable data copied to RAM (~200 KiB)
4. TLB handlers allocated in RAM
5. RAM used: ~200 KiB for data only

RAM Savings Breakdown

ComponentTraditionalXIPSavings
Kernel text (decompressed)~2304 KiB0 KiB (in flash)2304 KiB
Kernel data~200 KiB~200 KiB0 KiB
TLB handlers0 (in .text)~4 KiB-4 KiB
Trampolines0~0.5 KiB-0.5 KiB
Total~2508 KiB~204 KiB~2304 KiB

Kernel Patches

Five patches against Linux 6.12.34 enable XIP on MIPS. Each addresses a specific challenge.

Patch 1 arch/mips/Kconfig

Kconfig Additions

Adds CONFIG_XIP_KERNEL (bool) and CONFIG_XIP_PHYS_ADDR (hex) to MIPS architecture options. Dependencies: 32BIT && !RELOCATABLE && !MAPPED_KERNEL && !SMP.

Patch 2 arch/mips/kernel/vmlinux-xip.lds.S

XIP Linker Script (261 lines)

The largest patch. Defines ROM region at XIP_VIRT_ADDR and RAM region at LINKER_LOAD_ADDRESS. Introduces XIP_AT() macro to override asm-generic address macros. Moves .data..ro_after_init to RAM.

Patch 3 arch/mips/kernel/head.S

XIP Data Copy

Adds a copy loop in head.S that copies writable data from flash LMA to RAM VMA before BSS clearing. Mirrors ARM and RISC-V XIP implementations.

Patch 4 arch/mips/kernel/setup.c

Memblock Accounting

Three changes: reserves only [_sdata, _end) under XIP, validates only RAM-resident sections, sets data_resource.start = _sdata.

Patch 5 arch/mips/mm/tlb-funcs.S, page-funcs.S, tlbex.c

TLB Handlers + Uasm RAM Buffers

The hardest part. MIPS TLB handlers are uasm-generated at boot into .text buffers — impossible under XIP. Solution: move patchable buffers to RAM section, add ROM trampolines with cross-segment register jumps.

XIP Linker Script

The linker script is the most complex patch (261 lines). It splits the kernel into ROM and RAM regions.

Key Directives

ld
/* XIP virtual address = KSEG0 + physical offset */
XIP_VIRT_ADDR = 0x80000000 + CONFIG_XIP_PHYS_ADDR;

/* ROM region — executes directly from flash */
SECTIONS {
    .text XIP_VIRT_ADDR : AT(CONFIG_XIP_PHYS_ADDR) {
        _text = .;
        /* ... code sections ... */
        _etext = .;
    }

    /* RAM region — writable data */
    .data LINKER_LOAD_ADDRESS : {
        _sdata = .;
        /* ... data sections ... */
        _edata = .;
    }

    .bss : {
        _sbss = .;
        /* ... BSS sections ... */
        _ebss = .;
    }

    /* Special section for uasm-generated TLB handlers */
    .data..xip_patchable_text : {
        /* RAM buffers that kernel writes at boot */
    }
}

/* Custom macro to override asm-generic AT() */
#define XIP_AT(addr) (addr) - LOAD_OFFSET + XIP_PHYS_OFFSET

XIP_AT() Macro Explained

The Problem: asm-generic macros hardcode AT(ADDR(x) - LOAD_OFFSET), which would reset RAM-section load addresses to incorrect flash offsets. The XIP_AT() macro overrides this to produce correct LMA values.

TLB Handlers

The TLB (Translation Lookaside Buffer) handler patch is the most technically challenging part of the XIP implementation.

The Problem

MIPS TLB exception handlers (handle_tlbl, handle_tlbs, handle_tlbm) are generated at boot using uasm — a MIPS assembly DSL. The kernel writes machine code into buffers in .text. Under XIP, .text is ROM — writes are silently dropped.

The Solution

asm
/* ROM trampoline in .text (original symbol location) */
handle_tlbl:
    PTR_LA  t9, xip_handle_tlbmiss_handler_setup_pgd
    jr      t9
    nop

/* RAM buffer in .data..xip_patchable_text */
xip_handle_tlbmiss_handler_setup_pgd:
    /* uasm-generated code lives here — writable RAM */

Cross-Segment Jump Problem

The jal/R_MIPS_26 instruction cannot cross the 256 MiB jump segment between flash (0x9fcxxxxx) and RAM (0x80xxxxxx). Solution: trampolines use lui/addiu/jr sequences for register-indirect jumps.

Memory Layout

Physical Memory Map

text
Physical Address Space
┌────────────────────────────────────────────┐ 0x20000000 (512 MiB)
│  End of KSEG0/KSEG1 direct-mapped region   │
├────────────────────────────────────────────┤
│  ...                                       │
├────────────────────────────────────────────┤ 0x1FC00000
│  SPI-NOR Flash (16 MiB)                    │
│  ├── 0x1FC00000: Boot Shim (4 KiB)         │
│  ├── 0x1FC01000: Kernel .text (XIP)        │
│  ├── 0x1FDxxxxx: Kernel .rodata (XIP)      │
│  ├── 0x1FExxxxx: Kernel .init.text (XIP)   │
│  └── 0x1FFxxxxx: Data LMA (copied to RAM)  │
├────────────────────────────────────────────┤ 0x1E000000
│  RAM (8–16 MiB)                            │
│  ├── 0x80000000: Kernel .data              │
│  ├── 0x800xxxxx: Kernel .bss               │
│  ├── 0x801xxxxx: TLB handlers (RAM)        │
│  ├── 0x802xxxxx: Free memory               │
│  └── 0x81FFFFFF: End of 32 MiB window      │
└────────────────────────────────────────────┘

Virtual Address Mapping

SegmentVirtual RangePhysicalCachedUse
KSEG00x80000000–0x9FFFFFFF0x00000000–0x1FFFFFFFYesRAM, XIP kernel
KSEG10xA0000000–0xBFFFFFFF0x00000000–0x1FFFFFFFNoI/O registers
KSEG20xC0000000–0xFFFFFFFFmappedYesVMalloc, kernel modules

IoT Overview

The Internet of Things (IoT) connects embedded devices to networks for data collection, monitoring, and control. This project's XIP technology is directly applicable to IoT devices running on flash-constrained hardware.

IoT Device Categories

📡

Sensor Nodes

Temperature, humidity, motion sensors. Ultra-low power, periodic wake-and-report. XIP saves RAM for sensor buffers.

🏠

Smart Home

Lighting, locks, thermostats. Local processing + cloud connectivity. XIP enables larger applications on same hardware.

🏭

Industrial IoT

PLC controllers, predictive maintenance. Deterministic timing requirements. XIP eliminates decompression latency.

🌐

Edge Routers

Gateway devices bridging IoT protocols to internet. This project targets exactly this category.

IoT + XIP Benefits

IoT Protocols

Common protocols used in IoT deployments, many of which can run on this XIP-enabled embedded Linux platform.

ProtocolLayerTransportUse CaseRAM Footprint
MQTTApplicationTCPPub/sub messaging~10 KiB
CoAPApplicationUDPREST for constrained devices~5 KiB
HTTP/1.1ApplicationTCPWeb APIs, OTA updates~20 KiB
WebSocketApplicationTCPReal-time bidirectional~15 KiB
LwM2MApplicationCoAPDevice management~25 KiB
DHCPNetworkUDPIP address assignment~8 KiB
mDNSNetworkUDPLocal service discovery~12 KiB
802.15.4Data LinkLow-power wireless (Zigbee/Thread)~15 KiB
LoRaWANNetworkLong-range, low-power~30 KiB

Router Firmware

Building custom router firmware is one of the primary use cases for this XIP kernel project. The LWRT (Lightweight Router Toolkit) configuration demonstrates a full networking stack.

Router Firmware Components

Kernel
XIP + networking drivers + nftables
Network Stack
IPv4/IPv6, bridge, VLAN, NAT
Userspace
procd, netifd, dnsmasq, hostapd
Web UI
LuCI or custom admin interface

OpenWrt Build Process

bash
# Standard OpenWrt build (for reference)
git clone https://git.openwrt.org/openwrt/openwrt.git
cd openwrt
./scripts/feeds update -a
./scripts/feeds install -a
make menuconfig  # Select Target → MediaTek Ralink → MT76x8
make -j$(nproc)  # Builds full firmware image

# Output: bin/targets/ramips/mt76x8/openwrt-ramips-mt76x8-*.bin

Custom Firmware with XIP

To build router firmware with XIP kernel:

  1. Build the XIP kernel using this project's Makefile
  2. Stage an LWRT root filesystem with build-userspace.sh
  3. Configure LWRT_ROOTFS=/path/to/staged/rootfs
  4. The build system generates a combined initramfs image
  5. Flash the resulting xip-bios.bin to the router's SPI-NOR

Network Stack

The LWRT defconfig enables a full IPv4 networking stack suitable for router firmware.

Network Subsystems Enabled

SubsystemConfigPurpose
IPv4CONFIG_INET=yCore Internet Protocol
BridgingCONFIG_BRIDGE=yLAN port bridging (switch functionality)
VLANCONFIG_VLAN_8021Q=yVirtual LAN tagging
MulticastCONFIG_IP_MULTICAST=yMulticast routing support
Advanced routingCONFIG_IP_ADVANCED_ROUTER=yPolicy routing, multiple tables
PCnet NICCONFIG_PCNET32=yQEMU malta network driver
ConntrackCONFIG_NF_CONNTRACK=yConnection tracking for NAT
nftablesCONFIG_NF_TABLES=yModern packet filtering

Firewall & NAT

The nftables-based firewall provides stateful packet inspection and NAT for router deployments.

Netfilter Components

kconfig
# Connection tracking
CONFIG_NF_CONNTRACK=y

# NAT
CONFIG_NF_NAT=y
CONFIG_NF_NAT_MASQUERADE=y

# nftables framework
CONFIG_NF_TABLES=y
CONFIG_NFT_CT=y          # Connection tracking in nftables
CONFIG_NFT_NAT=y         # NAT in nftables
CONFIG_NFT_MASQ=y        # Masquerading in nftables

Typical Router Firewall Rules

nftables
#!/usr/sbin/nft -f
flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        iif lo accept
        ct state established,related accept
        tcp dport 22 accept           # SSH
        icmp type echo-request accept # Ping
    }

    chain forward {
        type filter hook forward priority 0; policy drop;
        ct state established,related accept
        iif "lan" oif "wan" accept    # LAN → WAN
    }

    chain output {
        type filter hook output priority 0; policy accept;
    }
}

table inet nat {
    chain prerouting {
        type nat hook prerouting priority -100;
    }

    chain postrouting {
        type nat hook postrouting priority 100;
        oif "wan" masquerade          # NAT for LAN clients
    }
}

QEMU Testing

QEMU provides a safe, reproducible testing environment that closely mirrors real hardware without risk of bricking devices.

Why QEMU Malta?

QEMU Launch Command

bash
qemu-system-mipsel \
  -M malta \
  -m 256M \
  -nographic \
  -bios build/out/xip-bios.bin \
  -serial mon:stdio \
  -no-reboot \
  -device pci-testdev 2>&1 | tee build/boot.log

Expected Serial Output

text
Linux version 6.12.34 (builder@x86) (clang 18.0) #1 SMP ...
...
Memory: 245760K/262144K available (1024K kernel code, ...)
...
Freeing unused kernel image(s): 32K freed
XIP-USERSPACE-OK
XIP-POWEROFF: requesting power off

Real Hardware

Deploying to real hardware requires additional considerations beyond QEMU testing.

Deployment Checklist

Serial Console Setup

bash
# Connect USB-to-serial adapter to router UART pins
# TX → RX, RX → TX, GND → GND

# Linux
sudo screen /dev/ttyUSB0 115200

# Or with minicom
sudo minicom -D /dev/ttyUSB0 -b 115200

# Or with picocom
sudo picocom -b 115200 /dev/ttyUSB0

Test Infrastructure

Two test scripts verify the XIP implementation: static layout assertions and dynamic boot testing.

Static Verification (verify-layout.py)

Uses readelf and System.map to verify ELF layout without booting:

  1. A LOAD segment with VMA == LMA == 0x9FC01000, flags R E (true XIP)
  2. A RAM segment whose LMA lies inside the ROM image (data shipped in flash)
  3. All uasm buffers at RAM addresses
  4. All trampolines in ROM
  5. ROM image fits within flash budget

Boot Smoke Test (smoke-test.py)

Boots the image in QEMU and asserts serial markers appear in order:

python
#!/usr/bin/env python3
"""QEMU boot smoke test for XIP kernel."""
import subprocess, sys, time

MARKERS = [
    "Linux version",        # kernel alive
    "Memory:",              # memblock accounting sane
    "Freeing unused kernel",# init memory reclaim worked
    "XIP-USERSPACE-OK",     # PID 1 ELF executed
    "XIP-POWEROFF:",        # userspace reached poweroff
]

def test():
    proc = subprocess.Popen(
        ["qemu-system-mipsel", "-M", "malta", "-m", "256M",
         "-nographic", "-bios", "build/out/xip-bios.bin",
         "-no-reboot", "-serial", "mon:stdio"],
        stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
        text=True
    )
    output = []
    found = 0
    deadline = time.time() + 30

    while time.time() < deadline:
        line = proc.stdout.readline()
        if not line:
            break
        output.append(line)
        if found < len(MARKERS) and MARKERS[found] in line:
            found += 1
        if "XIP-POWEROFF" in line:
            break

    proc.wait(timeout=10)
    output_text = "".join(output)

    if found == len(MARKERS) and proc.returncode == 0:
        print("PASS: all markers found, clean poweroff")
        sys.exit(0)
    else:
        print(f"FAIL: found {found}/{len(MARKERS)} markers")
        sys.exit(1)

CI/CD Pipeline

GitHub Actions runs the full build + test pipeline on every push and pull request.

Pipeline Stages

1
Setup

Install clang, lld, llvm, binutils-mipsel-linux-gnu, qemu-system-mips, flex, bison on ubuntu-24.04

2
Cache

Cache kernel tarball to avoid re-downloading

3
Build

make — full pipeline

4
Verify

make verify — static assertions

5
Test

make test — QEMU boot smoke test

6
Artifacts

Upload xip-bios.bin and boot.log

ci.yml Configuration

yaml
name: Build & Test
on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: |
          sudo apt update
          sudo apt install -y clang lld llvm \
            binutils-mipsel-linux-gnu \
            qemu-system-mips flex bison bc

      - name: Cache kernel tarball
        uses: actions/cache@v4
        with:
          path: build/linux-*.tar.xz
          key: kernel-${{ hashFiles('patches/*') }}

      - name: Build
        run: make

      - name: Verify layout
        run: make verify

      - name: Boot test
        run: make test

      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        with:
          name: xip-kernel
          path: |
            build/out/xip-bios.bin
            build/boot.log

Verification

Verification happens at multiple levels to ensure correctness.

Verification Levels

LevelToolWhat It ChecksWhen
Compile-timegcc/clang warningsSyntax, type errors, warningsDuring build
Link-timeld assertionsShim size ≤ 4 KiB, symbol resolutionDuring link
Static layoutverify-layout.pyELF segments, XIP addresses, ROM/RAM splitAfter build
Boot testsmoke-test.pyKernel boots, init runs, poweroff succeedsAfter build
CIGitHub ActionsFull pipeline on every pushOn commit

File Reference

Complete reference of all project files.

FileLinesPurpose
Makefile41Top-level build orchestration
README.md154Project documentation
configs/xip_qemu_malta_defconfig82Minimal XIP kernel config
configs/lwrt_qemu_malta_defconfig123LWRT router config with networking
patches/0001-mips-add-xip-kconfig.patch53Kconfig additions
patches/0002-mips-xip-linker-script.patch293XIP linker script (largest patch)
patches/0003-mips-head-xip-data-copy.patch36head.S data copy loop
patches/0004-mips-setup-xip-memblock.patch52Memblock accounting fixes
patches/0005-mips-mm-xip-patchable-text.patch265TLB handlers + RAM buffers
scripts/build-kernel.sh53Kernel download/patch/compile
scripts/build-image.sh59Flash image assembly
scripts/build-userspace.sh62Initramfs generation
scripts/run-qemu.sh8Interactive QEMU launch
shim/shim.S86Boot shim assembly
shim/shim.ld224 KiB linker script
userspace/init.c65Freestanding PID 1
tests/verify-layout.py106Static ELF layout verification
tests/smoke-test.py109QEMU boot smoke test
.github/workflows/ci.yml46GitHub Actions CI pipeline

Glossary

XIP (Execute-In-Place)
Executing code directly from non-volatile storage without copying to RAM first.
SPI-NOR Flash
Serial Peripheral Interface NOR flash memory. Common in embedded systems for firmware storage.
UASM
MIPS assembly DSL (Domain-Specific Language) used by Linux kernel to generate TLB handlers at boot.
TLB (Translation Lookaside Buffer)
Hardware cache for virtual-to-physical address translations in MIPS MMU.
KSEG0 / KSEG1
MIPS direct-mapped address segments. KSEG0 is cached, KSEG1 is uncached. Both map to physical 0x0–0x1FFFFFFF.
GT-64120
Galileo GT-64120 system controller used in MIPS Malta development boards (and QEMU emulation).
YAMON
Yet Another Monitor — MIPS bootloader/debugger. The boot shim fakes its protocol.
Memblock
Linux kernel's early memory allocator. Tracks available and reserved memory regions.
Initramfs
Initial RAM filesystem — a cpio archive loaded into memory at boot, used as the root filesystem.
nftables
Modern Linux packet filtering framework, replacing iptables.
LWRT
Lightweight Router Toolkit — a minimal router firmware userspace for embedded Linux.
OpenWrt
Linux distribution for embedded networking devices. Provides package management and web interface.
Device Tree (DTB)
Hardware description data structure passed to the kernel at boot, describing board configuration.
U-Boot
Universal Bootloader — the most common bootloader for embedded Linux systems.
DTB (Device Tree Blob)
Binary format of the device tree, passed from bootloader to kernel.

Frequently Asked Questions

On devices with 8–16 MiB RAM, a compressed kernel still needs ~2.3 MiB for the decompressed copy. XIP eliminates this entirely, freeing that RAM for applications, network buffers, and sensor data. The tradeoff is slightly slower execution from flash vs. RAM.

XIP works best with SPI-NOR flash because it supports random access and has consistent read latency. SPI-NAND flash requires ECC management and bad-block handling that makes XIP impractical. Parallel NOR flash also supports XIP but is less common in modern designs.

Yes. The LWRT defconfig demonstrates enabling networking, nftables, and drivers while keeping XIP. Use make menuconfig in the kernel source tree to toggle features. Each addition increases kernel size and RAM usage.

1) Create a new defconfig for your board. 2) Modify the boot shim to initialize your board's system controller. 3) Update CONFIG_XIP_PHYS_ADDR to match your flash's physical address. 4) Adjust the linker script if your board has different memory geometry. 5) Test with QEMU first if possible.

Flash reads are slower than RAM (~50 MB/s vs ~800 MB/s for DDR1). However, most kernel code is execution-bound, not bandwidth-bound. Real-world benchmarks show 5–15% slowdown for typical router workloads (NAT, firewall rules, packet forwarding). The RAM savings often outweigh this cost.

Currently no. The XIP implementation requires !SMP because TLB handler generation and cross-segment trampolines assume single-core execution. Adding SMP support would require significant additional work for cache coherency and TLB shootdown across cores.