Extending Raspberry Pi OS Images for Cloud-Init

A Deep Dive into Disk Image Manipulation

When deploying Raspberry Pi clusters for Kubernetes or other distributed systems, manual configuration quickly becomes tedious. Cloud-init offers an elegant solution for automated provisioning, but Raspberry Pi OS doesn’t include it by default.

This article explores a script that modifies Raspberry Pi OS images to install cloud-init, providing detailed explanations of the low-level Linux tools that make this magic possible.

Prerequisites and Setup

In this setup, Ubuntu 22.04 x86 is running on the host.

Tools

Before diving into the script, you’ll need several tools installed on your host system:

The tools provide the following functionality:

• qemu-utils: Provides disk image manipulation tools, particularly qemu-img for resizing images
• qemu-user-static: Enables execution of ARM binaries on x86 systems through binary translation, allowing us to chroot into ARM filesystems
• binfmt-support: Registers binary formats with the kernel, enabling automatic detection and execution of ARM binaries via QEMU
• jq: a command-line tool for parsing, filtering, and transforming JSON data.
• xz-utils: Handles XZ-compressed files (the format Raspberry Pi OS images use)
• cloud-utils: Provides cloud image utilities
• util-linux: Contains essential system utilities like losetup, mount, and sfdisk

Finally, restarting systemd-binfmt.service ensures the kernel recognizes ARM binary formats immediately.

# install packages
sudo apt-get install -qy \
    binfmt-support \
    cloud-utils \
    jq \
    qemu-user-static \
    qemu-utils \
    util-linux
    xz-utils \

# start systemd-binfmt service
sudo systemctl restart systemd-binfmt.service

Download Raspberry Pi OS

Download the Raspberry Pi OS disk image using curl.

curl -sfSLO https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-04-07/2022-04-04-raspios-bullseye-arm64-lite.img.xz

The 2022-04-04-raspios-bullseye-arm64-lite.img.xz file is compressed using xz.

Checkout the index for all available images.

Phase 1: Image Decompression and Preparation

image=2022-04-04-raspios-bullseye-arm64-lite.img.xz
xz --decompress --keep $image.img.xz
mv $image.img $image-cloudinit.img

The script decompresses the XZ-compressed disk image, and renames it to indicate it will contain cloud-init. The --keep flag preserves the original compressed file.

Phase 2: Expanding the Disk Image

Raspberry Pi OS images are minimal by default. Installing cloud-init and later software like K3s requires additional space. This is where disk image manipulation becomes crucial.

Using qemu-img for Non-Destructive Expansion

qemu-img resize -f raw $image-cloudinit.img +1G

qemu-img is QEMU’s Swiss Army knife for disk images. The resize command extends the image file by 1GB without touching existing data.

At this point, the image file is larger, but the partition table and filesystem don’t know about the extra space yet.

Phase 3: Partition Manipulation with Loopback Devices

Inspect the disk image

Use fdisk to inspect the contents of the disk image.

fdisk -l $image-cloudinit.img

The output shows that there are two partitions. We want to extend the second partition, since it is the root filesystem where Raspberry Pi OS is installed.

Disk 2022-04-04-raspios-bullseye-arm64-lite.img: 1.86 GiB, 2000683008 bytes, 3907584 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0ee3e8a8

Device                                      Boot  Start     End Sectors  Size Id Type
2022-04-04-raspios-bullseye-arm64-lite.img1        8192  532479  524288  256M  c W95 FAT32 (LBA)
2022-04-04-raspios-bullseye-arm64-lite.img2      532480 3907583 3375104  1.6G 83 Linux

Create a Loopback Device

device=$(sudo losetup --show --find --partscan $image-cloudinit.img)

losetup creates loopback devices, which make files appear as block devices to the system. This is essential because partition tools expect to work with devices like /dev/sda, not regular files.

The flags do the following:

• --show: Prints the assigned device name (e.g., /dev/loop0)
• --find: Automatically finds an unused loopback device
• --partscan: Scans for partitions and creates sub-devices (e.g., /dev/loop0p1, /dev/loop0p2)

Now our image file appears as a real disk to Linux, complete with accessible partitions.

Resizing the Partition with parted

parted is a partition editor that can modify partition tables on both real disks and loopback devices. The command resizepart 2 100% extends partition 2 (the root filesystem) to use all available space on the disk.

This updates the partition table but doesn’t yet resize the filesystem itself.

sudo parted $device resizepart 2 100%

Phase 4: Filesystem Checking and Resizing

Ensuring Filesystem Integrity with e2fsck

e2fsck is a checker and repair tool for ext2/ext3/ext4 filesystems. The -f flag forces a check even if the filesystem appears clean. This is critical before resizing because any corruption could be magnified during the resize operation, potentially causing data loss.

Appending p2 to the device name gives the second partition.

device_part="${device}p2"
sudo e2fsck -f $device_part

Expanding the Filesystem with resize2fs

resize2fs resizes the filesystem. Without size arguments, it expands the filesystem to fill the entire partition.

This is the final step that makes the added space actually usable within the filesystem.

sudo resize2fs $device_part

Cleanup

Detach the loopback device to releasing system resources. Always detach loopback devices when finished to prevent resource leaks.

sudo losetup -d $device

Phase 5: Mounting and Chroot Environment Setup

Calculating Partition Offsets

The fdisk -l command output above, shows the sizes and offsets of each partition within the disk image. This steps calculates the offset of the second partition, which starts at 532480, so that it can be mounted.

Conveniently, sfdisk dumps partition information in JSON format, so jq can be used to extract the start sector of the second partition (index 1). Since sectors are 512 bytes, we multiply to get the byte offset needed for mounting.

start=$(sfdisk --json $image-cloudinit.img | jq '.partitiontable.partitions[1].start')
offset=$(($start * 512))

Mounting the Root Filesystem

Create a temporary directory for mounting the root filesystem partition.

tmpdir=$(mktemp -d)
sudo mount -o loop,rw,sync,offset=$offset $image-cloudinit.img "${tmpdir}"

The mount command with the loop option mounts a file as if it were a block device.

Key options:

• rw: Mount read-write
• sync: Write operations are immediately flushed to disk (safer but slower)
• offset=$offset: Skip to the specified byte position, starting at the root partition

Bind Mounting System Directories

Bind mounts are required for a successful chroot into the ARM filesystem.

• /dev: Device files for hardware access
• /sys: Kernel and device information
• /proc: Process and system information

Without these, many programs inside the chroot would fail or behave unpredictably.

sudo mount --bind /dev "${tmpdir}/dev"
sudo mount --bind /sys "${tmpdir}/sys"
sudo mount --bind /proc "${tmpdir}/proc"

Phase 6: Chroot and Cloud-Init Installation

Entering the Chroot Environment

chroot changes the apparent root directory for a process and its children. This creates an isolated environment where the ARM filesystem appears as /, allowing us to run commands as if we booted into the Raspberry Pi OS.

sudo chroot "${tmpdir}" /bin/bash --norc --noprofile

The --norc --noprofile flags prevent bash from loading user configuration, ensuring a clean, predictable environment.

Installing Cloud-Init

Within the chroot, the following commands will install and configure cloud-init. The execution of each command can be slow, since the ARM64 architecture is being emulated with Qemu.

export DEBIAN_FRONTEND=noninteractive
export DEBCONF_NONINTERACTIVE_SEEN=true

apt-get update -q
apt-get install -qy --no-install-recommends cloud-init

The environment variables prevent interactive prompts during package installation. The --no-install-recommends flag keeps the installation minimal by skipping suggested packages.

Enabling Cloud-Init Services

systemctl enable creates symlinks so these services start automatically at boot. Cloud-init’s initialization happens in stages:

1. cloud-init-local: Runs early, before networking, to configure network interfaces
2. cloud-init-network: Waits for network availability
3. cloud-init-main: Retrieves user data and runs configuration
4. cloud-config: Executes cloud config modules

systemctl enable cloud-init-local.service
systemctl enable cloud-init-main.service
systemctl enable cloud-init-network.service
systemctl enable cloud-config.service

Disabling Conflicting Services

The userconfig.service performs first-boot user setup, which conflicts with cloud-init’s user provisioning.

Furthermore, the apt timers are disabled to prevent automatic updates from interfering with cloud-init’s initial configuration.

systemctl disable userconfig.service
systemctl disable apt-daily.timer
systemctl disable apt-daily-upgrade.timer

Cleanup

These commands remove unnecessary packages and clean package caches, reducing the final image size.

apt-get autoremove -qy
apt-get autoclean
apt-get clean

Exit the chroot

The installation is complete, so you can exit the chroot shell with the exit command or pressing Ctrl+D.

Phase 7: Unmounting and Cleanup

Unmounting must happen in reverse order—first the bind mounts, then the root filesystem. Forgetting to unmount can lead to filesystem corruption or prevent the loopback device from detaching.

sudo umount "${tmpdir}/dev"
sudo umount "${tmpdir}/sys"
sudo umount "${tmpdir}/proc"
sudo umount "${tmpdir}"

Next Steps

The resulting cloud-init enabled image can be flashed to SD cards, together with the cloud-init configuration, typically provided with a user-data file within the boot volume, to automatically provision on first boot.

See the cloud-init documentation for more information on the configuration.