Lightweight Development Sandboxes with systemd-nspawn on Linux

Why systemd-nspawn for Development Containers?

If you've ever wanted to let an AI code agent or experimental script run wild without worrying about it wreaking havoc on your system, you need isolation. While Docker is the go-to solution for many, systemd-nspawn offers a compelling alternative that's built right into systemd-based Linux distributions.

What Makes systemd-nspawn Different?

systemd-nspawn is a lightweight container manager that comes pre-installed with systemd. Think of it as "chroot on steroids" - it provides OS-level virtualization without the overhead of full virtual machines or even Docker's daemon architecture. Here's why it's particularly suited for development sandboxes:

Advantages:

Perfect Use Cases:

This guide demonstrates the setup on Arch Linux, but the concepts apply to any systemd-based distribution (Ubuntu, Debian, Fedora, etc.) with minor command adjustments.

Prerequisites

Before starting, ensure you have:

Install prerequisites on Arch Linux:

sudo pacman -S arch-install-scripts

Step-by-Step Setup Guide

Step 1: Create Container Directory Structure

We'll create the container in your home directory for easy access, then symlink it to the standard systemd-nspawn location:

mkdir -p ~/containers/mycontainer
sudo ln -s ~/containers/mycontainer /var/lib/machines/mycontainer

Why this approach?

Directory Structure Explained:

Step 2: Install Base System with pacstrap

Bootstrap a minimal Arch Linux installation into the container:

sudo pacstrap -K -c ~/containers/mycontainer base openssh

Command breakdown:

What's being installed? This creates a minimal bootable Linux system (~150-200 MB) with:

Time estimate: 2-5 minutes depending on your internet speed.

Step 3: Set Root Password

Enter the container and set a root password for SSH access:

sudo systemd-nspawn -D ~/containers/mycontainer
passwd
logout

What's happening here:

Security note: You're setting a password inside the container, not on your host. This isolation is key to the security model.

Step 4: Enable SSH and Networking in Container

Boot the container with a full init system to properly enable services:

sudo systemd-nspawn -bD ~/containers/mycontainer

Command breakdown:

After it boots, log in as root with your password, then run:

systemctl enable sshd
systemctl enable systemd-networkd
sed -i 's/^#*PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
poweroff

What each command does:

  1. systemctl enable sshd - Start SSH server automatically on boot

  2. systemctl enable systemd-networkd - Enable networking on boot (for DHCP)

  3. The sed command modifies SSH config to allow root login (needed for initial access)

  4. poweroff cleanly shuts down the container

Networking Architecture: The container will get a virtual network interface (host0) that connects to a virtual Ethernet pair on the host. This provides network isolation while still allowing internet access via NAT.

Step 5: Configure Static Network on Host

The host needs systemd-networkd to provide network connectivity for containers. First enable systemd-networkd:

sudo systemctl enable --now systemd-networkd

Then create a static network configuration for the container's virtual interface:

sudo vim /etc/systemd/network/50-ve-mycontainer.network

Add the following configuration:

[Match]
Name=ve-mycontainer
Driver=veth

[Network]
Address=192.168.200.1/24
LinkLocalAddressing=yes
DHCPServer=no
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge

Restart the networking service to apply the configuration:

sudo systemctl restart systemd-networkd

What this configuration does:

Step 6: Start Container with machinectl

Now use systemd's container management tool to start your container:

sudo machinectl start mycontainer

About machinectl: machinectl is systemd's high-level tool for managing containers and VMs. It handles: - Starting/stopping containers - Network setup (creates virtual Ethernet pairs) - Integration with systemd's journal (logging) - Resource limits and cgroups

What's happening behind the scenes: 1. systemd creates a virtual Ethernet pair (veth) 2. One end goes into the container (host0), one stays on host (ve-mycontainer) 3. The container boots with systemd as init 4. The host interface gets configured with static IP 192.168.200.1/24 5. SSH server starts automatically 6. We'll configure the container with static IP 192.168.200.200/24 in the next step

Step 7: Configure Container Static Network

Shell into the container and configure the static network:

sudo machinectl shell mycontainer

Create the static network config inside the container:

cat > /etc/systemd/network/80-container-host0.network << 'EOF'
[Match]
Name=host0

[Network]
DHCP=no
Address=192.168.200.200/24
Gateway=192.168.200.1
DNS=192.168.200.1

[Link]
RequiredForOnline=yes
EOF

Restart networking inside the container:

systemctl restart systemd-networkd

Check the result:

ip -4 addr show host0

Expected output:

2: host0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.200.200/24 brd 192.168.200.255 scope host0

Exit the container:

exit

Reboot the container to ensure all services start properly:

sudo machinectl reboot mycontainer

Step 8: SSH into Your Container

Connect to the container via SSH from your host using the static IP:

ssh root@192.168.200.200

What you can do now:

Example use case - AI code agent:

# From your host, give an AI agent access to the container
ssh root@192.168.200.200 "cd /workspace && ai-agent-run --dangerous-mode"

If something breaks, just destroy and recreate the container in seconds!

Container Management Commands

Here's your complete toolkit for managing containers:

# Start a container (if stopped)
sudo machinectl start mycontainer

# Stop a running container (graceful shutdown)
sudo machinectl stop mycontainer

# Restart a container
sudo machinectl restart mycontainer

# Enable container to start at boot
sudo machinectl enable mycontainer

# Disable auto-start
sudo machinectl disable mycontainer

# Check container status and IP address
sudo machinectl status mycontainer

# Get an interactive shell (without SSH)
sudo machinectl shell mycontainer

# Login to console (like physical machine access)
sudo machinectl login mycontainer

# Execute a single command in container
sudo machinectl shell mycontainer /usr/bin/command

# List all containers
sudo machinectl list

# Show container properties
sudo machinectl show mycontainer

# Power off immediately (hard stop)
sudo machinectl poweroff mycontainer

# Remove/unregister a container (doesn't delete files)
sudo machinectl remove mycontainer

# Copy files to container
sudo machinectl copy-to mycontainer /local/file /container/path

# Copy files from container
sudo machinectl copy-from mycontainer /container/file /local/path

Advanced Tips and Troubleshooting

Resource Limits

Limit container resources using systemd slices:

# Limit CPU (50%)
sudo systemctl set-property systemd-nspawn@mycontainer.service CPUQuota=50%

# Limit RAM (1GB)
sudo systemctl set-property systemd-nspawn@mycontainer.service MemoryMax=8G

# Limit I/O
sudo systemctl set-property systemd-nspawn@mycontainer.service IOWeight=100

How it works:

When machinectl start mycontainer runs, systemd creates a systemd-nspawn@mycontainer.service unit

The @ syntax indicates it's an instantiated service template

systemd-nspawn@ is the template, mycontainer is the instance name

You can verify the service name with:

sudo systemctl status systemd-nspawn@mycontainer.service

Persistent Storage

Mount host directories into containers:

# Start with bind mount
sudo systemd-nspawn -bD ~/containers/mycontainer \
  --bind=/home/user/projects:/workspace

Or configure in /etc/systemd/nspawn/mycontainer.nspawn:

[Files]
Bind=/home/user/projects:/workspace

Creating Multiple Containers

Easily spin up multiple isolated environments:

# Create second container
mkdir -p ~/containers/testbox
sudo ln -s ~/containers/testbox /var/lib/machines/testbox
sudo pacstrap -K -c ~/containers/testbox base openssh
# Repeat configuration steps...

Cleanup and Reset

Destroy a container completely:

# Stop container
sudo machinectl stop mycontainer

# Remove from systemd
sudo machinectl disable mycontainer

# Delete filesystem
sudo rm -rf ~/containers/mycontainer
sudo rm /var/lib/machines/mycontainer

Then recreate from Step 1 for a fresh start!

Common Issues

Container won't start:

No network in container:

SSH connection refused:

Security Considerations

Isolation Level:

Hardening Options:

# Start with additional security
sudo systemd-nspawn -bD ~/containers/mycontainer \
  --private-network \    # No network access
  --read-only \          # Read-only root filesystem
  --drop-capability=all  # Drop all Linux capabilities

Best Practices:

Choose systemd-nspawn when:

Choose Docker when:

Conclusion

systemd-nspawn provides a lightweight, powerful way to create isolated development environments without the complexity of full virtualization or container orchestration platforms. It's ideal for:

The tight integration with systemd means you get enterprise-grade container management with tools you already have installed. Start simple, experiment freely, and destroy/recreate at will!

Further Reading