Setting up unprivileged containers with LXC on Fedora 38
Published on
Updated on
LXC is a containerization technology that allows us to create system containers. We can set it up so that we can SSH into a container and perform many of the same tasks we would on a regular Linux box. I currently have two uses cases for this.
First, these system containers allows me to follow instruction documentation for projects that do not treat docker/podman as a first class distribution method. Maybe the project relies on being able to access systemd, or perhaps I don’t want the additional burden of ensuring the upgrade path of a custom docker-based method.
Second, it allows me to provide a lightweight virtualized environment for my friends and family. With this approach, we don’t have to set caps on the amount of CPU and memory usage from the start (as opposed to a VM), and instead impose them later if others are violating fair use.
In this guide, we’ll focus on setting up unprivileged containers. These are containers that are started up and maintained by a user other than root.
This technology is moving fast. It’s likely that this guide will be out of date by the time you’re reading this. I do hope, however, that this can serve a as a good jumping off point for other queries.
Setup
First, let’s install lxc
sudo dnf install lxc lxc-templates
To setup networking for our containers, we’ll also need to install dnsmasq.
sudo dnf install dnsmasq
We now need to tell LXC that our user is allowed to create a certain number of network devices on our lxcbr0 bridge that LXC configures for us.
echo $(id -un) veth lxcbr0 50 | sudo tee -a /etc/lxc/lxc-usernet
I can’t imagine myself creating more than 50 system containers, but do adjust that number as you see fit.
The last command we’ll need to run as root on the host system is to enable and start the lxc-net service.
sudo systemctl enable --now lxc-net
For our user on the host, we need to setup the LXC configuration file. Here, we’re going to map the user UID/GIDs into our container.
This set of commands were taken from LXC getting started.
mkdir -p ~/.config/lxc
cp /etc/lxc/default.conf ~/.config/lxc/default.conf
MS_UID="$(grep "$(id -un)" /etc/subuid | cut -d : -f 2)"
ME_UID="$(grep "$(id -un)" /etc/subuid | cut -d : -f 3)"
MS_GID="$(grep "$(id -un)" /etc/subgid | cut -d : -f 2)"
ME_GID="$(grep "$(id -un)" /etc/subgid | cut -d : -f 3)"
echo "lxc.idmap = u 0 $MS_UID $ME_UID" >> ~/.config/lxc/default.conf
echo "lxc.idmap = g 0 $MS_GID $ME_GID" >> ~/.config/lxc/default.conf
Now we can download and create our container.
systemd-run --unit=my-unit --user --scope -p "Delegate=yes" -- lxc-create -t download -n test
At the time of writing, we unfortunately have to wrap the lxc-create command within a systemd-run because of a semi-recent change in how cgroups work.
After running that command, you should see a large table of distributions and questions about which one to choose. Of course this is personal preference, but I selected ubuntu jammy amd64.
Once downloaded, we can then start our container.
systemd-run --unit=my-unit --user --scope -p "Delegate=yes" -- lxc-start test
If we see an error message here, then we can add a log file to check for details.
systemd-run --unit=my-unit --user --scope -p "Delegate=yes" -- lxc-start test --logfile=~/lxc.log
The error I commonly saw when setting this up was:
lxc-start test 20231130034412.168 ERROR start - start.c:print_top_failing_dir:99 - Permission denied - Could not access /home/brandon. Please grant it x access, or add an ACL for the container root
This means we need to give $MS_UID access to open the /home/brandon folder. Though what it’s really trying to do is access /home/brandon/.local/share/lxc for it’s rootfs and config.
setfacl -m u:$MS_UID:x /home/rozek
Then try running the lxc-start command from before again. Sometimes when setting this up it worked from here, other times it wanted me to add the +x permission to other folders.
If you run it and don’t see a bunch of errors, then it’s hopefully a success! Check that it’s running:
[brandon@host ~]$ lxc-ls --fancy
NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
test RUNNING 0 - 10.0.3.43 - true
We can then attach to our container to get a shell within it.
lxc-attach test
If the above command doesn’t work, you might need to wrap it in a systemd-run
systemd-run --unit=my-unit2 --user --scope -p "Delegate=yes" -- lxc-attach test
Auto-starting LXC containers at boot
When we start a container with the systemd-run command, it’s tied to that particular terminal session. If we want to start these containers when the machine boots up, we can rely on systemd.
First let’s create a service file under ~/.config/systemd/user/lxc@.service
[Unit]
Description=Start LXC container
After=lxc-net.service
Wants=lxc-net.service
[Service]
Type=oneshot
Delegate=yes
RemainAfterExit=yes
ExecStart=/usr/bin/lxc-start %i
ExecStop=/usr/bin/lxc-stop %i
TimeoutStartSec=0
[Install]
WantedBy=default.target
Then we can start our test container
systemctl --user start lxc@test
Enable it on bootup
systemctl --user enable lxc@test
Since this is a user service, we need to make sure linger is on for it to respect the enable on boot setting.
sudo loginctl enable-linger rozek
Other Resources
https://linuxcontainers.org/lxc/getting-started/