Introduction
I’ve always liked a clean, slim, lightweight, and robust OS on my laptop (which is my only PC) – I’ve been running the i3 window manager for years, with some custom configuration to enable the Fn keys and set up my preferred desktop session layout. Initially on Ubuntu, for the last two and a half years under Fedora (since I moved to Red Hat). I started with a minimal server install and then had a post-install script that installed the packages that I need, restore my /etc files from git, and some other minor bits.
But over time there’s always some cruft that accumulates – that quick and
dirty sudo make install to test some upstream project, some global pip
install, or simply the increasing delta coming from upgrading through many OS
releases.
Immutable OS installs with atomic upgrades are very appealing to me, as they are always “clean”, and more importantly, they enforce not taking shortcuts during development and scribbling over /usr. But the known ones – Fedora Silverblue or Ubuntu Core – demand using Flatpaks or snaps, which I really don’t believe in and want to avoid.
Build it yourself!
But Fedora Silverblue is just a particular pre-made configuration for building an OSTree image with rpm-ostree.
It is quite easy to build your own rpm-ostrees, so on last weekend’s rainy Sunday I finally took some time to do this.
I took the original Fedora Silverblue config and created my own repo/branch from it. My original Fedora post-install script was a nice basis for the package list and config. (See the OSTree tree file options documentation for details.)
OSTrees include files in /etc, and since most stuff in /etc/ is not machine specific, it makes sense to put these machine independent customizations right into the OSTree instead of maintaining it separately in git.
This has another great benefit – It is now really easy to see what bits in
/etc/ are really machine specific and worth backing up – things like
machine-id, SSH host-keys, human users/passwords, or NetworkManager’s
system-connections:
sudo diff -Nur --no-dereference /usr/etc/ /etc/|view -
This makes managing/backing up of /etc a lot simpler and robust.
Building the tree itself is very easy – in the workstation-ostree-config.git
checkout, just run
sudo ./compose.sh.
This is a tiny shell script for ostree init and rpm-ostree compose to
build the OSTree repository in /srv/ostree/repo, using /srv/ostree/cache/ as a
cache dir (so that subsequent builds don’t need to re-download all the RPMs,
and thus just take a few minutes).
Building the OSTree can happen on any system that has ostree and rpm-ostree installed, i. e. Fedora classic, Atomic, CoreOS, or Silverblue.
Installation
I didn’t find a way to install the built tree directly onto my laptop or even a VM. I played around with the Fedora CoreOS Assembler but was too stupid make it build images which actually install successfully. But that’s not a big hurdle – I just installed the standard Fedora Silverblue desktop, and then switched over to my own repo:
sudo ostree remote add i3 file:///srv/ostree/repo --no-gpg-verify
sudo rpm-ostree rebase i3:pitti-desktop
Then rebooting lands in my minimal lxdm+i3 desktop. As this is OSTree, this is a clean switch, and you can even clean up the remaining on-disk bits from the original GNOME OSTree:
sudo rpm-ostree cleanup --rollback
sudo rpm-ostree cleanup --repomd
Or keep it around for a while, then you can switch between GNOME and i3.
As most stuff is now contained in the OSTree, there are now pleasantly few things to do after installation.
Maintenance
Building your own tree of course means that you don’t have to do this just
once, but regularly, to pick up security and bug fix updates. So make sure to
rebuild (sudo ./compose.sh) and update (sudo rpm-ostree update) your tree
often, maybe once a week. This can be done in a cronjob without trouble and
interference – updating to a tree won’t change any visible files on disk, the
new tree only gets active on reboot. And if something breaks, there’s always
the previous version to roll back to (with sudo rpm-ostree rollback or pick
the previous version in the Grub menu).
Development in chroots
Both the official Silverblue images and also my minimal i3 desktop only ship the kernel, drivers, desktop, and browser, but no compilers, headers, development documentation, etc. This is by intent – I’d find it too cumbersome to build a new OSTree just because I quickly need some new -devel package to build something. It would also quickly turn my system back into a “fat” one.
Hence I much prefer to do development work (building, uploading, running tests, etc.) in chroots. This has the added benefit that you can use one and the same set of tools, workflows, and muscle memory to develop for a variety of OSes – in my case, Fedora stable and rawhide, Debian stable and unstable, and also some Ubuntu work (mostly for Cockpit backports).
Despite all the hype around containers, VMs, and things like
Silverblue’s toolbox,
I still prefer working with chroots for that– I decidedly don’t want the
additional isolation of containers or VMs, as I want to be able to share my home
directory, /tmp, pid and network namespace, $DISPLAY and so on with the
chroots. I want to do things like starting a local cockpit web server out of
the build tree and talking to it from my host’s browser, or run virt-viewer in
the chroot.
The tools of choice for chroot based development are mock for Fedora, and schroot for Debian/Ubuntu. Both are packaged in Fedora, although sbuild isn’t (which contains the convenient scripts for building a chroot – but it’s not that hard to do it with plain debootstrap).
I created a
build-devmock
script which creates a chroot for the default release (the host’s, i. e. Fedora
30 at the moment) with a few build dependencies and development packages that I
need day to day. It also does some tricks to use my own $HOME instead of
/builddir for interactive shells (but not for e. g. fedpkg build).
I also need a few settings in ~/.config/mock.cfg to disable network isolation
and bind-mount my home directory and /dev/kvm into the chroot:
config_opts['use_nspawn'] = False
config_opts['rpmbuild_networking'] = True
config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/home/martin', '/home/martin'))
config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/dev/kvm', '/dev/kvm'))
Finally I define some aliases:
alias m='mock -r ${OS:-default} --quiet --chroot --unpriv --cwd=`pwd`'
alias ms='mock -r ${OS:-default} --quiet --shell --unpriv -- bash -l'
With that, I can run one-off commands in the chroot with e. g. m make. To do
more stuff, I run ms to get a shell in the chroot.
You’ll notice that all of these look at a $OS environment variable. With that
I can use exactly the same tools to e. g. build, test, and run my project in a
Fedora rawhide (or CentOS 7, OpenSUSE, RHEL 8 beta, etc.) chroot:
export OS=fedora-rawhide-x86_64
build-devmock
m './configure && make'
Conclusion
I’ve used this new installation for a full work week now. This surely took some adjustments to both the OSTree config and the build-devmock script to add a few missing packages and tweaks, but I really like the setup. It’s easy and clean to replicate, update, and backup, while staying sufficiently flexible for development, and then containing the cruft in chroots without contamining my OS. This also led me to finding better ways to run system-level bits out of the build tree without copying stuff over to /usr (and first cleaning up our build system to actually allow that).
I suppose the main drawback with that is that it makes it much harder to use
IDEs, as supposedly most of them wouldn’t be able to prefix their build/run
commands with a chroot executor like mock --chroot or schroot -r. I’ve
pretty much always used an “edit in vim in one terminal, run/test/debug in
another terminal’ approach, where this fits right in – I just need to run ms
once in the second terminal.