tag/linux

A QEMU question regarding user mode networking

Sun, 08 Dec 2024 09:40:51 +0000

Solved: this works: -net nic,model=virtio -net user

My goal

I want to run a Debian cloud image with qemu-system-x86_64 so that the guest operating system has network access using user mode networking.

Context

I want to use this as part of my CI system (Ambient). I would prefer to not use TUN/TAP networking, or to set up a bridge on the host. I'm aware that user mode networking with QEMU is constrained and limited, and I'm OK with that.

I will explore the other options if user mode networking proves to be impossible.

What I've done

I've attached the script I've been experimenting with. To run, give it two arguments: the URL to the cloud image published by Debian, and the local filename where to store that.

You'll need the following installed:

wget
qemu-img
qemu-system-x86_64
genisoimage

You can run the script as an unprivileged user. If will run faster if you can use the Linux kernel kvm module, but it isn't required. On my laptop the device takes about three minutes to run, assuming the image has been downloaded already.

What the script does is set up cloud-init to run ip a, and then run a VM with the cloud image and the cloud-init configuration, with two virtual serial ports directed to files console.log and run.log. The ip command output goes to the second one.

What I want is for there to be a virtual Ethernet device. I've not been able to make that happens. The ip output in run.log, for me, is:

xyzzy from bootcmd to ttyS1
xyzzy from runcmd to ttyS1
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever

This lists only the lo or loopback network interface. I'd like that to list eth0 or another device via which the guest operating system can connect to the public Internet. For now, I'd be happy just to have the device: once I have that, I can tackle the problem of getting to have an IP address and outgoing Internet connectivity.

(The xyzzy lines are just for debugging. You can ignore them.)

My specific request

How should I change the script so that the ip output lists the network device I want?

I've tried many different variations of network related QEMU options, but nothing seems to give me what I want. I'm sure I'm doing something wrong, and I'm hoping it's obvious to someone else.

Can you help me?

You can respond to by email (liw@liw.fi), or on this fediverse thread, or in any other way you can reach me.

The script

#!/bin/bash
#
# Usage: scripts/qemu.sh https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2 /tmp/debian12.qcow2

set -euo pipefail

# I use the generic Debian 12 image from
# https://cloud.debian.org/images/cloud/bookworm/latest/
url="$1"
image="$2"
if [ ! -e "$image" ]; then
    wget -c "$url" -O "$image"
    chmod a-w "$image"
fi

tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT

# Make a copy-on-write image, on top of the base image. This allows us
# to modify things in the file system inside the running VM, without
# affecting the base image.
qemu-img create -b "$image" -F qcow2 -f qcow2 "$tmp/vm.qcow2"

# Create a cloud-init local data source: an ISO image of a specific
# form. The user-data runcmd writes to the run log (ttyS1) what
# network interfaces are known, with the ip command.
mkdir "$tmp/cloud-init"
cat <<EOF >"$tmp/cloud-init/user-data"
#cloud-config
bootcmd:
- echo xyzzy from bootcmd to ttyS0 > /dev/ttyS0
- echo xyzzy from bootcmd to ttyS1 > /dev/ttyS1
runcmd:
- echo xyzzy from runcmd to ttyS0 > /dev/ttyS0
- echo xyzzy from runcmd to ttyS1 > /dev/ttyS1
- ip a > /dev/ttyS1
- poweroff
EOF

cat <<EOF >"$tmp/cloud-init/meta-data"
hostname: ambient
EOF

cat <<EOF >"$tmp/cloud-init/network-config"
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
EOF

genisoimage -quiet -volid CIDATA -joliet -rock -output "$tmp/cloud-init.iso" "$tmp/cloud-init"

# Run the VM.
qemu-system-x86_64 \
    -m 2048 \
    -display none \
    -serial file:console.log \
    -serial file:run.log \
    -drive format=qcow2,if=virtio,file="$tmp/vm.qcow2" \
    -cdrom "$tmp/cloud-init.iso" \
    -nic user

Framework 13 AMD, first impressions

Sun, 08 Sep 2024 16:52:36 +0000

I've bought a new laptop to more efficiently work when I'm away from home. It's a Framework 13 AMD, with plenty of RAM and storage. Some first impressions:

I bought the DIY version, which meant some assemble was required. I've built desktop PCs from parts and I wasn't scared. The laptop came with the necessary screwdriver, and the Framework website has an excellent guide. With barely a quiver, I started the quest. The assembly consisted of:
- install RAM modules (easy, just snap in)
- install m.2 SSD (easy, one screw)
- connect touch pad cable (easy, unless you get confused about 3D geometry or only have two arms)
- install the keyboard/touch pad (easy, magnets)
- install the display bezel (easy, magnets)
- screw in the back plate (easy, five screws)
- install extension modules (easy, push into slot)
The whole assembly took less than fifteen minutes. It would've been faster if I hadn't taken photographs of every step.
The hardware seems nice. It's thin and lightweight, so it should be easy to travel with.
I quite like the screen. It's bright.
I don't like the keyboard, but I don't like any laptop keyboards. (I'm spoiled by a Keyboardio Model100, which comes with a travel case.).
The touch pad seems OK.
The lid hinge seems OK, and general build quality is fine. It doesn't feel as solid as the ones in the ThinkPads that have been my only personal laptops since 2009, but few things do.
The extension modules seem great. I got a selection.

After that, I installed Linux on it. The Debian in-development installer works, but I used my own installer (v-i), because I'm an old and grumpy and like my own toys. I'm prepared to re-install at the drop of a hat.

The Debian installation passes my checklist for my primary laptop. WiFi, suspend, graphics, it all works so far. (I may make a more detailed review once I've used the machine a while, like anarcat has done. I'll collect my notes and links on my own page.)

The biggest hurdle I had was this isn't a ThinkPad. I had to learn things about Linux configuration I've not had to think about in fifteen years: the WiFi driver isn't called iwlwifi, and the GPU needs a firmware blob.

I specifically needed to install Debian trixie (which will become Debian 13), not Debian 12 (bookworm), because the laptop hardware is sufficiently new that it needs newer kernel and device drivers than what the current Debian stable release has. The trixie version is expected to be released next year, and I can cope with running an in-development version of my operating system until then.

(Those who prefer other Linux distributions that make more frequent releases than Debian does can just use a recent release of their preferred distribution. I like Debian, but I don't judge your for your preference; please don't judge me for mine. I got tired to distro hopping in 1995.)

I've got the laptop basically working, and haven't really used it yet for real. I expect I'll be doing a lot of tweaking to its setup in the coming weeks. For those interested, I've published the v-i spec and Ansible playbook, though given they're peculiar to my way of doing things, they may not be all that useful for others.

I call the machine cantata, after Cantata Sangui, a former Finnish heavy metal band that a friend of mine used to be part of. (My naming theme for my computers is Finnish heavy metal bands).

Rant: year of Linux on the desktop

Thu, 29 Dec 2022 17:45:39 +0000

A rant about "year of Linux on the desktop" from a tired old man. I've been part of the Linux community since before Linux was called Linux. Over the years there's been many people telling me directly that Linux is silly or wrong or imperfect, or that free and open source software is foolish or pointless. A lot more people have, of course, pontificated along those lines in public, and not directed it at me. I'm not claiming to be targeted at that, but I've been around and active for long enough that things accumulate. It's the end of a long year for me, and I though I'd let off some steam myself. Hence this rant.

Over time, the goal posts of success keep being moved by the naysayers. I'm too tired to dig up all important milestones and dates, or references, but here's highlights of the timeline as I have experienced it (years may be a little off):

1991: It's not possible for a hobbyist to write their own operating system kernel.
1992: Well OK, you have a kernel, but there's no networking or a graphical desktop, so it's not actually useful. Anyway, ain't nobody got time to compile all their software.
1993: Fine, there's some kind of desktop, but it doesn't support drag and drop so it's a toy. Also, it won't go anywhere unless it fully and equally supports every graphics card ever made for the PC. Oh, and these pre-build distributions are insecure. Don't download code from the Internet and run that, that's just stupid.
1994: Linux is pointless, since it only runs on the PC and not on any other kind of computer.
1997: All these Linux users are just hobbyists, nobody will use it for anything important.
1998: Corporations will never put money into developing this free software thing.
2000: Linux and free software are a cancer on the IT industry.
2001: The dotcom bubble burst, so Linux will now die, since nobody can afford to continue to develop it, as there's no profit in free software.
2003: Uh, okay, Linux didn't die. But it's too hard to install.
2005: Ubuntu is just a toy.
2006: So Linux is used a lot, but only on servers. It will never work on phones or embedded devices.

(skipping ahead so I don't drag up too many bad memories)

2022: Linux runs all top 500 super computers, billions of personal devices, most servers on the Internet, on all continents, on all oceans, in the air, in orbit, and on Mars. Oh, and in the air on Mars. All big corporations use open source in some form.
Also 2022: Linux will always be a hobbyist toy, unless solves all these new problems we've just thought about.

Next year, 2023, will be my thirtieth year of Linux on the desktop, and the thirtieth year of being told it's not possible to use Linux on the desktop, or to only use Linux on the desktop. Some of the people telling me this weren't born when I started using Linux on the desktop.

Despite everything, it's been fun. I've been lucky to have been able to take part of this journey.

A billion files

Thu, 24 Sep 2020 08:04:00 +0000

I'm sometimes cruel to my tools to learn how they behave in unusual circumstances. For example, what is the fastest way to delete a very large number of files?

If the circumstances are just right, the answer may be mkfs to create a new, empty file system. But it's not always the appropriate answer.

The more obvious answer is sudo rm -rf—or so you'd think. It turns out, however, that sometimes find -delete is a lot faster, or so I learned when I did a little speed testing some years ago. I've been told that rsync -delete is even faster than that. What is the current situation?

How would you test that? I would start by creating a large number of files in a directory tree, and then delete them. How many should I create?

Performance tends to only be interesting for large values of N. Thus, I've created a file system billion files. Since I pay for my own storage, I created empty files.

I did this twice: once using ext4, once with btrfs. Those are the file systems I currently care about. The disks were overwritten with zeroes first, and afterwards I stashed a way a copy compressed with xzip. The ext4 one is 11 GiB, the btrfs one 31 GiB.

I've unpacked the disk images to an LV attached to a VM, then run various commands to delete the file.

command	ext4 (s)	btrfs (s)
rm -rf	18647	19783
find -delete	18964	19702
rsync --del	24279	53516

Based this entirely unscientific benchmark, the fastest way to delete files is either rm of find, with not much difference between them. rsync seems to be significantly slower.

I didn't measure memory use or other factors. If someone knows of a CS student who could do this more formally for class, I'm sure it'd be a fascinating project. Have at it.

Links:

https://events.static.linuxfound.org/slides/2010/linuxcon2010_wheeler.pdf
- slides of talk from 2010 about creating file systems with a billion files: back then it was a challenge to do that, now it's not
https://lwn.net/Articles/400629/
- LWN article about the talk
http://git.liw.fi/billion-files/
- the scripts I used for this
- they're stupid, but simple so they can be easily reviewed
- they run during my work day or overnight

Linux 25 jubilee symposium

Mon, 22 Aug 2016 15:03:38 +0000

I gave a talk about the early days of Linux at the jubilee symposium arranged by the University of Helsinki CS department. Below is an outline of what I meant to speak about, but the actual talk didn't follow it exactly. You can compare these to the video once it comes online.

Linus and I met at uni, the only 2 Swedish speaking new students that year, so we naturally migrated towards each other.
After a year away for military service, got back in touch, summer of
C & Unix course fall of 1990; Minix.
Linus didn't think atime updates in real time were plausible, but I showed him; funnily enough, atime updates have been an issue in Linux until fairly recently, since they slow things down (without being particularly useful)
Jan 5, 1991 bought his first PC (i386 + i387 + 4 MiB RAM and a small hard disk); he had a Sinclair QL before that.
Played Prince of Persia for a couple of months.
Then wanted to learn i386 assembly and multitasking.
A/B threading demo.
Terminal emulation, Usenet access from home.
Hard disk driver, mistaking hard disk for a modem.
More ambition, announced Linux to the world for the first time
first ever Linux installation.
Upload to ftp.funet.fi, directory name by Ari Lemmke.
Originally not free software, licence changed early 1992.
First mailing list was created and introduced me to a flood of email (managed with VAX/VMS MAIL and later mush on Unix).
I talked a lot with Linus about design at this time, but never really participated in the kernel work (partly because disagreeing with Linus is a high-stress thing).
However, I did write the first sprintf for the kernel, since Linus hadn't learnt about varargs functions in C; he then ruined it and added the comment "Wirzenius wrote this portably..." (add google hit count for wirzenius+fucked).
During 1992 Linux grew fast, and distros happened, and a lot of packaging and porting of software; porting was easier because Linus was happy to add/change things in the kernel to accomodate software
A lot of new users during 1992 as well.
End of 1992 I and a few others founded the Linux Documentation Project to help all the new users, some of who didn't come from a Unix background.
In fact, things progressed so fast in 1992 that Linus thought he'd release 1.0 very soon, resulting in a silly sequence of version numbers: 0.12, 0.95, 0.96, 0.96b, 0.96c, 0.96c++2.
X server ported to Linux; almost immediate prediction of the year of the Linux desktop never happening unless ALL the graphics cards were supported immediately.
Linus was of the opinion that you needed one process (not thread) per window in X; I taught him event driven programming.
Bug in network code, resulting in ban on uni network.
Pranks in the shared office room.
We released 1.0 in an event at the CS dept in March, 1994; this included some talks and a ritual compilation of the release version during the event.

Matthew Garret and Intel and the so-called gamergate

Fri, 03 Oct 2014 13:55:38 +0000

Kudos to Matthew for taking a stance. It has, not surprisingly, provoked a lot of comments and feedback, most of it unpleasant.

If I did anything that was directly related to Intel, I'd join him, but I do very, very little architecture dependent stuff anymore.

I will, however, say this: Even if the "gamergate" were actually about good journalism and ethics (and it's clear it isn't), if your reaction to a differing opinion is abuse, harrassment, and other kinds of psychological violence, you're not making anything better, you're making it all worse.

Reasonable people can handle disagreement without any kind of violence.

Thinkpad X200s scroll wheel emulation

Mon, 11 Mar 2013 20:50:59 +0000

Daniel told me how to set up scroll wheel emulation on a Thinkpad. Presumably it works on other systems as well, but the magic numbers for xinput may be different. I wrote a script to set it up, and configure my GNOME session to run it upon login.

The result: pressing the middle mouse button and moving the trackpoint up/down or left/right results in the applications receiving events as if a scroll wheel had been used.

Very, very handy. Thanks, Daniel.

#!/bin/sh

set -eu

id=$(xinput list | sed -n '/TPPS\/2 IBM TrackPoint/s/.*id=\([0-9]\+\).*/\1/p')
emu=$(xinput list-props "$id" | 
    sed -n '/Evdev Wheel Emulation (/s/.*(\([0-9]\+\)).*/\1/p')
but=$(xinput list-props "$id" | 
    sed -n '/Evdev Wheel Emulation Button (/s/.*(\([0-9]\+\)).*/\1/p')
axs=$(xinput list-props "$id" | 
    sed -n '/Evdev Wheel Emulation Axes (/s/.*(\([0-9]\+\)).*/\1/p')
xinput set-int-prop "$id" "$emu" 8 1
xinput set-int-prop "$id" "$but" 8 2
#xinput set-int-prop "$id" "$axs" 8 6 7 4 5

What is a Linux distribution?

Sun, 08 Apr 2012 12:15:00 +0000

I got into a discussion recently about what a Linux distribution actually is. To some people, a distro is just a convenient way of source of binaries from which to pick what you install, either on your own computer or when creating an embedded system, or some other specific kind of system. To others it's a way of life, and the realization of the ideals of software freedom. To yet others, a distro is an outdated concept that will go away in the near-ish future, to be replaced by ... something.

Here's my take: a Linux distribution is defined by the following things:

It's a specific, chosen set of upstream projects. The selection criteria vary wildly between distros, and may be based on things like purpose, quality, licenses, etc. Debian, for example, chooses pretty much everything that is free software, but a distro targeted at providing NAS devices would choose a much smaller set.
It's a set of changes made to the upstream code, and specific configurations of the upstream code, in order make all the software work well together. For example, if the distro supports multiple mail transport agents, they might all need to be patched and configured to store incoming mail the same way by default.
It's the tools, processes, policies, and workflows used to develop the distribution.
It's the tools provided to the people using the distribution to install and manage their systems.
It's the people and companies who develop the distribution and support its users, and their shared values and purpose. This may be a real community, or just the kind of community that exists as a web page.

Historically, there has been a lot of variation in all aspects mentioned above. Some of this variation may today be unnecessary. For example, it might be helpful for more distributions to share work on common tools, so that not everyone needs to re-invent and re-implement the same kinds of tools. However, my gut feeling is that due to the inherent interconnectedness of all things, it may be very difficult to have one set of tools to support well all workflows, policies, processes, and communities.

The most important differentiating factors between the various distros are not the tools, but the community values, the perceived purpose of the distro, and the resultant criteria for choosing what to include and the workflow for developing the distro.

Even when two distros use the same tools, there can be a lot of difference in the end result. Fedora and RHEL, for example, use a lot of the same infrastructure, but their purpose is entirely different. Debian and Ubuntu use mostly the same packages, even, and still there's a huge difference between the two distros: one aims at a high quality system based on free software, the other aims to support the business of one particular company.

For this reason, there will always be a lot of Linux distributions around, and that is good.

On the future of Linux distributions

Fri, 28 Oct 2011 19:17:05 +0000

Executive summary

Debian should consider:

less fine-grained software packaging for overall simplification of the system and its development
less flexibility at the lower levels of the software stack, again for overall simplicity
reducing friction in the development workflow, especially by making it possible to branch and merge at the whole system level

Introduction

I've taken a job with Codethink, as part of a team to develop a new embedded Linux system, called Baserock. Baserock is not a distribution as such, but deals with many of the same issues. I thought it might be interesting to write out my new thoughts related to this, since they might be useful for proper distributions to think about too.

I'll hasten to add that many of these thoughts are not originally mine. I shamelessly adopt any idea that I think is good. The core ideas for Baserock come from Rob Taylor, Daniel Silverstone, and Paul Sherwood, my colleagues and bosses at Codethink. At the recent GNOME Summit in Montreal, I was greatly influenced by Colin Walters.

This is also not an advertisment for Baserock, but since my of my current thinking come from that project, I'll discuss things in the context of Baserock.

Finally, I'm writing this to express my personal opinion and thoughts. I'm not speaking for anyone else, least of all Codethink.

On the package abstraction

Baserock abandons the idea of packages for all individual programs. In the early 1990s, when Linux was new, and the number of packages in a distribution could be counted on two hands in binary, packages were a great idea. It was feasible to know at least something about every piece of software, and pick the exact set of software to install on a machine.

It is still feasible to do so, but only in quite restricted circumstances. For example, picking the packages to install a DNS server, or an NFS server, or a mail server, by hand, without using meta packages or tasks (in Debian terminology), is still quite possible. On embedded devices, there's also usually only a handful of programs installed, and the people doing the install can be expected to understand all of them and decide which ones to install on each specific device.

However, those are the exceptions, and they're getting rarer. For most people, manually picking software to install is much too tedious. In Debian, we've realised this a great many years ago, and developed meta packages (whose only purpose is to depend on other packages) and tasks (solves the same problem, but differently). These make it possible for a user to say "I want to have the GNOME desktop environment", and not have to worry about finding every piece that belongs in GNOME, and install that separately.

For much of the past decade, computers have had sufficient hard disk space that it is no longer necessary to be quite so picky about what to install. A new cheap laptop will now typically come with at least 250 gigabytes of disk space. An expensive one, with an SSD drive, will have at least 128 gigabytes. A fairly complete desktop install uses less than ten gigabytes, so there's rarely a need to pick and choose between the various components.

From a usability point of view, choosing from a list of a dozen or two options is much easier than from a list of thirty five thousand (the number of Debian packages as I write this).

This is one reason why Baserock won't be using the traditional package abstraction. Instead, we'll collect programs into larger collections, called strata, which form some kind of logical or functional whole. So there'll be one stratum for the core userland software: init, shell, libc, perhaps a bit more. There'll be another for a development environment, one for a desktop environment, etc.

Another, equally important reason, to move beyond packages is the problems caused by the combinatorial explosion of packages and versions. Colin Walters talks about this very well. When every system has a fairly unique set of packages, and versions of them, it becomes much harder to ensure that software works well for everyone, that upgrades work well, and that any problems get solved. When the number of possible package combinations is small, getting the interactions between various packages right is easier, QA has a much easier time to test all upgrade paths, and manual test coverage improves a lot when everyone is testing the same versions. Even debugging gets easier, when everyone can easily run the same versions.

Grouping software into bigger collections does reduce flexibility of what gets installed. In some cases this is important: very constrainted embedded devices, for example, still need to be very picky about what software gets installed. However, even for them, the price of flash storage is low enough that it might not matter too much, anymore. The benefit of a simpler overall system may well outweigh the benefit of fine-grained software choice.

Everything in git

In Baserock, we'll be building everything from source in git. It will not be possible to build anything, unless the source is committed. This will allow us to track, for each binary blob we produce, the precise sources that were used to bulid it.

We will also try to achieve something a bit more ambitious: anything that affects any bit in the final system image can be traced to files committed to git. This means tracking also all configuration settings for the build, and the whole build environment, in git.

This is important for us so that we can reproduce an image used in the field. When a customer is deploying a specific image, and needs it to be changed, we want to be able to make the change with the minimal changes compared to the previous version of the image. This requires that we can re-create the original image, from source, bit for bit, so that when we make the actual change, only the changes we need to make affect the image.

We will make it easy to branch and merge not just individual projects, but the whole system. This will make it easy to do large changes to the system, such as transitioning to a new version of GNOME, or the toolchain. Currently, in Debian, such large changes need to be serialised, so that they do not affect each other. It is easy, for example, for a GNOME transition to be broken by a toolchain transition.

Branching and merging has long been considered the best available solution for concurrent development within a project. With Baserock, we want to have that for the whole system. Our build servers will be able to build images for each branch, without requiring massive hardware investment: any software that is shared between branches only gets built once.

Launchpad PPAs and similar solutions provide many of the benefits of branching and merging on the system level. However, they're much more work than "git checkout -b gnome-3.4-transition". I believe that the git based approach will make concurrent development much more efficient. Ask me in a year if I was right.

Git, git, and only git

There are a lot of version control systems in active use. For the sake of simplicity, we'll use only git. When an upstream project uses something else, we'll import their code into git. Luckily, there are tools for that. The import and updates to it will be fully automatic, of course.

Git is not my favorite version control system, but it's clearly the winner. Everything else will eventually fade away into obscurity. Or that's what we think. If it turns out that we're wrong about that, we'll switch to something else. However, we do not intend to have to deal with more than one at a time. Life's too short to use all possible tools at the same time.

Tracking upstreams closely

We will track upstream version control repositories, and we will have an automatic mechanism of building our binaries directly from git. This will, we hope, make it easy to follow closely upstream development, so that when, say, GNOME developers make commits, we want to be able to generate a new system image which includes those changes the same day, if not within minutes, rather than waiting days or weeks or months.

This kind of closeness is greatly enhanced by having everything in version control. When upstream commits changes to their version control system, we'll mirror them automatically, and this then triggers a new system image build. When upstream makes changes that do not work, we can easily create a branch from any earlier commit, and build images off that branch.

This will, we hope, also make it simpler to make changes, and give them back to upstream. Whenever we change anything, it'll be done in a branch, and we'll have system images available to test the change. So not only will upstream be able to easily get the change from our git repository, they'll also be easily verify, on a running system, that the change fixes the problem.

Automatic testing, maybe even test driven development

We will be automatically building system images from git commits for Baserock. This will potentially result in a very large number of images. We can't possibly test all of them manually, so we will implement some kind of automatic testing. The details of that are still under discussion.

I hope to be able to start adding some test driven development to Baserock systems. In other words, when we are requested to make changes to the system, I want the specification to be provided as executable tests. This will probably be impossible in real life, but I can hope.

I've talked about doing the same thing for Debian, but it's much harder to push through such changes in an established, large project.

Solving the install, upgrade, and downgrade problems

All mainstream Linux distributions are based on packages, and they all, pretty much, do installations and upgrades by unpacking packages onto a running system, and then maybe running some scripts from the packages. This works well for a completely idle system, but not so well on systems that are in active use. Colin Walters again talks about this.

For installation of new software, the problem is that someone or something may invoke it before it is fully configured by the package's maintainer script. For example, a web application might unpack in such a way that the web server notices it, and a web browser may request the web app to run before it is configured to connect to the right database. Or a GUI program may unpack a .desktop file before the executable or its data files are unpacked, and a user may notice the program in their menu and start it, resulting in an error.

Upgrades suffer from additonal problems. Software that gets upgraded may be running during the upgrade. Should the package manager replace the software's data files with new versions, which may be in a format that the old program does not understand? Or install new plugins that will cause the old version of the program to segfault? If the package manager does that, users may experience turbulence without having put on their seat belts. If it doesn't do that, it can't install the package, or it needs to wait, perhaps for a very long time, for a safe time to do the upgrade.

These problems have usually been either ignored, or solved by using package specific hacks. For example, plugins might be stored in a directory that embeds the program's version number, ensuring that the old version won't see the new plugins. Some people would like to apply installs and upgrades only at shutdown or bootup, but that has other problems.

None of the hacks solve the downgrade problem. The package managers can replace a package with an older version, and often this works well. However, in many cases, any package maintainer scripts won't be able to deal with downgrades. For example, they might convert data files to a new format or name or location upon upgrades, but won't try to undo that if the package gets downgraded. Given the combinatorial explosion of package versions, it's perhaps just as well that they don't try.

For Baserock, we absolutely need to have downgrades. We need to be able to go back to a previous version of the system if an upgrade fails. Traditionally, this has been done by providing a "factory reset", where the current version of the system gets replaced with whatever version was installed in the factory. We want that, but we also want to be able to choose other versions, not just the factory one. If a device is running version X, and upgrades to X+1, but that version turns out to be a dud, we want to be able to go back to X, rather than all the way back to the factory version.

The approach we'll be taking with Baserock relies on btrfs and subvolumes and snapshots. Each version of the system will be installed in a separate subvolume, which gets cloned from the previous version, using copy-on-write to conserve space. We'll make the bootloader be able to choose a version of the system to boot, and (waving hands here) add some logic to be able to automatically revert to the previous version when necessary. We expect this to work better and more reliably than the current package based one.

Making choices

Debian is pretty bad at making choices. Almost always, when faced with a need to choose between alternative solutions for the same problem, we choose all of them. For example, we support pretty much every init implementation, various implementations of /bin/sh, and we even have at least three entirely different kernels.

Sometimes this is non-choice is a good thing. Our users may need features that only one of the kernels support, for example. And we certainly need to be able to provide both mysql and postresql, since various software we want to provide to our uses needs one and won't work with the other.

At other times, the inability to choose causes trouble. Do we really need to support more than one implemenation of /bin/sh? By supporting both dash and bash for that, we double the load on testing and QA, and introduce yet another variable to deal with into any debugging situation involving shell scripts.

Especially for core components of the system, it makes sense to limit the flexibility of users to pick and choose. Combinatorial explosion déjà vu. Every binary choice doubles the number of possible combinations that need to be tested and supported and checked during debugging.

Flexibility begets complexity, complexity begets problems.

This is less of a problem at upper levels of the software stack. At the very top level, it doesn't really matter if there are many choices. If a user can freely choose between vi and Emacs, and this does not add complexity at the system level, since nothing else is affected by that choice. However, if we were to add a choice between glibc, eglibc, and uClibc for the system C library, then everything else in the system needs to be tested three times rather than once.

Reducing the friction coefficient for system development

Currently, a Debian developer takes upstream code, adds packaging, perhaps adds some patches (using one of several methods), builds a binary package, tests it, uploads it, and waits for the build daemons and the package archive and user-testers to report any problems.

That's quite a number of steps to go through for the simple act of adding a new program to Debian, or updating it to a new version. Some of it can be automated, but there's still hoops to jump through.

Friction does not prevent you from getting stuff done, but the more friction there is, the more energy you have to spend to get it done. Friction slows down the crucial hack-build-test cycle of software development, and that hurts productivity a lot. Every time a developer has to jump through any hoops, or wait for anything, he slows down.

It is, of course, not just a matter of the number of steps. Debian requires a source package to be uploaded with the binary package. Many, if not most, packages in Debian are maintained using version control systems. Having to generate a source package and then wait for it to be uploaded is unnecessary work. The build daemon could get the source from version control directly. With signed commits, this is as safe as uploading a tarball.

The above examples are specific to maintaining a single package. The friction that really hurts Debian is the friction of making large-scale changes, or changes that affect many packages. I've already mention the difficulty of making large transitions above. Another case is making policy changes, and then implementing them. An excellent example of that is in Debian is the policy change to use /usr/share/doc for documentation, instead of /usr/doc. This took us many years to do. We are, I think, perhaps a little better at such things now, but even so, it is something that should not take more than a few days to implement, rather than half a decade.

On the future of distributions

Occasionally, people say things like "distributions are not needed", or that "distributions are an unnecessary buffer between upstream developers and users". Some even claim that there should only be one distribution. I disagree.

A common view of a Linux distribution is that it takes some source provided by upstream, compiles that, adds an installer, and gives all of that to the users. This view is too simplistic.

The important part of developing a distribution is choosing the upstream projects and their versions wisely, and then integrating them into a whole system that works well. The integration part is particularly important. Many upstreams are not even aware of each other, nor should they need to be, even if their software may need to interact with each other. For example, not every developer of HTTP servers should need to be aware of every web application, or vice versa. (It they had to be, it'd be a combinatorial explosion that'd ruin everything, again.)

Instead, someone needs to set a policy of how web apps and web servers interface, what their common interface is, and what files should be put where, for web apps to work out of the box, with minimal fuss for the users. That's part of the integration work that goes into a Linux distribution. For Debian, such decisions are recorded in the Policy Manual and its various sub-policies.

Further, distributions provide quality assurance, particularly at the system level. It's not realistic to expect most upstream projects to do that. It's a whole different skillset and approach that is needed to develop a system, rather than just a single component.

Distributions also provide user support, security support, longer term support than many upstreams, and port software to a much wider range of architectures and platforms than most upstreams actively care about, have access to, or even know about. In some cases, these are things that can and should be done in collaboration with upstreams; if nothing else, portability fixes should be given back to upstreams.

So I do think distributions have a bright future, but the way they're working will need to change.

Linux at 20

Thu, 18 Aug 2011 10:37:03 +0000

Wrote up some of my memories about the early history of Linux. See http://liw.fi/linux20/, but be warned that it's long and rambling.

Linux 20 years: anecdotes from 1998

Wed, 10 Aug 2011 19:34:00 +0000

Linux will be 20 years old this year. I've been drafting an outline for an article of my memories of the early days, but that's going to take a lot of writing before it is ready.

Just in case I don't finish the article, I'll point at Linux Anecdotes, the transcript of a talk I gave in 1998, when Linux was only seven years old. It might still be entertaining to some. If nothing else, you can laugh at my naivete about the future, now our past.

rm is too slow

Sun, 17 Jul 2011 08:42:02 +0000

Some weeks ago I noticed that one of my backup servers was not removing the old backup generations it was making. It had over 18 months worth of daily backups, done with rsync and "cp -al". This is not the world's greatest backup system, but that's entirely irrelevant for this rant.

The cron job that would do that did not exist: I had forgotten to enable it.

So I ran it by hand. Then fixed it, when I noticed it was buggy, and didn't remove any old generations prior to 2010. After that I removed years-old backups of machines that haven't existed for years.

Disk usage:

       Size  Used Avail Use% 
before 5.4T  2.6T  2.6T  51% 
after  5.4T  892G  4.3T  18%

Inodes:

          Inodes    IUsed     IFree IUse%
before 732577792 77645915 654931877   11%
after  732577792 16871789 715706003    3%

That's about 61 million files removed, over the course of three weeks, of which the machine was down due to crashes for about one week.

It should not take two weeks to copy a couple of terabytes (it's less than two megabytes per second). It should definitely not take two weeks to delete it. Almost all the time it was up, it was removing files. There were a couple of times when it finished one run and was idle until I woke up and started the next run. Say a day in total.

Why is "sudo rm -rf" on a large directory tree, or set of large directory trees, so slow? I suspect part of the problem is that the system call API is beautiful, but simplistic: you need to call unlink or rmdir on every file separately, and that means doing an inode lookup separately for each file. Having a system call to delete everything in a directory would have a chance of removing things more efficiently, I suspect. However, since I've not done any profiling, nor even analyzed the relevant source code in the kernel, I might be entirely wrong.

I wonder if anyone else needs to remove large directory trees frequently? I do that a lot, because of my backup software development hobby. But I may be in a unique position, and nobody else would benefit from fast removal.

(I'll note that my backup server is using ext3, and that ext4 is much faster at removing files, particularly large files, but it's still not fast enough.)

Beep be gone

Sat, 07 May 2011 11:35:49 +0000

It is a fundamental part of my belief structure that computers should be silent, unless told otherwise. A default install of the operating system should not make any sounds, unless there is a really good reason to. Thus, I object to the following:

GDM beeping when it is ready to let me log in
login sounds
sound effects whenever I do anything, such as open or close windows
sound effects whenever the computer wants to tell me something that isn't particularly important, such as new e-mail, IRC or IM messages to me, the networking going up or down, or a hard disk going missing
a beep when I reboot or shutdown the computer

I'm sure there are situations when it would be OK for my computer to make noise to alert me. For example, supposed I'm asleep, and a zombie has penetrated my defensive perimeters and is about to chop off the top of my skull so it can devour my brain. In that case, I would be OK with my computer beeping. Once.

Some remedies:

With gdm3, edit /etc/gdm3/greeter.gconf-defaults to set /desktop/gnome/sound/event_sounds to false
Configure GNOME's sound theme to be "No sounds". (It should not be necessary to set "alert sounds" to muted, but it won't hurt either.)
Run amixer -q sset Beep off to shut up wall and other such tools. (I made an init.d script to make that happen automatically, and to make sure I don't forget it when I install new machines for myself.)

Now, it's possible that other people might like or need their computer to make sounds. I imagine, for example, that the GDM login-ready beep is to help people with vision problems. I'm all for that. However, I wish it be easier to shut things up in one simple place for those of us who don't need it.

Edit: Thanks to NP237, noshadow, lindi-, flightplan, and others on #debian-devel with help on shutting up wall.

Linux-tekijä 2008

Mon, 29 Nov 2010 09:18:04 +0000

I'm on the jury for the "Linux-tekijä 2008" award. Roughly translated that means "Linux maker 2008". It's an award for a person or organization who has significantly advanced Linux development or improvided the conditions for Linux users in Finland.

I'm on the jury as a Debian person, since the Finnish Debian community received an honorary mention two years ago.

The nomination form is open, so anyone who has any a suggestion, send it in.

Manuaalisivujen kirjoittaminen – esitelmä Linux-teemaillassa

Mon, 29 Nov 2010 09:18:04 +0000

Pidin 19.2. esitelmän FLUGin ja muiden järjestämässä Linux-teemaillassa, Helsingin Kumpulassa. Käyttämäni kalvot eivät ole järin hyödyllisiä ilman esitystä, mutta ehkä niistä on iloa jollekulle paikalla olleelle.

RAID0 on gytha

Mon, 29 Nov 2010 09:18:04 +0000

I've just more than doubled my desktop's disk I/O speed. Neat, eh?

My desktop machine has four identical hard disks. I keep backups of anything I care about, and I run some disk-intensive things from time to time. Thus, I want the disk subsystem to be as fast as possible, and I don't care about using RAID to recover from disk problems.

When I initially installed the system, however, I couldn't coax the Ubuntu installer to create a RAID0 for me, so it used plain LVM only. Yesterday I spent a day to re-install the system, and got it done. Not entirely straightforward, though.

The alternate installer lets you create RAID systems. However, it fails to install a boot loader if you do that on the entire disks. The final partitioning scheme I ended up with uses LVM for the root and swap partitions, and then RAID0 for a big scratch partition. I also have /boot on a separate partition, to get the boot loader (GRUB or LILO) to work.

In other words, every disk has three partitions:

a 10 GiB LVM physical volume
a 500 MiB volume
the rest as a RAID0 physical volume

The 500 MiB volume on the first disk is used for /boot, and is unused on the three other disks, wasting 1.5 GiB, but that's small enough that I don't care.

Was this worth it? I think so. Russell Coker's bonnie++ gives me the following results for the non-striped LVM volume, i.e., essentially for one disk:

     ------Sequential Output------ --Sequential Input- --Random-
     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
 16G 54932  78 58243  14 28037   6 58412  67 61662   6 261.2   0

In comparison, the RAID0 gets the following results:

     ------Sequential Output------ --Sequential Input- --Random-
     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
 16G 68732  95 143030  29 65331  14 76887  97 152345  17 394.6   1

That's a speedup of about 2.5 times. I'm sure it could be better: in theory, a speedup of 4 times would be possible, but that's the theory. I'm going to be satisified with this, for now.

Renaming a Linux host

Mon, 29 Nov 2010 09:18:04 +0000

I wanted to set up a virtual machine running Ubuntu jaunty (which is undergoing heavy development right now). In order to avoid having to install one from scratch, I copied an existing virtual machine running intrepid, and upgrading that.

Upgrading from release to release (even to one under development) is, of course, one of the things Debian and Ubuntu do very well. The hitch here was that was I wanted to keep the original virtual machine, since I also want to have somewhere to test things on intrepid.

In the end I did this:

virt-clone --connect=qemu:///system -o achmed -n brutha -f VM/brutha.img
Change the hostname of the new machine.

I wrote down the steps to rename the host as a checklist. If I find the list if incomplete, I'll to it. (It's a wiki page, so if you find something to add, please do.)

(Obviously, those steps probably apply to most Unix based operating systems, but I only have Linux to test them on. Perhaps I should set up a few virtual machines with various BSD flavors, some day.)

Screensavers are evil

Mon, 29 Nov 2010 09:18:04 +0000

A screensaver is, in theory, a program to prevent a static image to burn onto a screen. This was could happen with CRT screens. What would happen is that when the same image is displayed for a long time, it would "burn" into the phosphor.

To prevent that, screensavers were invented. They blanked the screen if the computer hasn't been used for a while, and unblanked it when the user pressed a key or moved the mouse.

One day someone came up with the idea of putting an animation on the screen instead of just blanking it.

Turns out, this is evil. It makes the computer use more power. Screensavers cause pollution without having any benefit than a very minor amusement effect.

Also, amusingly, the more a screen is powered on, the shorter it lasts. Thus, screensaver animations actually do the opposite from saving screens.

To prevent that, modern desktop environments have controls to specify how long the screensaver animation works before the screen is turned off. That's an unnecessary bit of complexity.

If I had my way, screensaver animations would be removed from all operating systems. People running their computers off of solar power could add them, if they wanted, but everyone else would be better off without them.

Brother P-touch QL-550 with Ubuntu 9.10 (karmic)

Fri, 13 Nov 2009 13:40:57 +0000

I recently set up my Brother P-touch QL-550 label printer to work with CUPS on my laptop, which runs Ubuntu 9.10 (karmic). This required installing ptouch-driver, with some patches, and then hand-editing the PPD file that gets created.

I made a very crude .deb package to install the ptouch-driver. I have no idea how to package CUPS stuff properly, so I didn't even try. I don't intend to maintain this in Debian or Ubuntu, I just needed it for myself, so I didn't go to much effor, I'm afraid. Perhaps it's useful anyway. I filed a "Request for Package" bug in Debian (555892) and talked with Till Kamppeter from Ubuntu about this, and he has taken a look at this (see comments in the bug).

With my package, the generated PPD has some margins that make it almost impossible to fit anything on the label. The margins are only a few millimeters, but on labels that matters a lot. I fixed this by hand-editing the PPD, Till seems to have fixed in a better way.

I hope this is useful to someone. The printer itself is pretty nice, too bad Brother doesn't provide free drivers.

Kernel oops message compression

Tue, 11 Aug 2009 07:22:52 +0000

I recently attended a talk where a kernel hacker told about debugging techniques when the Linux kernel is having problems. One of the things he mentioned was that kernel oops messages can sometimes be so long they won't fit on a screen. When users submit them, e.g. by photographing the screen, they don't show enough information to be useful.

It struck me that it might be possible to encode the message in various ways to make them use less space so they don't scroll off the screen. My first idea was a 2D bar graph, but those can be difficult in a text-only situation. A second idea is to use gzip and base64 encoding, and a refinement of that is to use base85 encoding.

This has probably been thought of before, but I thought it'd be fun to prototype it anyway. I used a free ascii85 library for Python:

#!/usr/bin/python

import zlib, bz2, base64, sys, tempfile, os, ascii85

msg = sys.stdin.read()
print "msg:", len(msg)
compressed = zlib.compress(msg, 9)
print "compressed:", len(compressed)
fd, compressed_name = tempfile.mkstemp()
os.write(fd, compressed)
os.close(fd)
compressed_f = file(compressed_name)
ascii85.encode(compressed_f, sys.stdout)

The output looks like this:

msg: 1634
compressed: 875
<~GhS.A>Ar7S'Z],0.1WSK(0_bV2p94FTa66f6^JQWJ.4:CK&Xc39T^5bfu^_\Tq&&u<XaD\6
lu::c_'C[5Mge#qt9CE#'^1ebXmM5]cY70)>\-TVY0RhHe#H5XE"-6f[R9^fc!k:qXQdpBILR
6-RZ.C_!]`gG]i0_,]6jbo(CuY*or^o"ci=l"*@htS%n/RF(:Jgj<5p6&:F5]&dboUE&elsfd
(Q>r9iZ[<<TZLdD7flXWfQDgJ!O=E[hn,#9sJ0_F*M9!*pLQ#:GL4bWjmeR!dh="[NORpL"BF
OgCfThPUJ>0e2C=Obgq4J@,HKB#6Wr!Z+@=_@(ucapa-?HiFl=c"6F,Qo2QK(X'kRhemB#?^r
pBRcu?kXCBo9UdEqEaUhnW/r!4.oagXhWh)9Y>RZ?M=[Db_+)Nb88)_KecZ61*?GU=10IDTNZ
@h3T<[tqqj4<[3j+[I6!?OXO0qBWQ0<YT-K+]1e.[N,Q?6$4F(pQ7kcUCJ69(R48PNhTQ6LQR
>;[?u=6M!-4PRrFlO$PJ_>td@h^!-H"HZk_>Z8a9\I:.%7"s6[U+.X)e]4M\-ARjM!0-D:>%0
eFfc3Fc1<$sYqHgbA#+hOK6k9X1*BWOX^I.!.@D^OAsXk<XOUm9]JB3hYYDPjok,F&KuVOYo5
jUn!'=jNq-Rp8b0dr+]li21>EeIRagC'/t^Y_UQY+,XX9/EmmP;VEaUW0^]7@VM:>=akU:$fB
@N%:0J2YqsmA!S\\"W@G`pph9sm23(,kcah7N#Oi3NDUm^]nM8!#ZV<8R#ljg^o/XkT9)5(r\
khNf<p3Dg0*6OunTq%DR]IEL_Yc1L7C%^bPTs@>Rr-QcRu_Kqr#bmu/*cGH<O?:W;@fbN1SiY
`R&"a>KcXOi6kW\+ee9JqMiUWW>UPW.UjWGIh/ipoFIX()/L'M$-9>>k4*gN[<-8@B2460k.l
p,KdJ7>F,O<%]=$A=5.lEMZ-bl[@Hmt&XFYRkrm]9bjOF2f6ZKNWKW[)b)c:,H4.6h+2CCuDa
T@DIM&oI6i'nj&5G9Lqf<F@4K@"0@kZh1P/>[:WP!J_E\T:#c;Y)"?g.\t#.Xt&'IYkT##I2]
T~>

Gzip seems to compress things to about half the original size, and ascii85 inflates that by about 25%. However, the result is then all mashed up on long lines, so use of vertical space if fairly small. Thus, this might be acceptable, perhaps as an optional feature.

Decoding that using OCR from a digital photograph should be possible, and is left as an excercise for the reader, as is an implementation of the encoder that is actually suitable for the kernel. (Read: not in Python.)

It might be possible to write a custom compressor: oops messages seem fairly regular in shape, after all.

There you are. If this is useful for anyone, at any time, please do use the idea.

On mono

Wed, 01 Jul 2009 08:47:16 +0000

There is a fair bit of controversy on Mono, the free software implementation of Microsoft's .NET stack. See, for example, RMS and Ubuntu position statement for two contradicting opinions.

To me, the patent issue is moot. All interesting software violates some patent, I am sure, since the USPTO and EPO conspire to ruin innovation by granting ridiculous patents to megacorporations and patent trolls. Basically, anything and everything can be patented, even if it is patented already.

(The reality is better than that, but not in a significant way.)

The real problem with Mono is that it implements .NET, and needs to be compatible with .NET. This means it needs to follow what Microsoft does, and cannot innovate freely.

Microsoft leads, free software follows. That is not the way to go.

I'm personally fine with people writing software using Mono, and with those apps being included in Linux distributions by default. I think it's stupid to do that, but not so stupid it needs to be stopped, as long as only very few important programs are using Mono. What we cannot allow is to make Mono an essential part of the free software experience, with much of the software requiring it.

That's why I won't write any software in Mono myself, and why I'm wary of using any such software.

Time machine for logging

Fri, 22 May 2009 16:51:58 +0000

Log files from servers and other programs can be an invaluable debugging tool. Really good and detailed logging can make it very easy to pin-point the trigger for a bug.

Unless there is a bug, log files are a useless waste of computing resources. A colleague told me that an app he uses produces 20+ megabytes of log files per day in normal use. That's insane.

What is needed is a way to have log files have happened when a bug manifests itself, but otherwise not.

I should perhaps become a quantum physicist to solve this problem properly, but meanwhile it might be enough to have really fast rotation of debug logs. Say, once an hour.

Libc is broken. What now?

Tue, 05 May 2009 18:10:18 +0000

Today I woke up and read the news that the Ubuntu hardy C library (libc6) had been broken. The bug is so bad that the system no longer boots. Tempers are running high. What to do?

There is a discussion thread on the ubuntu-devel mailing list, where someone flames the person who did the broken upload, and someone else flames the first flamer, and then other people flame the second flamer. It's a regular flamefest. It's not the most constructive reaction from any of the involved people (which is why I'm not linking to it).

This is not Ubuntu specific in any way. Any system that is being developed is going to have bugs of various sorts, some really bad like this. When that happens, people are going to be annoyed.

I'd like to offer the following as lessons to be learned from these kinds of episodes:

Developers must have the freedom to make mistakes without retaliation. Flaming counts as retaliation. Fear makes people less productive.
People who suffer from the bug need to vent. Preferably they will do this in private rather than in public, and without aiming it at specific people, but the need is real, and suppressing it won't improve things. Back in the old days, alt.flame worked pretty well for Usenet, perhaps that's a model that could still apply.
Many problems can be found via automatic testing. Perhaps testing all uploads with, say, piuparts, before they are installed into the package archive could catch some future problems. (Disclaimer: piuparts can't currently test libc or other essential packages.)
Even with rigorous testing, there will be catastrophic bugs. Linux could benefit from a way to recover from them that is easier than fiddling with installation media and tweaking stuff manually. There's various approaches that could apply, perhaps we should pick one and push it. "We" being not just Ubuntu, but also Debian and other distributions.