Advertise with Googlier.com tag/debian https://blog.liw.fi/tag/debian/ blog.liw.fi ikiwiki Sat, 30 Aug 2025 13:46:28 +0000 Why I use Debian https://blog.liw.fi/posts/2025/why-debian/ https://blog.liw.fi/posts/2025/why-debian/ debian Sat, 30 Aug 2025 12:03:32 +0000 2025-08-30T13:46:28Z <p>I've been using the Debian Linux distribution since the mid-1990s. I still use it. I had a brief exploration of Linux distributions, early on. It was brief partly, because there were so few of them.</p> <ul> <li><p>My first Linux installation was by Linus, to develop and test the installation method. He'd never installed Linux, because it grew on his PC on top of an existing Minix installation. He used my PC to figure out how to install Linux. This was in 1991.</p></li> <li><p>I then tried subsequent boot+root floppy images, by Linus or others, and the MCC Interim Linux distribution, and SLS. Possibly one or two others.</p></li> <li><p>Then in 1993, Debian was announced. I think I tried it first in 1994, and was hooked. Debian was interesting in particular because it was a community project: I could join and help. So I did. I became a Debian developer in 1996.</p></li> </ul> <p>I've since used Ubuntu (while working for Canonical), and Baserock (while working for Codethink), and I've looked at several others, but I always return to Debian.</p> <p>I like Debian for several reasons:</p> <ul> <li>I know it very well.</li> <li>I know the community of Debian developers.</li> <li>I trust the community of Debian developers.</li> <li>I trust the Debian project to follow its <a href="https://www.debian.org/social_contract">Social Contract</a>.</li> <li>I trust the Debian project to vet software freedom of what they package.</li> <li>I trust the Debian project to update its packages for security fixes.</li> <li>I trust the Debian project to keep the privacy of its users in mind.</li> </ul> <p>The key word here is trust. Over thirty years, I've built very strong trust in Debian doing the right thing, from my point of view. That's a pretty high bar for any other distribution to clear.</p> Why is Debian the way it is? https://blog.liw.fi/posts/2023/debian-reasons/ https://blog.liw.fi/posts/2023/debian-reasons/ debian Fri, 11 Apr 2025 15:33:41 +0000 2025-04-11T15:33:41Z <p>Debian is a large, complex operating system, and a huge open source project. It's thirty years old now. To many people, some of its aspects are weird. Most such things have a good reason, but it can be hard to find out what it is. This is an attempt to answer some such questions, without being a detailed history of the project.</p> <h1>What Debian wants to be</h1> <p>Debian wants to be a high-quality, secure general purpose operating system that consists only of free and open source software that runs on most kinds of computers that are in active use in the world.</p> <p>By general purpose I mean Debian should be suitable for most people for most purposes. There will always be situations where it's not suitable, for whatever reason, but it's a good goal to aim for. Some other distributions aim for specific purposes: a desktop, a server, playing games, doing scientific research, etc. It's fine to aim to be general purpose, or specific purpose, but the choice of goal leads to different decisions along the way.</p> <p>For Debian, aiming to be general purpose means that Debian doesn't choose what to package based on the purpose of the software. The only real choice Debian makes here is on whether the software is free and whether it's plausible for Debian to maintain a high quality package.</p> <h1>The constitution, power structure, governance</h1> <p>Debian is one of the more explicitly democratic open source organizations. It has well-defined processes for making decisions, and elects a project leader every year. Further, the powers of the project leader are strictly constrained, and most powers usually associated with leadership are explicitly delegated to other people.</p> <p>The historic background for this is that the first Debian project leaders were implicitly all-powerful dictators until they chose to step down. Then one project leader went too far, and a revolt threw them out, and democracy was introduced. As part of this, the project got a formal <a href="https://www.debian.org/devel/constitution">constitution</a>, which defines rules for the project.</p> <p>The reason Debian has the rules it has, is because less rules, and less bureaucracy, didn't work for Debian earlier in its history.</p> <h1>Social contract and Debian free software guidelines</h1> <p>In the mid-1990s, before the term open source had been introduced, what was "free software" was defined by the Free Software Foundation, but in a way that left much to be interpreted. Debian wanted to have clearer rules, and came up with the Debian Free Software Guidelines, and made them part of its <a href="https://www.debian.org/social_contract">Social Contract</a>.</p> <p>The social contract is Debian's promise to itself and to the world at large about what Debian is and does. The DFSG is part of that. This is a foundation document for Debian, and changing it is intentionally made difficult in the Debian constitution.</p> <p>The more detailed rules have made it clearer what Debian will accept, and have simplified discussions about this. There is still a lot to discuss, of course.</p> <p>The DFSG was later the basis of the <a href="https://opensource.org/osd/">Open Source Definition</a>.</p> <h1>Self-contained</h1> <p>Debian insists on being self-contained. Anything that is packaged in Debian, by Debian, must be built (compiled) using only dependencies in Debian. Also, everything in Debian must be built by Debian. This can cause a lot of extra work. For example, current programming language tooling often assumes it can download dependencies from online repositories at build time, and that is not acceptable to Debian.</p> <p>The main reason for this is that a dependency might not be available later. Debian has no control over third party package repositories, and if a package, or entire repository, goes away, it might be impossible for Debian to rebuild the package. Debian needs to rebuild to upgrade to a new compiler, to fix a security problem, to port to a new architecture, or just to make some change to the packaged software, including bug fixes.</p> <p>If Debian weren't self-contained, it would be at the mercy of any of the tens of thousands of packages it has, and all their dependencies, being available when an urgent security fix needs to be released. This is not acceptable to Debian, and so Debian chooses to do the work of packaging all dependencies.</p> <p>That means, of course, that for Debian to package something can be a lot of work.</p> <h1>No bundled libraries</h1> <p>Debian avoids using copies of libraries, or other dependencies, that are bundled with the software it packages. Many upstream projects find it easier to bundle or "vendor" dependencies, but for Debian, this means that there can be many copies of some popular libraries. When there is a need to fix a security or other severe problem in such a library, Debian would have to find all copies to fix them. This can be a lot of work, and if the security problem is urgent, it wastes valuable time to have to do that.</p> <p>As an example: the zlib is used by a very large number of projects. By its nature, it needs to process data that may be constructed to exploit a vulnerability in the library. This has happened. At one point, Debian found dozens of bundled copies of zlib in its archive, and spent considerable effort making sure only the packaged version of zlib is used by packages in Debian.</p> <p>Thus, Debian chooses to do the work up front, before it's urgent, while packaging the software, and make sure the package in Debian uses the version of the library packaged in Debian.</p> <p>This is not always appreciated by upstream developers, who would prefer to only have to deal with the version of the library they bundle. That's the version they've verified their own software with. This sometimes leads to friction with Debian.</p> <h1>Membership process</h1> <p>Given the size and complexity of Debian as an operating system, and its popularity, the project needs to trust its members. This especially means trusting those who upload new packages. Because of technical limitations in Linux in the 1990s, every Debian package has full root access during its installation. In other words, every Debian developer can potentially become the root user on any machine running Debian. With tens of millions of machines running Debian, that is potentially a lot of power.</p> <p>Debian vets its new members in various ways. Ideally, every new member has been part of the Debian development community sufficiently long that they are known to others, and they've built trust within the community.</p> <p>The process can be quite frustrating to those wanting to join Debian, especially to someone used to a smaller open source project.</p> <h1>Release code names</h1> <p>Debian assigns a code name for its each major release. This was originally done to make mirroring the Debian package archive less costly.</p> <p>In the mid-1990s, when Debian was getting close to making its 1.0 release, code names weren't used. Instead, the archive had a directory for each release, named after its version. Developing a new release takes a while, so the directory "1.0" was created well ahead of time. Unfortunately, a publisher of CD-ROMs, prematurely mass-produced a disc they labeled 1.0, before Debian had actually finished making 1.0. This meant that people who got the Debian 1.0 CD-ROM got something that wasn't actually 1.0.</p> <p>An obvious solution to prevent this from happening again would have been to prepare the release in a directory called "1.0-not-released", and rename the directory to "1.0" after the release was finished. However, this would've meant that all the mirrors would've had to re-download the release when the name of the directory changed. That would've been costly, given the massive size of Debian (hundreds of packages! tens of megabytes!). Thus, Debian chose to use code names instead.</p> <p>Later, the "pool" structure was added to the Debian archive. With this, the files for all releases are in the same directory tree, and metadata files specify what files belong to each release. This makes mirroring easier. It might be possible to drop the code names and stick to versions, now, but I don't know if Debian would be interested in that.</p> <h1>Changing slowly</h1> <p>As implied above, Debian is huge. It's massive. It's enormous, It's really not very small at all, any more.</p> <p>Large ships stop slowly. Large projects change slowly. Any change in Debian that affects large portion of its packages may require hundreds of volunteers to do work. That is not going to happen quickly.</p> <p>Sometimes the work can be done with just a small number of people, and Debian has processes to enable that. As an example, if a new version of the GNU C compiler is uploaded, the work of finding out what fixes in other packages need to be made can usually be done by a handful of people.</p> <p>Often a change takes time because there's a need to build consensus, and that requires extensive discussion, which takes time and can only rarely be short-circuited.</p> <p>This all also means Debian developers tend to be conservative in technical decisions. They often prefer solutions that don't require large scale changes.</p> <hr /> <p>To comment publicly, please use <a href="https://toot.liw.fi/@liw/111198682212279688">this fediverse thread</a>.</p> <p>2025 update: I've posted a followup <a href="https://blog.liw.fi/posts/2025/why-debian-changes/">Why does Debian change software?</a>.</p> Why does Debian change software? https://blog.liw.fi/posts/2025/why-debian-changes/ https://blog.liw.fi/posts/2025/why-debian-changes/ debian Fri, 11 Apr 2025 11:13:58 +0000 2025-04-11T11:14:49Z <p>When I wrote <a href="https://blog.liw.fi/posts/2023/debian-reasons/">Why is Debian the way it is?</a>, a year and a half ago, I was asked to also cover why Debian changes the software it packages. Here's a brief list of examples of why that happens:</p> <ul> <li><p>Software in Debian needs to follow certain policies as set by Debian over the years, and documented in the <a href="https://www.debian.org/doc/debian-policy/">Debian Policy Manual</a>. These are mostly mundane things like system wide configuration being in <code>/etc</code>, documentation in <code>/usr/share/doc</code>, and so on. Some of this is more intricate, like when names of executables can be the same in different packages.</p></li> <li><p>Programs included in Debian need to work together in other ways. This might mean require changing one or both. As an example, they might need to agree where Unix domain socket exists, or what Unix user account they should run under.</p></li> <li><p>Debian will remove code that "calls home" or tries to update software in a way that bypasses the Debian packaging system. This is done both for privacy reasons, and because updating software without going via the packaging system is usually problematic from a functional point of view, and always problematic from a security point of view.</p></li> <li><p>Debian may fix bugs before they're fixed in upstream, or may backport a bug fix to an earlier version. The goal here is to make life better for users of Debian. Debian does this especially for fixes to security problems, but also for other problems.</p></li> <li><p>Debian avoids including anything in the main part of its package archive it can't legally distribute. This applies to the source packages. This means, Debian may strip out those parts software that it doesn't think are free according to the Debian Free Software Guidelines. The stripped-out parts might be moved to another package in the "non-free" part of Debian. An example might a manual that is licensed under the GNU Free Documentation License with immutable parts, or a logo that can't be changed.</p></li> <li><p>Debian has often added a manual page when the upstream doesn't provide one.</p></li> </ul> <p>Thank you to Jonathan McDowell for help with this list. Opinions and mistakes are mine. Mine, I say!</p> Debian Git monorepo https://blog.liw.fi/posts/2024/monorepo/ https://blog.liw.fi/posts/2024/monorepo/ announcement becrueltoyourtools debian git Mon, 01 Apr 2024 05:31:27 +0000 2024-04-01T07:41:43Z <p><a href="https://liw.fi/">Lars Wirzenius Consulting Ltd</a> has created a Git repository with the source code of Debian 12 (bookworm), the <code>main</code> component. It's one repository for all the source code, as one commit. It's a <a href="https://en.wikipedia.org/wiki/Monorepo">monorepo</a>.</p> <p>To clone the repository (but read below before you run this):</p> <pre><code>git clone https://monorepo.liw.fi/git/debian </code></pre> <p>Please be kind to the proof-of-concept Git server and avoid cloning if 42 or more people are already cloning. The checkout will take a few hours and use about 500 GiB disk space, and have about 15 million files outside <code>.git</code>.</p> <p>This is a read-only repository that will need to be moved onto Debian infrastructure before it's actually usable for collaboration.</p> <h2>Motivation</h2> <p>The monorepo will greatly simplify and improve the development process of Debian:</p> <ul> <li><p>Simpler collaboration: every package uses the same process, and the same tools, and it'll be easier than ever to help with other people's packages.</p></li> <li><p>Transitions are easier: all changes will be prepared in one branch and merged atomically, rather than uploading each source package separately.</p></li> <li><p>Enables distribution-wide changes in general: With all the source code for everything in one tree, in on repository, it's feasible to make changes to Debian that affect many packages. For example, back in the day Debian took seven years to migrate <code>/usr/doc</code> to <code>/usr/share/doc</code>, and that can now be done in one commit.</p></li> <li><p>Easier to see what has changed: the release notes maintainers will especially appreciate this. In the future, Debian release notes will be excruciatingly detailed. They will no longer merely say "bug fixes and improvements".</p></li> <li><p>Better non-free management: if a package in main is ever found to contain non-free code, it can be removed using <code>git filter-branch</code>.</p></li> <li><p>Better quality control: merges will only be happen after CI tests pass, using the <a href="https://bors.rust-lang.org/"><code>bors</code></a> tool from the developers of the Rust language. Currently packages are tested only after they've already been uploaded to the Debian archive. The change will reduce problems for people running the unstable branch of Debian in production. Soon there will be no bugs in unstable.</p></li> <li><p>More development speed: because Debian developers will no longer need to spend time fixing bugs in unstable, and maintaining the Debian bug tracker, they will have more time and energy to update packages to newer versions. This, in turn, will motivate large enterprise users of Debian to fund Debian development more, as their primary complaint of the slow pace of change will finally be solved.</p></li> </ul> <h1>Future of ".dsc"</h1> <p>There will be no further need for the legacy <code>.dsc</code> source package format. Instead, a copy of the monorepo can be shipped.</p> <h1>Resource use</h1> <p>The monorepo is not small. However, given how important Debian is to their business, companies like Apple, Microsoft, Amazon, and Google, are already chomping at the bit to donate hardware and cloud resources to enable use of the monorepo. Expect an announcement on <code>debian-devel-announce</code> as soon as details have been worked out.</p> <h1>Implementation</h1> <p>The program <a href="https://app.radicle.xyz/nodes/radicle.liw.fi/rad:zgYpM7b29D6wTMjEUxxzBjcF9EvK">unpack-debian-sources</a> is available on the <a href="https://radicle.xyz/">Radicle</a> network as repository ID <code>rad:zgYpM7b29D6wTMjEUxxzBjcF9EvK</code>. Running it took about seven hours on my home computer. After that finished, I created the repository with the following commands:</p> <pre><code>git init . git add -v . git commit -m "Debian 12 (bookworm), main" </code></pre> <p>The <code>git add</code> took about 3.5 hours and <code>git commit</code> took another 2.5 hours. This required 16 GiB RAM to work. Since I've already done this, nobody else needs to. Everyone else can just clone, from my server or each other. The commit is:</p> <pre><code>commit 8ae3129b9f222534d6ed20dd0489fb8f2f1d1a97 (HEAD -&gt; main, origin/main, origin/HEAD) Author: Lars Wirzenius &lt;liw@liw.fi&gt; Date: Tue Mar 19 11:05:57 2024 +0000 Debian 12 main </code></pre> <p>Have fun!</p> <h1>Disclaimer</h1> <p>This blog post is an April Fool's joke. Debian is not moving to a monorepo. I have not been a member of Debian since 2018. I don't speak for Debian. I would be very surprised if Debian would abandon its current development model of being developer and package centric. I don't even think it'd be sensible for Debian to have a monorepo. The monorepo is real, though it will be deleted by the end of April.</p> <p>The real motivation for all this is that I sometimes like to be cruel to my tools. This time, I'm cruel to Git: can it handle a repository of this size? In 2009 it could not. In 2024 it can.</p> DPL elections: candidate counts https://blog.liw.fi/posts/dpl-election-candidates/ https://blog.liw.fi/posts/dpl-election-candidates/ debian Mon, 22 Aug 2022 04:52:09 +0000 2022-08-22T04:52:09Z <p>Out of curiosity, and because it is Sunday morning and I have a cold and can't get my brain to do anything tricky, I counted the number of candidates in each year's DPL elections.</p> <table> <thead> <tr> <th>Year </th> <th> Count </th> <th> Names</th> </tr> </thead> <tbody> <tr> <td>1999 </td> <td> 4 </td> <td> Joseph Carter, Ben Collins, <strong>Wichert Akkerman</strong>, Richard Braakman</td> </tr> <tr> <td>2000 </td> <td> 4 </td> <td> Ben Collins, <strong>Wichert Akkerman</strong>, Joel Klecker, Matthew Vernon</td> </tr> <tr> <td>2001 </td> <td> 4 </td> <td> Branden Robinson, Anand Kumria, <strong>Ben Collins</strong>, Bdale Garbee</td> </tr> <tr> <td>2002 </td> <td> 3 </td> <td> Branden Robinson, Raphaël Hertzog, <strong>Bdale Garbee</strong></td> </tr> <tr> <td>2003 </td> <td> 4 </td> <td> Moshe Zadka, Bdale Garbee, Branden Robinson, <strong>Martin Michlmayr</strong></td> </tr> <tr> <td>2004 </td> <td> 3 </td> <td> <strong>Martin Michlmayr</strong>, Gergely Nagy, Branden Robinson</td> </tr> <tr> <td>2005 </td> <td> 6 </td> <td> Matthew Garrett, Andreas Schuldei, Angus Lees, Anthony Towns, Jonathan Walther, <strong>Branden Robinson</strong></td> </tr> <tr> <td>2006 </td> <td> 7 </td> <td> Jeroen van Wolffelaar, Ari Pollak, Steve McIntyre, <strong>Anthony Towns</strong>, Andreas Schuldei, Jonathan (Ted) Walther, Bill Allombert</td> </tr> <tr> <td>2007 </td> <td> 8 </td> <td> Wouter Verhelst, Aigars Mahinovs, Gustavo Franco, <strong>Sam Hocevar</strong>, Steve McIntyre, Raphaël Hertzog, Anthony Towns, Simon Richter</td> </tr> <tr> <td>2008 </td> <td> 3 </td> <td> Marc Brockschmidt, Raphaël Hertzog, <strong>Steve McIntyre</strong></td> </tr> <tr> <td>2009 </td> <td> 2 </td> <td> Stefano Zacchiroli, <strong>Steve McIntyre</strong></td> </tr> <tr> <td>2010 </td> <td> 4 </td> <td> <strong>Stefano Zacchiroli</strong>, Wouter Verhelst, Charles Plessy, Margarita Manterola</td> </tr> <tr> <td>2011 </td> <td> 1 </td> <td> <strong>Stefano Zacchiroli</strong> (no vote yet)</td> </tr> </tbody> </table> <p>Winner indicate by boldface. I expect Zack to win over "None Of The Above", so I went ahead and boldfaced him already, even if there has not been a vote for this year.</p> <p>Median number of candidates is 4.</p> v-i version 0.2: non-interactive Debian installer for bare metal machines https://blog.liw.fi/posts/2022/08/07/v-i/ https://blog.liw.fi/posts/2022/08/07/v-i/ announcement debian v-i Sun, 07 Aug 2022 13:21:32 +0000 2022-08-07T14:04:35Z <p>I've just released version 0.2 of v-i, my non-interactive, fairly fast, unofficial, alternative installer of Debian for physical computers ("bare metal"). It's what I use to install Debian on my PCs now. I <a href="https://blog.liw.fi/posts/2022/01/01/v-i">blogged about it previously</a>.</p> <p>Below is a transcript of me installing Debian to my Thinkpad T480 laptop, from my desktop machine.</p> <pre><code class="sh">$ scp exolobe1-spec.yaml root@v-i: exolobe1-spec.yaml 100% 228 190.6KB/s 00:00 $ ssh root@v-i Linux v-i 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Sun Aug 7 13:10:02 2022 from 10.1.1.54 root@v-i:~# time ./v-i exolobe1-spec.yaml OK, done real 1m11.003s user 0m30.671s sys 0m8.645s root@v-i:~# reboot Connection to v-i closed by remote host. Connection to v-i closed. $ </code></pre> <p>The <code>exolobe1-spec.yaml</code> file contains:</p> <pre><code class="yaml">hostname: exolobe1 drive: /dev/sda extra_lvs: - name: home size: 300G mounted: /home ansible_vars: user_pub: | ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPQe6lsTapAxiwhhEeE/ixuK+5N8esCsMWoekQqjtxjP liw personal systems </code></pre> <p>To summarize what happened above:</p> <ul> <li>I wrote the installer image to a USB drive, and booted the laptop with that drive</li> <li>I logged into the installer, running on the laptop, using SSH, to run the installer</li> <li>the installer asks no questions: I give it a file with all the information it needs to install Debian</li> <li>the installation command took 1 minute 11 seconds <ul> <li>Disclaimer: this is the second install with the same USB drive. The first install takes about four minutes longer, because it runs <code>debootstrap</code>. Later installs use a cache of its output.</li> </ul> </li> <li>that time doesn't include the time to write the USB drive, or to boot the target PC to the installer, or to boot the installed system</li> <li>the installed system is fairly minimal, and does not have, say, a desktop system, a software stack to run web applications, or anything much else that's actually useful</li> <li>the installed system is, however, accessible over SSH, and can be provisioned with a configuration management system such as Ansible</li> </ul> <p>See <a href="https://files.liw.fi/v-i/0.2/">https://files.liw.fi/v-i/0.2/</a> for the installer image I used above, and some other bits and bobs. See <a href="https://gitlab.com/larswirzenius/v-i">git repository</a> to open issues or to contribute.</p> v-i 0.1, a Debian installer using vmdb2 https://blog.liw.fi/posts/2022/03/06/v-i/ https://blog.liw.fi/posts/2022/03/06/v-i/ announcement debian v-i vmdb2 Sun, 06 Mar 2022 17:16:08 +0000 2022-12-28T17:57:56Z <p>I've published release 0.1 of v-i, my installer for Debian based on vmdb2. It's entirely non-interactive, dangerous, unhelpful, but fairly fast. It <em>will</em> remove all data from your drives. See the <a href="https://blog.liw.fi/posts/2022/01/01/v-i/">initial announcement</a> for more, or the <a href="https://gitlab.com/larswirzenius/v-i/-/blob/main/README.md">README</a>. There is now a pre-built image available, see the <a href="https://gitlab.com/larswirzenius/v-i/-/blob/main/tutorial.md">tutorial</a>.</p> <p>I'd be curious to hear if anyone else is interested in this.</p> <p>This is the target spec file I use for my VM host:</p> <pre><code class="yaml">drive: /dev/nvme0n1 extra_drives: - /dev/nvme1n1 hostname: exolobe5 extra_playbooks: - exolobe5-ansible.yml ansible_vars: user_pub: | ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPQe6lsTapAxiwhhEeE/ixuK+5N8esCsMWoekQqjtxjP liw personal systems extra_lvs: - name: vms size: 1T fstype: ext4 mounted: /mnt/vms </code></pre> Debian disappointment https://blog.liw.fi/posts/2021/04/20/debian/ https://blog.liw.fi/posts/2021/04/20/debian/ debian vmdb2 Tue, 20 Apr 2021 09:07:19 +0000 2021-04-20T09:07:19Z <p>I retired from Debian a couple of years ago (see <a href="https://blog.liw.fi/posts/2018/11/18/retiring_from_debian/">1</a> and <a href="https://blog.liw.fi/posts/2019/12/21/why_debian_isn_t_fun_for_me/">2</a>). It's not gotten healthier. The campaigning around the recent <a href="https://www.debian.org/vote/2021/vote_002">Debian vote</a> showed that clearly to me. I've been questioning my own use of Debian, and the sense of developing the <a href="https://vmdb2.liw.fi/">vmdb2</a> software for making hard disk images with Debian installed.</p> <p>After much internal struggle, I've landed on the following compromise: I like the Debian operating system, but I object to the behavior of a number of people involved with Debian. As an vehicle for promoting software freedom in practice, Debian is very valuable. I will continue to use Debian myself, and continue to maintain vmdb2, but I will argue against bad behavior in Debian, or elsewhere in free software movement.</p> Alternative Debian installer based on vmdb2: v-i https://blog.liw.fi/posts/2020/02/29/alternative_debian_installer_based_on_vmdb2_v-i/ https://blog.liw.fi/posts/2020/02/29/alternative_debian_installer_based_on_vmdb2_v-i/ debian v-i vmdb2 Sat, 29 Feb 2020 20:12:00 +0000 2020-02-29T18:52:00Z <p>I wrote an alternative Debian installer as a toy, called v-i. One of the following two bullet points is correct:</p> <ul> <li><p>v-i can install a very rudimentary Debian onto exactly one computer in the world: my very own spare Thinkpad x220 laptop. It might not work on your x220. v-i almost certainly won't work on any other kind of computer. If you try, it will probably delete all your data. Make sure your backups work.</p></li> <li><p>v-i is perfect in every way. There are not even any typos in the manual. There are no bugs, and all features are fully implemented. Every possible use case is supported. Not only is there no danger to your data, v-i will prevent it from ever disappering. Even your hardware will never break again. Your laptop will have infinite battery life, and your screen resolution will require 64 bit integers to express.</p></li> </ul> <p>The v-i installer is based on the <a href="https://vmdb2.liw.fi/">vmdb2</a> tool, which I also wrote. It has nothing to do with debian-installer, which is the official Debian installer, also known as d-i. I use d-i, but have a couple of things I wanted to change:</p> <ul> <li><p>I'd like something I can easily modify. d-i requires building special udeb packages for any software that's to be part of the installer. v-i is happy with normal debs.</p></li> <li><p>Debian in general uses preseeding for automating an installation. Preseeding means providing answers, in a file, to questions the package may ask during its installation. This is fine, if a little cumbersome, but only helps when the packages ask the right questions. v-i lets you have the full power of Ansible during initial installation, which is much more flexible.</p></li> </ul> <p>On the other hand, d-i is mature software and tested by a very large number of people, on a very large number of different hardware. v-i is not. v-i might, at best, be the beginning of something useful for a small number of people.</p> <p>I can now install Debian onto my x220 with v-i. It's a very basic install, without LVM2, full-disk encryption, or a graphical desktop, but it does have sshd and I can configure the laptop further with Ansible from another host. I've installed the GNOME desktop that way, after rebooting into a v-i installed system. (In theory, I could install GNOME directly from v-i. In practice, there are bugs in packages and/or how vmdb2 runs Ansible.)</p> <p>The installed system is also highly configured to my needs and preferences. It uses Finnish locales, and requires my SSH key to log in. The root account has no password. All of this could be made better with a bit of work.</p> <p>The code is at <a href="https://gitlab.com/larswirzenius/v-i">https://gitlab.com/larswirzenius/v-i</a>. Check the README for more instructions if you're curious. If you do give it a try, I'd love to hear from you, unless you just lost all your data. Please don't lose all your data.</p> <p>If you'd like to help build a more viable installer from v-i, please talk to me. I dream of a future where I can install a bare metal machine as easily as I can create and configure a VM.</p> <p>PS. A 128 GB USB3 flash drive can be had for as little as 20 euros, and that has enough disk space for v-i and a Debian mirror.</p> <p>If you want to respond to this blog post, please email me (liw@liw.fi) or respond to <a href="https://toot.liw.fi/@liw/103743493062316650">this fediverse post</a>.</p> Why Debian isn't fun for me https://blog.liw.fi/posts/2019/12/21/why_debian_isn_t_fun_for_me/ https://blog.liw.fi/posts/2019/12/21/why_debian_isn_t_fun_for_me/ debian Sat, 21 Dec 2019 10:13:00 +0000 2019-12-21T08:32:55Z <p>I retired from Debian as a developer a year ago. I said then that it was because Debian wasn't fun anymore, but I didn't unpack that much. It's been long enough that I feel I can do that. I should've done it back then, but I wasn't strong enough.</p> <p>A big part of Debian not being fun is that there's so much hatred in the project. There's people attacking others for who they are, be it women or trans or non-binary. There's people standing up to defend the attackers. Debian is just now going through another bout of that. It's sad and it's disgusting. And it reaffirms that I made the right decision getting out.</p> <p>People denying other people their humanity, their very right to exist, is something Debian should not tolerate. I think Debian should exclude people who do that from the project. Likewise, people defending the right to deny others their humanity should equally unacceptable.</p> <p>De-humanizing rhetoric isn't the only reason Debian stopped being fun. Everything else seems irrelevant, though. If people don't want others to even exist, there's no point in discussing minor points like improving a consensus building culture, paying off at least a noticeable part of the technical debt Debian carries from the past quarter century, or smoothing away some of the worst sources of friction in the development process of the project.</p> <p>Stop the hatred. The good will follow.</p> Free software development doesn't have to be awful https://blog.liw.fi/posts/2019/11/03/free_software_development_doesn_t_have_to_awful/ https://blog.liw.fi/posts/2019/11/03/free_software_development_doesn_t_have_to_awful/ debian gnu governance rms software-freedom tolerance Sun, 03 Nov 2019 09:43:00 +0000 2019-11-25T14:27:02Z <p>I want to develop free software with people who lift up each other, and aren't arseholes.</p> <p>A year ago I left Debian. The process is called retiring in Debian, and it's not final: if you do it in an orderly manner, you can come back again, and be re-instated as a Debian developer with a faster, more lightweight process than is used for entirely new developers. This was the third time I retired. The reasons were different than previously.</p> <p>The first two times I retired because I was pursuing other passions, and did not feel I could give Debian even the minimal attention and effort required to keep a few minor, leaf packages maintained. This time, I retired because Debian was not fun; it was in fact becoming awful, and I didn't want to participate anymore.</p> <p>Debian had stopped being fun for me in several ways. One was that the tools, file formats, and workflows Debian uses are getting a little archaic, and generally sub-optimal. Even mundane, everyday tasks involved much more friction than they should. Another is that making any large changes in Debian is too much of an effort, these days, partly because of inertia, partly because it involves so many people.</p> <p>All of that could have been tolerable, if not for the people. Some of the nicest, most competent people I know work on Debian. It has been a privilege and a joy to work with them.</p> <p>A few of the other people in Debian I don't want to be associated with in any way, any more.</p> <p>Debian has some vocal people who treat other people in ways that I don't want to accept. I don't want to go into specifics, or names, because that's not going help me move forward.</p> <p>This is of course not a new thing. Debian has had problems with people behaving badly for years. I may have contributed to that, passively if not actively. However, as I get older, the friction from dealing with abrasive people is sanding off what thick skin I may have had when younger.</p> <p>As I get older, I am also learning that some of the things I thought were OK when I was younger, are in fact harmful to other people. I don't want to harm other people, and I don't want to participate in a project where some of the people insist on what I think is harmful behaviour, because they feel it's their right.</p> <p>Long after I left Debian, RMS managed to collapse the reality distortion field that's been surrounding and protecting him for many years. The triggering event for this was comments he made in a context involving Jeffrey Epstein. The comments caused a public uproar, and as a result RMS resigned from role of president of the Free Software Foundation, which he founded. He is currently still the leader of the GNU project. A lot of people are religously defending RMS and attacking his detractors. I find this to be problematic.</p> <p><a href="https://lwn.net/Articles/802985/">LWN</a> has an excellent article on the topic. RMS has been behaving in problematic ways for a long time. He's not been publicly confronted about it before, at the scale he has been now.</p> <p>RMS has done some awesome things that he should be honoured for. He started the GNU project and gave it, and the world, a vision of being able to use computers whose entire software stack is free, and inventing copyleft, a legal tool to protect software freedom, as well as writing large amounts of the initial code in the GNU project. He has worked hard and long to help drive the vision of freedom into reality. For this, he shall always be remembered and revered.</p> <p>That doesn't excuse bad behaviour, such as insisting on abortion jokes, making women feel unwelcome in the GNU project, or various other things. I'm not going to make a list of his shortcomings, because this isn't a critique of RMS specifically. The problem I want to discuss isn't RMS or his personal behaviour.</p> <p>The problem I do want to discuss is that almost everywhere in the free and open source development communities there's a lot of harmful behavour, and tolerance of it.</p> <p>Harmful behaviour comes in many forms. Some people, for example, say outright that they don't want women involved in free software development. Others attack gay, lesbian, trans, queer, black, old, young, Christian, Muslim, atheist, any other other group of people identified by whatever attribute the attacker happens to dislike. Yet others are more subtle, not attacking directly, but not giving people in the group they dislike the same chance to participate, learn, grow, and generally be the best person they can be in the context of free software development.</p> <p>This doesn't just harm the groups of people being targeted. It harms others, who see it happen, and think they might be targeted too, later, maybe for some other reason. It harms reaching the vision of software freedom, because it shoves large parts of humanity outside the software freedom movement, robbing the movement from many voices and much effort. This makes it harder to achieve the vision.</p> <p>Excluding people from the movement for irrelevant reasons also harms humanity in general. It propagates the hate, hurt, and harm that is emblematic of life and politics around the world. While the software freedom movement can't solve all of those problems, we can and should at least not make it worse.</p> <p>What should we in the software freedom movement do about all this? I've come to a few conclusions so far, though my process to think about this is ongoing.</p> <ul> <li><p>Most importantly, we need to stop being tolerant of intolerance and bad behaviour. It's time for all project, groups, and organisations in the movement to have and enforce at least a minimal level of civil behaviour. We are a movement consisting of many communities, and each community may want or need their own norms, and that's OK. Some norms may even be in conflict. That's also OK, if unfortunate.</p> <p>Some people react to this kind of suggestion with hyperbolic claims and conspiracy theories. I don't want to debate them. It's possible to discuss community norms in a civil and constructive way. I know this, because I've seen it happen many times. However, it requires all participants to at least agree that there's behaviour that's unwelcome, and not reject the notion of community norms outright.</p></li> <li><p>I am by nature averse to conflicts. I will try to confront bad behaviour in the future, rather than slinking away and going elsewhere. I will at least speak out and say I think something is unacceptable, when I see it.</p></li> <li><p>I think the era for dictatorial models of governance for large free software projects is over. For small projects, it's unavoidable, because there's only one person doing any development, but when a project grows, it doesn't work to have one person, or a small anointed group, making all decisions. It's time to have more democratic governance.</p> <p>There are technical decisions that probably can't be done well by letting everyone vote on them. However, every substantial project will have other decisions that the whole community around the project should have a say in. Voting on how to fix a bug may not be workable, but voting on the minimum criteria for determining if a bug fix is acceptable is. Should the project require adding a regression test for any bug found in production? Should any such test and bug fix be subjected to code review? Should such bugs and their fixes be documented, even announced, publicly or kept secret? How should security problems be handled?</p> <p>In Debian, one of the guiding principles is that those who do, decide. It seems time to involve those who use in the decision making process as well.</p></li> </ul> <p>I can't force all communities in the software freedom movement to agree with me. Obviously not. I won't even try. I will, however, in the future be wary of joining dictatorial projects where bad behaviour is tolerated. I'm hoping this will have at least some effect.</p> <p>What about you?</p> <p>(My blog does not have comments, but you can respond to this <a href="https://en.wikipedia.org/wiki/Fediverse">fediverse</a> thread: <a href="https://toot.liw.fi/@liw/103080486083100970">https://toot.liw.fi/@liw/103080486083100970</a>.)</p> Debian and the SKS signature flooding attack https://blog.liw.fi/posts/2019/07/02/debian_and_the_sks_signature_flooding_attack/ https://blog.liw.fi/posts/2019/07/02/debian_and_the_sks_signature_flooding_attack/ debian keyserver pgp Tue, 02 Jul 2019 17:18:00 +0000 2019-07-02T14:32:34Z <p>Debian uses PGP keys of its uploading members to authenticate package uploads. The keys are stored in a keyring maintained by the keyring-maint team. The team also maintains a PGP keyserver, and new keys and signatures are uploaded there. The Debian keyserver gets new signatures from other keyservers, but only accepts signatures by keys already in the Debian keyring. New keys are accepted into the keyring manually, after a vetting process of the person. Updates from the Debian keyserver to the Debian keyring happen manually, roughly once a month. Only package uploads signed by keys in the Debian keyring get accepted by the Debian package archive.</p> <p>The Debian package archive is signed by another key, securely maintained by the Debian FTP team. A copy of the public key is installed on every Debian machine, and updates to the key, as well as new keys, are distributed via package updates. This happens routinely in Debian.</p> <p>Because of these reasons, Debian seems immune to the current attack on the SKS keyserver network, since Debian isn't getting those signatures, and the Debian package archive and updates to machines running Debian aren't getting the fraudulent signatures either.</p> <p>(I've checked the above text with the keyring-maint team and they are OK with it.)</p> Retiring from Debian https://blog.liw.fi/posts/2018/11/18/retiring_from_debian/ https://blog.liw.fi/posts/2018/11/18/retiring_from_debian/ debian retiring Sun, 18 Nov 2018 18:32:00 +0000 2018-11-18T16:59:31Z <p>I've started the process of retiring from Debian. Again. This will be my third time. It'll take a little while I take care of things to do this cleanly: uploading packages to set Maintainer to QA, removing myself from Plant Debian, sending the retirement email to -private, etc.</p> <p>I've had a rough year, and Debian has also stopped being fun for me. There's a number of Debian people saying and doing things that I find disagreeable, and the process of developing Debian is not nearly as nice as it could be. There's way too much friction pretty much everywhere.</p> <p>For example, when a package maintainer uploads a package, the package goes into an upload queue. The upload queue gets processed every few minutes, and the packages get moved into an incoming queue. The incoming queue gets processed every fifteen minutes, and packages get imported into the master archive. Changes to the master archive get pushed to main mirrors every six hours. Websites like lintian.debian.org, the package tracker, and the Ultimate Debian Database get updated at time. (Or their updates get triggered, but it might take longer for the update to actually happen. Who knows. There's almost no transparency.)</p> <p>The developer gets notified, by email, when the upload queue gets processed, and when the incoming queue gets processed. If they want to see current status on the websites (to see if the upload fixed a problem, for example), they may have to wait for many more hours, possibly even a couple of days.</p> <p>This was fine in the 1990s. It's not fine anymore.</p> <p>That's not why I'm retiring. I'm just tired. I'm tired of dragging myself through high-friction Debian processes to do anything. I'm tired of people who should know better tearing open old wounds. I'm tired of all the unconstructive and aggressive whinging, from Debian contributors and users alike. I'm tired of trying to make things better and running into walls of negativity. (I realise I'm not being the most constructive with this blog post and with my retirement. I'm tired.)</p> <p>I wish everyone else a good time making Debian better, however. Or whatever else they may be doing. I'll probably be back. I always have been, when I've retired before.</p> Idea for a Debian QA service: monitoring install size with dependencies https://blog.liw.fi/posts/2018/10/24/idea_for_a_debian_qa_service_monitoring_install_size_with_dependencies/ https://blog.liw.fi/posts/2018/10/24/idea_for_a_debian_qa_service_monitoring_install_size_with_dependencies/ debian Wed, 24 Oct 2018 10:42:00 +0000 2018-10-24T07:55:50Z <p>This is an idea. I don't have the time to work on it myself, but I thought I'd throw it out in case someone else finds it interesting.</p> <p>When you install a Debian package, it pulls in its dependencies and recommended packages, and those pull in theirs. For simple cases, this is all fine, but sometimes there's surprises. Installing mutt to a base system pulls in libgpgme, which pulls in gnupg, which pulls in a pinentry package, which can pull in all of GNOME. Or at least people claim that.</p> <p>It strikes me that it'd be cool for someone to implement a QA service for Debian that measures, for each package, how much installing it adds to the system. It should probably do this in various scenarios:</p> <ul> <li>A base system, i.e., the output of debootstrap.</li> <li>A build system, with build-essentian installed.</li> <li>A base GNOME system, with gnome-core installed.</li> <li>A full GNOME system, with gnome installed.</li> <li>Similarly for KDE and each other desktop environment in Debian.</li> </ul> <p>The service would do the installs regularly (daily?), and produce reports. It would also do alerts, such as notify the maintainers when installed size grows too large compared to installing it in stable, or a previous run in unstable. For example, if installing mutt suddenly installs 100 gigabytes more than yesterday, it's probably a good idea to alert interested parties.</p> <p>Implementing this should be fairly easy, since the actual test is just running debootstrap, and possibly apt-get install. Some experimentation with configuration, caching, and eatmydata may be useful to gain speed. Possibly actual package installation can be skipped, and the whole thing could be implemented just by analysing package metadata.</p> <p>Maybe it even exists, and I just don't know about it. That'd be cool, too.</p> On flatpak, snap, distros, and software distribution https://blog.liw.fi/posts/2018/10/11/on_flatpak_snap_distros_and_software_distribution/ https://blog.liw.fi/posts/2018/10/11/on_flatpak_snap_distros_and_software_distribution/ debian distribution flatpak Thu, 11 Oct 2018 10:12:00 +0000 2018-10-11T08:24:01Z <p>I don't think any of <a href="https://flatpak.org/">Flatpak</a>, <a href="https://en.wikipedia.org/wiki/Snappy_(package_manager)">Snappy</a>, traditional Linux distros, non-traditional Linux distros, containers, online services, or other forms of software distribution are a good solution for all users. They all fail in some way, and each of them requires continued, ongoing effort to be acceptable even within their limitations.</p> <p>This week, there's been some discussion about Flatpak, a software distribution approach that's (mostly) independent of traditional Linux distributions. There's also, Snappy, which is <a href="https://www.canonical.com/">Canonical</a>'s similar thing.</p> <p>The discussion started with the launch of a new website attacking Flatpak as a technology. I'm not going to link to it, since it's an anonymous attack and rant, and not constructive. I'd rather have a constructive discussion. I'm also not going to link to rebuttals, and will just present my own view, which I hope is different enough to be interesting.</p> <p>The website raises the issue that Flatpak's sandboxing is not as good as it should be. This seems to be true. Some of Flatpak's defenders respond that it's an evolving technology, which seems fair. It's not necessary to be perfect; it's important to be better than what came before, and to constantly improve.</p> <p>The website also raises the point that a number of flatpaks themselves contain unfixes security problems. I find this to be more worrying than an imperfect sandbox. A security problem inside a perfect sandbox can still be catastrophic: it can leak sensitive data, join a distributed denial of service attack, use excessive CPU and power, and otherwise cause mayhem. The sandbox may help in containing the problem somewhat, but to be useful for valid use, the sandbox needs to allow things that can be used maliciously.</p> <p>As a user, I want software that's...</p> <ul> <li>easy to install and update</li> <li>secure to install (what I install is what the developers delivered)</li> <li>always up to date with security fixes, including for any dependencies (embedded in the software or otherwise)</li> <li>reasonably up to date with other bug fixes</li> <li>sufficiently up to date with features I want (but I don't care a about newer features that I don't have a use for)</li> <li>protective of my freedoms and privacy and other human rights, which includes (but is not restricted to) being able to self-host services and work offline</li> </ul> <p>As a software developer, I additionally want my own software to be...</p> <ul> <li>effortless to build</li> <li>automatically tested in a way that gives me confidence it works for my users</li> <li>easy to deliver to my users</li> <li>easy to debug</li> <li>not be broken by changes to build and runtime dependencies, or at least make such changes be extremely obvious, meaning they result in a build error or at least an error during automated tests</li> </ul> <p>These are requirements that are hard to satisfy. They require a lot of manual effort, and discipline, and I fear the current state of software development isn't quite there yet. As an example, the Linux kernel development takes great care to never break userland, but that requires a lot of care when making changes, a lot of review, and a lot of testing, and a willingness to go to extremes to achieve that. As a result, upgrading to a newer kernel version tends to be a low-risk operation. The glibc C library, used by most Linux distributions, has a similar track record.</p> <p>But Linux and glibc are system software. Flatpak is about desktop software. Consider instead LibreOffice, the office suite. There's no reason why it couldn't be delivered to users as a Flatpak (and indeed it is). It's a huge piece of software, and it needs a very large number of libraries and other dependencies to work. These need to be provided inside the LibreOffice Flatpak, or by one or more of the Flatpak "runtimes", which are bundles of common dependencies. Making sure all of the dependencies are up to date can be partly automated, but not fully: someone, somewhere, needs to make the decision that a newer version is worth upgrading to right now, even if it requires changes in LibreOffice for the newer version to work.</p> <p>For example, imagine LO uses a library to generate PDFs. A new version of the library reduces CPU consumption by 10%, but requires changes, because the library's API (programming interface) has changed radically. The API changes are necessary to allow the speedup. Should LibreOffice upgrade to the new version of not? If 10% isn't enough of a speedup to warrant the effort to make the LO changes, is 90%? An automated system could upgrade the library, but that would then break the LO build, resulting in something that doesn't work anymore.</p> <p>Security updates are easier, since they usually don't involve API changes. An automated system could upgrade dependencies for security updates, and then trigger automated build, test, and publish of a new Flatpak. However, this is made difficult by there is often no way to automatically, reliably find out that there is a security fix released. Again, manual work is required to find the security problem, to fix it, to communicate that there is a fix, and to upgrade the dependency. Some projects have partial solutions for that, but there seems to be nothing universal.</p> <p>I'm sure most of this can be solved, some day, in some manner. It's definitely an interesting problem area. I don't have a solution, but I do think it's much too simplistic to say "Flatpaks will solve everything", or "the distro approach is best", or "just use the cloud".</p> Building Debian packages in CI (ick) https://blog.liw.fi/posts/2018/07/19/building_debian_packages_in_ci_ick/ https://blog.liw.fi/posts/2018/07/19/building_debian_packages_in_ci_ick/ debian ick Thu, 19 Jul 2018 18:57:00 +0000 2018-07-19T15:58:04Z <p>I've recently made the first release of <a href="https://ick.liw.fi/">ick</a>, my CI engine, which was built by ick itself. It went OK, but the process needs improvement. This blog post is some pondering on how the process of building Debian packages should happen in the best possible taste.</p> <p>I'd appreciate feedback, preferably by email (liw@liw.fi).</p> <h1>Context</h1> <p>I develop a number of (fairly small) programs, as a hobby. Some of them I also maintain as packages in Debian. All of them I publish as Debian packages in my own APT repository. I want to make the process for making a release of any of my programs as easy and automated as possible, and that includes building Debian packages and uploading them to my personal APT repository, and to Debian itself.</p> <p>My personal APT repository contains builds of my programs against several Debian releases, because I want people to have the latest version of my stuff regardless of what version of Debian they run. (This is somewhat similar to what OEMs that provide packages of their own software as Debian packages need to do. I think. I'm not an OEM and I'm extrapolating wildly here.)</p> <p>I currently don't provide packages for anything but Debian. That's mostly because Debian is the only Linux distribution I know well, or use, or know how to make packages for. I could do Ubuntu builds fairly easily, but supporting Fedora, RHEL, Suse, Arch, Gentoo, etc, is not something I have the energy for at this time. I would appreciate help in doing that, however.</p> <p>I currently don't provide Debian packages for anything other than the AMD64 (x86-64, "Intel 64-bit") architecture. I've previously provided packages for i386 (x86-32), and may in the future want to provide packages for other architectures (RISC-V, various Arm variants, and possibly more). I want to keep this in mind for this discussion.</p> <h1>Overview</h1> <p>For the context of this blog post, let's assume I have a project Foo. Its source code is stored in <code>foo.git</code>. When I make a release, I tag it using a signed git tag. From this tag, I want to build several things:</p> <ul> <li><p>A <strong>release tarball</strong>. I will publish and archive this. I don't trust git, and related tools (tar, compression programs, etc) to be able to reproducibly produce the same bit-by-bit compressed tarball in perpetuity. There's too many things that can go wrong. For security reasons it's important to be able to have the exact same tarball in the future as today. The simplest way to achive this is to not try to reproduce, but to archive.</p></li> <li><p>A <strong>Debian source package</strong>.</p></li> <li><p>A <strong>Debian binary package</strong> built for each target version of Debian, and each target hardware architecture (CPU, ABI, possibly toolchain version). The binary package should be built from the source package, because otherwise we don't know the source package can be built.</p></li> </ul> <p>The release tarball should be put in a (public) archive. A digital signature using my personal PGP key should also be provided.</p> <p>The Debian source and binary packages should be uploaded to one or more APT repositories: my personal one, and selected packages also the Debian one. For uploading to Debian, the packages will need to be signed with my personal PGP key.</p> <p>(I am not going to give my CI access to my PGP key. Anything that needs to be signed with my own PGP key needs to be a manual step.)</p> <h2>Package versioning</h2> <p>In Debian, packages are uploaded to the "unstable" section of the package archive, and then automatically copied into the "testing" section, and from there to the "stable" section, unless there are problems in a specific version of a package. Thus all binary packages are built against unstable, using versions of build dependencies in unstable. The process of copying via testing to stable can take years, and is a core part of how Debian achieves quality in its releases. (This is simplified and skips consideration like security updates and other updates directly to stable, which bypass unstable. These details are not relevant to this discussion, I think.)</p> <p>In my personal APT repository, no such copying takes place. A package built for unstable does not get copied into section with packages built for a released version of Debian, when Debian makes a release.</p> <p>Thus, for my personal APT repository, there may be several builds of the any one version of Foo available.</p> <ul> <li>foo 1.2, built for unstable</li> <li>foo 1.2, built for Debian 9</li> <li>foo 1.2, built for Debian 8</li> </ul> <p>In the future, that list may be expanded by having builds for several architectures:</p> <ul> <li>foo 1.2, built for unstable, on amd64</li> <li>foo 1.2, built for Debian 9, on amd64</li> <li><p>foo 1.2, built for Debian 8, on amd64</p></li> <li><p>foo 1.2, built for unstable, on riscv</p></li> <li>foo 1.2, built for Debian 9, on riscv</li> <li>foo 1.2, built for Debian 8, on riscv</li> </ul> <p>When I or my users upgrade our Debian hosts, say from Debian 8 to Debian 9, any packges from my personal APT archive should be updated accordingly. When I upgrade a host running Debian 8, with foo 1.2 built for Debian 8, gets upgraded to Debian 9, foo should be upgraded to the version of 1.2 built for Debian 9.</p> <p>Because the Debian package manager works on combinations of package name and package version, that means that the version built for Debian 8 should have a different, and lesser, version than the one built for Debian 9, even if the source code is identical except for the version number. The easiest way to achieve this is probably to build a different source package for each target Debian release. That source package has no other differences than the debian/changelog entry with a new version number, so it doesn't necessarily need to be stored persistently.</p> <p>(This is effectively what Debians "binary NMU" uploads do: use the same source package version, but do a build varying only the version number. Debian does this, among other reasons, to force a re-build of a package using a new version of a build depenency, for which it is unnecessary to do a whole new sourceful upload. For my CI build purposes, it may be useful to have a new source package, for cases where there are other changes than the source package. This will need further thought and research.)</p> <p>Thus, I need to produce the following source and binary packages:</p> <ul> <li><code>foo_1.2-1.dsc</code> &mdash; source package for unstable</li> <li><code>foo_1.2-1.orig.tar.xz</code> &mdash; upstream tarball</li> <li><code>foo_1.2-1.debian.tar.xz</code> &mdash; Debian packaging and changes</li> <li><code>foo_1.2-1_amd64.deb</code> &mdash; binary package for unstable, amd64</li> <li><p><code>foo_1.2-1_riscv.deb</code> &mdash; binary package for unstable, riscv</p></li> <li><p><code>foo_1.2-1~debian8.dsc</code> &mdash; source package for Debian 8</p></li> <li><code>foo_1.2-1~debian8.debian.tar.xz</code> &mdash; Debian packaging and changes</li> <li><code>foo_1.2-1~debian8_amd64.deb</code> &mdash; binary package for Debian 8, amd64</li> <li><p><code>foo_1.2-1~debian8_riscv.deb</code> &mdash; binary package for Debian 8, riscv</p></li> <li><p><code>foo_1.2-1~debian9.dsc</code> &mdash; source package for Debian 9</p></li> <li><code>foo_1.2-1~debian9.debian.tar.xz</code> &mdash; Debian packaging and changes</li> <li><code>foo_1.2-1~debian9_amd64.deb</code> &mdash; binary package for Debian 9, amd64</li> <li><code>foo_1.2-1~debian9_riscv.deb</code> &mdash; binary package for Debian 9, riscv</li> </ul> <p>The <code>orig.tar.xz</code> file is a bit-by-bit copy of the upstream release tarball. The <code>debian.tar.xz</code> files have the Debian packaging files, plus any Debian specific changes. (For simplicity, I'm assuming a specific Debian source package format. The actual list of files may vary, but the <code>.dsc</code> file is crucial, and references the other files in the source package. Again, these details don't really matter for this discussion.)</p> <p>To upload to Debian, I would upload the <code>foo_1.2-1.dsc</code> source package from the list above, after downloading the files and signing them with my PGP key. To upload to my personal APT repository, I would upload all of them.</p> <h1>Where should Debian packaging be stored in version control?</h1> <p>There seems to be no strong consensus in Debian about where the packaging files (the debian/ subdirectory and its contents) should be stored in version control. Several approaches are common. The examples below use git as the version control system, as it's clearly the most common one now.</p> <ul> <li><p>The "upstream does the packaging" approach: upstream's <code>foo.git</code> also contains the Debian packaging. Packages are built using that. This seems to be especially common for programs, where upstream and the Debian package maintainer are the same entity. That's also the OEM model.</p></li> <li><p>The "clone upstream and add packaging" approach: the Debian package maintainer clonse the upstream repository, and adds the packaging files in a separate branch. When upstream makes a release, the master branch in the packaging repository is updated to match the upstream's master branch, and the packaging branch is rebased on top of that.</p></li> <li><p>The "keep it separate" approach: the Debian packager puts the packaging files in their own repository, and the source tree is constructed from botht the upstream repository and the packaging repository.</p></li> </ul> <p>For my own use, I prefer the "upstream does packaging" approach, as it's the least amount of friction for me. For ick, I want to support any approach.</p> <p>There are various tools for maintaining package source in git (e.g., dgit and git-buildpackage), but those seem to not be relevant to this blog post, so I'm not discussing them in any detail.</p> <h1>The build process</h1> <p>Everything starts from a signed git tag in the <code>foo.git</code> plus additional tags in any packaging repository. The tags are made by the upstream developers and Debian package maintainers. CI will notice the new tag, and build a release from that.</p> <ul> <li><p>Create the upstream tarball (<code>foo-1.2.tar.gz</code>).</p></li> <li><p>Manully download and sign the upstream tarball with PGP.</p></li> <li><p>Manully publish the upstream tarball and its signature in a suitable place.</p></li> <li><p>Create the Debian source package for unstable (<code>foo_1.2-1.dsc</code>), using a copy of the upstream tarball, renamed.</p></li> <li><p>Using the Debian source package, build a Debian binary package for unstable for each target architecture (<code>foo_1.2-1_amd64.deb</code> etc).</p></li> <li><p>For each target Debian release other than unstable, create a new source package by unpacking the source package for unstable, and adding a <code>debian/changelog</code> entry with <code>~debianN</code> appended to the version number. If there is a need, make any additional Debian release specific changes to the source package.</p></li> <li><p>Build each of those source packages for each target architecture, in a build environment with the target Debian release. (<code>foo_1.2-1~debianN_amd64.deb</code> etc).</p></li> <li><p>Upload all the Debian source and binary packages to an APT repository that allows upload by CI. Have that APT repository sign the resulting Packages file with its own PGP key.</p></li> <li><p>Manully download the Debian packages and sign the unstable build to Debian, and upload it to Debian. (Source package only, except in cases where the binary package also needs to be uploaded, such as for new packages.)</p></li> </ul> dpkg maintainer script containerisation https://blog.liw.fi/posts/2018/03/05/dpkg_maintainer_script_containerisation/ https://blog.liw.fi/posts/2018/03/05/dpkg_maintainer_script_containerisation/ debian Mon, 05 Mar 2018 12:58:00 +0000 2018-03-05T11:14:13Z <p>Random crazy Debian idea of today: add support to dpkg so that it uses containers (or namespaces, or whatever works for this) for running package maintainer scripts (pre- and postinst, pre- and postrm), to prevent them from accidentally or maliciously writing to unwanted parts of the filesystem, or from doing unwanted network I/O.</p> <p>I think this would be useful for third-party packages, but also for packages from Debian itself. You heard it here first! Debian package maintainers have been known to make mistakes.</p> <p>Obviously there needs to be ways in which these restrictions can be overridden, but that override should be clear and obvious to the user (sysadmin), not something they notice because they happen to be running strace or tcpdump during the install.</p> <p>Corollary: dpkg could restrict where a .deb can place files based on the origin of the package.</p> <p>Example: Installing chrome.deb from Google installs a file in <code>/etc/apt/sources.list.d</code>, which is a surprise to some. If dpkg were to not allow that (as a file in the .deb, or a file created in postinst), unless the user was told and explicitly agreed to it, it would be less of a nasty surprise.</p> <p>Example: Some <a href="https://contributors.debian.org/contributor/lars@debian/">stupid</a> Debian package maintainer is very busy at work and does Debian hacking when they should really be sleeping, and types the following into their postrm script, while being asleep:</p> <pre><code>#!/bin/sh PKG=perfectbackup LIB="/var/lib/ $PKG" rm -rf "$LIB" </code></pre> <p>See the mistake? Ideally, this would be found during automated testing <em>before</em> the package gets uploaded, but that assumes said package maintainer uses tools like piuparts.</p> <p>I think it'd be better if we didn't rely only infallible, indefatigable people with perfect workflows and processes for safety.</p> <p>Having dpkg make the whole filesystem read-only, except for the parts that clearly belong to the package, based on some sensible set of rules, or based a suitable override, would protect against mistakes like this.</p> What is Debian all about, really? Or: friction, packaging complex applications https://blog.liw.fi/posts/2018/02/17/what_is_debian_all_about_really_or_friction_packaging_complex_applications/ https://blog.liw.fi/posts/2018/02/17/what_is_debian_all_about_really_or_friction_packaging_complex_applications/ debian opinion Sat, 17 Feb 2018 15:05:00 +0000 2018-02-17T17:13:18Z <h1>Another weekend, another big mailing list thread</h1> <p>This weekend, those interested in Debian development have been having a <a href="https://lists.debian.org/debian-devel/2018/02/msg00295.html">discussion</a> on the debian-devel mailing list about "What can Debian do to provide complex applications to its users?". I'm commenting on that in my blog rather than the mailing list, since this got a bit too long to be usefully done in an email.</p> <p>directhex's recent <a href="https://apebox.org/wordpress/linux/1229">blog post</a> "Packaging is hard. Packager-friendly is harder." is also relevant.</p> <h1>The problem</h1> <p>To start with, I don't think the email that started this discussion poses the right question. The problem not really about complex applications, we already have those in Debian. See, for example, LibreOffice. The discussion is really about how Debian should deal with the way some types of applications are developed upstream these days. They're not all complex, and they're not all big, but as usual, things only get interesting when <em>n</em> is big.</p> <p>A particularly clear example is the whole nodejs ecosystem, but it's not limited to that and it's not limited to web applications. This is also not the first time this topic arises, but we've never come to any good conclusion.</p> <p>My understanding of the problem is as follows:</p> <blockquote><p>A current trend in software development is to use programming languages, often interpreted high level languages, combined with heavy use of third-party libraries, and a language-specific package manager for installing libraries for the developer to use, and sometimes also for the sysadmin installing the software for production to use. This bypasses the Linux distributions entirely. The benefit is that it has allowed ecosystems for specific programming languages where there is very little <em>friction</em> for using libraries written in that language to be used by developers, speeding up development cycles a lot.</p></blockquote> <h1>When I was young(er) the world was horrible</h1> <p>In comparison, in the old days, which for me means the 1990s, and before Debian took over my computing life, the cycle was something like this:</p> <blockquote><p>I would be writing an application, and would need to use a library to make some part of my application easier to write. To use that library, I would download the source code archive of the latest release, and laboriously decipher and follow the build and installation instructions, fix any problems, rinse, repeat. After getting the library installed, I would get back to developing my application. Often the installation of the dependency would take hours, so not a thing to be undertaken lightly.</p></blockquote> <h1>Debian made some things better</h1> <p>With Debian, and apt, and having access to hundreds upon hundreds of libraries packaged for Debian, this become a much easier process. But only for the things packaged for Debian.</p> <p>For those developing and publishing libraries, Debian didn't make the process any easier. They would still have to publish a source code archive, but also hope that it would eventually be included in Debian. And updates to libraries in the Debian stable release would not get into the hands of users until the next Debian stable release. This is a lot of friction. For C libraries, that friction has traditionally been tolerable. The effort of making the library in the first place is considerable, so any friction added by Debian is small by comparison.</p> <h1>The world has changed around Debian</h1> <p>In the modern world, developing a new library is much easier, and so also the friction caused by Debian is much more of a hindrance. My understanding is that things now happen more like this:</p> <blockquote><p>I'm developing an application. I realise I could use a library. I run the language-specific package manager (pip, cpan, gem, npm, cargo, etc), it downloads the library, installs it in my home directory or my application source tree, and in less than the time it takes to have sip of tea, I can get back to developing my application.</p></blockquote> <p>This has a lot less friction than the Debian route. The attraction to application programmers is clear. For library authors, the process is also much streamlined. Writing a library, especially in a high-level language, is fairly easy, and publishing it for others to use is quick and simple. This can lead to a virtuous cycle where I write a useful little library, you use and tell me about a bug or a missing feature, I add it, publish the new version, you use it, and we're both happy as can be. Where this might have taken weeks or months in the old days, it can now happen in minutes.</p> <h1>The big question: why Debian?</h1> <p>In this brave new world, <strong>why would anyone bother with Debian anymore?</strong> Or any traditional Linux distribution, since this isn't particularly specific to Debian. (But I mention Debian specifically, since it's what I now best.)</p> <p>A number of things have been mentioned or alluded to in the <a href="https://lists.debian.org/debian-devel/2018/02/msg00295.html">discussion</a> mentioned above, but I think it's good for the discussion to be explicit about them. As a computer user, software developer, system administrator, and software freedom <em>enthusiast</em>, I see the following reasons to continue to use Debian:</p> <ul> <li><p>The freeness of software included in Debian has been vetted. I have a <strong>strong guarantee</strong> that software included in Debian is free software. This goes beyond the licence of that particular piece of software, but includes practical considerations like the software can actually be built using free tooling, and that I have access to that tooling, because the tooling, too, is included in Debian.</p> <ul> <li><p>There was a time when Debian debated (with itself) whether it was OK to include a binary that needed to be built using a proprietary C compiler. We decided that it isn't, or not in the main package archive.</p></li> <li><p>These days we have the question of whether "minimised Javascript" is OK to be included in Debian, if it can't be produced using tools packaged in Debian. My understanding is that we have already decided that it's not, but the discussion continues. To me, this seems equivalent to the above case.</p></li> </ul> </li> <li><p>I have a <strong>strong guarantee</strong> that software in a stable Debian release won't change underneath me in incompatible ways, except in special circumstances. This means that if I'm writing my application and targeting Debian stable, the library API won't change, at least not until the next Debian stable release. Likewise for every other bit of software I use. Having things to continue to work without having to worry is a good thing.</p> <ul> <li>Note that a side-effect of the low friction of library development current ecosystems sometimes results in the library API changing. This would mean my application would need to change to adapt to the API change. That's friction for my work.</li> </ul> </li> <li><p>I have a <strong>strong guarantee</strong> that a dependency won't just disappear. Debian has a large mirror network of its package archive, and there are easy tools to run my own mirror, if I want to. While running my own mirror is possible for other package management systems, each one adds to the friction.</p> <ul> <li><p>The nodejs NPM ecosystem seems to be especially vulnerable to this. More than once packages have gone missing, resulting other projects, which depend on the missing packages, to start failing.</p></li> <li><p>The way the Debian project is organised, it is almost impossible for this to happen in Debian. Not only are package removals carefully co-ordinated, packages that are depended on on by other packages aren't removed.</p></li> </ul> </li> <li><p>I have a <strong>strong guarantee</strong> that a Debian package I get from a Debian mirror is the official package from Debian: either the actual package uploaded by a Debian developer or a binary package built by a trusted Debian build server. This is because Debian uses cryptographic signatures of the package lists and I have a trust path to the Debian signing key.</p> <ul> <li><p>At least some of the language specific package managers fail to have such a trust path. This means that I have no guarantees that the library package I download today, was the same code uploaded by library author.</p></li> <li><p>Note that https does not help here. It protects the transfer from the package manger's web server to me, but makes absolutely no guarantees about the validity of the package. There's been enough cases of the package repository having been attacked that this matters to me. Debian's signatures protect against malicious changes on mirror hosts.</p></li> </ul> </li> <li><p>I have a <strong>reasonably strong guarantee</strong> that any problem I find can be fixed, by me or someone else. This is not a strong guarantee, because Debian can't do anything about insanely complicated code, for example, but at least I can rely on being able to rebuild the software. That's a basic requirement for fixing a bug.</p></li> <li><p>I have a <strong>reasonably strong guarantee</strong> that, after upgrading to the next Debian stable release, my stuff continues to work. Upgrades may always break, but at least Debian tests them and treats it as a bug if an upgrade doesn't work, or loses user data.</p></li> </ul> <p>These are the reasons why I think Debian and the way it packages and distributes software is still important and relevant. (You may disagree. I'm OK with that.)</p> <h1>What about non-Linux free operating systems</h1> <p>I don't have much personal experience with non-Linux systems, so I've only talked about Linux here. I don't think the BSD systems, for example, are actually all that different from Linux distributions. Feel free to substitute "free operating system" for "Linux" throughout.</p> <h1>What is it Debian tries to do, anyway?</h1> <p>The previous section is one level of abstraction too low. It's important, but it's beneficial take a further step back and consider what it is Debian actually tries to achieve. Why does Debian exist?</p> <blockquote><p>The <strong>primary goal</strong> of Debian is to enable its users to use their computers using only free software. The freedom aspect is fundamentally important and a principle that Debian is not willing to compromise on.</p> <p>The primary approach to achieve this goal is to produce a "distribution" of free software, to make installing a free software operating system and applications, and to maintain such a computer, a feasible thing for our users.</p></blockquote> <p>This leads to secondary goals, such as:</p> <ul> <li><p>Making it easy to install Debian on a computer. (For values of easy that should be compared to toggling boot sector bytes manually.)</p> <p>We've achieved this, though of course things can always be improved.</p></li> <li><p>Making it easy to install applications on a computer with Debian. (Again, compared to the olden days, when that meant configuring and compiling everything from scratch, with no guidance.)</p> <p>We've achieved this, too.</p></li> <li><p>A system with Debian installed is reasonably secure, and easy to keep reasonably secure.</p> <p>This means Debian will provide security support for software it distributes, and has ways in which to install security fixes. We've achieved this, though this, too, can always be improved.</p></li> <li><p>A system with Debian installed should keep working for extended periods of time. This is important to make using Debian feasible. If it takes too much effort to have a computer running Debian, it's not feasible for many people to that, and then Debian fails its primary goal.</p> <p>This is why Debian has stable releases with years of security support. We've achieved this.</p></li> </ul> <h1>The disconnect</h1> <p>On the one hand, we have Debian, which pretty much has achieved what I declare to be its primary goal. On the other hand, a lot of developers now expect much less friction than what Debian offers. This is a disconnect that is cause, I believe, the debian-devel discussion, and variants of that discussion all over the open source landscape.</p> <p>These discussions often go one of two ways, depending on which community is talking.</p> <ul> <li><p>In the distribution and more old-school communities, the low-friction approach of language-specific package managers is often considered to be a horror, and an abandonment of all the good things that the Linux world has achieved. "Young saplings, who do they think they are, all agile and bendy and with no principles at all, get off our carefully cultivated lawn."</p></li> <li><p>In the low-friction communities, Linux distributions are something only old, stodgy, boring people care about. "Distributions are dead, they only get in the way, nobody bothers with them anymore."</p></li> </ul> <p>This disconnect will require effort by both sides to close the gap.</p> <p>On the one hand, so much new software is being written by people using the low-friction approach, that Linux distributions may fail to attract new users and especially new developers, and this will hurt them and their users.</p> <p>On the other hand, the low-friction people may be sawing the tree branch they're sitting on. If distributions suffer, the base on which low-friction development relies on, will wither away, and we'll be left with running low-friction free software on proprietary platforms.</p> <h1>Things for low-friction proponents to improve</h1> <p>Here's a few things I've noticed that go wrong in the various communities oriented towards the low-friction approach.</p> <ul> <li><p>Not enough care is given to copyright licences. This is a boring topic, but it's the legal basis that all of free software and open source is based on. If copyright licences are violated, or copyrights are not respected, or copyrights or licences are not expressed well enough, or incompatible licences are mixed, the result is very easily not actually either free software or open source.</p> <p>It's boring, but be sufficiently pedantic here. It's not even all that difficult.</p></li> <li><p>Do provide actual source. It seems quite a number of Javascript projects only distribute "minimised" versions of code. That's not actually source code, any more than, say, Java byte code is, even if a de-compiler can make it kind of editable. If source isn't available, it's not free software or open source.</p></li> <li><p>Please try to be careful with API changes. What used to work should still work with a new version of a library. If you need to make an API change that breaks compatibility, find a way to still support those who rely on the old API, using whatever mechanisms available to you. Ideally, support the old API for a long time, years. Two weeks is really not enough.</p></li> <li><p>Do be careful with your dependencies. Locking down dependencies on a specific version makes things difficult for distributions, because they often can only provide one or a very small number of versions of any one package. Likewise, avoid embedding dependencies in your own source tree, because that explodes the amount of work distributions have to do to patch security holes. (No, distributions can't rely on tends of thousands of upstream to each do the patching correctly and promptly.)</p></li> </ul> <h1>Things for Debian to improve</h1> <p>There are many sources of friction that come from Debian itself. Some of them are unavoidable: if upstream projects don't take care of copyright licence hygiene, for example, then Debian will impose that on them and that can't be helped. Other things are more avoidable, however. Here's a list off the top of my head:</p> <ul> <li><p>A lot of stuff in Debian happens over email, which might happen using a web application, if it were not for historical reasons. For example, the Debian bug tracking system (bugs.debian.org) requires using email, and given delays caused by spam filtering, this can cause delays of more than fifteen minutes. This is a source of friction that could be avoided.</p></li> <li><p>Likewise, Debian voting happens over email, which can cause friction from delays.</p></li> <li><p>Debian lets its package maintainers use any version control system, any packaging helper tooling, and packaging workflow they want. This means that every package is, to some extent, a new territory for someone other than its primary maintainers. Even when the same tools are used, they can be used in variety of different ways. Consistency should reduce friction.</p></li> <li><p>There's too little infrastructure to do things like collecting copyright information into <code>debian/control</code>. This really shouldn't be a manual task.</p></li> <li><p>Debian packaging uses arcane file formats, loosely based on email headers. More standard formats might make things easier, and reduce friction.</p></li> <li><p>There's not enough automated testing, or it's too hard to use, making it too hard to know if a new package will work, or a modified package doesn't break anything that used to work.</p></li> <li><p>Overall, making a Debian package tends to require too much manual work. Packaging helpers like <code>dh</code> certainly help, but not enough. I don't have a concrete suggestion how to reduce it, but it seems like an area Debian should work on.</p></li> <li><p>Maybe consider supporting installing multiple versions of a package, even if only for, say, Javascript libraries. Possibly with a caveat that only specific versions will be security supported, and a way to alert the sysadmin if vulnerable packages are installed. Dunno, this is a difficult one.</p></li> <li><p>Maybe consider providing something where the source package gets automatically updated to every new upstream release (or commit), with binary packages built from that, and those automatically tested. This might be a separate section of the archive, and packages would be included into the normal part of the archive only by manual decision.</p></li> <li><p>There's more, but mostly not relevant to this discussion, I think. For example, Debian is a big project, and the mere size is a cause of friction.</p></li> </ul> <h1>Comments?</h1> <p>I don't allow comments on my blog, and I don't want to debate this in private. If you have comments on anything I've said above, please post to the debian-devel mailing list. Thanks.</p> <h1>Baits</h1> <p>To ensure I get <em>some</em> responses, I will leave these bait here:</p> <p>Anyone who's been programming less than 12332 days is a young whipper-snapper and shouldn't be taken seriously.</p> <p>Depending on the latest commit of a library is too slow. The proper thing to do for really fast development is to rely on the version in the unsaved editor buffer of the library developer.</p> <p>You shouldn't have read any of this. I'm clearly a troll.</p> Debian and the GDPR https://blog.liw.fi/posts/2017/10/10/debian_and_the_gdpr/ https://blog.liw.fi/posts/2017/10/10/debian_and_the_gdpr/ debian gdpr privacy Tue, 10 Oct 2017 18:08:00 +0000 2017-10-10T15:11:50Z <p><a href="https://en.wikipedia.org/wiki/General_Data_Protection_Regulation">GDPR</a> is a new EU regulation for privacy. The name is short for "General Data Protection Regulation" and it covers all organisations that handle personal data of EU citizens and EU residents. It will become enforceable May 25, 2018 (<a href="https://en.wikipedia.org/wiki/Towel_Day">Towel Day</a>). This will affect Debian. I think it's time for Debian to start working on compliance, mainly because the GDPR requires sensible things.</p> <p>I'm not an expert on GDPR legislation, but here's my understanding of what we in Debian should do:</p> <ul> <li><p>do a privacy impact assessment, to review and <strong>document</strong> what data we have, and collect, and what risks that has for the people whose personal data it is if the data leaks</p></li> <li><p>only collect personal information for specific purposes, and only use the data for those purposes</p></li> <li><p>get explicit consent from each person for all collection and use of their personal information; archive this consent (e.g., list subscription confirmations)</p></li> <li><p>allow each person to get a copy of all the personal information we have about them, in a portable manner, and let them correct it if it's wrong</p></li> <li><p>allow people to have their personal information erased</p></li> <li><p>maybe appoint one or more data protection officers (not sure this is required for Debian)</p></li> </ul> <p>There's more, but let's start with those.</p> <p>I think Debian has at least the following systems that will need to be reviewed with regards to the GDPR:</p> <ul> <li>db.debian.org - Debian project members, "Debian developers"</li> <li>nm.debian.org</li> <li>contributors.debian.org</li> <li>lists.debian.org - <strong>at least</strong> membership lists, maybe archives</li> <li>possibly irc servers and log files</li> <li>mail server log files</li> <li>web server log files</li> <li>version control services and repositories</li> </ul> <p>There may be more; these are just off the top of my head.</p> <p>I expect that mostly Debian will be OK, but we can't just assume that.</p> Enabling TRIM/DISCARD on Debian, ext4, luks, and lvm https://blog.liw.fi/posts/2017/08/05/enabling_trim_discard_on_debian_ext4_luks_and_lvm/ https://blog.liw.fi/posts/2017/08/05/enabling_trim_discard_on_debian_ext4_luks_and_lvm/ debian discard luks ssd sysadmin Sat, 05 Aug 2017 20:58:00 +0000 2017-08-05T18:37:16Z <p>I realised recently that my laptop isn't set up to send <a href="https://en.wikipedia.org/wiki/Trim_(computing)">TRIM</a> or DISCARD commands to its SSD. That means the SSD firmware has a harder time doing garbage collection (see whe linked Wikipedia page for more details.)</p> <p>After some searching, I found two articles by <a href="https://blog.christophersmart.com/about/">Christopher Smart</a>: <a href="https://blog.christophersmart.com/2013/06/05/trim-on-lvm-on-luks-on-ssd/">one</a>, <a href="https://blog.christophersmart.com/2016/05/12/trim-on-lvm-on-luks-on-ssd-revisited/">update</a>. Those, plus some addition reading of documentation, and a little experimentation, allowed me to do this. Since the information is a bit scattered, here's the details, for Debian stretch, as much for my own memory as to make sure this is collected into one place.</p> <ul> <li>Append <code>,discard</code> to the fourth column on relevant lines in <code>/etc/crypttab</code>. For me, this means the fourth column should be <code>luks,discard</code>.</li> <li>Change in <code>/etc/lvm/lvm.conf</code> that says <code>issue_discards</code> to enable it (assign 1 instead of 0).</li> <li>Append <code>rd.luks.options=discard</code> to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> value in <code>/etc/default/grub</code></li> <li>Run <code>sudo update-grub</code></li> <li>Run <code>sudo update-initramfs -u</code></li> <li>Reboot.</li> <li>Run <code>sudo fstrim -av</code> - if this works, you're good! If it gives you errors, then you get to debug. I have no idea what I'm talking about.</li> <li>Copy <code>/usr/share/doc/util-linux/examples/fstrim.*</code> to <code>/etc/systemd/system</code> and run <code>sudo systemctl enable fstrim.timer</code>. This will tell systemd to run fstrim every week. (If you don't use systemd you'll have to adapt the systemd bits mentioned here. I've no idea how.)</li> </ul> <p>Note that it seems to be a possible information leak to TRIM encryped devices. I don't know the details, but if that bothers you, don't do it.</p> <p>I don't know of any harmful effects for enabling TRIM for everything, except the crypto bit above, so I wonder if it wouldn't make sense for the Debian installer to do this by default.</p> Dropping Yakking from Planet Debian https://blog.liw.fi/posts/2017/07/19/dropping_yakking_from_planet_debian/ https://blog.liw.fi/posts/2017/07/19/dropping_yakking_from_planet_debian/ debian yakking Wed, 19 Jul 2017 08:53:00 +0000 2017-07-19T05:54:05Z <p>A couple of people objected to having Yakking on Planet Debian, so I've removed it.</p> Debian miniconf in Cambridge https://blog.liw.fi/posts/2016/11/22/debian_miniconf_in_cambridge/ https://blog.liw.fi/posts/2016/11/22/debian_miniconf_in_cambridge/ conference debian qvarn talk vmdebootstrap Tue, 22 Nov 2016 19:08:00 +0000 2016-11-22T17:20:19Z <p>I spent a few days in Cambridge for a minidebconf. This is a tiny version of the full annual Debconf. We had a couple of days for hacking, and another two days for talks.</p> <p>I spent my hacking time on thinking about vmdebootstrap (my tool for generating disk images with an installed Debian), and came to the conclusion I need to atone my sins for writing such crappy code by rewriting it from scratch to be nicer to use. I gave a talk about this, too. The <a href="https://lists.debian.org/debian-cloud/2016/11/msg00050.html">mailing list post</a> has the important parts, and <a href="http://meetings-archive.debian.net/pub/debian-meetings/2016/miniconf_cambridge16/">meetings-archive</a> has a video.</p> <p>I haven't started the rewrite, and it's not going to make it for stretch.</p> <p>I also gave two other talks, on the early days of Linux, and <a href="http://qvarn.org">Qvarn</a>, the latter being what I do at work.</p> <p>Thank you to ARM, for sponsoring the location, and the other sponsors for sponsoring food. These in-real-life meetings between developers are important for the productivity and social cohesion of Debian.</p> 20 years ago I became a Debian developer https://blog.liw.fi/posts/dd20/ https://blog.liw.fi/posts/dd20/ debian Tue, 16 Aug 2016 15:47:27 +0000 2016-08-16T15:47:27Z <p>Today it is 23 years ago since Ian Murdock published his intention to develop a new Linux distribution, Debian. It also about 20 years since I became a Debian developer and made my first package upload.</p> <p>In the time since:</p> <ul> <li><p>I've retired a couple of times, to pursue other interests, and then un-retired.</p></li> <li><p>I've maintained a bunch of different packages, most importantly the PGP2 software in the 90s. (I now only maintain software for which I'm also upstream, in order to make jokes about my upstream being an unco-operative jerk, and my packager being unhelpful in the extreme.)</p></li> <li><p>Got kicked out from the Debian mailing lists for insulting another developer. Not my proudest moment. I was allowed back later, and I've tried to be polite ever since. (See also <a href="http://liw.fi/rules/">rules 6</a>.)</p></li> <li><p>I've been to a few Debconfs (3, 5, 6, 9, 10, 15). I'm looking forward to going to many more in the future. It's clear that seeing many project members at least every now and then has a very big impact on project cohesion.</p></li> <li><p>I had a gig where I was paid to improve the technical quality of Debian. After a few months of bug fixing (which isn't my favourite pastime), I wrote piuparts in order to find new bugs. (I gave that project away many years ago, but it seems to still be going strong.)</p></li> <li><p>I've almost ran for DPL twice, but I'm glad I didn't actually. I've carefully avoided any positions of power or responsibility in the project. (I live in fear that someone decides to nominate me for something where I'd actually have make important decisions.)</p> <p>Not being responsible means I can just ignore the project for a while when something annoying happens. (Or retire again.) With such a large project, eventually something really annoying does happen.</p></li> <li><p>Came up with the DEP process with Zack and Dato. I also ran the second half of the DEP5 process to get the debian/copyright machine readable format accepted. (I'm no longer involved, though, and I don't think DEP is much now.)</p></li> <li><p>I've taught several workshops about Debian packaging, including online for Debian-Women. It's always fun when others "get" how easy packaging really is, despite all the efforts of the larger variety in tooling and random web pages go to to obscure the fundamental simplicity.</p></li> <li><p>Over the years Í've enjoyed many of the things developed within Debian (without claiming any credit for myself):</p> <ul> <li><p>the policy manual, perhaps the most important technical achievement of the project</p></li> <li><p>the social contract and Debian free software guidelines, unarguably the most important non-technical achievements of the project</p></li> <li><p>the whole package management system, but especially apt</p></li> <li><p>debhelper's dh, which made the work of packaging simple cases so easy it's nearly a no-brainer</p></li> <li><p>d-i made me not hate installing Debian (although I think time is getting ripe to replace d-i with something new; catch me in a talkative mood at party to hear more)</p></li> <li><p>Debian-Women made an almost immediate improvement to the culture of the larger project (even if there's still much too few women developers)</p></li> <li><p>the diversity statement made me a lot happier about being a project member.</p></li> </ul> <p>I'd like to thank everyone who's worked on these and made them happen. These are important milestones in Debian.</p></li> <li><p>I've opened my mount in a lot of places over the years, which means a lot of people know of me, but nobody can actually point at anything useful I've actually done. Which is why when I've given talks at, say, FOSDEM, I get introduced as "the guy who shared an office with Linus Torvalds a long time ago".</p></li> <li><p>I've made a number of friends via participation in Debian. I've found jobs via contacts in Debian, and have even started a side business with someone.</p></li> </ul> <p>It's been a good twenty years. And the fun ain't over yet.</p> Two-factor auth for local logins in Debian using U2F keys https://blog.liw.fi/posts/u2f-pam/ https://blog.liw.fi/posts/u2f-pam/ debian pam u2f Fri, 15 Jul 2016 08:57:07 +0000 2016-07-15T11:19:28Z <p><strong>Warning:</strong> This blog post includes instructions for a procedure that can lead you to lock yourself out of your computer. Even if everything goes well, you'll be hunted by dragons. Keep backups, have a rescue system on a USB stick, and wear flameproof clothing. Also, have fun, and tell your loved ones you love them.</p> <p>I've recently gotten two <a href="https://en.wikipedia.org/wiki/Universal_2nd_Factor">U2F</a> keys. U2F is a open standard for authentication using hardware tokens. It's probably mostly meant for website logins, but I wanted to have it for local logins on my laptop running Debian. (I also offer a line of stylish aluminium foil hats.)</p> <p>Having two-factor authentication (2FA) for local logins improves security if you need to log in (or unlock a screen lock) in a public or potentially hostile place, such as a cafe, a train, or a meeting room at a client. If they have video cameras, they can film you typing your password, and get the password that way.</p> <p>If you set up 2FA using a hardware token, your enemies will also need to lure you into a cave, where a dragon will use a precision flame to incinerate you in a way that leaves the U2F key intact, after which your enemies steal the key, log into your laptop and leak your cat GIF collection.</p> <p>Looking up information for how to set this up, I found a blog post by <a href="http://seabre.github.io/blog/2015/10/17/local-two-factor-authentication-with-u2f-on-ubuntu-14-dot-04/">Sean Brewer</a>, for Ubuntu 14.04. That got me started. Here's what I understand:</p> <ul> <li><p>PAM is the technology in Debian for handling authentication for logins and similar things. It has a plugin architecture.</p></li> <li><p>Yubico (maker of Yubikeys) have written a PAM plugin for U2F. It is packaged in Debian as <code>libpam-u2f</code>. The package includes documentation in <code>/usr/share/doc/libpam-u2f/README.gz</code>.</p></li> <li><p>By configuring PAM to use <code>libpam-u2f</code>, you can require both password and the hardware token for logging into your machine.</p></li> </ul> <p>Here are the detailed steps for Debian stretch, with minute differences from those for Ubuntu 14.04. If you follow these, and lock yourself out of your system, it wasn't my fault, you can't blame me, and look, squirrels! Also not my fault if you don't wear sufficient protection against dragons.</p> <ol> <li>Install <code>pamu2fcfg</code> and <code>libpam-u2f</code>.</li> <li>As your normal user, <code>mkdir ~/.config/Yubico</code>. The list of allowed U2F keys will be put there.</li> <li>Insert your U2F key and run <code>pamu2fcfg -u$USER &gt; ~/.config/Yubico/u2f_keys</code>, and press the button on your U2F key when the key is blinking.</li> <li>Edit <code>/etc/pam.d/common-auth</code> and append the line <code>auth required pam_u2f.so cue</code>.</li> <li>Reboot (or at least log out and back in again).</li> <li>Log in, type in your password, and when prompted and the U2F key is blinking, press its button to complete the login.</li> </ol> <p><code>pamu2fcfg</code> reads the hardware token and writes out its identifying data in a form that the PAM module understands; see the <a href="https://developers.yubico.com/pam-u2f/">pam-u2f</a> documentation for details. The data can be stored in the user's home directory (my preference) or in <code>/etc/u2f_mappings</code>.</p> <p>Once this is set up, anything that uses PAM for local authentication (console login, GUI login, sudo, desktop screen lock) will need to use the U2F key as well. ssh logins won't.</p> <p>Next, add a second key to your <code>u2f_keys</code>. This is important, because if you lose your first key, or it's damaged, you'll otherwise have no way to log in.</p> <ol> <li>Insert your second U2F key and run <code>pamu2fcfg -n &gt; second</code>, and press the second key's button when prompted.</li> <li>Edit <code>~/.config/Yubico/u2f_keys</code> and append the output of <code>second</code> to the line with your username.</li> <li>Verify that you can log in using your second key as well as the first key. Note that you should have only one of the keys plugged in at the same time when logging in: the PAM module wants the first key it finds so you can't test both keys plugged in at once.</li> </ol> <p>This is not too difficult, but rather fiddly, and it'd be nice if someone wrote at least a way to manage the list of U2F keys in a nicer way.</p> Not-platform for Debian project leader elections 2016 https://blog.liw.fi/posts/dpl-2016-not-platform/ https://blog.liw.fi/posts/dpl-2016-not-platform/ debian Sat, 12 Mar 2016 07:40:40 +0000 2016-03-12T07:40:40Z <p>After some serious thinking, I've decided <strong>not</strong> to nominate myself in the Debian project leader elections for 2016. While I was doing that, I wrote the beginnings of a platform, below. I'm publishing it to have a record of what I was thinking, in case I change my mind in the future, and perhaps it can inspire other other people to do something I would like to happen.</p> <p>Why not run? I don't think I want to deal with the stress. I already have more than enough stress in my life, from work. I enjoy my obscurity in Debian. It allows me to go away for long periods of time, and to ignore any discussions, topics, and people that annoy or frustrate me, if I don't happen to want to tackle them at any one time. I couldn't do that if I was DPL.</p> <h1>NOT a platform for Debian project leader election, 2016</h1> <p>Apart from what the Debian constitution formally specifies, I find that the important duties of the Debian project leader are:</p> <ul> <li>Inspire, motivate, and enable the Debian community to make Debian better: to be the grease that makes the machinery run smoother. It is not important that the DPL do things, except to make sure other people can do.</li> <li>Delegate what work can be delegated. The DPL is only one person, and should not be a bottleneck. Anything that can reasonably be delegated should be delegated.</li> <li>Deal with mundane management tasks. This includes spending Debian money when it makes things easier, such as by funding sprints.</li> <li>Represent the project in public, or find people to do that in specific cases.</li> <li>Help resolve conflicts within the project.</li> </ul> <p>I do not feel it is the job of the DPL to set goals for the project, technical or otherwise, any more than any other member of the project. Such goals tend to best come from enthusiastic individual developer who want something and are willing to work on it. The DPL should enable such developers, and make sure they have what they need to do the work.</p> <h1>My plan, if elected</h1> <ol> <li><p>Keep Debian running. Debian can run for a long time effectively on autopilot, even if the DPL vanishes, but not indefinitely. At minimum, the DPL should delegate the secretary and technical committee members, and decide on how money should be spent. I will make sure this minimum level is achieved.</p></li> <li><p>While I have no technical goals to set for the project, I have an organisational one. I believe it is time for the project to form a social committee whose mandate is to step in and help resolve conflicts in their early stages, before they grow big enough that the DPL, the tech-ctte, listmasters, or the DAM needs to involved. See below for more details on this. If I am elected, I will do my best to get a social committee started, and I will assume that any vote for me is also a vote for a social committee.</p></li> </ol> <h1>Social committee</h1> <p>(Note: It's been suggested that this is a silly name, but I haven't had time to come up with anything better. I already rejected "nanny patrol".)</p> <p>We are a big project now. Despite our reputation, we are a remarkably calm project, but there are still occasional conflicts, and some of them spill out into our big mailing lists. We are not very good, as a project, in handling such situations.</p> <p>It is not a new idea, but I think its time has come, and I propose that we form a new committee, a social committee, whose job is to help de-escalate conflict situations while they are still small conflicts, to avoid them growing into big problems, and to help resolve big conflicts if they still happen.</p> <p>This is something the DPL has always been doing. People write to the DPL to ask for mediation, or other help, when they can't resolve a situation by themselves. We also have the technical committee, listmasters, GRs, and the expulsion process defined by the DAM. These are mostly heavy-weight tools and by the time it's time to consider their use, it's already too late to find a good solution.</p> <p>Having the DPL do this alone puts too much pressure on one person. We've learnt that important tasks should generally be handled by teams rather than just one person.</p> <p>Thus, I would like us to have a social committee that:</p> <ul> <li>is reasonably small, like the technical committee</li> <li>attempts to resolve conflicts at an early stage</li> <li>is delegated by the DPL to have authority to do this</li> <li>doesn't necessarily have direct authority to remove people from mailing lists, IRC channels, or expel them from the project, but has authority suggest such measures to the appropriate team</li> <li>may do other things, such as educate people on how to resolve conflicts constructively themselves, or deal with chronic conflicts in addition to acute flare-ups</li> </ul> <h1>About me</h1> <p>I've been a Debian developer since 1996. I've been retired twice, while I spent large amounts of time on other things. I haven't been a member of any important team in Debian, but I've been around long enough that I know many people, and have a reasonable understanding of how the projects works.</p> Update on 'Backups as a service in Debian': no news is bad news https://blog.liw.fi/posts/backups-as-a-default-service-update/ https://blog.liw.fi/posts/backups-as-a-default-service-update/ debian Sun, 31 Jan 2016 10:50:17 +0000 2016-01-31T10:55:16Z <p>At Debconf15 I gave a talk on topic of having backups be a default service on Debian machines. In that talk, I proposed that we create infrastructure to be included in a default Debian install to manage backups.</p> <p>I still think this is a good idea, but over the past several months, I've had nearly no time at all to actually do this.</p> <p>I'm afraid I have to say now that I won't be able to work on this in time for the stretch release. I would be very happy for others to do that, however.</p> <p>The Debian wiki page <a href="https://wiki.debian.org/Backup">https://wiki.debian.org/Backup</a> acts as a central point of information for this. If you're interested in working on this, you can just do it.</p> On Norbert Preining, Sage Sharp, and Debian https://blog.liw.fi/posts/preining/ https://blog.liw.fi/posts/preining/ debian rant Thu, 08 Oct 2015 16:58:27 +0000 2022-08-22T06:07:24Z <p>I disagree with pretty much everything Norbert Preining says in his <a href="http://www.preining.info/blog/2015/10/looking-at-the-facts-sarah-sharps-crusade/">blog post about Sage Sharp</a> in response to Sharp's blog post <a href="http://sarah.thesharps.us/2015/10/05/closing-a-door/">Closing a door</a> about leaving Linux kernel development. I think Preining mis-characterises what happens on the Linux kernel mailing list, and in Debian, in free software development in general, and what Sage Sharp has said and done.</p> <p>I do not wish to participate in yet another discussion on the <a href="https://www.debian.org/code_of_conduct">Debian Code of Conduct</a>, respect, appropriate behaviour, or whether these things result in worse software. I won't spend any further of my time on debating, point-by-point, the arguments and straw-men Preining makes. Those discussions have happened often enough, without any effect on Preining and those who think like him. Likewise, despite the many times he makes the arguments, I continue to think he is wrong about everything on this topic.</p> <p>I do feel it is important to make it clear to the people reading Planet Debian, where both Preining's and my blogs are published, that his opinions are not mainstream in the Debian project, and that despite what he says, Debian development continues to be fun.</p> <p><em>Amended in 2019 to not use Sharp's old name. They changed it in 2017. Justification: if people refer to this blog post now, it'd be nice if the post didn't lead people to using the old name.</em></p> Jessie release part in Helsinki https://blog.liw.fi/posts/jessie-release-party/ https://blog.liw.fi/posts/jessie-release-party/ announcement debian Wed, 22 Apr 2015 16:49:34 +0000 2015-04-22T16:49:34Z <p>There will be a gathering of Debian people to celebrate the release of jessie this Saturday in Helsinki. For details, see the <a href="https://wiki.debian.org/ReleasePartyJessie/Finland/Helsinki">wiki page</a>. Welcome, everyone.</p> DPL elections for 2015 are going on https://blog.liw.fi/posts/dpl-2015/ https://blog.liw.fi/posts/dpl-2015/ debian Thu, 09 Apr 2015 15:51:03 +0000 2015-04-09T15:51:03Z <p>The Debian Project Leader electsions are going on. This is the yearly election for the leader, where members of the project vote for a new leader for a year. The debate this year seemed to me to be quite quiet, and voting activity seems to not be very high, either. Pity. Many years ago, the election period used to be quite energetic, bringing up some quite good viewpoints.</p> <p>There seems to also not have been the usual repeat of the voting announcement, not sure what's going there. There's time until next Tuesday midnight (in the UTC time zone) to vote. Below are links to the vote page (with instructions for voting) and the (corrected) initial announcment.</p> <ul> <li><a href="https://www.debian.org/vote/2015/vote_001">https://www.debian.org/vote/2015/vote_001</a></li> <li><a href="https://lists.debian.org/debian-devel-announce/2015/04/msg00000.html">https://lists.debian.org/debian-devel-announce/2015/04/msg00000.html</a></li> </ul> <p>I voted for Neil as my top candidate. I think he's got the best background and personality for being the leader of this project of ours.</p> A vision of backups in Debian https://blog.liw.fi/posts/debian-backups-by-defaut/ https://blog.liw.fi/posts/debian-backups-by-defaut/ backups debian obnam vision Mon, 10 Nov 2014 15:35:11 +0000 2014-11-10T15:35:11Z <p>Meet Alfred. Alfred is a Debian user. He has a laptop with Debian and a desktop environment running on it. Alfred does a lot of impotant things on his computer: his hobby is to photograph his cat, and also he works for a non-governmental organisation that investigates and reports on human rights violations. His job involves a lot of travel to many parts of the world, and he needs to handle a lot of very sensitive information. His laptop uses full-disk encryption, and it's generally speaking very well secured against the various security threats that are due to his job.</p> <p>He is worried about losing important data. He's not too worried that the sensitive information he has will leak if his laptop is stolen, but it might be impossible to re-create the data if the laptop is gone. If he interviews a whistleblower for a slave-trading corporation, and his laptop is stolen after that, it might be impossible to ever meet with the whistleblower again.</p> <p>Alfred wants backups of his data. He gets a USB thumb drive, and plugs it in. The laptop has never seen the drive before, so it asks Alfred if the drive should be used for backups. Alfred says yes.</p> <p>The laptop formats the thumb drive, again with full-disk encryption, and then runs a backup. The backup automatically picks up all the files from Alfred's home directory, and some system confguration files that may be necessary as well. (Read: /home and /etc.) Files that are usually not very precious, such as web browser caches, are automatically excluded.</p> <p>Later, when Alfred wants to update the backup, he plugs in the same drive again. The system recognises the drive, and runs the backup. While the backup is running, Alfred has an indicator in his desktop status bar. If Alfred leaves the drive plugged in, and changes anything in his home directory, that gets immediately backed up to the backup drive. Until the changes have been backed up, the indicator stays on Alfred's status bar.</p> <p>This isn't good enough, however. Alfred needs to carry the USB drive with him, and if he's mugged, he might lose both the laptop and the backup drive. Therefore, the system administrator at Alfred's NGO, Janet, sets up an account on an online backup server, and e-mails Alfred a configuration file, which Alfred drops into the backup system's configuration tool.</p> <p>From then on, whenever Alfred's laptop is online, and can see the backup server (identified by an SSH host key), any changes Alfred makes are backed up as soon as possible. For the next interview, as soon as the interview is finished and Alfred closes the laptop lid to suspend it, the backup has already finished, both to the online server and the USB thumb drive.</p> <p>Alfred is now happy, and no longer fears for the safety of his data.</p> <p>Janet, however, is still a little worried, because the online backup server is an attractive target for attacks. She asks Alfred to configure the backup service on the laptop to encrypt and digitally sign the backups, and sends the master backup public key with the request. Janet keeps the corresponding private key in a secure location.</p> <p>Alfred goes into the configuration dialog, ticks the right box, and drops in the server public key. The backup software generates a new public key for the laptop to use for encrypting the backups, and Alfred e-mails that to Janet, using PGP encrypted and signed e-mail. He also puts the laptop backup encryption keys on a couple of USB thumb drives, which he stores in safe places (in his sock drawer and coffee jar, but don't tell anyone that).</p> <p>Alfred's online backups are now encrypted with public keys so that both Alfred and Janet can decrypt them, but only they can do that. The backups are digitally signed so that if the server is hacked, the backups can't be altered without it being detectable.</p> <p>Some time passes.</p> <p>Alfred needs to go to speak to the general assembly of the Cat Conference, about how awesome his cat is. This requires him to travel to the US, and he's worried that the US authorities will confiscate his laptop and try to get at his work files that way. He deletes all his work files, ssh keys, and other files that aren't necessary to show his cat pictures at the conference.</p> <p>The conference goes fine, and when Alfred comes back home, he gets the USB thumb drive that contains his backup encryption key. He plugs it in, tells the backup configuration software to import it. Alfred can then open his backups on the online backup server in his file browser, and can restore back his files by copying them with drag and drop.</p> <p>However, the next day Alfred's cat, upset at how much he travels, pees on the laptop. It is ruined. Everything is lost.</p> <p>Alfred gets a new laptop from Janet, and installs Debian on it. During installation, Alfred gives the installer the USB backup drive, and the installer restores all of Alfred's own files, and also restores system configuration. After a little while, Alfred has a newly installed laptop with all his usual software and all of his files.</p> <hr /> <p>This is a summary of a vision for backups being a service in a default Debian install in the future. It is currently just a vision, and nobody is currently working on making it reality. Would you like to work on this for the release after jessie?</p> <p>(No cats were harmed in the production of this vision.)</p> Thoughtful, inspiring words from Russ https://blog.liw.fi/posts/rra-init-system/ https://blog.liw.fi/posts/rra-init-system/ debian Mon, 30 Dec 2013 20:24:00 +0000 2013-12-30T20:24:00Z <p>Russ Allbery has written up summaries of the research he's done in preparation of a Debian Technical Committee vote on default init system in Debian's next release. The mails are long, and well worth reading completely if the topic interests you at all, but I'd like to quote separately a few paragraphs that I found thoughtful, and inspiring, in the greater Debian context and not just for the init system discussion.</p> <p>From <a href="http://lists.debian.org/debian-ctte/2013/12/msg00234.html">"init system other points, and conclusion"</a>:</p> <blockquote><p>Here again, I think we have an opportunity for Debian to be more innovative and forward-looking in what we attempt to accomplish in the archive by adopting frameworks that let us incorporate the principles of least privilege and defense in depth into our standard daemon configurations.</p></blockquote> <p>And later, from the same mail:</p> <blockquote><p>I think we should make wise decisions about which areas we want to invest project effort. I dislike investing significant project effort in catch-up efforts that, when complete, merely get us back to where we would have been if we'd chosen a different solution. I don't think that's wise stewardship of project resources. I want to see Debian focus its efforts on places where we can make a real difference, where we can be leaders. That means adopting the best-of-breed existing solutions and building on top of them, not reinventing wheels and thereby starting from a trailing position.</p></blockquote> <p>From <a href="http://lists.debian.org/debian-ctte/2013/12/msg00258.html">"loose ends for init system decision"</a>:</p> <blockquote><p>It is not, in general, necessary to justify what you want to do in Debian. It doesn't matter if it's going to be used by thousands of people or two people. If you can do your work to the standards that we expect to be maintained across the archive, and without negative impact on Debian's other contributors, you can work on whatever you love. And, furthermore, we all support each other in our passions. Debian is built on a culture of deference to other people's work, and reasonable accomodation to the projects that other people want to work on.</p> <p>Now, there is a fine balance here. Deference to other people's work does not mean a requirement to join their work. Reasonable accomodation does not mean that every Debian developer is required to test their software on non-Linux ports. The goal is that all of us should be empowered to work on the things we are passionate about, which implicitly includes being empowered to not work on the things that are not of interest to us. Therefore, for some efforts, and specifically for the non-Linux port efforts, the work is mostly born by the porters. They're expected to test, diagnose problems, and submit patches. The deference and reasonable accomodation that we expect of Debian contributors is to merge those patches when they are reasonable, to not undermine the porting effort when there is a reasonable approach that preserves it, and to be aware of the implications of their packaging for those ports in the broad sense (such as qualifying build-dependencies with [linux-any] where appropriate).</p> <p>We do not expect Debian contributors to reject significant new upstream versions or significant new integrations because they will affect the non-Linux ports, or, for that matter, other projects in Debian. We do expect those changes to follow the standards that we expect of Debian as a whole, and that porting efforts will be treated with deference and reasonable accomodation.</p></blockquote> <p>I won't offer a comment on these quotes&mdash;I prefer to let them speak for themselves&mdash;but I will say I find these to be among the wisest things said within Debian all year.</p> Debian: developing it wrong https://blog.liw.fi/posts/debian-developing-it-wrong/ https://blog.liw.fi/posts/debian-developing-it-wrong/ debian presentation Sat, 16 Nov 2013 17:56:47 +0000 2013-11-16T18:34:31Z <p>This is an essay form of a talk I have given today at the Cambridge Mini-debconf. The talk was videoed, so will presumably show up in the Debconf video archive eventually.</p> <h1>Abstract</h1> <p>Debian has a long and illustrious history. However, some of the things we do perhaps no longer make as much sense as they used to do. It's a new millennium, and we might find better ways of doing things. Things that used to be difficult might now be easy, if we dare look at things from a fresh perspective. I have been doing that at work, for the Baserock project, and this talk is a compilation of observations based on what I've learnt, concentrating on things that affect the development workflow of package maintainers.</p> <h1>Introduction and background</h1> <p>I have been a Debian developer since August, 1996 (modulo a couple of retirements), and have used it a little bit longer. I have done a variety of things for Debian, from maintaining PGP 2 packages to writing <code>piuparts</code> to blathering excessively on the mailing lists.</p> <p>My day job is to develop the <a href="http://wiki.baserock.org/">Baserock</a> system at <a href="http://www.codethink.co.uk/">Codethink</a>. Baserock is a set of tools and workflows for developing embedded and appliance Linux systems. If you squint, it looks a bit like a source-based Linux distribution. I have worked on Baserock since September, 2011.</p> <p>Some of Baserock's design has been influenced by my experience with Debian. With Baserock I have the chance to fix all the things that are wrong in Debian, and this talk is me giving back to Debian by pointing out some the things I feel should be fixed. I don't have solutions for these problems: this is a bug report, not a patch. It's also perhaps a bit of a rant.</p> <p>I am specifically concentrating here on technical and tooling issues that affect the development process of Debian. I excluding social issues.</p> <p>I am also not trying to get Debian to switch to Baserock. Baserock is targeting embedded and appliance systems, and makes simplifying assumptions based on those targets, which Debian does not get to do. I am pointing out problems, and I am outlining solutions as implemented in Baserock, when I think the concept carries over well to Debian.</p> <h1>Build tools should be intelligent, packaging should be dumb</h1> <p>In Debian, the fundamental tool for compiling the upstream code and assembling a binary package is <code>dpkg-buildpackage</code>. It uses <code>debian/rules</code>, which is a Makefile with targets with specific names and semantics. By executing the right targets in the right order, <code>dpkg-buildpackage</code> tells the source package to build the binary package.</p> <p>On the one hand, this is a nice design, because it abstracts away the large variety of upstream build systems into one API for <code>dpkg-buildpackage</code> to use. On the other hand, it puts all the intelligence for how packages are built into the packages.</p> <p>Making packaging intelligent, rather than the build tool, means packagers need to do more work, and there's more work to do when, say, the Debian policy changes, or when there are other changes that affect a large number of packages. If packaging is intelligent, then every package needs changes. If the build tool is intelligent, then you change the build tool and re-build everything.</p> <p>In Baserock we put as much intelligence as we can into the Morph tool, which drives the build process. It turns out that today, unlike in 1995, most upstream projects use one of a handful well-known build systems: autotools, cpan, cmake, Python distutils, etc. With just a little extra logic in Morph we avoid having any delta against upstream in Baserock. This doesn't work for quite all upstream projects, of course, but we've spent the 2% (two, not twenty) of effort that solves 80% of the problem.</p> <p>In recent years, the <code>dh</code> approach to packaging has made a lot of packages be able to have only a minimal, 3-line <code>debian/rules</code> file. This is excellent. Wouldn't it be nice if even that wasn't needed? It'd save thousands of files in source packages across the archive, at least. It would be easy to: if the file is missing, have <code>dpkg-buildpackage</code> assume the 3-line version by default.</p> <p>Getting rid of a single file is, of course, not a particularly big win. The big win is the change in mindset: rather than dealing with all new issues in development by adding yet more fields to <code>debian/control</code> and more optional, competing tooling outside the core toolset, if you improve the tools everyone uses, then everyone's packages gets better.</p> <p>The goal should, in my opinion, be that for the large number of packages where upstream uses a well-known, well-behaved build system, and uses it in a reasonably sensible way, the Debian source package should not require anything added to make the package build. There will still be a need to add some stuff, such as the <code>debian/copyright</code> file, to make a good Debian package, but just getting the package to build should require nothing extra. (Side note: wouldn't it be nice if there was a well-known, widely used way to mark up copyright information so that <code>debian/copyright</code> could be constructed and updated automatically?)</p> <h1>Configuration file handling on upgrades: add ucf to dpkg already</h1> <p>In the 1990s, <code>dpkg</code> had excellent handling of configuration files and merging local changes with changes from the new package version, but it was excellent only because it tried to do that at all, and mostly nothing else did. It hasn't changed much, since, and it's not excellent on any absolute scale.</p> <p>We have the <code>ucf</code> tool, which can do a better job, but requires to be added to each package wanting to use it. Why don't we make <code>dpkg</code> smarter, instead? If <code>ucf</code> is not not good enough to be merged wholesale into <code>dpkg</code>, let's write something better.</p> <p>Making every package maintainer use <code>ucf</code> manually is just wasteful. This is not the kind of thing that should be done for each package separately: the package manager should be smart so that packaging can be stupid.</p> <p>The goal should be that <code>dpkg</code> is smart enough in its configuration file handling that having the package do it is a very rare special case.</p> <h1>Clean building shouldn't be hard</h1> <p>The basic tool for building a package is <code>dpkg-buildpackage</code>. It is ever so slightly cumbersome to use, so there are some wrappers, most importantly <code>debuild</code>. However, if you're making a build intended to be uploaded to the Debian archive, you should be doing a clean build. This means having to learn, configure, and use yet more tools.</p> <p>A clean build is important: security updates, development, debugging, quality control, porting, and user support become more difficult if we don't know how a package was built, and can't reproduce the build. It gets harder to keep build dependencies correct, making it harder for everyone to build things.</p> <p>Luckily, Debian has solved the clean build problem. Unluckily, it has solved it multiple times, creating the problem of having to choose the clean building approach you want to use. The default way of building is not clean, so then you have to remember to use the non-standard way. You also get to spend time maintaining the clean build environments yourself, since that doesn't seem to be fully automated. None of this is hard, as such, but it's extra friction in the development workflow.</p> <p>The primary approaches for clean building in Debian seem to be <code>pbuilder</code>, <code>cowbuilder</code>, and <code>sbuild</code>. I happen to use <code>pbuilder</code> myself, because it's what's been around the longest, but I make no claim of having made an informed choice.</p> <p>That is part of the problem here: why should I have to spend the effort to become informed to make a choice well? Why is the default way of building not clean? Don't say performance: Morph sets up a clean staging area in seconds, and does not offer you a choice of not doing so. It's a chroot, with everything but the build tree hardlinked from cached, unpacked build dependencies, and protected using read-only bind mounts.</p> <p>What's more, this approach avoids having to maintain the <code>pbuilder</code> base tarballs or <code>sbuild</code> chroots manually. It's all automatic, and up to date, for every build. Furthermore, the staging area contains only the specified build dependencies, and not anything else, meaning a build fails if a build dependency is missing, rather than succeeding because it happens to be in the default set of packages the build tool installs.</p> <h1>Mass building shouldn't be hard</h1> <p>Suppose you want to try out a large-scale change in Debian. It might be trying out a new version of GCC, or using llvm's <code>clang</code> as the default C compiler, or updating glibc, or doing a large library transition such as a new version of GTK+, or trying a new <code>dpkg</code> version, or even something more exploratory such as changing the default optimisation flags for all packages to see what breaks.</p> <p>All of these will require you to at least rebuild all affected packages. Ideally you'd test the built packages as well, but let's concentrate on the building for now.</p> <p>Here's what you do. You make the changes you want to try out, and build those packages. You create a dedicated APT repository, and upload your packages there. You configure your build environment to favour that APT repository. You make a list of all reverse build dependencies of the packages you changed. You write a script to build all of those, preferably so that if you change A, and rebuild B, then you also rebuild C which build depends on B. Each rebuilt package you also upload to your APT repository. You keep track of the build log, and success or failure, of each build.</p> <p>The people in Debian who do this kind of stuff regularly presumably have tools for doing it. It shouldn't be a rare, special thing, though. If my package has reverse build dependencies, I should at least consider testing building when I'm making changes. Otherwise, it might take years until the reverse build dependencies are rebuilt, and the problem is only found then, making it harder to fix.</p> <p>To be fair, building a lot of packages takes a lot of resources. It's not feasible to rebuild everything in Debian every time there's any change to, say, eglibc. However, it's feasible to do it, for large subsets of the archive, without huge hardware investments.</p> <h1>One VCS to rule them all</h1> <p>In 1995 there was really only one relevant version control system: CVS. It was not a great tool. In 2000, another contender existed: Subversion. It fixed some problems in CVS, but still wasn't a great tool. In 2005, there was a great upheaval and distributed version control systems started to become mainstream. There were a large handful of them. In 2010, it was becoming pretty clear that <code>git</code> had won. It's ugly, but it's powerful.</p> <p>I'm not going to debate the relative merits of different version control systems. Until recently, I was a Bazaar boy, and all of my personal projects were kept in Bazaar. (I have recently switched everything to git.)</p> <p>There are, however, strong benefits from everyone using the same system. Developers don't need to learn a dozen version control systems. Tools that operate on many repositories are easier to write and maintain. Workflows become simpler if one system can be assumed.</p> <p>Debian has a strong historical tendency to choose every option. This is sometimes a good thing, and sometimes a bad thing. For keeping source packages in version control I believe it is to be a bad thing. The status quo is that a Debian source package may not be in version control at all, or it might be in any version control system.</p> <p>This is acceptable when everyone only ever needs to maintain their own packages. However, in a distribution the size of Debian, that is not the case. NMUs, security support, archive wide transitions, and other situations arise when I might need to change yours, or you might need to change mine.</p> <p>We try to work around this by having a complicated source package format, using <code>quilt</code> to maintain patches to upstream semi-manually in a <code>debian/patches</code> directory. This is an awkward workflow. It's a workflow that trips up those that are not used to it. (I know <code>quilt</code> is a patch management system, not a version control system. I think git does it much better anyway.)</p> <p>It would be oh so much easier if everyone kept their source packages in the same, real version control system. I don't even care what it is, as long as it is powerful enough to handle the use cases we have.</p> <p>Imagine a world where every Debian source package is kept, for argument's sake, <code>git</code>, and everyone also uses the same layout and roughly the same workflow to maintain it. What would this mean?</p> <p>It would mean that if you want to inspect the history of your package, you know how to do that. If you want to merge in some bugfix from upstream code, you know how to do that, without having to figure out which of the several source package formats are in use.</p> <p>It would make feasible the development of more powerful, more higher level tooling. For example, it would allow Debian to have what we call system branches in Baserock. In Debian we have stable, testing, unstable, and experimental. We may get something like Ubuntu's PPAs, or perhaps an improved version of those. These are very poor versions of system branches, just like quilt is a poor way to manage patches and deltas against upstream. For example, you can upload an experimental version of <code>gcc</code>to <code>experimental</code>, but then nobody else can upload another experimental version. You can set up your own PPA for this, but you'll still be affected by all the uploads to <code>unstable</code> while you're working.</p> <p>A Baserock system branch is a branch of the entire system, or the entire distribution in the case of Debian. It is isolated from other system branches. In branch in an individual repository is a well-known concept. A system branch is conceptually like branching every repository in the distribution at once. The actual implementation is more efficient, of course.</p> <p>This would be possible to implement without standardising on one version control system, but it would be much harder to implement, and would have to live with the lowest common denominator for features. CVS and Subversion, for example, don't really to merges, where Bazaar, Mercurial, and git do. Possible does not mean feasible.</p> <p>Any work you do in a system branch is isolated. Your work doesn't affect others, and theirs doesn't affect yours, until a merge happens. This is a simple, but very powerful tool.</p> <p>Cheap system branches, and powerful merging, makes it possible to do experiments safely, with little fuss. Combine that with being able to build everything cleanly and quickly, and get into a situation where there's no need to have make technical decisions based on arguments on mailing lists, and instead they can be done by choosing by looking at working code.</p> <p>I don't know how this could be implemented in Debian, but think about it. If Debian could have this, it might many archive-scale changes easier.</p> <h1><code>debian/rules clean</code>: really?</h1> <p>One of the silliest things we require of packages is that they have a <code>debian/rules clean</code> rule that cleans up after a build perfectly so that we can do repeated builds in the same source tree.</p> <p>Let's just use <code>git clean -fdx</code> instead.</p> <p>This is a problem that is superbly well suited for automation. There is no point whatsover making packagers do any manual work for this.</p> <h1>Large scale semi-mechanical changes require too much effort</h1> <p>About a decade ago, we decided to follow a new version of the Filesystem Hierarchy Standard and transition from <code>/usr/doc</code> to <code>/usr/share/doc</code>. This was an almost entirely mechanical change: in many cases, a mere rebuild would fix it, and in almost every other case it was just a minor tweak to the packaging. A one-line change. It took us seven years to do this.</p> <p>Seven years. Think about it.</p> <p>In a recent discussion about building R binary data files from source at package build time it was suggested that we take 2-3 release cycles to get this done. That's four to six years. Think about it.</p> <p>These are not isolated cases. Every time we need to make a change that affects more than a small handful of packages, it becomes a major undertaking. Most of the time all the people involved in this are agreeable to the change, and welcome it. The change takes long because it requires co-ordinating a large number of people. Sometimes people are just busy. Sometimes they've left the project, but haven't properly orphaned their packages. Waiting for a timeout for answers about packages drags the process longer.</p> <p>Mechanical changes, or semi-mechanical ones, which are very easy and very quick to do should not take years. They should take an evening, no more.</p> <p>There's no end of changes we might want to do like this. In 2005 and 2006 I filed a few hundred bugs from failed <code>piuparts</code> runs. They're still being fixed, even when the fix is simple, such as adding a <code>postrm</code> script to remove, when the package is purged, a configuration file created by postinst, or starting the service with <code>invoke-rc.d</code> rather than running an <code>init.d</code> script directly. Mostly one-line changes.</p> <p>There's more mechanical changes that might happen. For example, changing the value of the <code>VCS-Browser</code> field when the Debian version control server changes domain names.</p> <p>It's not just about simple, mechanical changes, either. Transitions of important library packages, for example, which require changes to reverse dependencies due to a changed API are another source of pain. What should be a few evenings of build fixes can drag out to months of co-ordination.</p> <p>This is caused partly by technical issues, and partly due to social issues. The main social issue is that we have quite a strong ownership of packages, and NMUs are to be done carefully, and only in some cases. This is both good and bad, and I won't discuss that aspect now. The technical issues are that our tools are primarily intended for maintaining individual packages, rather than all of them together, making it harder to make even simple changes in large number of packages.</p> <p>In addition to easy mass building and system branches, as outlined above, large-scale changes would require testing tools so that you don't build the new package versions, but also test things automatically. Essentially, CI at the distribution level.</p> <h1>Conclusion</h1> <p>I've listed above a small variety of problems I see in the Debian development processes and tools. They're not the important part of this talk. The important part is that it we, the Debian developers, should look at our tools, and workflows, critically, and improve them when we can. Even small improvements are very useful, when they affect each of our twenty thousand source packages. The important change I argue for here is one in mindset, rather than any specific problem in any tool. We need a mindset of constant, incremental improvement for our tools.</p> Debian discussion culture: what doesn't work https://blog.liw.fi/posts/discussion-what-works-or-not/ https://blog.liw.fi/posts/discussion-what-works-or-not/ debian Sun, 02 Sep 2012 14:31:09 +0000 2012-09-02T14:31:09Z <p>I claim that the way we conduct discussions on debian-devel has gone quickly downhill this year. The discussions around the default init implementation are particularly sad examples of how Debian debates technical topics.</p> <p>I've whined about this before, and people have suggested various things. Here's a list of things that I think we can conclude do not work:</p> <ul> <li>"Do not feed the troll" does not work, because there is always someone who can't resist responding. Also, some of the bad behavior is not trolling. Further, having everyone ignore people who behave badly does not give negative feedback for bad behavior, and does not build shared social norms for what is acceptable behavior.</li> <li>Unorganized private feedback (people mailing in response to public list posts) sometimes works, but it's a bit hit-and-miss, and again does not help build shared social norms. In other words, new members of the community see that others behave badly, and then nothing happens as far as they can see, whereas in private there may be quite a lot of activity.</li> <li>Up/down-voting a la Reddit or Stack Overflow works on those sites, but it is not the only thing that keeps their sites in shape. It also works badly on mailing lists, I think, and we can't very well switch away from the lists for all sorts of reasons. Voting can also result in a stronger "echo chamber", where the popular people or opinions are upvoted, regardless of technical merit.</li> </ul> <p>Things that can work:</p> <ul> <li>Real-life meetings between participants. Debconf, sprints, FOSDEM, other such conferences. Unfortunately, this is expensive, and we can't reach everyone.</li> <li>Active public, but constructive confrontation when people behave badly. This needs good social skills to be done well.</li> </ul> <p>Things that might be worthwhile to try:</p> <ul> <li>Changes to how our mailing list software works. Not sure what they would be, but Clay Shirky's <a href="http://www.shirky.com/writings/group_user.html">Group as User: Flaming and the Design of Social Software</a> may be a good source of inspiration.</li> <li>If not technical changes to the list software, possibly ways in which we use the tools we have. For example, a friend of mine suggested a while ago that when there is a topic on which consensus is difficult to achieve, each side of the debate (there may be any number) writes up their position and arguments for it and against the others on a wiki page. This might avoid going in circles in the discussions, repeating the same arguments over and over, during weeks and months of discussion.</li> <li>Public, but anonymous, feedback tools, a la Reddit and Stack Overflow. Perhaps as simple as adding "+1" and "-1" buttons to the web archive pages for messages, along the "Spam" button, and then publishing aggregate statistics? (A voting system, if you wish.)</li> <li>A consensus on the rough bounds of acceptable behavior and some group of people who are delegated the authority to enforce that.</li> </ul> <p>The current situation, where a few hard-nosed, uncompromising people can prevent any rational discussion about init systems cannot be allowed to continue.</p> Dabbling in diplomacy https://blog.liw.fi/posts/diplomacy/ https://blog.liw.fi/posts/diplomacy/ debian Sat, 01 Sep 2012 22:41:58 +0000 2012-09-01T22:41:58Z <p>For about the past three months I've been conducting a small experiment. When I've seen people misbehave on the Debian development mailing list, I've sent a private e-mail to them suggesting they should avoid some specific behavior, in the hope of getting them to stop.</p> <p>My success has been varied. A few have thanked me, and changed their behavior. A couple have taken offense. Some have ignored me.</p> <p>I've noticed a correlation between how much effort I put into the letter and how successful it. Broadly speaking, if I send anything earlier than the sixth version, it fails. Getting the message exactly right is not easy, for me, and takes quite some effort and time.</p> <p>I'm happy about my successes, but on the whole, I fear my experiment is a failure. It's too much effort to do this in the long term, and it's too chancy. Someone else might be able to do it better, but this is not something I can keep on doing.</p> <p>I think we need some better solution to keep discussions in Debian on track and not have them blow up. We have a systemic problem where contentious issues, such as choice of init system, is quite hard for us to deal with. We're also not very good at dealing with situations where a few individuals are dominating the discussion by being loud, insistent, and unwilling to budge or to give any credence to opposing views. I don't know what to do about that, but we clearly need social and possibly technical tools for this.</p> Quality of discussion in free software development https://blog.liw.fi/posts/quality-of-discussions/ https://blog.liw.fi/posts/quality-of-discussions/ debian discourse free-software Mon, 07 May 2012 10:08:56 +0000 2012-05-07T10:08:56Z <p><a href="http://theonlinephotographer.typepad.com/the_online_photographer/2012/05/richard-john.html">The Online Photographer</a> has a meta-article on some discussion in the photography world. Summary: someone wrote an opinion piece on one site, and people on the discussion forum of another site got his name wrong, possibly repeatedly. And the quality of the discussion went down from there.</p> <p>The quality of the discourse of free software development is frequently of some concern. Debian has a reputation as being a host to, er, particularly vigorous discussions. That reputation is not unwarranted, but, I think, we've improved a lot since 2005. The problem is hardly restricted to Debian, however.</p> <p>How can we improve this? I don't know. As a community, I'm not even sure we agree what the problems are. Here's my list.</p> <ul> <li>unshakeable, dogmatic opinions; an unwillingness to consider others' points of view or their justifications; willful ignorance of anything that contradicts with the way one wants things to be; an uncompromising, winner-takes-all, last-poster-wins attitude to debates; in short, a lack of respect for anyone who isn't on one's own side</li> <li>an (unintended?) emphasis on discussion speed, leading to short missives, written quickly, without much thought, and without giving even a glimpse of how the conclusion or opinion was formed; this further leads to discussions that are hard to follow, because there are so many messages to read (the total word count would probably be about the same if everyone only wrote one or two essays)</li> <li>few good ways of dealing with bad behavior, unless it fits into some clear categories of bad behavior; no clear community consensus of what is acceptable behavior, outside of a small core that is obvious (there's probably several PhD's worth of reasons for this, and it's not just because of "geeks don't understand social interaction" or "everyone is from a different cultural background")</li> </ul> <p>Insults, personal attacks, and other such outrageously bad behavior is uncommon. It crosses the line so clearly it becomes easy to deal with; I don't think handling this needs much attention.</p> <p>What can we do about this? I'm not sure. I have, for the time being, abandonded Debian mailing lists as a way to influence what goes on in the project, but that's just a way for me to clear some space in my head and time in my day to actually do things.</p> <p>My pet hypothetical solution of the day is that mailing lists might raise the quality of the debates by limiting the number of messages written by each person per day in each thread. This might, I think, induce people to write with more thought and put more effort into making each message count.</p> Obnam 1.0 planning https://blog.liw.fi/posts/obnam-1.0-planning/ https://blog.liw.fi/posts/obnam-1.0-planning/ debian obnam Sun, 22 Apr 2012 16:24:53 +0000 2012-04-22T16:24:53Z <p>Obnam is my backup program. See <a href="http://liw.fi/obnam/">http://liw.fi/obnam/</a> for details.</p> <p>Debian is hopefully going to be freezing the current testing version in June, in preparation for a release late this year. I'd like Obnam 1.0 to be included. In preparation for that I reviewed the currently open bugs and tagged the ones that I think are blockers.</p> <p>See <a href="http://liw.fi/obnam/bugs-1.0-blockers/">list of 1.0 blocker bugs</a>.</p> <p>Did I miss anything? Is anything there that isn't really a blocker?</p> <p>Other than that, the <a href="http://liw.fi/obnam/roadmap-for-1.0/">roadmap for 1.0</a> has the following things that aren't done yet:</p> <ul> <li>Verification test</li> <li>Performance</li> </ul> <p>Performance is one of those things that's always going to be a problem. I'm hoping to optimize the sftp performance a big, but apart from that, I don't consider it to be a blocker for 1.0. The verification test I'm confident I will pass, but it needs to actually be performed. (See the roadmap for details of the test.)</p> <p>Anyone else have opinions of what needs to be done for Obnam 1.0?</p> <p>PS. There's a <a href="http://vlists.pepperfish.net/cgi-bin/mailman/listinfo/obnam-flarn.net">mailing list</a> for Obnam now.</p> What is a Linux distribution? https://blog.liw.fi/posts/what-is-a-distro/ https://blog.liw.fi/posts/what-is-a-distro/ debian linux Sun, 08 Apr 2012 12:15:00 +0000 2012-04-08T12:15:00Z <p>I got into a discussion recently about what a Linux distribution actually is. To some people, a distro is just a convenient way of source of binaries from which to pick what you install, either on your own computer or when creating an embedded system, or some other specific kind of system. To others it's a way of life, and the realization of the ideals of software freedom. To yet others, a distro is an outdated concept that will go away in the near-ish future, to be replaced by ... something.</p> <p>Here's my take: a Linux distribution is defined by the following things:</p> <ul> <li>It's a specific, chosen set of upstream projects. The selection criteria vary wildly between distros, and may be based on things like purpose, quality, licenses, etc. Debian, for example, chooses pretty much everything that is free software, but a distro targeted at providing NAS devices would choose a much smaller set.</li> <li>It's a set of changes made to the upstream code, and specific configurations of the upstream code, in order make all the software work well together. For example, if the distro supports multiple mail transport agents, they might all need to be patched and configured to store incoming mail the same way by default.</li> <li>It's the tools, processes, policies, and workflows used to develop the distribution.</li> <li>It's the tools provided to the people using the distribution to install and manage their systems.</li> <li>It's the people and companies who develop the distribution and support its users, and their shared values and purpose. This may be a real community, or just the kind of community that exists as a web page.</li> </ul> <p>Historically, there has been a lot of variation in all aspects mentioned above. Some of this variation may today be unnecessary. For example, it might be helpful for more distributions to share work on common tools, so that not everyone needs to re-invent and re-implement the same kinds of tools. However, my gut feeling is that due to the inherent interconnectedness of all things, it may be very difficult to have one set of tools to support well all workflows, policies, processes, and communities.</p> <p>The most important differentiating factors between the various distros are not the tools, but the community values, the perceived purpose of the distro, and the resultant criteria for choosing what to include and the workflow for developing the distro.</p> <p>Even when two distros use the same tools, there can be a lot of difference in the end result. Fedora and RHEL, for example, use a lot of the same infrastructure, but their purpose is entirely different. Debian and Ubuntu use mostly the same packages, even, and still there's a huge difference between the two distros: one aims at a high quality system based on free software, the other aims to support the business of one particular company.</p> <p>For this reason, there will always be a lot of Linux distributions around, and that is good.</p> On the future of Linux distributions https://blog.liw.fi/posts/future-of-linux-distros/ https://blog.liw.fi/posts/future-of-linux-distros/ baserock debian linux Fri, 28 Oct 2011 19:17:05 +0000 2011-10-28T19:17:05Z <h2>Executive summary</h2> <p>Debian should consider:</p> <ul> <li>less fine-grained software packaging for overall simplification of the system and its development</li> <li>less flexibility at the lower levels of the software stack, again for overall simplicity</li> <li>reducing friction in the development workflow, especially by making it possible to branch and merge at the whole system level</li> </ul> <h2>Introduction</h2> <p>I've taken a job with <a href="http://www.codethink.co.uk/">Codethink</a>, as part of a team to develop a new embedded Linux system, called <a href="http://wiki.baserock.org/">Baserock</a>. Baserock is not a distribution as such, but deals with many of the same issues. I thought it might be interesting to write out my new thoughts related to this, since they might be useful for proper distributions to think about too.</p> <p>I'll hasten to add that many of these thoughts are not originally mine. I shamelessly adopt any idea that I think is good. The core ideas for Baserock come from Rob Taylor, Daniel Silverstone, and Paul Sherwood, my colleagues and bosses at Codethink. At the recent GNOME Summit in Montreal, I was greatly influenced by Colin Walters.</p> <p>This is also not an advertisment for Baserock, but since my of my current thinking come from that project, I'll discuss things in the context of Baserock.</p> <p>Finally, I'm writing this to express my personal opinion and thoughts. I'm not speaking for anyone else, least of all Codethink.</p> <h2>On the package abstraction</h2> <p>Baserock abandons the idea of packages for all individual programs. In the early 1990s, when Linux was new, and the number of packages in a distribution could be counted on two hands in binary, packages were a great idea. It was feasible to know at least something about every piece of software, and pick the exact set of software to install on a machine.</p> <p>It is still feasible to do so, but only in quite restricted circumstances. For example, picking the packages to install a DNS server, or an NFS server, or a mail server, by hand, without using meta packages or tasks (in Debian terminology), is still quite possible. On embedded devices, there's also usually only a handful of programs installed, and the people doing the install can be expected to understand all of them and decide which ones to install on each specific device.</p> <p>However, those are the exceptions, and they're getting rarer. For most people, manually picking software to install is much too tedious. In Debian, we've realised this a great many years ago, and developed meta packages (whose only purpose is to depend on other packages) and tasks (solves the same problem, but differently). These make it possible for a user to say "I want to have the GNOME desktop environment", and not have to worry about finding every piece that belongs in GNOME, and install that separately.</p> <p>For much of the past decade, computers have had sufficient hard disk space that it is no longer necessary to be quite so picky about what to install. A new cheap laptop will now typically come with at least 250 gigabytes of disk space. An expensive one, with an SSD drive, will have at least 128 gigabytes. A fairly complete desktop install uses less than ten gigabytes, so there's rarely a need to pick and choose between the various components.</p> <p>From a usability point of view, choosing from a list of a dozen or two options is much easier than from a list of thirty five thousand (the number of Debian packages as I write this).</p> <p>This is one reason why Baserock won't be using the traditional package abstraction. Instead, we'll collect programs into larger collections, called strata, which form some kind of logical or functional whole. So there'll be one stratum for the core userland software: init, shell, libc, perhaps a bit more. There'll be another for a development environment, one for a desktop environment, etc.</p> <p>Another, equally important reason, to move beyond packages is the problems caused by the combinatorial explosion of packages and versions. Colin Walters talks about this very well. When every system has a fairly unique set of packages, and versions of them, it becomes much harder to ensure that software works well for everyone, that upgrades work well, and that any problems get solved. When the number of possible package combinations is small, getting the interactions between various packages right is easier, QA has a much easier time to test all upgrade paths, and manual test coverage improves a lot when everyone is testing the same versions. Even debugging gets easier, when everyone can easily run the same versions.</p> <p>Grouping software into bigger collections does reduce flexibility of what gets installed. In some cases this is important: very constrainted embedded devices, for example, still need to be very picky about what software gets installed. However, even for them, the price of flash storage is low enough that it might not matter too much, anymore. The benefit of a simpler overall system may well outweigh the benefit of fine-grained software choice.</p> <h2>Everything in git</h2> <p>In Baserock, we'll be building everything from source in git. It will not be possible to build anything, unless the source is committed. This will allow us to track, for each binary blob we produce, the precise sources that were used to bulid it.</p> <p>We will also try to achieve something a bit more ambitious: anything that affects any bit in the final system image can be traced to files committed to git. This means tracking also all configuration settings for the build, and the whole build environment, in git.</p> <p>This is important for us so that we can reproduce an image used in the field. When a customer is deploying a specific image, and needs it to be changed, we want to be able to make the change with the minimal changes compared to the previous version of the image. This requires that we can re-create the original image, from source, bit for bit, so that when we make the actual change, only the changes we need to make affect the image.</p> <p>We will make it easy to branch and merge not just individual projects, but the whole system. This will make it easy to do large changes to the system, such as transitioning to a new version of GNOME, or the toolchain. Currently, in Debian, such large changes need to be serialised, so that they do not affect each other. It is easy, for example, for a GNOME transition to be broken by a toolchain transition.</p> <p>Branching and merging has long been considered the best available solution for concurrent development within a project. With Baserock, we want to have that for the whole system. Our build servers will be able to build images for each branch, without requiring massive hardware investment: any software that is shared between branches only gets built once.</p> <p>Launchpad PPAs and similar solutions provide many of the benefits of branching and merging on the system level. However, they're much more work than "git checkout -b gnome-3.4-transition". I believe that the git based approach will make concurrent development much more efficient. Ask me in a year if I was right.</p> <h2>Git, git, and only git</h2> <p>There are a lot of version control systems in active use. For the sake of simplicity, we'll use only git. When an upstream project uses something else, we'll import their code into git. Luckily, there are tools for that. The import and updates to it will be fully automatic, of course.</p> <p>Git is not my favorite version control system, but it's clearly the winner. Everything else will eventually fade away into obscurity. Or that's what we think. If it turns out that we're wrong about that, we'll switch to something else. However, we do not intend to have to deal with more than one at a time. Life's too short to use all possible tools at the same time.</p> <h2>Tracking upstreams closely</h2> <p>We will track upstream version control repositories, and we will have an automatic mechanism of building our binaries directly from git. This will, we hope, make it easy to follow closely upstream development, so that when, say, GNOME developers make commits, we want to be able to generate a new system image which includes those changes the same day, if not within minutes, rather than waiting days or weeks or months.</p> <p>This kind of closeness is greatly enhanced by having everything in version control. When upstream commits changes to their version control system, we'll mirror them automatically, and this then triggers a new system image build. When upstream makes changes that do not work, we can easily create a branch from any earlier commit, and build images off that branch.</p> <p>This will, we hope, also make it simpler to make changes, and give them back to upstream. Whenever we change anything, it'll be done in a branch, and we'll have system images available to test the change. So not only will upstream be able to easily get the change from our git repository, they'll also be easily verify, on a running system, that the change fixes the problem.</p> <h2>Automatic testing, maybe even test driven development</h2> <p>We will be automatically building system images from git commits for Baserock. This will potentially result in a very large number of images. We can't possibly test all of them manually, so we will implement some kind of automatic testing. The details of that are still under discussion.</p> <p>I hope to be able to start adding some test driven development to Baserock systems. In other words, when we are requested to make changes to the system, I want the specification to be provided as executable tests. This will probably be impossible in real life, but I can hope.</p> <p>I've talked about doing the same thing for Debian, but it's much harder to push through such changes in an established, large project.</p> <h2>Solving the install, upgrade, and downgrade problems</h2> <p>All mainstream Linux distributions are based on packages, and they all, pretty much, do installations and upgrades by unpacking packages onto a running system, and then maybe running some scripts from the packages. This works well for a completely idle system, but not so well on systems that are in active use. Colin Walters again talks about this.</p> <p>For installation of new software, the problem is that someone or something may invoke it before it is fully configured by the package's maintainer script. For example, a web application might unpack in such a way that the web server notices it, and a web browser may request the web app to run before it is configured to connect to the right database. Or a GUI program may unpack a <code>.desktop</code> file before the executable or its data files are unpacked, and a user may notice the program in their menu and start it, resulting in an error.</p> <p>Upgrades suffer from additonal problems. Software that gets upgraded may be running during the upgrade. Should the package manager replace the software's data files with new versions, which may be in a format that the old program does not understand? Or install new plugins that will cause the old version of the program to segfault? If the package manager does that, users may experience turbulence without having put on their seat belts. If it doesn't do that, it can't install the package, or it needs to wait, perhaps for a very long time, for a safe time to do the upgrade.</p> <p>These problems have usually been either ignored, or solved by using package specific hacks. For example, plugins might be stored in a directory that embeds the program's version number, ensuring that the old version won't see the new plugins. Some people would like to apply installs and upgrades only at shutdown or bootup, but that has other problems.</p> <p>None of the hacks solve the downgrade problem. The package managers can replace a package with an older version, and often this works well. However, in many cases, any package maintainer scripts won't be able to deal with downgrades. For example, they might convert data files to a new format or name or location upon upgrades, but won't try to undo that if the package gets downgraded. Given the combinatorial explosion of package versions, it's perhaps just as well that they don't try.</p> <p>For Baserock, we absolutely need to have downgrades. We need to be able to go back to a previous version of the system if an upgrade fails. Traditionally, this has been done by providing a "factory reset", where the current version of the system gets replaced with whatever version was installed in the factory. We want that, but we also want to be able to choose other versions, not just the factory one. If a device is running version X, and upgrades to X+1, but that version turns out to be a dud, we want to be able to go back to X, rather than all the way back to the factory version.</p> <p>The approach we'll be taking with Baserock relies on btrfs and subvolumes and snapshots. Each version of the system will be installed in a separate subvolume, which gets cloned from the previous version, using copy-on-write to conserve space. We'll make the bootloader be able to choose a version of the system to boot, and (waving hands here) add some logic to be able to automatically revert to the previous version when necessary. We expect this to work better and more reliably than the current package based one.</p> <h2>Making choices</h2> <p>Debian is pretty bad at making choices. Almost always, when faced with a need to choose between alternative solutions for the same problem, we choose all of them. For example, we support pretty much every <code>init</code> implementation, various implementations of <code>/bin/sh</code>, and we even have at least three entirely different kernels.</p> <p>Sometimes this is non-choice is a good thing. Our users may need features that only one of the kernels support, for example. And we certainly need to be able to provide both mysql and postresql, since various software we want to provide to our uses needs one and won't work with the other.</p> <p>At other times, the inability to choose causes trouble. Do we really need to support more than one implemenation of <code>/bin/sh</code>? By supporting both dash and bash for that, we double the load on testing and QA, and introduce yet another variable to deal with into any debugging situation involving shell scripts.</p> <p>Especially for core components of the system, it makes sense to limit the flexibility of users to pick and choose. Combinatorial explosion déjà vu. Every binary choice doubles the number of possible combinations that need to be tested and supported and checked during debugging.</p> <p>Flexibility begets complexity, complexity begets problems.</p> <p>This is less of a problem at upper levels of the software stack. At the very top level, it doesn't really matter if there are many choices. If a user can freely choose between vi and Emacs, and this does not add complexity at the system level, since nothing else is affected by that choice. However, if we were to add a choice between glibc, eglibc, and uClibc for the system C library, then everything else in the system needs to be tested three times rather than once.</p> <h2>Reducing the friction coefficient for system development</h2> <p>Currently, a Debian developer takes upstream code, adds packaging, perhaps adds some patches (using one of several methods), builds a binary package, tests it, uploads it, and waits for the build daemons and the package archive and user-testers to report any problems.</p> <p>That's quite a number of steps to go through for the simple act of adding a new program to Debian, or updating it to a new version. Some of it can be automated, but there's still hoops to jump through.</p> <p>Friction does not prevent you from getting stuff done, but the more friction there is, the more energy you have to spend to get it done. Friction slows down the crucial hack-build-test cycle of software development, and that hurts productivity a lot. Every time a developer has to jump through any hoops, or wait for anything, he slows down.</p> <p>It is, of course, not just a matter of the number of steps. Debian requires a source package to be uploaded with the binary package. Many, if not most, packages in Debian are maintained using version control systems. Having to generate a source package and then wait for it to be uploaded is unnecessary work. The build daemon could get the source from version control directly. With signed commits, this is as safe as uploading a tarball.</p> <p>The above examples are specific to maintaining a single package. The friction that really hurts Debian is the friction of making large-scale changes, or changes that affect many packages. I've already mention the difficulty of making large transitions above. Another case is making policy changes, and then implementing them. An excellent example of that is in Debian is the policy change to use <code>/usr/share/doc</code> for documentation, instead of <code>/usr/doc</code>. This took us many years to do. We are, I think, perhaps a little better at such things now, but even so, it is something that should not take more than a few days to implement, rather than half a decade.</p> <h2>On the future of distributions</h2> <p>Occasionally, people say things like "distributions are not needed", or that "distributions are an unnecessary buffer between upstream developers and users". Some even claim that there should only be one distribution. I disagree.</p> <p>A common view of a Linux distribution is that it takes some source provided by upstream, compiles that, adds an installer, and gives all of that to the users. This view is too simplistic.</p> <p>The important part of developing a distribution is choosing the upstream projects and their versions wisely, and then integrating them into a whole system that works well. The integration part is particularly important. Many upstreams are not even aware of each other, nor should they need to be, even if their software may need to interact with each other. For example, not every developer of HTTP servers should need to be aware of every web application, or vice versa. (It they had to be, it'd be a combinatorial explosion that'd ruin everything, again.)</p> <p>Instead, someone needs to set a policy of how web apps and web servers interface, what their common interface is, and what files should be put where, for web apps to work out of the box, with minimal fuss for the users. That's part of the integration work that goes into a Linux distribution. For Debian, such decisions are recorded in the Policy Manual and its various sub-policies.</p> <p>Further, distributions provide quality assurance, particularly at the system level. It's not realistic to expect most upstream projects to do that. It's a whole different skillset and approach that is needed to develop a system, rather than just a single component.</p> <p>Distributions also provide user support, security support, longer term support than many upstreams, and port software to a much wider range of architectures and platforms than most upstreams actively care about, have access to, or even know about. In some cases, these are things that can and should be done in collaboration with upstreams; if nothing else, portability fixes should be given back to upstreams.</p> <p>So I do think distributions have a bright future, but the way they're working will need to change.</p> Obnam 0.21 released (backup software) https://blog.liw.fi/posts/obnam-0.21/ https://blog.liw.fi/posts/obnam-0.21/ debian obnam Tue, 23 Aug 2011 22:03:47 +0000 2011-08-23T22:03:47Z <p>I've today released version 0.21 of <a href="http://braawi.org/obnam/">Obnam</a>, my backup application. This is the first version that also got uploaded to Debian. It is currently waiting for manual processing in the NEW queue.</p> <p>From the NEWS file:</p> <p>USER VISIBLE CHANGES:</p> <ul> <li>Obnam will now unlock the repository if there's an error during a backup. For the most part, the <code>force-lock</code> operation should now be unnecessary, but it's still there in case it's useful some day.</li> </ul> <p>BUG FIXES:</p> <ul> <li>Negative timestamps for files now work. Thanks to Jamil Djadala for reporting the bug.</li> <li>The documentation for --checkpoint units fixed. Thanks, user weinzwang from IRC.</li> <li>The connections to the repository and live data filesystem are now properly closed. This makes benchmark read/write statistics be correct.</li> </ul> Missing roles in software: build engineers https://blog.liw.fi/posts/sw-roles-build-engineers/ https://blog.liw.fi/posts/sw-roles-build-engineers/ debian programming rant Sun, 07 Aug 2011 10:22:55 +0000 2011-08-07T10:22:55Z <p>The construction, deployment, and use of software involves a number of roles. I do not have an exhaustive list, but I have a couple in my list that seem to be surprising to some developers.</p> <p>In the traditional Unix (mainframe) model, there are three roles:</p> <ul> <li><strong>developers</strong> actually create the software; they include architects, coders, documenters, testers, maintenance programmers, etc; their deliverable is a release tarball</li> <li><strong>sysadmins</strong> install the software on a host, and install new versions when appropriate; often, the sysadmin is expected to carefully read <code>README</code> files, and construct or tweak configuration files accoriding to the specific needs of their users; their deliverable is some files in <code>/usr/local</code> and <code>/etc</code>, plus perhaps an announcement e-mail to an internal mailing list</li> <li><strong>users</strong> use the software; they have no deliverable in this context</li> </ul> <p>In my personal reality distortion bubble, this model is badly broken, and has been since the middle of the 1990s. Here's an updated model:</p> <ul> <li><strong>users</strong> control their own machines, and are not experts on the software, or on using computers at all, and requiring them to be would be to limit their freedom in important ways; the big change is that users now have much more control of their computing environment than before</li> <li><strong>sysadmins</strong> mainly work for companies or other organizations; they need not be experts on what their users do, and in fact often aren't, but they know how to handle systems that are by nature too complicated or hard for the layman to handle, because of performance, reliability, security, or other such needs</li> <li><strong>distro developers</strong> create a system consisting of thousands or tens of thousands of components (mostly provided by upstream developers), and integrate and adjust things so that they work together; they also monitor upstream projects and other news sources for updates, and provide those</li> <li><strong>upstream developers</strong> actually create the software; they include architects, coders, documenters, testers, maintenance programmers, etc;<br/> their deliverable is a release tarball (no change from earlier)</li> </ul> <p>A common misconception is that distro developers merely compile software and shove the binaries into glorified tarballs. That is, of course, part of the job, but it's also the part that most distro's tools completely automate. There's a lot more to do, though. For example, just keeping track of known security problems among thirty thousand packages, and making or finding fixes for them, and testing the fixes, and releasing updates for them, takes up quite a bit of effort.</p> <p>Another thing distro developers have to spend effort on is dealing with the multitude of build systems upstream projects use. The freedom to experiment with different solutions is one of the cornerstones of free software development. However, when it comes to build systems, many of them are not serious experiments, and instead are quick hacks thrown together to imperfectly solve problems that sometimes should not exist at all.</p> <p>This highlights one role prominently still missing: <strong>build engineer</strong>. Their task is to solve the problem of taking the upstream source code, dealing with build-time configuration issues, actually building the software, and then installing it in whatever location the person installing the software wants it installed. Further, their task is to provide the software to do this in a sensible manner.</p> <p>Build engineers already exist, of course. For example, the developers of the autotools suite are such. Every upstream project also has someone who sets up autotools for their project, or some other build tools, or writes them from scratch for that project.</p> <p>However, what's lacking, perhaps, is recognition that build engineering is an important part of software development. Mostly, build systems are set up (or, worse, written) by upstream developers to fit their narrow view of how things should work, a view anchored in the old, broken model of roles.</p> <p>This attitude matters, because for the most part, build systems do not seem to be written to the same quality standards as the rest of the upstream software. For example, when an upstream developer writes a <code>Makefile</code>, and does not consider distro development, they'll often make it hard to differentiate between install-time location and run-time location of various components of the software. Thus, it might be hard for the Debian packager to get the software installed into <code>debian/tmp</code> rather than <code>/usr/local</code>, because the build system hardcodes <code>debian/tmp/usr/lib/foo/plugins</code> as the location of the plugins, since that's where the software gets installed, even if, in the Debian scenario, it's not where it gets run from.</p> <p>Luckily for much software these problems have been solved or avoided. Indeed, as crufty as autotools is, it's one of the things that has kept upstreams sane about their build systems, by providing such good tools that it's more work to write a new, bad one than to use autotools.</p> <p>We currently have roving bands of translators, who move from project to project and translate them into various natural languages. I would like to see a future where there are people dedicated to build engineering, who go from project to project and set up or fix their build problems. A good build system, once set up, should require a minimum of work to keep working.</p> <p>I'd also like to see much improvement in the build software. As good as autotools has been to free software, I'm sure there's a lot of room for improvement. For example, autotools requires setting up a bunch of boilerplate files in each project, which seems like something that could be avoided by relying on specific conventions. However, those improvements are a separate topic. I might go into them some day, if I decide to start developing build tools myself, but until then, I'll let those who do that decide how they want to do it.</p> <p>PS. There are lots more missing roles in my model of free software development, of course. For example, security engineers (find and fix problems, but also provide ways to prevent problems, or limit their scope), internal politicians (keep the project's human wheels from squeaking too much), diplomats (keep different projects from clashing too much), release engineers, etc. Someone should make an exhaustive list.</p> Using OpenPGP subkeys in Debian development https://blog.liw.fi/posts/subkeys/ https://blog.liw.fi/posts/subkeys/ debian Fri, 05 Aug 2011 14:36:47 +0000 2011-08-05T14:36:47Z <p>I don't want to keep my GPG private keys on my laptop, so I've been keeping them on an encrypted USB drive instead, and loading them onto the laptop only when needed. That's cumbersome, when making Debian packages, and needing to sign the packages frequently. So I'm going to start using subkeys instead. However, I couldn't find good documentation for that, so after figuring things out, I wrote something that will hopefully be useful for others.</p> <p>It's at <a href="http://wiki.debian.org/subkeys">http://wiki.debian.org/subkeys</a> for your fixing pleasure.</p> Chunkwise Debian package downloads https://blog.liw.fi/posts/chunkwise-debian-downloads/ https://blog.liw.fi/posts/chunkwise-debian-downloads/ debian idea Sat, 18 Jun 2011 18:39:39 +0000 2011-06-18T18:39:39Z <p>Note: This is an idea I had, not something I'm working on. If it interests you, please work on it. I may come back to this some day, but not anytime soon, but I thought I'd share it anyway.</p> <p>When downloading updates for a Debian package, often much of the data doesn't actually change, or at least not much. Because of this, the <strong>debdelta</strong> service is a wonderful idea (and Debian should adopt it officially).</p> <p>However, a different way of achieving that might be like this:</p> <ul> <li>split up contents of Debian binary packages into chunks of suitable size (which may vary depending on the contents)</li> <li>each chunk has an identifier</li> <li>there is a mapping from checksums of chunks to their identifiers, and vice versa</li> <li>a .deb consists of a list of chunk ids, instead of the chunks themselves</li> <li>there are also "patch chunks", which say how to modify another chunk (e.g., modify offsets in compiled code when it's been moved in a new version, but is otherwise the same)</li> </ul> <p>When downloading updates, apt would look up the chunks ids and checksums, and see if it already has such chunks installed locally. If so, it can reuse those chunks. (Note that apt can look for chunks in the installed files, and does not need to cache things separately.)</p> <p>Now, I have not done the research to verify that this is actually worthwhile. Someone should experiment with different sizes of chunks to see if enough bandwidth could be saved to make this worthwhile, in an upgrade from one Debian release to another (lenny to squeeze, for example).</p> <p>Any takers?</p> Test driven distribution development https://blog.liw.fi/posts/debian-tdd/ https://blog.liw.fi/posts/debian-tdd/ debian tddd Wed, 08 Jun 2011 08:06:34 +0000 2011-06-08T08:06:34Z <blockquote><p>Disclaimer: There is a huge discussion going on right now in the <a href="http://lists.debian.org/debian-devel/2011/05/threads.html">debian-devel</a> mailing list, about different ways to rearrange how Debian development happens, and maybe to provide a Debian users with an additional release. This blog entry is not a comment on that discussion, which I am ignoring. The discussion did, however, prompt a <a href="http://identi.ca/notice/72416032">dent</a> about "test driven development". That prompted a short exchange of thoughts with Tom Marble, and this blog post is a cleaned up version of my part in that discussion.</p></blockquote> <p>Currently Debian development happens roughly like this: all packages are uploaded to a part of the Debian archive called <strong>unstable</strong>. Once they've been in use for a while without serious problems, they get automatically copied into another part called <strong>testing</strong>. Every year or two, testing is frozen and any remaining problems are fixed, after which all of testing gets copied into yet another part of the archive called <strong>stable</strong>, and that's the actual release.</p> <p>There is a little bit of automatic testing, but only a little. Almost all testing is done by people, by them using unstable or testing on their usual computers. If they have problems, they report them.</p> <p>Contrast this with development using Test Driven Development, or <a href="http://en.wikipedia.org/wiki/Test-driven_development">TDD</a>, and other modern development methodologies. Here's a rough summary of what I do when I write (much) of my software.</p> <ul> <li>first write one or more automatic tests (unit or functional ones)</li> <li>write the actual code, enough to make all tests pass</li> <li>add more tests</li> <li>write more code, or change existing code</li> <li>repeat this until tests describe all the behavior desired of the code</li> </ul> <p>In addition, I measure <strong>coverage</strong> to make sure all parts of the code gets tested. I usually aim for 100% coverage, except for those parts that very hard or quite pointless to test. (That's easier to achieve than you'd think, but that's the topic of another blog post.)</p> <p>This all sounds like a lot of bureaucratic nonsense, but what I get out of this is this: once all tests pass, I have a strong confidence that the software works. As soon as I've added all the features I want to have for a release, I can push a button to push it out. (Well, actually there's a <a href="http://liw.fi/release/">little bit</a> more to it.)</p> <p>This does mean I never have bugs in my software. Of course I do. However, there's a lot fewer of them. In fact, there's so few of them, after tests pass, that I would almost be happy to make an automatic release after every successful run of "make check".</p> <p>Another aspect of the way I do development is distributed version control. The relevant feature is powerful branching and merging. Most things I develop happen in short-lived single-purpose branches: whenever I start work on a feature or bugfix, I create a branch for it. (Unless I'm feeling particularly lazy, which happens more often than you'd think.)</p> <p>When I've finished the feature, or fixed the bug, I merge the branch back into the <strong>trunk</strong> branch.</p> <p>This way, the trunk is always in releaseable state. It might not have all the features I want for the next release, but the features that are there, do work. The branch probably has bugs, but if I've written tests that are good enough, I know the software works well enough.</p> <p>Or that's the idea. Sometimes things go wrong. Then I write more tests and the next time goes better. (It doesn't have to be perfect, as long as it gets better every time.)</p> <p>See the contrast between automatic tests with good coverage, and the Debian style of relying on user feedback? There's no need to wait for user feedback with automatic tests. This speeds up development, makes releasing easier, but most importantly takes away any reason to fear making changes, even big changes.</p> <p>Automatic tests and good test coverage are easy achieve in small projects. For a system as huge as Debian, good test coverage is quite hard to achieve.</p> <p>The thing about automatic tests, though, is that even a little bit of it is helpful, and after you have a few tests, it gets easier to add more. The first test is the hardest to get done, since you need to set up all the infrastructure for it.</p> <p>Debian does do a bit of automatic testing already. (See lintian, piuparts, autopkgtest, edos, etc.) I don't want to belittle that, but I think we could do better.</p> <p>Here's what I would love to see:</p> <ul> <li>we have a part of the archive that corresponds to a trunk branch, i.e., it is always in a releasable state (the "testing" area was originally meant to be that; we could make it so now)</li> <li>releaseable state is determined by two things: <ul> <li>an automatic test suite</li> <li>user feedback, particularly in the form of bug reports (as now)</li> </ul> </li> <li>whenever changes are made, they happen in "branches" <ul> <li>the branch is not affected by changes elsewhere in the archive, except by manual synching ("merge from trunk")</li> <li>individual package uploads, as well as groups of packages such as for transitions, are each in their own branch</li> <li>this is sort of similar to a PPA, but more powerful</li> </ul> </li> <li>when a branch is to be merged into trunk, the automatic tests must first pass (or at least no new failures in them can be introduced) <ul> <li>tests can also be run for any other branch, of course, so that those developing the branch know if they're ready to push their changes into trunk</li> </ul> </li> <li>there's a culture of writing tests for bugs (whenever that is feasible), and for new features <ul> <li>particularly release goals should be expressed as automatic test suites</li> </ul> </li> <li>there's a culture of sharing tests with upstreams, and with other distributions</li> </ul> <p>Since full test coverage is going to be impossible for Debian, some subset should be targeted. Perhaps something like this would suffice to start with:</p> <ul> <li>a version of debian-installer is generated from a branch</li> <li>test installation of a new system from scratch <ul> <li>with a large set of tasks selected</li> </ul> </li> <li>test upgrade from previous stable release to the branch <ul> <li>with a large set of packages installed (as many as possible, actually)</li> </ul> </li> <li>test upgrade from trunk branch to branch to be merged <ul> <li>ditto large set of packages</li> </ul> </li> <li>test with lintian, reject on specific tests failing</li> <li>test with piuparts</li> <li>test the whole system for specific functionality <ul> <li>ssh access from outside</li> <li>sudo access for a logged in user</li> <li>sending mail with SMTP</li> <li>web server</li> <li>possibly test specific web applications</li> <li>possibly test Samba and NFS services</li> <li>possibly test printing services (CUPS)</li> <li>possibly test essential desktop functionality (automatic login, at least)</li> </ul> </li> <li>if package comes with package specific tests, run those too</li> </ul> <p>These tests would not guarantee that a set of changes would not break Debian, but they would give a high confidence that at least basic stuff would still work after the changes.</p> <p>Now, obviously implementing all of this I'm dreaming of (it is just past midnight, after all) is going to be impossible. There's way too much work to do, there's not enough tools for writing tests, and it would require too much computing power to run, and so on and so forth. But it's late, and I've had a bad day, and I might as well dream of something nice.</p> <p>Anyway, anyone interested in these things should perhaps help drive <a href="http://dep.debian.net/deps/dep8/">DEP8</a>.</p> TDDD systest tool proof of concept https://blog.liw.fi/posts/tddd-systest/ https://blog.liw.fi/posts/tddd-systest/ debian systest tddd Tue, 07 Jun 2011 14:16:39 +0000 2011-06-07T14:16:39Z <p>Continuing my earlier musings about test driven distro development, and the tools that would require.</p> <p>I imagine something like this:</p> <ul> <li>create a virtual machine with a particular configuration of stable</li> <li>run a test suite verifying the VM works</li> <li>upgrade the VM to testing and reboot</li> <li>re-run test suite, verifying things work in testing</li> </ul> <p>We might also want this scenario:</p> <ul> <li>install testing directly in a VM (instead of upgrading from stable)</li> <li>run test suite</li> </ul> <p>There might be other scenarios that would be useful to test as well. Even from these two, it's clear there's a need for at least three separate tools:</p> <ul> <li>create a machine image with a particular configuration <ul> <li>might be based on stable or testing or unstable</li> <li>might also be using some other package sources</li> <li>might want particular packages installed</li> <li>might want particular configuration settings</li> </ul> </li> <li>run a suite of tests against the running image <ul> <li>which tests to run will depend on the scenario</li> </ul> </li> <li>run various scenarios using the above two tools <ul> <li>needs reasonably easy ways to specify scenarios</li> <li>needs to support tests specific to scenarios</li> </ul> </li> </ul> <p>I've written a rudimentary version of the first tool: vmdebootstrap. I've since learned there's a bunch of others that might also work. There's room for more: someone should write (or find) a tool to make a snapshot of a real system and create a VM image that mimicks it, for example. Anyway, for now, I'll assume that one of the existing tools is good enough to get started.</p> <p>For the second tool, I wrote a quick-and-dirty proof-of-concept thing, see <a href="https://blog.liw.fi/posts/tddd-systest/systest.py">systest.py</a>. Here's a sample of how it might be used:</p> <pre><code>liw@havelock$ ./systest -v --target 192.168.122.139 --user tomjon test 1/6: cat test 2/6: only-ssh-port ERROR: Assertion failed: ['22/tcp', '139/tcp', '445/tcp'] != ['22/tcp'] [status 1] liw@havelock$ ./systest -v --target 192.168.122.139 --user tomjon \ cat ping6-localhost ping-localhost simple-dns-lookup ssh-login test 1/5: cat test 2/5: ping6-localhost test 3/5: ping-localhost test 4/5: simple-dns-lookup test 5/5: ssh-login liw@havelock$ </code></pre> <p>The first run failed, because the VM I'm testing against has some extra ports open. Some of the tests will require logging into the machine, via ssh, and for that one needs to specify the user to use.</p> <p><code>systool</code> may overlap heavily on system monitoring tools, and possibly the production implementation should be based on those.</p> <p>I think it's best to design such a tool for the more general purpose of testing whether a system currently works rather than as an integrated part of a more specific larger tool. This lets the tool be useful for other things than just testing specific things about Debian. (The production implementation would then need to not have all the tests hardcoded, of course. SMOP.)</p> <p>The third tool I have not spent a lot of time on yet. One thing at a time.</p> <p>Given these tools, one would then need to decide how to use them. The easiest way would be to use them like lintian and piuparts: run them frequently on whatever packages happen to be in testing or unstable or experimental, and put links to the test reports to the PTS, and hope that people fix things.</p> <p>That is the easiest way to start things.</p> <p>Once there's a nice set of test cases and scenarios, it may be interesting to think about more aggressive ways: for example, preventing packages from migrating to testing unless the test suite passes with them. If the tests do not pass, one of four things is broken:</p> <ul> <li>the package or packages in question</li> <li>other packages already in testing</li> <li>the tests themselves</li> <li>the test environment</li> </ul> <p>If things are set up properly, the last one should be rare. The other three always require manual inspection: it is not possible to automatically know whether the test itself, or the code it tests, is at fault. it is, however, enough to know that something is wrong. If the tests are written well, they should be robust enough to not be the culprits very often.</p> <p>(Someone wanting to make a rolling distrbution, or even better, making a monthly mini-release, might benefit from this sort of automated testing.)</p> TDDD, more thoughts https://blog.liw.fi/posts/debian-tdd-2/ https://blog.liw.fi/posts/debian-tdd-2/ debian tddd Mon, 16 May 2011 16:26:57 +0000 2011-05-16T16:26:57Z <p>My <a href="https://blog.liw.fi/posts/debian-tdd/">previous blog entry</a> about Test Driven Distro Development (TDDD) was rather rambling. Here is the condensed version, plus some further thoughts.</p> <p><strong>I would like to automatically determine whether "testing" is minimally ready to be released, at any given point in time.</strong></p> <p>To implement this, I would like to see Debian get tools that would allow us to define specific minimal functionality that should work in a Debian release, and automatically verify that it does. I would like us to use these tools to determine whether a release is ready for release, with manual override from the release team. Ideally, we would use the tools to prevent breakages from moving into "testing", so that it gets closer to always being releasable.</p> <p>Getting test coverage sufficiently high that we can replace the release team's judgement abour releasablity will never happen: Debian is rather too large and and complex for that. However, we can reap much benefit from even relatively small coverage.</p> <p>Here is my initial suggestion for the minimal criteria that should be satisfied, at all times, in "testing". They are aimed at making sure that even if something else breaks, at least the basic stuff works well enough that one can log in and fix things.</p> <ul> <li>there are no open ports in a base install</li> <li>logging in via getty works</li> <li>logging in via ssh works, if openssh-server is installed</li> <li>su, sudo, ls, cp, mv, rm, and vi work at least in some typical use cases</li> <li>web pages can be accessed from outside the system, given suitably configured apache2</li> <li>after automatic login into a graphical desktop environment, all the crucial per-session programs start up, as well as web browsers and other GUI programs that almost everyone expects to be part of a desktop environment <ul> <li>if there are good GUI testing tools, we may want to test in more detail, but that may have to wait for later</li> </ul> </li> </ul> <p>These are <strong>minimal</strong> criteria. They're not meant to verify that everything in Debian works. After these work, adding more coverage is not just possible, but reasonably easy in many cases. Want to make sure DNS gets configured right? Write a test that logs in and does a lookup inside the test system. After the tools have evolved, writing a simple test like that should be only a line or two.</p> <p>The tests I am talking here are system tests, or acceptance tests for the whole system, not just for individual packages. Unit tests would test individual functions or methods or classes; integration tests would test groups of them, or possibly entire programs; system tests test groups of programs/packages.</p> <p>The point is not to verify that every package works in every possible case, but that the system as a whole works. Most package-specific functional tests should be part of the package's own test suite. We should also run those, but that's somewhat orthogonal to the vision I'm trying to articulate here.</p> <p>At least the following scenarios should be tested (the above tests should be run after each scenario):</p> <ul> <li>install "stable" and upgrade to "testing"</li> <li>install "testing" using latest d-i release, and upgrade to current "testing"</li> <li>use an install from the previous successful test run, and upgrade to the current "testing" <ul> <li>this answers the question "if testing worked yesterday, does it still work if we add these new packages"</li> </ul> </li> </ul> <p>It may be useful to run the tests against "unstable" as well as "testing". However, rejecting an upload to "unstable" may be less wise than preventing packages from being copied from "unstable" to "testing". Exactly how this the development processes should adopt TDDD is an interesting question, but a reasonable toolset should be adaptable to anything we want to do.</p> <p>(Indeed, it would be possible to structure the toolset so that it isn't specific to developing a distro: a "system-test" tool that a sysadmin could use to verify that a server works would probably be useful to some people. Perhaps such a tool exists and could be adapted for TDDD?)</p> <p>Running these tests for security updates to "stable" would obviously also be a possibility that should perhaps be considered.</p> <p>Running these tests would happen something like this:</p> <ul> <li>generate a suitable virtual machine disk image, configured in the proper way for the test</li> <li>run the image under virtualization (or on real hardware if that can be remote-controlled suitably)</li> <li>run the tests against the image</li> <li>if tests fail, archive the image (or sufficient information to reproduce the start image) <ul> <li>also, automatically file bugs against all packages involved in the test, and e-mail debian-devel-announce with the names of the people who uploaded the new versions of packages compared to the previous successful test run, Cc'ing their spouses and parents for further shaming, and also Cc'ing the Debconf organization team so they can stood up before an audience armed with rotten tomatoes and eggs (this bullet point is here only to see if anyone actually reads my blog posts)</li> </ul> </li> </ul> <p>I would further like to see Debian make it possible to have isolated branches for development. This would help with, for example, large transitions that take a long time. The transition could be prepared in a branch of its own, and when it's ready to be pushed into unstable, the automatic tests outlined above could be used to verify the transition doesn't break anything too critical. (If the tests fail, the push fails; if necessary, that can be overridden manually, but that should be rare.)</p> <p>In my view, branches would greatly benefit Debian development, but they're not necessary for starting to implement TDDD.</p> <p>My condensed versions are still too long.</p> vmdebootstrap and FreedomBoxLite https://blog.liw.fi/posts/fboxlite/ https://blog.liw.fi/posts/fboxlite/ debian freedombox vmdebootstrap Sun, 15 May 2011 17:21:41 +0000 2011-05-15T17:21:41Z <p>I played around some with with <a href="http://liw.fi/vmdebootstrap/">vmdebootstrap</a>, my little toy to generate a disk image with a pre-installed Debian system. I added some more features, and to show them off, created a simplistic, proof-of-concept image to demostrate what <a href="http://wiki.debian.org/FreedomBox">FreedomBox</a> might do.</p> <p>The image includes Apache and an SSH server, and a pre-configured user. You can use SFTP to put files on the server, and HTTP to access them. An instant file-sharing appliance.</p> <p>A real FreedomBox should, of course, do all sorts of other things, and this filesharing should probably be done differently, too. This is not a demonstration of the FreedomBox. It's a demonstration of vmdebootstrap.</p> <p>The following command builds the image. Passwords are fixed, so you'll want to not run the image on a public network.</p> <pre><code>sudo vmdebootstrap \ --log fboxlite.log \ --log-level debug \ --image fboxlite.img \ --hostname fboxlite \ --size 1G \ --verbose \ --enable-dhcp \ --package ssh \ --package apache2 \ --mirror http://mirror.lan/debian/ \ --customize=./fboxlite-customize \ --root-password=password1 </code></pre> <p>You'll need the following customization script as well:</p> <pre><code>#!/usr/bin/python import crypt import os import subprocess import sys def runcmd(argv, stdin='', ignore_fail=False, **kwargs): p = subprocess.Popen(argv, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = p.communicate(stdin) if p.returncode != 0: msg = 'command failed: %s\n%s\n%s' % (argv, out, err) if not ignore_fail: raise Exception(msg) return out print 'Customizing fboxlite' rootdir = sys.argv[1] # Create a fboxlite account. runcmd(['chroot', rootdir, 'adduser', '--gecos', 'Freedombox Lite', '--disabled-password', 'fboxlite']) encrypted = crypt.crypt('password1', '..') runcmd(['chroot', rootdir, 'usermod', '-p', encrypted, 'fboxlite']) # Create /srv/shared for holding shared files. shared = os.path.join(rootdir, 'srv', 'shared') os.mkdir(shared) runcmd(['chroot', rootdir, 'chown', 'fboxlite:fboxlite', 'srv/shared']) os.chmod(shared, 0775) # Configure Apache to serve /srv/shared as /shared. conf = os.path.join(rootdir, 'etc', 'apache2', 'sites-available', 'default') text = open(conf, 'r').read() lines = text.splitlines() new_lines = [ 'Alias /shared "/srv/shared"', '&lt;Directory "/srv/shared/"&gt;', 'Options Indexes', '&lt;/Directory&gt;', ] lines = lines[:-1] + new_lines + lines[-1:] text = ''.join('%s\n' % line for line in lines) open(conf, 'w').write(text) </code></pre> <p>If you're using libvirt with KVM, you can then use the following command to deploy the disk image:</p> <pre><code>virt-install --connect qemu:///system -n fboxlite -r 512 \ --os-type linux --os-variant debiansqueeze \ --disk $(pwd)/fboxlite.img --vnc --import </code></pre> <p>Log in via the emulated console, and run <code>ip addr</code> to see what the IP address of the virtual machine is (and if you can find a way to query libvirt what the address is, please tell me). Then you can use the following URLs (in, for example, Nautilus):</p> <pre><code>sftp://fboxlite@192.168.122.133/srv/shared/ http://192.168.122.133/shared/ </code></pre> <p>Replace the correct IP address in the above URLs and off you go.</p> vmdebootstrap, for real https://blog.liw.fi/posts/vmdebootstrap-2/ https://blog.liw.fi/posts/vmdebootstrap-2/ debian Fri, 13 May 2011 10:44:50 +0000 2011-05-13T10:44:50Z <p>Some months ago I wished for a <a href="https://blog.liw.fi/posts/vmdebootstrap/">tool to create virtual disk images</a>, something similar to <code>debootstrap</code> except it would create a disk image instead of a directory. It would be used like this:</p> <pre><code>sudo ./vmdebootstrap --image test.img --size 1G \ --log test.log --mirror http://mirror.lan/debian/ </code></pre> <p><code>debootstrap</code> is wonderful for creating a basic Debian system for use with <code>chroot</code>. In the modern world, something similar but for virtual machines (such as KVM) would be nice to have, I thought. Particularly if it is something that is not tied to a particular virtualization technology, and does not require running in an actual virtual machine (for speed and simplicity).</p> <p>I could imagine using a tool like this for automatically building appliance-like system images, for example for <a href="http://wiki.debian.org/FreedomBox">FreedomBox</a>. Being able to do that in a fully automatic manner would make development simpler and more easy for others to reproduce, and would allow for things like automatic integration testing: build image, boot it in a virtual machine, run automatic tests, report results.</p> <p>There are a bunch of tools that almost do this, but not quite. For example, <code>live-build</code> builds a disk image, but it's one that is aimed for live CDs and such. This brings in some limitations. By default, the image is not persistent. You can make a persistent image, but that requires you to provide a second partition (or disk) that holds the modified parts, using copy-on-write. It also means that if the bootloader or initramfs needs to be updated, then the read-only image needs to be updated. That's fine for live CDs, but not so good if you want a real installed system that you can upgrade to all future releases of Debian. (Ben Armstrong kindly helped me use <code>live-build</code> the right way. Thanks!)</p> <p>There's also <code>grml-debootstrap</code> which seems to work on a real system, but not installing onto a virtual disk image (and the difference is crucial to me).</p> <p>Based on feedback I received back then, and some experimentation I've done the past couple of days, I wrote a very rudimentary version of the tool I want. The <a href="https://gitorious.org/vmdebootstrap">code is on gitorius</a>. (You need my <a href="http://liw.fi/cliapp/">cliapp</a> library to run it.)</p> <p>Be warned: it really is a very rudimentary version. For example, it provides not flexibility in choice of partitioning, filesystem type, what packages get installed, etc. It does not configure networking, and the root password is disabled. Some of that flexibility would be quite easy to add.</p> <p>It also uses <code>extlinux</code> as the bootloader, since I utterly failed to get <code>grub2</code> to install onto the virtual disk. (I did, however, manage to overwrite my develpment machine's bootloader.)</p> <p>The command above works, and results in a RAW format disk image that boots under KVM. It might boot off a hard disk or USB flashdrive too, but I haven't tried that.</p> <p>I now have several questions:</p> <ul> <li>Does something like this already exist?</li> <li>Does anyone else find this useful?</li> <li>Is another approach better? Perhaps running a fully preseeded debian-installer in a virtual machine might be better.</li> <li>If no other tool exists for the same job, what should I do next with vmdebootstrap?</li> </ul> Beep be gone https://blog.liw.fi/posts/beepless/ https://blog.liw.fi/posts/beepless/ debian linux Sat, 07 May 2011 11:35:49 +0000 2011-05-07T11:35:49Z <p>It is a fundamental part of my belief structure that computers should be silent, unless told otherwise. A default install of the operating system should not make any sounds, unless there is a really good reason to. Thus, I object to the following:</p> <ul> <li>GDM beeping when it is ready to let me log in</li> <li>login sounds</li> <li>sound effects whenever I do anything, such as open or close windows</li> <li>sound effects whenever the computer wants to tell me something that isn't particularly important, such as new e-mail, IRC or IM messages to me, the networking going up or down, or a hard disk going missing</li> <li>a beep when I reboot or shutdown the computer</li> </ul> <p>I'm sure there are situations when it would be OK for my computer to make noise to alert me. For example, supposed I'm asleep, and a zombie has penetrated my defensive perimeters and is about to chop off the top of my skull so it can devour my brain. In that case, I would be OK with my computer beeping. Once.</p> <p>Some remedies:</p> <ul> <li>With gdm3, edit <code>/etc/gdm3/greeter.gconf-defaults</code> to set <code>/desktop/gnome/sound/event_sounds</code> to <code>false</code></li> <li>Configure GNOME's sound theme to be "No sounds". (It should not be necessary to set "alert sounds" to muted, but it won't hurt either.)</li> <li>Run <code>amixer -q sset Beep off</code> to shut up <code>wall</code> and other such tools. (I made an init.d script to make that happen automatically, and to make sure I don't forget it when I install new machines for myself.)</li> </ul> <p>Now, it's possible that other people might like or need their computer to make sounds. I imagine, for example, that the GDM login-ready beep is to help people with vision problems. I'm all for that. However, I wish it be easier to shut things up in one simple place for those of us who don't need it.</p> <p>Edit: Thanks to <code>NP237</code>, <code>noshadow</code>, <code>lindi-</code>, <code>flightplan</code>, and others on <code>#debian-devel</code> with help on shutting up <code>wall</code>.</p> Blackmail as an argumentation technique https://blog.liw.fi/posts/blackmail/ https://blog.liw.fi/posts/blackmail/ debian rant Wed, 06 Apr 2011 15:44:56 +0000 2011-04-06T15:44:56Z <p>Imagine a conversation where some people would like to try to make Debian better, and several people say things like this:</p> <blockquote><p>as soon as you make this particular change, I will stop using Debian and all my customers will stop using Debian too</p></blockquote> <p>Quite a number of different possible responses pop up in my head immediately.</p> <ul> <li>you're trying to blackmail us; we're better off without you, regardless of who you or your customers are</li> <li>you're about four years old</li> <li>you're delusional: Debian is too hard a habit to get rid of, so you're going to be using Debian anyway, regardless of what we do</li> </ul> <p>More seriously: should this kind of behavior be acceptable on the Debian mailing lists? Not only is it insulting and anti-social, but trying to influence purely technical decisions by threats is not going to make Debian better.</p> DPL nomination period for 2011 https://blog.liw.fi/posts/dpl-2011/ https://blog.liw.fi/posts/dpl-2011/ debian Sat, 05 Mar 2011 10:02:00 +0000 2011-03-05T10:02:00Z <p>The <a href="http://lists.debian.org/debian-devel-announce/2011/03/msg00002.html">nominations for Debian Project Leader elections for 2011</a> are open. As always, I'm tempted to nominate myself, but after thinking about it, I again conclude that I shouldn't.</p> <p>If I did nominate myself, there's a couple of things I'd like to push for.</p> <ul> <li>re-think the new maintainer process: if you are known among existing members of the project to do good work for the project, and to be trustworthy, then you should be offered membership, and that means voting rights (upload and login rights are less important)</li> <li>in general, attract more new contributors to the project</li> <li>more rotation for key positions: it is good for the project if positions like DAM, DSA, secretary, etc, get new people into these positions, so that the knowledge and skills get spread out more</li> <li>improve how we deal with conflict, especially internally: we've reduced general flame levels a lot over the past fix years or so, but we still have problems dealing with conflict or disagreements <ul> <li>this should probably include a more explicit consensus on what is acceptable and what is unacceptable behavior within the project (lists, irc, bts, etc), and how this should be enforced</li> <li>this could also include education and training of how to deal with conflict; perhaps even about dealing with cross-cultural issues</li> <li>could also include neutral "ambassadors", "diplomats", or "hostage negotiators" that step into a heating situation and attempt to defuse it before it blows up</li> </ul> </li> <li>regular online training sessions, e.g., for packaging in general, packaging of particular kinds of applications, etc <ul> <li>Debian Women has run a number of such sessions over IRC, and Ubuntu runs such sessions regularly</li> <li>both popular and useful</li> <li>realtime access to a teacher is qualitatively different from reading books or web pages</li> </ul> </li> </ul> <p>The DPL position is inherently a political one, which is why nothing in the list above is technical. On the other hand, I want to concentrate my Debian time on doing stuff that is fun, and right now that means writing code. I want to write code for Debian.</p> Growing the Debian development community https://blog.liw.fi/posts/growing-the-debian-development-community/ https://blog.liw.fi/posts/growing-the-debian-development-community/ debian Tue, 08 Feb 2011 09:45:15 +0000 2011-02-08T09:45:15Z <p>In an <a href="http://www.blaghag.com/2011/02/when-gender-goes-pear-shaped.html">article about gender issues among American atheists</a>, Sharon Moss and Lyz Liddell give a list of things to consider to be better community, not just for women:</p> <blockquote><ul> <li>Be a leader. Take responsibility for the tone of your group. If potential new members are being made to feel uncomfortable and aren’t coming back, you’re doing it wrong. As a leader, it is your job to prioritize the comfort of your attendees in programs, group dynamics, and communications. Try to put yourself in other members’ shoes and also ask for feedback.</li> <li>Promote a sense of community. Take the time to socialize and get to know each other. If you’re group isn’t primarily a social group, thinking about adding some social time. Go to breakfast before the protest, compose your letters to the editor over coffee, or grab a pizza and beer after that lecture. When you know each other, you have each others backs. Being a jerk isn’t tolerated.</li> <li>Moderate discussions. Make sure everyone has the opportunity to participate- new people, quiet people, etc. Don’t let conversation be dominated by one or two people who must “win.”</li> <li>Embrace and accept different ways of communicating. Whether someone is an aggressive debater or not, make sure they’re still welcome. The other atheist in the room isn’t your enemy.</li> <li>Encourage subgroups. Every event your group hosts doesn’t need to appeal to your entire membership. Many women appreciate women-only space to express their nonbelief and to connect with other atheist women. Here at the Humanist Community of Central Ohio, we started a subgroup book club called Reasonable Women. When it grew to have about 25 regular members and was beginning to be a little too large and unwieldy to function as a book club, we created a second group, Heathen Chicks, which is just a social group that meets at a local cafe. It isn’t just about creating women only space, for us, this has been a way to draw more women into other events our group hosts.</li> <li>Foster women in leadership. Groups with women in leadership positions tend to have more women. Encourage women in your group to be visible in leadership.</li> </ul> </blockquote> <p>These should work pretty well for Debian as well. I mean for the project in general, for getting all kinds of new people in, and getting them to stay, not just women, even if the gender imbalance is perhaps the biggest glaring problem we have.</p> <p>As a concrete suggestion, maybe it would be a good idea to continue the <a href="http://debian-party-line.branchable.com/">Debian party line</a> that <a href="http://kitenet.net/~joey/">Joey</a> set up for the squeeze release. We already have IRC, but having an audio chat with your Debian peers might be a nice addition. Specifically a social one, instead of something intended to boost productivity.</p> Idea: addsysuser https://blog.liw.fi/posts/addsysuser/ https://blog.liw.fi/posts/addsysuser/ debian idea Sun, 06 Feb 2011 12:35:00 +0000 2024-11-14T15:54:19Z <p>EDIT in 2024 (originally from 2011): Most of this has since been implemented in the <code>adduser</code> package in Debian. See <a href="https://zug.network/@Zugschlus/113482077988124370">Marc Haber's message</a>.</p> <p>Debian got squeeze released, and ideas are brewing for what to do during the next release cycle. Here's an idea I had lying around. I'm throwing it out there, since I won't have time to do anything about it for at least a little while, but perhaps someone else thinks it's a good idea? Or, better yet, you can all tell me in comments why this is a bad idea, or at least why it should be designed in a different way.</p> <pre><code>NAME addsysuser, delsysuser - create and remove systems user in Debian pack‐ ages SYNOPSIS addsysuser [options] username delsysuser [options] username DESCRIPTION addsysuser creates a system user, and delsysuser removes it. The pur‐ puse of these tools is to make it easier for Debian packages to create and remove system users as the pacakges are installed and purged. These are wrappers around adduser(8) and deluser(8). addsysuser will create the system user, unless it already exists. If the user already exists, it does nothing. delsysuser removes the user. However, the system administrator may configure it to never remove the user, or to remove only some users. Use the /etc/adduser.conf for the settings: DELETESYSTEMUSERS Set to true or false depending on whether delsysuser should remove the system user or not. Defaults to true. OPTIONS -p, --packagePACKAGE Record the user as being used by a specific package. Many pack‐ ages can use the same user. delsysuser will only remove the user when no packages are using it anymore. --no-act Only pretend to call adduser or deluser without actually doing so. This allows easier testing of command lines. FILES /etc/adduser.conf Configuration file for addsysuser and delsysuser. EXAMPLE In the postinst script: addsysuser --package hello hellouser In the postrm script: delsysuser --package hello hellouser SEE ALSO adduser(8), deluser(8). </code></pre> Edinburgh squeeze release party https://blog.liw.fi/posts/squeeze-release-party/ https://blog.liw.fi/posts/squeeze-release-party/ debian Fri, 04 Feb 2011 19:25:14 +0000 2011-02-04T19:25:14Z <p>Moray and I decided to celebrate the Debian squeeze release in Edinburgh, Scotland.</p> <p><a href="http://wiki.debian.org/ReleasePartySqueeze/Scotland/Edinburgh">http://wiki.debian.org/ReleasePartySqueeze/Scotland/Edinburgh</a> has the details, and will get updated as necessary, but here's a repeat:</p> <ul> <li>When: Saturday 5 February, 1800 onwards</li> <li>Where: <a href="http://www.cumberlandbar.co.uk/find-us.html">Cumberland Bar</a></li> <li>What: beer and food</li> <li>More info: moray@debian.org</li> </ul> <p>If you like Debian, please join us.</p> DEP5 is CANDIATE https://blog.liw.fi/posts/dep5-candidate/ https://blog.liw.fi/posts/dep5-candidate/ debian Thu, 06 Jan 2011 21:51:08 +0000 2011-01-06T21:51:08Z <p>I have just pushed out the CANDIDATE version of DEP5, the proposal for a machine-readable format for debian/copyright files.</p> UDW session on piuparts https://blog.liw.fi/posts/udw-piuparts/ https://blog.liw.fi/posts/udw-piuparts/ debian piuparts presentation ubuntu Mon, 29 Nov 2010 09:18:04 +0000 2010-11-29T09:18:04Z <blockquote><p>The following is the text on which I based my Ubuntu Developer Week session on <a href="http://wiki.debian.org/piuparts">piuparts</a>. The actual session transcript can be found at <a href="https://wiki.ubuntu.com/MeetingLogs/devweek0901/PackageTesting">https://wiki.ubuntu.com/MeetingLogs/devweek0901/PackageTesting</a>.</p></blockquote> <h1>What is piuparts?</h1> <p>Piuparts is a tool to test that a .deb package can be installed and removed, and installed, upgraded, and removed.</p> <p>It is primarily meant for people who maintain and upload packages.</p> <p>It works by constructing a chroot with a minimal Debian or Ubuntu installation, and then installing the package being tested into the chroot. It also installs the dependencies.</p> <p>After this is done, it removes and then purges the package, and any other packages that got installed.</p> <p>Finally, it compares the state of the chroot before the package and its dependencies were installed, and after they got purged. It checks that all files are still there, and haven't been changed, and that there are no new files.</p> <p>Obviously, some things will always change, such as log files. Piuparts ignores those.</p> <h1>Basic usage</h1> <p>The very basic way of using piuparts is as follows:</p> <pre><code>piuparts foo.deb </code></pre> <p>Note that this can take a long time, and can produce quite a lot of output. You probably don't want to run the command just now.</p> <p>Because piuparts builds a chroot, it must run as root. Prefix the command with sudo. To avoid people accidentally copypasting commands when they shouldn't, I'm not prefixing it in examples.</p> <p>This will run the install/remove/purge test. If the package is already in the archive, it will also test that the package can be upgraded from the archive to the .deb you give.</p> <p>The other way of running piuparts is as follows:</p> <pre><code>piuparts -a foo </code></pre> <p>Here foo is the name of a package. Piuparts will fetch it from the archive and test it. This will, obviously, not test upgrades since there is no newer package.</p> <p>You can also test upgrading from release to release:</p> <pre><code>piuparts -a -d dapper -d hardy -d intrepid -d jaunty foo </code></pre> <p>This will take even longer.</p> <h1>Interpreting the results</h1> <p>Piuparts outputs quite a lot of text when it runs. This is because when I was developing it, it was quite useful to me to see what was going on and figure out when things broke. I never got around to fixing things so that it was nice to use for other people.</p> <p>Since the output can be pretty large, up to several hundred kilobytes, it is best to capture it in a file:</p> <pre><code>piuparts -l foo.log foo.deb </code></pre> <p>The output will go to the named file, and also to the screen.</p> <p>Every line of output piuparts outputs is prefixed with a relative timestamp (minutes and seconds since piuparts started), and a log level: DUMP, DEBUG, INFO, WARNING, ERROR, FATAL. DUMP and DEBUG you can mostly ignore; unfortunately there is no switch to let you do that at the moment.</p> <p>The useful stuff is in INFO, and in ERROR if it is there. Therefore, the way to read a piuparts log file is to grep for INFO and ERROR. Then, if those don't make any sense, read DUMP and DEBUG as well.</p> <p>When piuparts finishes, it outputs one of two things: either a PASS or a FAIL message. They look like these:</p> <pre><code>2m11.0s INFO: PASS: All tests. 5m33.8s ERROR: Broken symlinks: </code></pre> <p>Obviously these were from different runs.</p> <h1>Speeding it up</h1> <p>It can take a long time to construct the chroot. Since most package maintainers use pbuilder, and it constructs a chroot too, piuparts provides the --pbuilder (-p) option to use the pbuiler tarball to create the chroot. This makes things much faster.</p> <p>On the server I used to use to run piuparts on all packages in Debian, a couple of years ago, piuparts took only a few seconds per typical small package.</p> <h1>Testing packages for newer releases</h1> <p>Piuparts can test packages for newer releases, if you use the debootstrap package from backports.</p> <h1>Gotchas</h1> <p>Piuparts works on a chroot. This means that it does not properly protect the host system. A malicious package, or a really unfortunately buggy package, may ruin the host system. In the three years since I wrote piuparts, I have had that happen, but it's possible. The solution, of course, is to fix piuparts to use KVM.</p> <p>Piuparts can use a lot of CPU, RAM, disk I/O, and network bandwidth. Be prepared for that.</p> <p>Some of the checks piuparts does often fail, but they don't matter much in real life. The manual page has lists all options, so you can look there if something annoys you. For example, piuparts checks that all symlinks point somewhere that actually exists. For some reason, this is triggered fairly often, so you can disable it with the --no-symlinks (-N) option.</p> <p>For some reason piuparts in Ubuntu defaults to gutsy, since it hasn't been updated since the hardy release. You can blame me for that. Actually, don't blame me, file a bug in Launchpad, and subscribe me to it (liw on Launchpad). You can work around this by specifying the distribution manually with the -d option:</p> <pre><code>piuparts -d jaunty foo </code></pre> <h1>Summary</h1> <p>Do this:</p> <pre><code>sudo piuparts -d jaunty -p -l foo.log foo.deb </code></pre> <p>and grep foo.log for INFO and ERROR, and read surrounding text.</p> Debian VM image creation tool? https://blog.liw.fi/posts/vmdebootstrap/ https://blog.liw.fi/posts/vmdebootstrap/ debian wishlist Sun, 28 Nov 2010 15:10:29 +0000 2010-11-28T15:10:29Z <p>It is Sunday, and I was bored, and wanted to experiment a bit with a possible future direction for piuparts. To do that, I needed to create a virtual machine image with Debian installed in it. I failed.</p> <p>I failed, because I am not familiar with all the stuff I need to be to pull this off. More shame to me.</p> <p>There's a lot of possible ways to do this, and I tried a few, but ran out of time before I could exhaust all opportunities. Somewhere out there is probably the magic rune I am looking for, and if so, please point it out to me. I might want to get back to this problem say day, and having a working rune would simplify things a lot.</p> <p>Here are things I tried:</p> <ul> <li>Create a base system with <code>debootstrap</code>, then convert that into a bootable image: create image with <code>qemu-img</code>, partition it with <code>parted</code> (so I can have a swap partition), create devices for the partitions with <code>kpartx</code>, create filesystem and copy chroot into the partition. Reason for failure: could not find a way to install kernel or bootloader into the image.</li> <li>Live-helper, with various options. It wants to create a live CD, but I was told if I can build an image without the <code>live-boot</code> and <code>live-config</code> packages installed, it would be what I want. Reason for failure: <code>--initramfs none</code> either causes the image build to fail or the resulting image to not have a bootloader at all. I assume that is the right option, but don't know for sure. (<a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=605255">#605255</a>)</li> <li><code>qemubuilder</code> might do what I want, but failed to build anything at all, even after I worked around a bug by specifying architecture explicitly. (<a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=440944">#440944</a>, open since 2007)</li> </ul> <p>I did not try running debian-installer with a pre-seed file under kvm. I might try that some day, but today I did not get there yet. Ideally, I would like to build the image without running things inside a VM, just so it is as independent of specific VM technologies as possible.</p> <p>Quite a lot of people want to create virtual machine images easily. It would be really nifty to have a tool like this:</p> <pre><code>vmdebootstrap squeeze foo.img http://cdn.debian.net/debian </code></pre> <p>It would be a fairly low-level tool, just like <code>debootstrap</code> is, but having just one place where all the trickiness to create the initial image happens would simplify the overall complexity of other tools.</p> <p>Making it simple to use would allow other stuff to be easily built on top of the new tool. For example, piuparts could use it to create a VM image in which to run tests, rather than using chroot, which would allow more code paths to be tested.</p> <p>But I'm not ready to write the tool today, alas.</p> Packaging tutorial for Debian-Women https://blog.liw.fi/posts/dw-packaging-tutorial/ https://blog.liw.fi/posts/dw-packaging-tutorial/ debian Sat, 20 Nov 2010 08:48:57 +0000 2010-11-20T08:48:57Z <p>On Thursday, I gave a packaging tutorial for Debian-Women. The logs are now available on the <a href="http://wiki.debian.org/IntroDebianPackaging">Debian wiki</a>, and also on <a href="http://meetbot.debian.net/debian-women/2010/debian-women.2010-11-18-20.05.log.html">the meetbot site</a>. This was similar to the <a href="https://wiki.ubuntu.com/UbuntuDeveloperWeek">Ubuntu Developer Week</a> sessions I've participated in the past.</p> <p>Giving these tutorials is fun. It's also a great way to share your knowledge and skills. While most of the session is going to be uni-directional, with the teacher speaking and everyone else listening, IRC is an interactive medium, so whenever there is something unclear, the students can ask questions or make comments.</p> <p>And that's what makes these sessions different from people just reading a web page on their own: the chance to interact with the teacher.</p> <p>It would be great to see other Debian people do their own sessions. There are a lot of topics, and they don't even have to be about teaching. For example, a session where the Debian kernel team answers questions from the audience might be nice. Or the X team, or the DPL, or whoever. Debian's done these things for DPL elections on several years. The election sessions are hectic and pretty stressful for the participants, but that's because they're elections, not because of the sessions themselves.</p> <p>Any other Debian people up to this?</p> My Plans for Debian https://blog.liw.fi/posts/debian-plans/ https://blog.liw.fi/posts/debian-plans/ debian Thu, 09 Sep 2010 06:28:05 +0000 2010-09-09T06:28:05Z <p>I am a Debian developer again, and here's what I plan to do at first:</p> <ul> <li>continue driving <a href="http://dep.debian.net/deps/dep5/">DEP5</a> to conclusion</li> <li>help develop piuparts</li> <li>participate in <a href="http://wiki.debian.org/FreedomBox">Freedom Box</a> development</li> <li>write some other tools to aid Debian development, perhaps one to help maintainer scripts create/remove system accounts more easily</li> </ul> <p>I don't expect to do a lot of packaging work. I'll package, or help package, stuff that I develop, but I'll not be specifically be looking for stuff to package.</p> Oops, I did it again https://blog.liw.fi/posts/dd-again/ https://blog.liw.fi/posts/dd-again/ debian Wed, 08 Sep 2010 08:09:00 +0000 2010-09-08T08:09:00Z <p>I'm a Debian Developer again. This is the third time I've become one.</p> Debian CUT https://blog.liw.fi/posts/cut-point/ https://blog.liw.fi/posts/cut-point/ debian Thu, 12 Aug 2010 17:22:46 +0000 2010-08-12T17:22:46Z <p><a href="http://kitenet.net/~joey/">Joey Hess</a> had a BOF about <a href="http://penta.debconf.org/dc10_schedule/events/681.en.html">Constantly Usable Testing</a> at <a href="http://debconf10.debconf.org/">Debconf10</a> last week. I like the idea a lot, and here's why: I run Debian's testing distribution on my laptop. I like it to be reasonably up to date with current upstream versions (if only because the lenny versions often don't work with the hardware), but I don't want to take the risk of breakage I get from running Debian unstable.</p> <p>In fact, I don't want to the take the risk of breakage I get with running Debian testing, either. CUT seems to me to be an opportunity to get the benefits of running testing without most of its risks.</p> <p>Now, the breakage I'm concerned with isn't the kind of breakage that makes the machine as a whole unusable. I am more worried about the kinds of small changes that will inevitably happen that I can work around, but that require more time and effort I like to spend.</p> <p>For example, it might be that when upgrading this morning, something changes in Firefox, which requires me to restart that. Or there's a new kernel that requires a reboot. Or the update breaks Evolution, and I'll need to wait for a fix from unstable for a week, or have to backport it myself, perhaps by installing the relevant half-dozen packages directly from unstable.</p> <p>None of that is a disaster, but it's all a hassle, and it's all a distraction from what I really want to do.</p> <p>Having resonably frequent pseudo-releases, cuts, based on testing, would mean I only need to worry about things breaking when I upgrade to the next cut release. Meanwhile, I will have months of stress-free Debian use.</p> <p>I could almost achieve that by only uprading my laptop every few months to whatever happens to be current in testing at that time. However, there's no security support in that case. The big, central, blinking improvement with cut releases (if I understand them correctly) is that I would have security updates if I stay within the cut release. The security update would possibly just require installing the relevant packages and their updated dependencies, but that's good enough.</p> <p>And that's why I like cut, at least as an idea.</p> Debconf10 https://blog.liw.fi/posts/debconf10/ https://blog.liw.fi/posts/debconf10/ debconf10 debian Sun, 08 Aug 2010 00:47:35 +0000 2010-08-08T00:47:35Z <p>Some of my personal highlights from Debconf10:</p> <ul> <li><a href="http://www.branchable.com/">Branchable</a>. It happened during and at Debconf10, even if it is not directly related to Debian. All free software, though, all the way, and Franklin Street Statement -free as well.</li> <li><a href="http://wiki.debian.org/FreedomBox">Freedom Box</a>.</li> <li>Meeting old friends, and some new people as well. Putting faces and names together.</li> <li><a href="http://liw.fi/swimming-upstream/">My own talk</a>, of course.</li> <li><a href="http://soap.docstory.fi/posts/top/">Helicopter ride</a>.</li> </ul> <p>Oh yeah, and I decided to re-join Debian, and get a kilt.</p> Should I re-join Debian? https://blog.liw.fi/posts/re-join-debian/ https://blog.liw.fi/posts/re-join-debian/ debconf10 debian Fri, 06 Aug 2010 16:57:37 +0000 2010-08-06T16:57:37Z <p>I've spent the past couple of weeks at Debconf10. As usual, Debconf has a huge motivational effect on me, and I am excited about doing all sorts of things for the project.</p> <p>I'm still not all that interested in doing distribution development, which is why I retired last year. However, there's the <a href="http://wiki.debian.org/FreedomBox">Freedom Box</a> project starting, which is distribution development, and I'm quite excited about it, but on the whole, I don't want to fix other people's bugs, I want to make my own.</p> <p>Should I re-join Debian? If I do, what would I do?</p> <p>I could develop tools for Debian to use. That doesn't technically require being a DD, but if I'm doing work for the project, I want voting rights.</p> <p>I could perhaps do some work on piuparts. Or write various small tools that other people would find useful, and would make their own development go smoother.</p> <p>(Ideally, I would find people willing to pay me to do that, but I have no idea how to find such people.)</p> <p>And lest I forget, Debian is my tribe.</p> Brother P-touch QL-550 with Ubuntu 9.10 (karmic) https://blog.liw.fi/posts/brother-ql-550/ https://blog.liw.fi/posts/brother-ql-550/ debian linux ubuntu Fri, 13 Nov 2009 13:40:57 +0000 2009-11-13T13:40:57Z <p>I recently set up my Brother P-touch QL-550 label printer to work with CUPS on my laptop, which runs Ubuntu 9.10 (karmic). This required installing <a href="http://www.diku.dk/hjemmesider/ansatte/panic/P-touch/">ptouch-driver</a>, with some patches, and then hand-editing the PPD file that gets created.</p> <p>I made a very crude .deb package to install the ptouch-driver. I have no idea how to package CUPS stuff properly, so I didn't even try. I don't intend to maintain this in Debian or Ubuntu, I just needed it for myself, so I didn't go to much effor, I'm afraid. Perhaps it's useful anyway. I filed a "Request for Package" bug in Debian (<a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=555892">555892</a>) and talked with Till Kamppeter from Ubuntu about this, and he has taken a look at this (see comments in the bug).</p> <p>With my package, the generated PPD has some margins that make it almost impossible to fit anything on the label. The margins are only a few millimeters, but on labels that matters a lot. I fixed this by hand-editing the PPD, Till seems to have fixed in a better way.</p> <p>I hope this is useful to someone. The printer itself is pretty nice, too bad Brother doesn't provide free drivers.</p> Retired from Debian https://blog.liw.fi/posts/retired-from-debian/ https://blog.liw.fi/posts/retired-from-debian/ debian Tue, 06 Oct 2009 09:44:34 +0000 2009-10-06T09:44:34Z <p>I have retired from Debian today, in order to concentrate on other things that are, currently, more fun, such as attempting to become an upstream kind of guy.</p> Downloading updates faster https://blog.liw.fi/posts/downloading-updates-faster/ https://blog.liw.fi/posts/downloading-updates-faster/ debian ubuntu Tue, 11 Aug 2009 06:07:28 +0000 2009-08-11T06:07:28Z <p>One of the things I've been doing this release cycle for Ubuntu is look at ways in which updates can be downloaded faster. The conclusion seems to be that there are three relatively easy things we can do:</p> <ol> <li>Use <a href="http://packages.qa.debian.org/d/debdelta.html">debdelta</a> for getting updates to packages.</li> <li>Use <a href="http://zsync.moria.org.uk/">zsync</a> for getting updated Packages files.</li> <li>Use <a href="http://zsync.moria.org.uk/">zsync</a> for getting updated ISO files during testing.</li> </ol> <p>Some further notes:</p> <ul> <li>zsync is an implementation of the rsync algorithm that pre-computes the signatures (stored into a .zsync file), and uses HTTP range requests to get data from the server. It does not require anything above a normal http server, unlike rsync.</li> <li>debdelta computes binary deltas between package versions. It is good for upgrading between specific versions. zsync would work between any versions, but it requires downloading a lot more data. Although debdelta is less convenient, it should save so much bandwidth it is worth it.</li> <li>zsync works really well for Packages files, however, especially on uncompressed ones, but the gzip-compressed ones are really good, too.</li> <li>zsync works really well also for downloading test versions of ISOs during testing: the ISOs don't change all that much from day to day. zsync is actually much more efficient than rsync, using about a quarter of the bandwidth in my brief tests.</li> </ul> <p>I'm next going to be working on getting these into actual use for Ubuntu, and hopefully this work can benefit Debian as well.</p> <p>Some discussions on the ubuntu-devel mailing list:</p> <ul> <li><a href="https://lists.ubuntu.com/archives/ubuntu-devel/2009-July/028529.html">https://lists.ubuntu.com/archives/ubuntu-devel/2009-July/028529.html</a></li> <li><a href="https://lists.ubuntu.com/archives/ubuntu-devel/2009-July/028568.html">https://lists.ubuntu.com/archives/ubuntu-devel/2009-July/028568.html</a></li> <li><a href="https://lists.ubuntu.com/archives/ubuntu-devel/2009-July/028600.html">https://lists.ubuntu.com/archives/ubuntu-devel/2009-July/028600.html</a></li> </ul> Semi-retiring from Debian https://blog.liw.fi/posts/debian-semi-retirement/ https://blog.liw.fi/posts/debian-semi-retirement/ debian Tue, 28 Jul 2009 12:57:24 +0000 2009-07-28T12:57:24Z <p>I have only a fairly small amount of free time to allocate for Debian each week. At the moment, <em>all</em> of that time goes into keeping track of what is happening in the project, by reading IRC, e-mail, and blogs. This is a FAIL.</p> <p>After Debconf9, I'm going to start leaving IRC channels and mailing lists (except debian-*announce and piuparts-devel), and possibly stop following Planet Debian as well. I've already requested removal of two of my packages, leaving me with three (two of which are co-maintained with others).</p> <p>I am not, however, intending to actually retire. Ironically, by semi-retiring, I expect to get more done for Debian. I hope that by not spending so much keeping track of what others do, I will have some extra time to actually do some coding for Debian. I hope to get some actual piuparts work done.</p> Using zsync for .deb downloads: initial benchmark results https://blog.liw.fi/posts/zsync-deb-updates/ https://blog.liw.fi/posts/zsync-deb-updates/ debian ubuntu Tue, 14 Jul 2009 16:30:46 +0000 2009-07-14T16:30:46Z <blockquote><p>This is a repost to my blog of the message I sent to ubuntu-devel; see the archive for the <a href="https://lists.ubuntu.com/archives/ubuntu-devel/2009-July/028529.html">original</a>.</p></blockquote> <p>I'm working on a spec to add apt-sync support for karmic. See <a href="https://wiki.ubuntu.com/AptSyncInKarmicSpec">https://wiki.ubuntu.com/AptSyncInKarmicSpec</a> for details.</p> <p>I've done some initial benchmarking of using zsync to download .debs for updates, to see if apt-sync can be worth it. For simplicity, I've done the benchmarking using the underlying zsync tool rather than apt-sync.</p> <p>Background explanation: A .deb is, essentially, a very thin wrapper around a couple of tar files, which can be compressed with gzip, bzip2, or lzma. Zsync uses the rsync algorithm and some HTTP features to implement rsync without having to have an rsync server running, making it feasible to use it for all mirrors. apt-sync is an implementation of the idea that when downloading updates to a package you already have installed, there is no point in re-downloading the unchanged parts of the package.</p> <p>My benchmark consists of downloading all the security updates to hardy. The goal of the benchmark is to get some figures for how good zsync (and therefore apt-sync) actually is, or can be, at downloading .debs.</p> <p>Here's the summary results:</p> <pre><code>scenario % saved comment ----------------------------------- plain 3.7 plain zsync, original .debs zsyncmakeZ 3.7 use zsync gzip magic, original .debs rsyncable 25 gzip magic, recompress . with --rsyncable rsyncable2 33 gzip magic, compress ar, not within tar rsyncable3 33 gzip magic, convert lzma, bzip2 to gzip uncompressed 50 uncompress tarballs within .debs </code></pre> <p>The percent saved is the number of bytes zsync did NOT need to download, i.e., how much it reused from the previous package.</p> <p>Further explanation and discussion:</p> <ul> <li><p>plain: This is using plain zsync, and original .debs.</p></li> <li><p>zsyncmakeZ: This uses the -Z option to zsyncmake when creating the .zsync file, in order to use zsync's magic gzip handling. Turns out that this doesn't help at all, compared to the plain scenario. Unless we change how the archive generates .debs, plain or zsyncmakeZ are the only options we can choose between, and they seem to be identical, and neither of them is likely to be worth the effort.</p></li> <li><p>rsyncable: This makes it easier for zsync to do magic things with gzip, by recompressing the gzipped tarballs within the .deb files with gzip --rsyncable. This provies a lot of improvement. Saving a quarter of the bandwidth is already fairly significant, especially since the size impact on the .debs is less than 1%.</p></li> <li><p>rsyncable2: This tests whether zsync's gzip magic works better if the gzip compression is the outermost layer. This is not a realistic option for the archive, but provides a data point for comparisons. Turns out, the differences are insignificant.</p></li> <li><p>rsyncable3: Some packages use lzma or bzip2 compression of the tarballs within the .deb. This benchmark converts those to be compressed with gzip --rsyncable. This improves things a bit compared to just rsyncable, at a 17% increase in size compared to rsyncable. Because most of the packages using lzma are OpenOffice.org related, it is probably not realistic to make them use gzip --rsyncable due to CD size limits, but it might be possible to use them for updates that don't get put into CDs.</p></li> <li><p>uncompressed: This uncompresses all tarballs within the .deb, to give a baseline for just how much zsync could save in an optimal situation.</p></li> </ul> <p>I suspect the 25% value is a bit optimistic, since it comes from a rather special case: security updates don't typically change the package all that much. Backports and updates within a development cycle are likely to change the packages much more. Upgrades from release to release are also likely to change so much that zsync won't save a whole lot.</p> <p>Before I continue working on this, I'd like to have some feedback on this: is a 25% reduction in download sizes worthwhile to pursue? It would seem to require changing dpkg to call the external gzip binary to use --rsyncable, rather than use the internal zlib library.</p> <p>What do other people think?</p> <p>PS. I've fully automated my benchmark, and am happy to share the scripts. If anyone wants to play with them, drop me a note, and I'll set up a public bzr branch. You'll need fast access to a mirror, since they download snapshots of hardy-security and the corresponding packages from hardy, for a total of about three gigabytes.</p> On the Okularm DRM feature https://blog.liw.fi/posts/okular-drm/ https://blog.liw.fi/posts/okular-drm/ debian rant Tue, 02 Jun 2009 07:27:40 +0000 2009-06-02T07:27:40Z <p><a href="http://changelog.complete.org/">John Goerzen</a> filed <a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=531221">a bug on Okular</a> for arbitrarily enforcing DRM in PDF files. Much hilarity resulted.</p> <p>This incident makes me angry. I am now going to blow off some steam.</p> <ol> <li>I fully agree with John on the core issue: Okular should <strong>not restrict</strong> its users regardless of what PDF authors and the PDF say.</li> <li>If Okular upstream disagrees, <strong>Debian should patch</strong> its version. Especially so since it is a <a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=531221#136">one-word patch</a>.</li> <li>The feature is a copy restriction, but it is <strong>not DRM</strong>. Calling it DRM is clearly a bug and only serves to advance the agenda of the copyright mafia.</li> <li>Making it easy to disable is nice. Requiring the user to realize it can be turned off, and having to navigate to the settings dialog to do it, is not onerous, but it is <strong>user-unfriendly.</strong></li> <li>It is claimed (without references) that some corporations require the feature for nebulous legal reasons. They don't seem to require it with xpdf or evince, though. If they do require it, <strong>the corporations</strong> should change things to suit them. Serving the supposed interests of unidentified corporations over the actual, stated needs of prominent users is simply wrong. It's so mind-bogglingly, hideously opposite of anything sensible that it is clear it's not a real reason.</li> <li>I claim, based on their behavior in the discussion, that the real reason Okular upstream and Debian KDE maintainers still oppose changing the default is that this became a battle of egos and they don't want to lose face by giving in now.</li> </ol> <p>The last point is what annoys me most: that we, in Debian, still dig trenches so deep we can't see the other side, that we twist the entire world to protect our egos. Clearly, John shares some blame: his initial bug report was needlessly aggressive, but that doesn't excuse everyone else.</p> <p>Including myself, for writing this blog entry. I have no excuse.</p> Declarative .deb maintainer scripts https://blog.liw.fi/posts/declarative-deb-maintainer-scripts/ https://blog.liw.fi/posts/declarative-deb-maintainer-scripts/ debian idea Tue, 05 May 2009 18:13:21 +0000 2022-08-22T06:07:24Z <p>Currently, maintainer scripts in .deb packages are typically generated by debhelper. Debhelper inserts snippets of shell script to perform specific tasks. If the snippet has a bug, all packages need to be rebuilt.</p> <p>It strikes me that a more declarative approach might work better:</p> <pre><code>#!/usr/bin/debian-postinst create-system-user httpd update-config-file /etc/blipblop.conf </code></pre> <p>Alternatively, the snippes could just be moved to external tools:</p> <pre><code>#!/bin/sh set -e case "$1" in purge) dpkg-remove-logfiles --package=blipblop esac </code></pre> Debian Developers per capita https://blog.liw.fi/posts/dd-per-capita/ https://blog.liw.fi/posts/dd-per-capita/ debian wishlist Tue, 05 May 2009 18:12:47 +0000 2022-08-22T06:07:24Z <p>I wish I knew enough LDAP to implement this, or at least had enough time to learn how to do it myself. But since I don't...</p> <p>It would be fun to have a regularly updated statistic showing how many Debian Developers there are in each country, per million inhabitants in that country.</p> <p>Right now, Finland seems to have 27 DDs, according to <a href="http://db.debian.org">http://db.debian.org</a>, and an estimated 5,312,358 people at the end of 2008, according to <a href="http://en.wikipedia.org/wiki/Finland">Wikipedia</a>. That's about 5.1 DDs per million.</p>