pages tagged tddd

A tiny step towards TDDD and FreedomBox-NAS

Wed, 29 Jun 2011 12:45:43 +0000

I've taken a tiny step towards getting together a prototype toolset for developing an initial development iteration of the NAS functionality of the FreedomBox.

Here's what I am trying to do:

create a disk image for a system that provides simplistic NAS functionality, with no configuration, including no authentication: just serve whatever disks are available over SMB
use a TDD approach to developing it

In essence, this means I want to specify the functionality of the system I am building using automated tests. As soon as those tests pass, I will write more tests, and make them pass. When I can't think of new relevant tests, I know I'm done.

I'm building a simple tool, systest, that will execute the tests. It is built on top of Python's unittest library, because that makes my implementation simple.

I've now developed systest far enough that I can almost automatically do this:

build a new image, using vmdebootstrap, plus a customization script
start the image with KVM controlled by libvirt
run the tests using systest

(The command line would be something like this: ./build-fbxnas && virsh start fbxnas && sleep 30 && ~/systest/systest --target=192.168.122.89 --user=tomjon ~/systest/tests-debian-base-install.py --user-ssh-private-key=tomjon.secret --log=foo.log --user-password=password1 where build-fbxnas is a wrapper script around vmdebootstrap. But I didn't actually run that command, and it's lacking the virt-install command to actually set up the virtual machine the first time.)

systest contains very little code, but to get that far, I had to shave some yaks, and that made writing it rather an unpleasant experience.

The tests so far really simple, and don't test NAS stuff at all, just that the basic Debian install works: only the ssh port is open, and basic networking and such things work. However, now that I've gotten this much to work, I can start writing NAS tests. (smbclient may be my next yak. This would be a lot easier if I had ever used Samba for anything real...)

systest and vmdebootstrap are on Gitorious. No .debs yet, this stuff is much too raw, but you can run them from the checkouts, no install necessary.

PS. If anyone knows how to query KVM or libvirt for the IP address assigned by their internal DHCP server to the virtual machine guest, automatically, without asking the actual virtual machine, please tell.

Test driven distribution development

Wed, 08 Jun 2011 08:06:34 +0000

Disclaimer: There is a huge discussion going on right now in the debian-devel mailing list, about different ways to rearrange how Debian development happens, and maybe to provide a Debian users with an additional release. This blog entry is not a comment on that discussion, which I am ignoring. The discussion did, however, prompt a dent about "test driven development". That prompted a short exchange of thoughts with Tom Marble, and this blog post is a cleaned up version of my part in that discussion.

Currently Debian development happens roughly like this: all packages are uploaded to a part of the Debian archive called unstable. Once they've been in use for a while without serious problems, they get automatically copied into another part called testing. Every year or two, testing is frozen and any remaining problems are fixed, after which all of testing gets copied into yet another part of the archive called stable, and that's the actual release.

There is a little bit of automatic testing, but only a little. Almost all testing is done by people, by them using unstable or testing on their usual computers. If they have problems, they report them.

Contrast this with development using Test Driven Development, or TDD, and other modern development methodologies. Here's a rough summary of what I do when I write (much) of my software.

first write one or more automatic tests (unit or functional ones)
write the actual code, enough to make all tests pass
add more tests
write more code, or change existing code
repeat this until tests describe all the behavior desired of the code

In addition, I measure coverage to make sure all parts of the code gets tested. I usually aim for 100% coverage, except for those parts that very hard or quite pointless to test. (That's easier to achieve than you'd think, but that's the topic of another blog post.)

This all sounds like a lot of bureaucratic nonsense, but what I get out of this is this: once all tests pass, I have a strong confidence that the software works. As soon as I've added all the features I want to have for a release, I can push a button to push it out. (Well, actually there's a little bit more to it.)

This does mean I never have bugs in my software. Of course I do. However, there's a lot fewer of them. In fact, there's so few of them, after tests pass, that I would almost be happy to make an automatic release after every successful run of "make check".

Another aspect of the way I do development is distributed version control. The relevant feature is powerful branching and merging. Most things I develop happen in short-lived single-purpose branches: whenever I start work on a feature or bugfix, I create a branch for it. (Unless I'm feeling particularly lazy, which happens more often than you'd think.)

When I've finished the feature, or fixed the bug, I merge the branch back into the trunk branch.

This way, the trunk is always in releaseable state. It might not have all the features I want for the next release, but the features that are there, do work. The branch probably has bugs, but if I've written tests that are good enough, I know the software works well enough.

Or that's the idea. Sometimes things go wrong. Then I write more tests and the next time goes better. (It doesn't have to be perfect, as long as it gets better every time.)

See the contrast between automatic tests with good coverage, and the Debian style of relying on user feedback? There's no need to wait for user feedback with automatic tests. This speeds up development, makes releasing easier, but most importantly takes away any reason to fear making changes, even big changes.

Automatic tests and good test coverage are easy achieve in small projects. For a system as huge as Debian, good test coverage is quite hard to achieve.

The thing about automatic tests, though, is that even a little bit of it is helpful, and after you have a few tests, it gets easier to add more. The first test is the hardest to get done, since you need to set up all the infrastructure for it.

Debian does do a bit of automatic testing already. (See lintian, piuparts, autopkgtest, edos, etc.) I don't want to belittle that, but I think we could do better.

Here's what I would love to see:

we have a part of the archive that corresponds to a trunk branch, i.e., it is always in a releasable state (the "testing" area was originally meant to be that; we could make it so now)
releaseable state is determined by two things:
- an automatic test suite
- user feedback, particularly in the form of bug reports (as now)
whenever changes are made, they happen in "branches"
- the branch is not affected by changes elsewhere in the archive, except by manual synching ("merge from trunk")
- individual package uploads, as well as groups of packages such as for transitions, are each in their own branch
- this is sort of similar to a PPA, but more powerful
when a branch is to be merged into trunk, the automatic tests must first pass (or at least no new failures in them can be introduced)
- tests can also be run for any other branch, of course, so that those developing the branch know if they're ready to push their changes into trunk
there's a culture of writing tests for bugs (whenever that is feasible), and for new features
- particularly release goals should be expressed as automatic test suites
there's a culture of sharing tests with upstreams, and with other distributions

Since full test coverage is going to be impossible for Debian, some subset should be targeted. Perhaps something like this would suffice to start with:

a version of debian-installer is generated from a branch
test installation of a new system from scratch
- with a large set of tasks selected
test upgrade from previous stable release to the branch
- with a large set of packages installed (as many as possible, actually)
test upgrade from trunk branch to branch to be merged
- ditto large set of packages
test with lintian, reject on specific tests failing
test with piuparts
test the whole system for specific functionality
- ssh access from outside
- sudo access for a logged in user
- sending mail with SMTP
- web server
- possibly test specific web applications
- possibly test Samba and NFS services
- possibly test printing services (CUPS)
- possibly test essential desktop functionality (automatic login, at least)
if package comes with package specific tests, run those too

These tests would not guarantee that a set of changes would not break Debian, but they would give a high confidence that at least basic stuff would still work after the changes.

Now, obviously implementing all of this I'm dreaming of (it is just past midnight, after all) is going to be impossible. There's way too much work to do, there's not enough tools for writing tests, and it would require too much computing power to run, and so on and so forth. But it's late, and I've had a bad day, and I might as well dream of something nice.

Anyway, anyone interested in these things should perhaps help drive DEP8.

TDDD systest tool proof of concept

Tue, 07 Jun 2011 14:16:39 +0000

Continuing my earlier musings about test driven distro development, and the tools that would require.

I imagine something like this:

create a virtual machine with a particular configuration of stable
run a test suite verifying the VM works
upgrade the VM to testing and reboot
re-run test suite, verifying things work in testing

We might also want this scenario:

install testing directly in a VM (instead of upgrading from stable)
run test suite

There might be other scenarios that would be useful to test as well. Even from these two, it's clear there's a need for at least three separate tools:

create a machine image with a particular configuration
- might be based on stable or testing or unstable
- might also be using some other package sources
- might want particular packages installed
- might want particular configuration settings
run a suite of tests against the running image
- which tests to run will depend on the scenario
run various scenarios using the above two tools
- needs reasonably easy ways to specify scenarios
- needs to support tests specific to scenarios

I've written a rudimentary version of the first tool: vmdebootstrap. I've since learned there's a bunch of others that might also work. There's room for more: someone should write (or find) a tool to make a snapshot of a real system and create a VM image that mimicks it, for example. Anyway, for now, I'll assume that one of the existing tools is good enough to get started.

For the second tool, I wrote a quick-and-dirty proof-of-concept thing, see systest.py. Here's a sample of how it might be used:

liw@havelock$ ./systest -v --target 192.168.122.139 --user tomjon
test 1/6: cat
test 2/6: only-ssh-port
ERROR: Assertion failed: ['22/tcp', '139/tcp', '445/tcp'] != ['22/tcp']
[status 1]
liw@havelock$ ./systest -v --target 192.168.122.139 --user tomjon \
    cat ping6-localhost ping-localhost simple-dns-lookup ssh-login
test 1/5: cat
test 2/5: ping6-localhost
test 3/5: ping-localhost
test 4/5: simple-dns-lookup
test 5/5: ssh-login
liw@havelock$

The first run failed, because the VM I'm testing against has some extra ports open. Some of the tests will require logging into the machine, via ssh, and for that one needs to specify the user to use.

systool may overlap heavily on system monitoring tools, and possibly the production implementation should be based on those.

I think it's best to design such a tool for the more general purpose of testing whether a system currently works rather than as an integrated part of a more specific larger tool. This lets the tool be useful for other things than just testing specific things about Debian. (The production implementation would then need to not have all the tests hardcoded, of course. SMOP.)

The third tool I have not spent a lot of time on yet. One thing at a time.

Given these tools, one would then need to decide how to use them. The easiest way would be to use them like lintian and piuparts: run them frequently on whatever packages happen to be in testing or unstable or experimental, and put links to the test reports to the PTS, and hope that people fix things.

That is the easiest way to start things.

Once there's a nice set of test cases and scenarios, it may be interesting to think about more aggressive ways: for example, preventing packages from migrating to testing unless the test suite passes with them. If the tests do not pass, one of four things is broken:

the package or packages in question
other packages already in testing
the tests themselves
the test environment

If things are set up properly, the last one should be rare. The other three always require manual inspection: it is not possible to automatically know whether the test itself, or the code it tests, is at fault. it is, however, enough to know that something is wrong. If the tests are written well, they should be robust enough to not be the culprits very often.

(Someone wanting to make a rolling distrbution, or even better, making a monthly mini-release, might benefit from this sort of automated testing.)

TDDD, more thoughts

Mon, 16 May 2011 16:26:57 +0000

My previous blog entry about Test Driven Distro Development (TDDD) was rather rambling. Here is the condensed version, plus some further thoughts.

I would like to automatically determine whether "testing" is minimally ready to be released, at any given point in time.

To implement this, I would like to see Debian get tools that would allow us to define specific minimal functionality that should work in a Debian release, and automatically verify that it does. I would like us to use these tools to determine whether a release is ready for release, with manual override from the release team. Ideally, we would use the tools to prevent breakages from moving into "testing", so that it gets closer to always being releasable.

Getting test coverage sufficiently high that we can replace the release team's judgement abour releasablity will never happen: Debian is rather too large and and complex for that. However, we can reap much benefit from even relatively small coverage.

Here is my initial suggestion for the minimal criteria that should be satisfied, at all times, in "testing". They are aimed at making sure that even if something else breaks, at least the basic stuff works well enough that one can log in and fix things.

there are no open ports in a base install
logging in via getty works
logging in via ssh works, if openssh-server is installed
su, sudo, ls, cp, mv, rm, and vi work at least in some typical use cases
web pages can be accessed from outside the system, given suitably configured apache2
after automatic login into a graphical desktop environment, all the crucial per-session programs start up, as well as web browsers and other GUI programs that almost everyone expects to be part of a desktop environment
- if there are good GUI testing tools, we may want to test in more detail, but that may have to wait for later

These are minimal criteria. They're not meant to verify that everything in Debian works. After these work, adding more coverage is not just possible, but reasonably easy in many cases. Want to make sure DNS gets configured right? Write a test that logs in and does a lookup inside the test system. After the tools have evolved, writing a simple test like that should be only a line or two.

The tests I am talking here are system tests, or acceptance tests for the whole system, not just for individual packages. Unit tests would test individual functions or methods or classes; integration tests would test groups of them, or possibly entire programs; system tests test groups of programs/packages.

The point is not to verify that every package works in every possible case, but that the system as a whole works. Most package-specific functional tests should be part of the package's own test suite. We should also run those, but that's somewhat orthogonal to the vision I'm trying to articulate here.

At least the following scenarios should be tested (the above tests should be run after each scenario):

install "stable" and upgrade to "testing"
install "testing" using latest d-i release, and upgrade to current "testing"
use an install from the previous successful test run, and upgrade to the current "testing"
- this answers the question "if testing worked yesterday, does it still work if we add these new packages"

It may be useful to run the tests against "unstable" as well as "testing". However, rejecting an upload to "unstable" may be less wise than preventing packages from being copied from "unstable" to "testing". Exactly how this the development processes should adopt TDDD is an interesting question, but a reasonable toolset should be adaptable to anything we want to do.

(Indeed, it would be possible to structure the toolset so that it isn't specific to developing a distro: a "system-test" tool that a sysadmin could use to verify that a server works would probably be useful to some people. Perhaps such a tool exists and could be adapted for TDDD?)

Running these tests for security updates to "stable" would obviously also be a possibility that should perhaps be considered.

Running these tests would happen something like this:

generate a suitable virtual machine disk image, configured in the proper way for the test
run the image under virtualization (or on real hardware if that can be remote-controlled suitably)
run the tests against the image
if tests fail, archive the image (or sufficient information to reproduce the start image)
- also, automatically file bugs against all packages involved in the test, and e-mail debian-devel-announce with the names of the people who uploaded the new versions of packages compared to the previous successful test run, Cc'ing their spouses and parents for further shaming, and also Cc'ing the Debconf organization team so they can stood up before an audience armed with rotten tomatoes and eggs (this bullet point is here only to see if anyone actually reads my blog posts)

I would further like to see Debian make it possible to have isolated branches for development. This would help with, for example, large transitions that take a long time. The transition could be prepared in a branch of its own, and when it's ready to be pushed into unstable, the automatic tests outlined above could be used to verify the transition doesn't break anything too critical. (If the tests fail, the push fails; if necessary, that can be overridden manually, but that should be rare.)

In my view, branches would greatly benefit Debian development, but they're not necessary for starting to implement TDDD.

My condensed versions are still too long.