pages tagged larch

Obnam 1.6.1 (backup software) and larch 1.20131130 (B-tree) releases: serious bug fixes

Sat, 30 Nov 2013 17:46:42 +0000

Backups are fun and exciting! Restores are exciting, in a terrifying and stressful way. Fixing serious bugs is gratifying, in a depressing way.

I am glad to announce two software releases: Obnam version 1.6.1 and larch version 1.20131130. Obnam is my backup application, larch is a Python copy-on-write B-tree implementation that Obnam uses. I've uploaded the new versions to Debian unstable and to my own apt repository at [code.liw.fi] (also for wheezy). (I don't have the energy to upload to Debian backports: help is welcome there.)

These are mainly bug fix releases, and minor improvements. The larch release fixes serious problems, and everyone should be upgrading.

NEWS for larch:

Serious bug fixed: the "KeyError" crash for reference counts. This was false memory use optimisation, which triggered a rare bug in related code. Repeatable test case by Rob Kendrick, and helpful analysis by Itamar Turing-Trauring.
Serious bug fixed: another "node missing" bug. This crash was caused by a bug that overwrote on-disk reference count groups with zeroes. Repeatable test case by Rob Kendrick.
Fixes to fsck from Antoine Brenner.

NEWS for Obnam (combining versions 1.6 and 1.6.1, both released today):

Fix Debian package dependencies correctly.
Stop logging paramiko exceptions that get converted into another type of exception by the SFTP plugin in Obnam.
obnam-benchmark can now use an installed version of larch. Patch by Lars Kruse.
Obnam has been ported to FreeBSD by Itamar Turner-Trauring of HybridCluster.
Backup progress reporting now reports scanned file data, not just backed up file data. This will hopefully be less confusing to people.
The list-keys, client-keys, and list-toplevels commands now obey a new option, --key-details, to show the usernames attached to each public key. Patch by Lars Kruse.
New option --ssh-command to set the command Obnam runs when invoking ssh. patch by Lars Kruse.
obnam clients can now be used without being an existing client. Patch by Itamar Turner-Trauring.
New option --ssh-host-keys-check to better specify how SSH host keys should be checked. Patch by Itamar Turner-Trauring.

Bug fixes:

Fix"obnam list-toplevels so it doesn't give an error when it's unable to read the per-client directory of another client, when encryption is used. Fix by Lars Kruse.
Fix the encryption plugin to give a better error message when it looks for client directories but fails to find them. Fix by Lars Kruse.
obnam list-toplevels got confused when the repository contained extra files, such as "lock" (left there by a previous, crashed Obnam run). It no longer does. Fix by Lars Kruse.
The SFTP plugin now handles another error code (EACCESS) when writing a file and the directory it should go into not existing. Patch by Armin Größlinger.
Obnam's manual page now explains about breaking long logical lines into multiple physical ones.
The /~/ path prefix in SFTP URLs works again, at least with sufficiently new versions of Paramiko (1.7.7.1 in Debian wheezy is OK). Reported by Lars Kruse.
The Nagios plugin to report errors in a way Nagios expects. Patch by Martijn Grendelman.
The Nagios plugin for Obnam now correctly handles the case where a backup repository for a client exists, but does not have a backup yet. Patch by Lars Kruse.
obnam ls now handles trailing slashes in filename arguments. Reported by Biltong.
When restoring a backup, Obnam will now continue past errors, instead of aborting with the first one. Patch by Itamar Turner-Trauring.

Obnam 1.5 (backup software) release

Thu, 08 Aug 2013 22:38:57 +0000

I've just pushed out the release files for Obnam version 1.5, my backup application, and Larch, my B-tree library, which Obnam uses. They are available via my home page (http://liw.fi/). These versions have alos been uploaded to Debian unstable.

NEWS for Obnam:

Terminal progress reporting now updated only every 0.1 seconds, instead of 0.01 seconds, to reduce terminal emulator CPU usage. Reported by Neal Becker.
Empty exclude patterns are ignored. Previously, a configuration file line such as "exclude = foo, bar," (note trailing comma) would result in an empty pattern, which would match everything, and therefore nothing would be backed up. Reported by Sharon Kimble.
A FUSE plugin to access (read-only) data from the backup repository has been added. Written by Valery Yundin.

NEWS for larch:

Bug fix in how Larch handles partly-comitted B-tree journals in read-only mode. Previously, this would result in a crash if, say, a node had been removed, but the B-tree metadata hadn't been committed.

Obnam 1.4 (backup software) release

Sat, 16 Mar 2013 19:26:54 +0000

I've just pushed out the release files for Obnam version 1.4, my backup application, and Larch, my B-tree library, which Obnam uses. They are available via my home page (http://liw.fi/). Since Debian is frozen, I am not uploading packages to Debian, but .deb files are available from my personal apt repository for intrepid explorers. (I will be uploading to Debian again after the freeze. I am afraid I'm too lazy to upload to experimental, or do backports. Help is welcome!)

From the Obnam NEWS file:

The`ls command now takes filenames as (optional) arguments, instead of a list of generations. Based on patch by Damien Couroussé.
Even more detailed progress reporting during a backup.
Add --fsck-skip-generations option to tell fsck to not check any generation metadata.
The default log level is now INFO, instead of DEBUG. This is to be considered a quantum leap in the continuing rise of the maturity level of the software. (Actually, the change is there just to save some disk space and I/O for people who don't want to be involved in Obnam development and don't want to have massive log files.)
The default sizes for the lru-size and upload-queue-size settings have been reduced, to reduce the memory impact of Obnam.
obnam restore now reports transfer statistics at the end, similarly to what obnam backup does. Suggested by "S. B.".

Bug fixes:

If listing extended attributes for a filesystem that does not support them, Obnam no longer crashes, just silently does not backup extended attributes. Which aren't there anyway.
A bug in handling stat lookup errors was fixed. Reported by Peter Palfrader. Symptom: AttributeError: 'exceptions.OSError' object has no attribute 'st_ino' in an error message or log file.
A bug in a restore crashing when failing to set extended attributes on the restored file was fixed. Reported by "S. B.".
Made it clearer what is happening when unlocking the repository due to errors, and fixed it so that a failure to unlock is also an error. Reported by andrewsh.
The dependency on Larch is now for 1.20121216 or newer, since that is needed for fsck to work.
The manual page did not document the client name arguments to the add-key and remove-key subcommands. Reported by Lars Kruse.
Restoring symlinks as root would fail. Reported and fixed by David Fries.
Only set ssh user/port if explicitily requested, otherwise let ssh select them. Reported by Michael Goetze, fixed by David Fries.
Fix problem with old version of paramiko and chdir. Fixed by Nick Altmann.
Fix problems with signed vs unsigned values for struct stat fields. Reported by Henning Verbeek.

Obnam 1.3 (backup software) and other releases

Mon, 17 Dec 2012 19:57:58 +0000

I've just pushed out the release files for Obnam version 1.3, my backup application, as well as Larch, my B-tree library, and cliapp, my Python framework for command line applications. They are available via my home page (http://liw.fi/). Since Debian is frozen, I am not uploading packages to Debian, but .deb files are available from my personal apt repository for the intrepid. (I will be uploading to Debian again after the freeze. I am afraid I'm too lazy to upload to experimental, or do backports. Help is welcome!)

From the Obnam NEWS file:

When creating files in the backup repository, Obnam tries to avoid NFS synchronisation problems by first writing a temporary file and then creating a hardlink to the actual filename. This works badly on filesystems that do not allow hard links, such as VFAT. If creating the hardlink fails, Obnam now further tries to use the open(2) system call with the O_EXCL flag to create the target file. This should allow things to work with both NFS and VFAT.
More detailed progress reporting during the backup.
Manual page now covers the diff subcommand. Patch by Peter Valdemar Mørch.
Speed optimisation patch for backing up files in inode numbering order, from Christophe Vu-Brugier.
A setuid or setgid bit is now not restored if Obnam is not used by root or the same user as the owner of the restored file.
Many new settings to control "obnam fsck", mainly to reduce the amount of checking being done in order to make it faster. However, fsck is has lost some features (checks), which will be added back in a future release.
More frequent fsck progress reporting. Some speed optimisations to fsck.

Bug fixes for Obnam:

Empty values for extended attributes are now backed up correctly. Previously they would cause an infinite loop.
Extended attributes without values are now ignored. This is different from attributes with empty values. Reported by Vladimir Elisseev.
An empty port number in sftp URLs is now handled correctly. Found based on report by Anton Shevtsov.
A bad performance bug when backing up full systems (starting from the filesystem root directory) has been fixed. At the beginning of each generation, Obnam removes any directories that are not part of the current backup roots. This is necessary so that if you change the backup roots, the old stuff doesn't hang around forever. However, when the backup root is the filesystem root, due to the now-fixed bug Obnam would first remove everything, and then back it up all over again. This "worked", but was quite slow. Thanks to Nix for reporting the problem.
Obnam now runs GnuPG explicitly with the "no text mode" setting, to override a "text mode" setting in the user's configuration. The files Obnam encrypts need to be treated as binary, not text files. Reported by Robin Sheat.
A shared B-tree concurrency bug has been fixed: If another instance of Obnam was modifying a shared B-tree, Obnam would crash and abort a backup, possibly leaving lock files lying around. Now a failure to look up a chunk via its checksum is ignored, and the backup continues.
Bugs in how Python OSError exceptions were being raises have been fixed. Error messages should now be somewhat clearer.
Unset or wrongly set variable "full" fixed in "obnam diff". Reported by ROGERIO DE CARVALHO BASTOS and patched by Peter Valdemar Mørch.
Setuid and setgid bits are now restored correctly, when restore happens as root. Reported by Pavel Kokolemin.
Obnam now complains if no backup roots have been specfied.

NEWS for Larch:

Make fsck progress reporting be a bit more fine grained.

NEWS for cliapp:

Options in option groups are now included in manual page SYNOPSIS and OPTIONS sections.
--log=syslog message format improvement by Daniel Silverstone. No longer includes a timestamp, since syslog adds it anyway. Also, the process name is now set on Linux.
Make the default subcommand argument synopsis be an empty string, instead of None. Reported by Sam Thursfield.
Meliae memory dumping support has been fixed. Reported by Joey Hess.
Memory profiling reports can now be done at minimum intervals in seconds, rather than every time the code asks for them. This can reduce the overhead of memory profiling quite a lot.
If there are any subcommands, cliapp now adds a subcommand called help, unless one already exists.
For every boolean setting foo, there will no be a --no-foo option to be used on the command line.

Larch 1.20120527 (Python B-tree library)

Sun, 27 May 2012 10:04:04 +0000

It has been a long time since I announced a version of Larch on my blog: about 14 months. A fair bit has happened since then: see the NEWS file for details. Below is the snippet for today's release.

This is an implementation of particular kind of B-tree, based on research by Ohad Rodeh. See "B-trees, Shadowing, and Clones" (copied here with permission of author) for details on the data structure. This is the same data structure that btrfs uses. Note that my implementation is independent from the btrfs one, and might differ from what the paper describes.

The distinctive feature of this B-tree is that a node is never modified (sort-of). Instead, all updates are done by copy-on-write. Among other things, this makes it easy to clone a tree, and modify only the clone, while other processes access the original tree. This is utterly wonderful for my backup application, and that's the reason I wrote larch in the first place.

See the home page and README for more information; the API docs may be amusingly sparse.

Version 1.20120527, released 2012-05-27

New version scheme. Thank you, Joey Hess.
The on-disk data structures and file formats are now declared frozen. An automatic test has been added to verify that things do not break.

Obnam version 0.15 and 0.15.1, and btree/larch 0.19

Mon, 21 Mar 2011 08:53:42 +0000

I've just made a couple of software releases:

larch version 0.19. This used to be known as btree, but others are using that name, so I changed it. This is my copy-on-write B-tree library for Python.
- see http://liw.fi/larch/ for more information
- see http://liw.fi/larch/NEWS/ for changes
obnam version 0.15, and then the obligatory brown paper bag version 0.15.1 to fix a build issue under pbuilder. Obnam is my backup program.
- see http://braawi.org/obnam/ for more information

This is the first release of Obnam this year. Below is a summary of the changes.

My apt repository has binary packages for amd64, i386, and armel. I've started mentoring someone to help me maintain the Debian packages, and hope to start uploading Obnam and its dependencies to Debian within a few weeks.

Obnam is still in ALPHA, but the only missing thing for moving to BETA is encryption support. There's a bunch of bugs that need to be fixed, but I'm not going to block the BETA on them. I'd be very interested to hear any feedback on this version.

Bugs fixed:

Manual page GPL copyright blurb is now properly marked up as a comment. (Thanks, Joey Hess.)
README now links to python-lru correctly. (Thanks, Erik Johansson.)

Improvements and other changes:

Filenames and directories are backed up in sorted order. This should make it easier to know how far obnam's gotten.
The location where backups are stored is now called the repository, instead of the store. Suggested by Joey Hess.
The repository and the target directory for restored data are now both created by Obnam, if they don't already exist. Suggested by Joey Hess.
Better control of logging, using the new --trace option.
Manual page now explains making backups a little better.
Default value for --lru-size reduced to 500, for great improvement in memory used, without, it seems, much decrease in speed.
obnam verify now reports success explicitly. Based on question from Joey Hess.
obnam verify now accepts both non-option arguments and the --root option. Suggested by Joey Hess.
obnam forget now accepts "generation specifiers", not just numeric generation ids. This means that obnam forget latest works.
I/O statistics are logged more systematically.
obnam force-lock introduced, to allow breaking a lock left behind if obnam crashes. But it never does, of course. (Well, except if there's a bug, like when a file changes at the wrong moment.)
obnam genids introduced, to list generation ids without any other data. The old command obnam generations still works, and lists other info about each generation as well, but that's sometimes bad for scripting.
The --dump-memory-profile option now accepts the value simple, for reporting basic memory use. It has such a small impact that it's the default.
Obnam now stores the version of the on-disk format in the repository. This should allow it to handle repositories created by a different version and act suitably (hopefully without wiping all your backups).