blueslugs.com

Bookmarks, 02021-09

stephen@blueslugs.com (Stephen Hahn) — Sat, 05 Sep 2026 09:18:23 +0000

One month’s bookmarks from pinboard.in:

Brightgate/product: Core Brightgate Software Stack The appliance and cloud software from Brightgate. MPL-2.0 licensed. Might be some useful pieces for future projects.

Install OmniOS with only USB3 ports

stephen@blueslugs.com (Stephen Hahn) — Sat, 01 Aug 2026 22:10:57 -0700

A challenging aspect of getting started with OmniOS, one of the Illumos-based distributions, is that support for USB 3.0 is limited and it is easy to get stuck at “Installation media not found” and see worrying messages like “Could not mount installation media”. If your system has a recognized network interface and a reasonable amount of memory, however, you can actually complete the install with a few extra steps.

Getting installation images. Keep your USB stick onto which you flashed the omnios-r151058.usb-dd image. You’ll still need this for the first part of installation, which will be a USB boot. In addition, you need the matching “PXE image” for that release, which will be named omnios-r151058.zfs.xz. (These release IDs will be different for future OmniOS releases; see the OmniOS Downloads page.) Our first step is to serve that compressed ZFS image via HTTP on our network.

Serving the PXE image over HTTP. If you don’t have an HTTP server already running on your network, you ean easily set up a simple HTTP server using Python 3 and uv:

$ uv venv rangehttpserver
Using Python 3.14.2 interpreter at: /home/linuxbrew/.linuxbrew/opt/python@3.14/bin/python3.14
Creating virtualenv at: rangehttpserver
Activate with: source rangehttpserver/bin/activate
$ . ./rangehttpserver/bin/activate
(rangehttpserver) $ uv pip install rangehttpserver
Resolved 1 package in 440ms
Prepared 1 package in 47ms
Installed 1 package in 2ms
+ rangehttpserver==1.4.0
(rangehttpserver) $ cd rangehttpserver
(rangehttpserver) $ mkdir data ~/bl/rangehttpserver

Now place the PXE image file and activate the HTTP server:

(rangehttpserver) $ mv /path/to/downloaded/omnios-r151058.zfs.xz data
(rangehttpserver) $ cd data
(rangehttpserver) $ python3 -m RangeHTTPServer 7000
Serving HTTP on 0.0.0.0 port 7000 (http://0.0.0.0:7000/) ...
...

(You can probably serve this 233 MiB file over the Internet, but it’s more efficient to have it locally.)

You need to capture the IP address of this server (or of your already running HTTP server), because your OmniOS target box doesn’t have name services in its boot image.

Begin USB installation. Next, start the USB-based install. UEFI services allow OmniOS to start booting off a ramdisk, but OmniOS won’t be able to find the USB stick with the rest of the needed media. And so you will see frustrating messages. Press Enter to return to the Kayak installer menu and enter the shell.

Activate network. Use dladm(8) to show the interfaces on this system:

$ dladm show-link
LINK CLASS MTU STATE BRIDGE OVER
rge0 phys 1500 unknown -- --

It is difficult for me to accept that a `*adm` command could be in any section but 1M.

On this Beelink EQmini N150, we have a Realtek Gigabit Ethernet device, which enumerates as rge0. Illumos has good support for this device, so let’s proceed.

We need to get that network interface connected to the network. Use ifconfig(8) to activate the interface and then request configuration via DHCP:

ifconfig rge0 plumb
ifconfig rge0 dhcp start

(If you do not have DHCP services, you will have to configure the interface statically.)

Construct media simulacrum. The result of not finding the media is that the installer can’t make progress. What we are going to do is use a loopback mount to place the needed files at the locations that the installer expects.

Our first step is to create a backing store, use that store to create a loopback device, create in turn a filesystem on the loopback device, and mount that filesystem in the appropriate location to populate:

mkfile 2g /tmp/vdisk
lofiadm -a /tmp/vdisk
newfs -v /dev/lofi/1
mount /dev/lofi/1 /.cdrom

(That 2GiB file on /tmp is why I mentioned having sufficient memory. If you have a smaller system, this technique may be too heavyweight to work.)

And now we can create the needed structures in the /.cdrom directory:

touch /.cdrom/.volsetid
mkdir /.cdrom/image

Retrieve installation image. Finally, we can use curl(1) to retrieve the PXE image we placed on our HTTP server earlier:

cd /.cdrom/image
curl -kO http://{ipv4-address}:7000/omnios-r151058.zfs.xz

(Adjust URL for your own HTTP server, of course.)

Install. You should be able now to exit the shell and complete the install normally. OmniOS provides an installation walkthrough.

restore-mediacast-assets

stephen@blueslugs.com (Stephen Hahn) — Mon, 13 Jul 2026 19:54:56 -0700

I was working through old hard drives and moving their contents into our main ZFS pool. As part of that process, I found a zip archive of content from mediacast.sun.com. I’ve used that data to correct some of the old articles that had broken links to that site; the files are now static content on this blog.

Bookmarks, 02021-09

stephen@blueslugs.com (Stephen Hahn) — Mon, 30 Jun 2025 08:01:36 +0000

One month’s bookmarks from pinboard.in:

Brightgate/product: Core Brightgate Software Stack The appliance and cloud software from Brightgate. MPL-2.0 licensed. Might be some useful pieces for future projects.

Deploying Anubis on nginx

stephen@blueslugs.com (Stephen Hahn) — Fri, 28 Mar 2025 11:12:11 -0700

Anubis is a recently released HTTP reverse proxy by Xe Iaso of Techaro. Anubis forces specified clients to do additional computational work prior to delivering the content, which can interfere with abusive site scanning.

Deployment examples provided with the Anubis source cover Caddy and Kubernetes. In this post, I work through how to deploy Anubis with nginx and systemd. (Tested and deployed on Debian 12 “Bookworm”.)

Nginx configuration

A typical nginx configuration looks somewhat like the following schematic:

┌───────────────────────────────────────────────────────────────────┐
│ listen 80 │
│ │
│ {302 redirect to port 443} │
└───────────────────────────────────────────────────────────────────┘
┌───────────────────────────────────────────────────────────────────┐
│ listen 443 │
│ │
│ SSL certificate and algorithm │
│ Content Security Policy │
│ │
│ root location │
│ specific redirects │
└───────────────────────────────────────────────────────────────────┘

It is probably worth mentioning that, since Anubis’s implementation requires use of the Subtle Crypto Web API, that its content must be served by HTTPS. Without HTTPS, the cryptographic calls will fail and requests will remain on the Anubis transitional page.

In its default configuration, Anubis listens on port 8923 for HTTP requests (and also on port 9090 for metrics requests) and proxies successful requests to the server listening at port 9090. We will split the listen 443 section into a front-side section, that contains the SSL configuration, the Content Security Policy, and the proxy_pass to Anubis at port 8923, and a back-side section, which contains the root directory specification, location /, and other content configuration. That structure looks like the following:

┌───────────────────────────────────────────────────────────────────┐
│ listen 80 │
│ │
│ {302 redirect to port 443} │
└───────────────────────────────────────────────────────────────────┘
┌───────────────────────────────────────────────────────────────────┐
│ listen 443 │
│ │
│ SSL certificate and algorithm │
│ Content Security Policy │
│ │
│ {proxy_pass to anubis localhost:8923} │
└───────────────────────────────────────────────────────────────────┘
┌───────────────────────────────────────────────────────────────────┐
│ listen localhost:3939 │
│ │
│ root location │
│ specific redirects │
└───────────────────────────────────────────────────────────────────┘

`proxy_pass`

From the configuration examples, we can derive the appropriate proxy_pass specification for the “front-side” section:

location / {
proxy_pass http://localhost:8923;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}

Content security policy

Anubis uses data: and blob: URIs to include some of the supporting content for its “working page”. If you have a Content Security Policy add_header line like

add_header Content-Security-Policy "default-src 'self'; ..." always;

then you must ensure that script-src includes blob: and that font-src includes data:, so that Anubis’s content is accepted by CSP-aware clients.

Service description

If you run anubis without arguments, it will listen on both port 8923 and port 9090 on all interfaces. In the service description below, we restrict those to localhost-only.

We also do not break out a separate policy file, but instead rely upon the default policy included with Anubis. You may need to develop your own policy if Anubis is blocking too many friendly consumers.

anubis.service: Service description for Anubis, systemd(1)-style.

I hope that’s sufficient structure and specific information to let you adjust your nginx configuration to run Anubis successfully.

Update, 3/28: I had to add an ALLOW clause to let the blueslugs.com Atom feed through, so there is now an explicit separate policy file which we place at /etc/anubis_policy.json. The clause uses the path_regex field to whitelist the feed’s path:

{
"name": "feed-atom",
"path_regex": "^/index.xml$",
"action": "ALLOW"
},

The above unit file has been amended to use -policy-fname as part of the invocation.

Acknowledgments

Thanks to Xe for developing and sharing this interesting program.

Bookmarks, 02021-08

stephen@blueslugs.com (Stephen Hahn) — Tue, 26 Nov 2024 13:00:34 +0000

One month’s bookmarks from pinboard.in:

ligurio/unreliablefs: A FUSE-based fault injection filesystem. Has a number of modes, including returning random errors on I/O operations.
clip - The command line illustration processor - clip DSL and implementation for generating line graphs. Renders to SVG.
Herbie: Automatically Improving Floating Point Accuracy An IEEE754-aware expression improver.

Monitoring

stephen@blueslugs.com (Stephen Hahn) — Mon, 21 Oct 2024 11:15:18 -0700

The shoreline gets active as sunset approaches. Dina took this picture of interspecies regard on our walk.

Ambitious

stephen@blueslugs.com (Stephen Hahn) — Thu, 17 Oct 2024 12:57:16 -0700

We occasionally buy succulents in small multiples—six pack, eight pack—at Costco. It’s never apparent to me whether a particular plant will remain as it appears in its pot, perhaps with some slight expansion as it settles into the larger soil volume, or transform into a distinct, mature presentation.

Most of these plants stay low to the ground, and are produce a small number of identical structures that make up their shape. But, every so often, a plant will generate more ambitious output:

Bookmarks, 02020-08

stephen@blueslugs.com (Stephen Hahn) — Wed, 24 Jul 2024 11:07:07 +0000

One month’s bookmarks from pinboard.in:

Mastering Critical Thinking “Educators can improve their students’ critical thinking skills with the right strategies. In particular, a growing body of research suggests that we can substantially improve our students’ abilities to understand, analyze, and evaluate arguments, which are key critical thinking skills. Two promising interventions are argument mapping and mastery learning.”
dotfiles/.macos at main · mathiasbynens/dotfiles · GitHub Sets up many preferences and other applications defaults as a script.

Bookmarks, 02020-08

stephen@blueslugs.com (Stephen Hahn) — Thu, 11 May 2023 18:18:38 +0000

One month’s bookmarks from pinboard.in:

Mastering Critical Thinking “Educators can improve their students’ critical thinking skills with the right strategies. In particular, a growing body of research suggests that we can substantially improve our students’ abilities to understand, analyze, and evaluate arguments, which are key critical thinking skills. Two promising interventions are argument mapping and mastery learning.”
dotfiles/.macos at main · mathiasbynens/dotfiles · GitHub Sets up many preferences and other applications defaults as a script.

Neighbours

stephen@blueslugs.com (Stephen Hahn) — Tue, 02 May 2023 23:09:16 -0700

This past spring, we have been listening to a family of Great Horned Owls hooting in the evenings from the trees in our front yard. Dina, with guidance from our human neighbours, was finally able to capture photographic evidence.

Suspended

stephen@blueslugs.com (Stephen Hahn) — Mon, 05 Dec 2022 16:12:19 -0800

The Clark Bridge, driving west across the Mississippi to Missouri.

Bookmarks, 02020-07

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Dec 2022 19:54:55 +0000

One month’s bookmarks from pinboard.in:

TinyPilot: Build a KVM Over IP for Under $100 · mtlynch.io “TinyPilot is my inexpensive, open-source device for controlling computers remotely. It works even before the operating system boots, so I use TinyPilot to install new OSes and debug boot failures on my bare metal homelab servers.”
Ebiten - A dead simple 2D game library for Go “Ebiten is an open source game library for the Go programming language. Ebiten’s simple API allows you to quickly and easily develop 2D games that can be deployed across multiple platforms.”
Pretty git log in one line Adds an alias, “git logline”, that gives most useful commit information on a single line.
Prometheus - Auto-deploying Consul and Exporters using Saltstack Part 5: illumos :: YetiOps — A view on the tech and open source world from a hairy human Service manifests for salt minion, Consul, and Node Exporter.

Enclosed

stephen@blueslugs.com (Stephen Hahn) — Wed, 19 Oct 2022 15:48:38 -0700

The atrium at the Cleveland Museum of Art, a space connecting the 1916 Building to the West and East Wings and the North Building.

Bookmarks, 02020-06

stephen@blueslugs.com (Stephen Hahn) — Sat, 01 Oct 2022 21:25:50 -0700

One month’s bookmarks from pinboard.in:

Create diagrams with code using Graphviz – ncona.com – Learning about computers Some nice graph examples.

Bookmarks, 02020-05

stephen@blueslugs.com (Stephen Hahn) — Fri, 29 Oct 2021 10:56:31 +0000

One month’s bookmarks from pinboard.in:

Long explanation of using plt subplots to create small multiples Works through the various API choices one might make.
AeroPress Recipes - over 88 recipes - AeroPrecipe Attractively presented collection.
The Last Coffee Grind Size Chart You’ll Ever Need Good grind discussion and chart.
Errata Security: Securing work-at-home apps “… don’t add gimmicky features, but instead, take this opportunity to do security things you should already be doing, starting with a ‘vulnerability disclosure program’ or ‘vuln program’.” Followed by a discussion of how to start such a program.
Modern CSS Solutions “Modern CSS Solutions for Old CSS Problems.”
Gigli: petit douteux - blueslugs.com @ahl @bcantrill @BaronDestructo Proof:
Amb - Rosetta Code “The Amb operator takes a variable number of expressions (or values if that’s simpler in the language) and yields a correct one which will satisfy a constraint in some future computation, thereby avoiding failure.”
Naomi Kritzer, “So Much Cooking” Issue 110, November 2015.
aleatoric - synthetic control: elon’s tweet tanked tesla’s stock “Anyway, I asked myself: can we causally attribute Tesla tanking stock price to this tweet? The answer is yes, yes, we absolutely can. In the following I use a synthetic control approach to estimate the causal impact of Musk’s tweet on the Tesla stock price…”

Bookmarks, 02020-04

stephen@blueslugs.com (Stephen Hahn) — Fri, 22 Oct 2021 20:27:33 +0000

One month’s bookmarks from pinboard.in:

mifi/editly: Slick, declarative command line video editing “Editly is a tool and framework for declarative NLE (non-linear video editing) using Node.js and ffmpeg. Editly allows you to easily and programmatically create a video from set of clips, images and titles, with smooth transitions between and music overlaid.” In the past, I’ve used make(1), ffmpeg(1) and ImageMagick commands, so this more unified approach might be easier.
Database schema templates by DrawSQL “Collection of real world database schemas from open-source packages and real-world apps that you can use as inspiration when architecting your app.”
Five improvements in the design of coronavirus coverage - PressThink Proposes an urgency index: summarized, ordered list of urgent topics.
Why False Claims About COVID-19 Refuse to Die - Issue 84: Outbreak - Nautilus “There is a pandemic of misinformation about COVID-19 spreading on social media sites. Some of this misinformation takes well-understood forms: baseless rumors, intentional disinformation, and conspiracy theories. But much of it seems to have a different character. In recent months, claims with some scientific legitimacy have spread so far, so fast, that even if it later becomes clear they are false or unfounded, they cannot be laid to rest. Instead, they become information zombies, continuing to shamble on long after they should be dead.” Discusses some specific cases of incorrect work and their metrics.
Study: Warming makes US West megadrought worst in modern age Summary of report in Science.
Ontario Academic Credit - Wikipedia Earlier in this timeline:
paulokuong/noaa: NOAA Weather Service Python SDK SDK for NOAA Weather Service REST API for getting recent and forecast data.

Bookmarks, 02020-03

stephen@blueslugs.com (Stephen Hahn) — Mon, 05 Apr 2021 21:24:08 +0000

One month’s bookmarks from pinboard.in:

mbrt/gmailctl: Declarative configuration for Gmail filters A file format for representing Gmail filters, and a Golang-based utility for manipulating them (and applying them to a Gmail account).
Signs You’re Following A Fake Twitter Account… – NixIntel Techniques to discover synthesized profile pictures and to assess posting habits consistency with other profile data.
Olympus TG-6 review: the best waterproof camera on the market: Digital Photography Review Review of the performance/behavior of Olympus’s waterproof small camera.
GitHub - mbugert/laserscad: A library for efficient lasercutting with OpenSCAD. “A library for efficient lasercutting with OpenSCAD.” Render to DXF with Make/Python.
GitHub - bmsleight/lasercut: Module for openscad, allowing 3d models to be created from 2d lasercut parts “Module for openscad, allowing 3d models to be created from 2d lasercut parts, with a flat file automatically generated.” Render to DXF via a shell script.
GitHub - monitoror/monitoror: Unified monitoring wallboard — Light, ergonomic and reliable monitoring for anything. “Unified monitoring wallboard — Light, ergonomic and reliable monitoring for anything.”

Bookmarks, 02020-02

stephen@blueslugs.com (Stephen Hahn) — Mon, 30 Nov 2020 13:53:57 +0000

One month’s bookmarks from pinboard.in:

Tempress Upgrade “Many prestigious manufacturers have used Grohe’s Tempress Pressure Balance rough valve as a starting point for their own trims. Manufacturers such as Dornbracht, Jado, Phylrich, Cifial, and many others have used - or are still using - the Tempress PBV. Sometimes replacement trim parts are difficult to find, or have been discontinued, or are difficult to identify.” Worked for us: upgraded two shower fixtures to Grohe.
2018 Hawaii false missile alert - Wikipedia Article on false alert communicated using the EAS.
CONELRAD Adjacent: Code Word “Hatefulness”: The Great EBS Scare of 1971 A blogpost on an EBS incident of 1971. Blog has other posts on related topics.
The EBS Authenticator Word List Site with a number of implementation documents concerning the EBS.
Emergency Alert System - Wikipedia “The Emergency Alert System (EAS) is a national warning system in the United States put into place on January 1, 1997 (approved by the Federal Communications Commission (FCC) in November 1994),[1] when it replaced the Emergency Broadcast System (EBS), which in turn replaced CONELRAD.”
Emergency Broadcast System - Wikipedia “The Emergency Broadcast System (EBS), sometimes called the Emergency Broadcasting System or the Emergency Action Notification System (EANS), was an emergency warning system used in the United States.” Successor to CONELRAD.
CONELRAD - Wikipedia “CONELRAD (Control of Electromagnetic Radiation) was a method of emergency broadcasting to the public of the United States in the event of enemy attack during the Cold War.”
wcventure/FuzzingPaper: Recent Fuzzing Paper Literature survey of recent fuzz testing work.
Python Tools for Record Linking and Fuzzy Matching - Practical Business Python How to match records across datasets, potentially fuzzily.

File serving over the Internet, for one

stephen@blueslugs.com (Stephen Hahn) — Thu, 29 Oct 2020 16:48:36 -0700

For personal notes and development, there’s a lot to be said for using a shared network file system. Dropbox has a collection of notification and sharing features that don’t naturally fit development use (where many files are intermediate products of a build process and not particularly interesting at any point of their lifecycle). In this post, I outline the steps to build a single server-to-multiple client file service configuration that might be suitable for text editing and relatively small software builds. (You always want one or more big servers with fast I/O for large builds.) Technologies used are Wireguard, MUNGE, and 9P; examples are given for Ubuntu.

Dropbox is great for managing files shared with others, and also convenient for some forms of archiving.

The goal is to set up a network share that can be used from various systems. In my initial deployment there will be a stationary workstation and a laptop. Both are running current Ubuntu. The server is running current Ubuntu as well.

In most situations, my network connectivity—even tethered—is better that what we had in university and at work. (MPK17 received hundred megabit Ethernet long after most of the other buildings on campus.) That means, as long as we can put up with some latency, a network file service might be sufficient.

Our network transport is a Wireguard link to the server from each client. This choice means that the authentication and file operations are encrypted as they pass between client and server.

We have set up Wireguard using dsnet for ease of administration. dsnet is a lightweight command line utility for managing a Wireguard server and its clients; in particular, it manages the IP address assignment for the clients. Follow the dsnet author’s instructional post for the initial configuration.

$ sudo apt install wireguard
$ wget https://github.com/naggie/dsnet/releases/download/v0.1/dsnet-linux-amd64
$ sudo cp dsnet-linux-amd64 /usr/local/bin/dsnet
$ sudo dsnet init
[ edit /etc/dsnetconfig.json as appropriate ]
$ sudo dsnet up

dsnet’s add subcommand can be used to generate the configuration for a new client.

$ sudo dsnet add glowy > dsnet-glowy.conf

(glowy is a Hades Canyon NUC with an illuminated logo on its top.)

Copy this configuration to the appropriate workstation. You’ll also need the IP address of the server’s Wireguard interface; we’ll refer to this as SERVERADDR below.

On Ubuntu, wg-quick requires resolvconf. One possible workaround is:

$ sudo ln -s /usr/bin/resolvectl /usr/local/bin/resolvconf

On that workstation, test the configuration using

$ sudo apt install wireguard
$ wg-quick up ./dsnet-glowy.conf

Both authentication and file service will use the Wireguard link for transport. The link is not configured to take all traffic from the client, although that is certainly possible.

Authentication for file service is Munge, which will securely communicate the user ID (UID) and group ID (GID) making each request. Your UIDs (and GIDs) should match across systems.

Munge reminds me somewhat of NIS, in that a shared secret is used to allow a group of machines to trust one another’s user and group IDs. Munge has superior security characteristics to NIS, benefiting from delivering a smaller set of features and using, well, cryptography.

On the server and each client install Munge.

$ sudo apt install munge

The postinstall script for this package creates a weak key to keep the installation time low, so on the server generate a better one:

$ sudo create-munge-key -f -r

You need to place a copy of this key in /etc/munge/munge.key on each client system. This file needs to be owned by the munge user and group. Updating the key will require it to be distributed to each participating system, client and server. (And restarting the service as well, once the key is in place.)

Finally, we use 9P for network file service. We are using the diod implementation.

On the server, we create a /sync directory, under which we will collect the directories we plan to share.

$ apt install diod
$ mkdir /sync/$USER

Then add /sync/$USER to /etc/diod.conf

You can invoke diod directly, with logging off and authentication on:

$ sudo diod -f

(There are command line options for debugging and for deactivating authentication.)

Finally set DIOD_ENABLE to true in /etc/default/diod

$ sudo vim /etc/default/diod

and ensure that the diod service starts successfully.

On the client, create a target directory for the mount

$ mkdir ~/sync

And then use diodmount to make the mount:

$ apt install diod
$ sudo diodmount -n $SERVERADDR:/sync/$USER ~/sync
$ ls ~/sync

Since this mount was the first time I had used 9P, I spent a brief period of time copying around ~80MB files, just to get a feel for the performance.

$ time cp $EIGHTY_MB_FILE /tmp
cp $EIGHTY_MB_FILE /tmp 0.00s user 0.14s system 66% cpu 0.222 total
$ time cp $EIGHTY_MB_FILE ~/sync
cp $EIGHTY_MB_FILE sync 0.00s user 0.23s system 0% cpu 2:38.76 total
$ stat $EIGHTY_MB_FILE
File: $EIGHTY_MB_FILE
Size: 81224528 Blocks: 158648 IO Block: 4096 regular file
Device: 802h/2050d Inode: 37748762 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 1001/ $USER) Gid: ( 50/ staff)
Access: 2020-10-27 18:05:41.399117077 -0700
Modify: 2019-06-06 06:58:17.680609774 -0700
Change: 2020-10-27 15:26:29.596553909 -0700
Birth: 2019-06-06 06:58:13.608637639 -0700

That’s an effective rate of ~511 kilobytes/second. I have a typical Comcast home cable link, so there’s a maximum of ~6 megabits/second available.

Linux has FS-Cache, a built-in caching subsystem for network file systems. We can enable caching for 9P by using the cache mount option:

$ sudo umount ~/sync
$ sudo diodmount -n $SERVERADDR:/sync/$USER ~/sync -o cache=fscache

If we rerun our copy operation, we see

$ time cp $EIGHTY_MB_FILE ~/sync
cp $EIGHTY_MB_FILE sync 0.01s user 0.18s system 1% cpu 16.538 total

Which is an effective speed of ~4911 kilobytes/second. I suspect most of this speedup is just making the operations asynchronous from the perspective of our cp(1) invocation.

There isn’t much recent work on 9P performance. Van Hensbergen and Minnich (2005) showed that 9P was generally competitive with NFS on a variety of workloads.

In an attempt to make detaching and reattaching the mount simple, I wrote a brief script, named toggle-sync. You’ll need to make sure HOME, SERVERADDR, and USER are defined.

#!/bin/bash -p
hostname=$(uname -n)
mountpoint=$HOME/sync
server=$SERVERADDR
serverdir=/sync/$USER
if ! which wg-quick; then
echo "may need apt install wireguard"
exit 1
fi
if ! which diodmount; then
echo "may need apt install diod"
exit 1
fi
if ! pgrep -af munged; then
echo "munged not running; may need apt install munge and completed key configuration"
fi
if ip addr | grep -q "dsnet-$hostname"; then
echo "dsnet-$hostname wireguard link present"
else
wg-quick up "./dsnet-$hostname.conf"
fi
if mount | grep -q /home/sch/sync; then
echo "$mountpoint mounted; unmounting"
sudo umount "$mountpoint"
else
echo "$mountpoint not mounted; mounting"
sudo diodmount -n "$server:$serverdir" "$mountpoint" -o cache=fscache
fi

So far this setup has been working well for notes and small files. This post (and its Hugo build) were written (and invoked) from ~/sync. It’s nice to be able to switch machines without losing my working state.

Bookmarks, 02020-01

stephen@blueslugs.com (Stephen Hahn) — Thu, 29 Oct 2020 16:39:09 +0000

One month’s bookmarks from pinboard.in:

Sanitized IEEE OUI Data (oui.txt) “The IEEE oui.txt file has become somewhat inconsistent over the years and subsequently requires subvention for use in network infrastructure documentation and reporting. In response to this we have produced a [daily] sanitized edition that is available for download in bzip2, gzip, Zip and ASCII text. Our sanitizing process performs Title Case conversion of legacy all UPPERCASE OUI records, fixes for vendor name inconsistencies, spelling corrections, removal of superfluous and erroneous data artifacts that have crept into the original daily generated IEEE oui.txt over time.”
GitHub - paperwm/PaperWM: Tiled scrollable window management for Gnome Shell GNOME extension that enables a tiled window manage experience.
ClickHouse — open source distributed column-oriented DBMS “ClickHouse is an open source column-oriented database management system capable of real time generation of analytical data reports using SQL queries.”

Bookmarks, 02019-12

stephen@blueslugs.com (Stephen Hahn) — Thu, 02 Jul 2020 07:10:48 +0000

One month’s bookmarks from pinboard.in:

GitHub - mthh/jenkspy: Compute Natural Breaks in Python (Fisher-Jenks algorithm) “Compute “natural breaks” (Fisher-Jenks algorithm) on list/tuple/numpy.ndarray of integers/floats.”
[1911.04768] Debugging Crashes using Continuous Contrast Set Mining “…. Debugging groups of crashes is a manually intensive process that requires deep domain expertise and close inspection of traces and code, often under time constraints. We use contrast set mining, a form of discriminative pattern mining, to learn what distinguishes one group of crashes from another. Prior works focus on discretization to apply contrast mining to continuous data. We propose the first direct application of contrast learning to continuous data, without the need for discretization.”

Didelphis virginiana afternoon prowl

stephen@blueslugs.com (Stephen Hahn) — Mon, 18 May 2020 14:56:12 -0700

Over the years, I’ve seen opossums crossing the road at dusk and assumed they were nocturnal or crepuscular. This fellow, however, went along our back fence a couple of times last week, during the middle of the afternoon. Perhaps it was able to adjust its routine—by covering more territory?—during the quiet of the shelter-in-place restrictions.

The opossum was poking its nose into the leaf litter every so often, although in the image it’s probably monitoring Rockie’s relative position. Some marsupials must prowl, right?

Commit to run a virtual event

stephen@blueslugs.com (Stephen Hahn) — Thu, 07 May 2020 11:06:36 -0700

I just committed to run the Triple Nessie Challenge. Dina’s already done a virtual event, but I was reluctant, since I had a best-in-a-decade 5K result in February.

Bookmarks, 02019-11

stephen@blueslugs.com (Stephen Hahn) — Fri, 01 May 2020 09:01:07 +0000

One month’s bookmarks from pinboard.in:

A Guide to Rate Limiting with Examples in JavaScript “Rate limiting is an effective and relatively easy way to mitigate security risks. It will not be the only thing you do secure your applications, and it might not even be the most important thing you do to secure your applications, but it should ALWAYS be in your toolbox.”
No Frils (AKA: Syntax Highlighting Off) “Anti-syntax highlighting theme for Vim. Strives to theme everything without gaps in a minimalist, syntax off manner.”
[1606.00318] Discovering Phase Transitions with Unsupervised Learning “We show that unsupervised learning techniques can be readily used to identify phases and phases transitions of many body systems. Starting with raw spin configurations of a prototypical Ising model, we use principal component analysis to extract relevant low dimensional representations the original data and use clustering analysis to identify distinct phases in the feature space.”
[1605.01735] Machine learning phases of matter “Neural networks can be used to identify phases and phase transitions in condensed matter systems via supervised machine learning. Readily programmable through modern software libraries, we show that a standard feed-forward neural network can be trained to detect multiple types of order parameter directly from raw state configurations sampled with Monte Carlo.”
The MGR Window System “MGR, sometimes said to be short for ‘ManaGeR’, sometimes short for ‘Munger’, is a simple network transparent window system. It was originally developed for the Sun 3 series of workstations by Stephen Uhler and colleagues beginning in 1984 while at Bellcore (later Telcordia, now part of Ericsson) and later enhanced by many others.”
[1908.03795] Eigenvectors from Eigenvalues “We present a new method of succinctly determining eigenvectors from eigenvalues. Specifically, we relate the norm squared of the elements of eigenvectors to the eigenvalues and the submatrix eigenvalues.”
How VCs Make Money “Digging into ‘Two and Twenty’ & VC compensation.” Example arithmetic about a 50M$ venture fund, plus other interesting analysis.
GitHub - lunaryorn/mdcat: cat for markdown “cat for Markdown.” Implemented in Rust; available in Homebrew.
Prophet | Prophet is a forecasting procedure implemented in R and Python. It is fast and provides completely automated forecasts that can be tuned by hand by data scientists and analysts. “Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data.” Implementations in Python and R.
Time Series Prediction - A short introduction for pragmatists · Blog · Liip “Are you trying to predict time series but don’t know where to start? This blog post will provide a comparison of the most prominent techniques and show you how to implement them.”
System design hack: Postgres is a great pub/sub & job server “If you’re making any project of sufficient complexity, you’ll need a publish/subscribe server to process events. This article will introduce you to Postgres, explain the alternatives, and walk you through an example use case of pub/sub and its solution.”
Separate Your Go Tests with Build Tags | Josh Michielsen (Jmickey) “Have you ever wanted to separate your unit tests from your integration or smoke tests? Go has a built-in mechanism for allowing you to logically separate your tests, in addition to adding conditionals (e.g. operating system or CPU architecture) with Build Tags.”

Bookmarks, 02019-10

stephen@blueslugs.com (Stephen Hahn) — Sat, 04 Jan 2020 07:22:12 +0000

One month’s bookmarks from pinboard.in:

Normform CSS improvements for forms.
Deciphering Glyph :: Tips And Tricks for Shipping a PyGame App on the Mac “A quick and dirty guide to getting that little PyGame hack you did up and running on someone else’s Mac.”
GitHub - terrycojones/daudin: A Python command-line shell “daudin is a UNIX command-line shell based on Python.”
Five new-ish Python things – Part 1 « Robin’s Blog Some Python utilities (including a MacOS previewer for Jupyter notebooks) I hadn’t heard of.
juruen/rmapi: Go app that allows you to access your reMarkable tablet files through the Cloud API Command line shell to manipulate reMarkable tablet contents.
Twitter Pulled out a classic on a cooler morning.
GoAccess - Visual Web Log Analyzer A terminal-based web log analyzer

Bookmarks, 02019-09

stephen@blueslugs.com (Stephen Hahn) — Sat, 04 Jan 2020 07:21:43 +0000

One month’s bookmarks from pinboard.in:

Twitter It is polite for an article on a report to link to the report. Possibly before the inevitable précis you felt you m…
Just Delete Me | A directory of direct links to delete your account from web services. Quick links (and assessment of deletion difficulty) for many web services.

Bookmarks, 02019-07

stephen@blueslugs.com (Stephen Hahn) — Tue, 12 Nov 2019 09:36:37 +0000

Last month’s bookmarks from pinboard.in:

A Mexico City Hotel for True Fans of Midcentury Modernism - WSJ Short review of a modern hotel and nearby attractions in Mexico City.
7 Gorgeous Free And Open-Source Typefaces And When To Use Them — Smashing Magazine Review of some recently published fonts.
Living without the modern browser – AN3223’s Blog – Linux and stuff Examples of how to use text browsers and other small clients instead of a web browser.
Assessing Contagion Risk in a Financial Network Contagion risk modeling.
ARIMA Model - Complete Guide to Time Series Forecasting in Python | ML+ Introduction to ARIMA models in Python

Bookmarks, 02019-06

stephen@blueslugs.com (Stephen Hahn) — Sat, 02 Nov 2019 08:48:02 +0000

Last month’s bookmarks from pinboard.in:

National Park Typeface “A TYPEFACE DESIGNED TO MIMIC THE NATIONAL PARK SERVICE SIGNS THAT ARE CARVED USING A ROUTER BIT.” (Upper case theirs.)

Community center demolition visible

stephen@blueslugs.com (Stephen Hahn) — Sat, 26 Oct 2019 07:00:00 +0000

Rockie and I have been regularly circling the Hillview Community Center as one of our early morning walks. When the construction fences went up, we adjusted our route. This Wednesday, we finally noticed demolition on a larger scale.

Bookmarks, 02019-05

stephen@blueslugs.com (Stephen Hahn) — Thu, 25 Jul 2019 06:06:05 +0000

Last month’s bookmarks from pinboard.in:

Modern SAT solvers: fast, neat and underused (part 3 of N) — The Coding Nest “In this part, we will look at how SAT solvers work and what heuristics and other tricks they employ in their quest for performance.”
TF-IDF in a nutshell – Manticore Search “In this article I’ll try to show in very clear examples how TF standing for ‘term frequency’ and it’s counterpart IDF standing for ‘inverse document frequency’ help to find what you’re looking for.”

Bookmarks, 02019-04

stephen@blueslugs.com (Stephen Hahn) — Thu, 25 Jul 2019 06:04:37 +0000

Last month’s bookmarks from pinboard.in:

Gauche - A Scheme Implementation “Gauche is an R7RS Scheme implementation developed to be a handy script interpreter, which allows programmers and system administrators to write small to large scripts for their daily chores.”
Indicator Stickynotes : Umang Varma A stickynotes application for Ubuntu.
Decode The Meaning Behind Your Apple Serial Number | BeetsBlog “The serial numbers of Apple products contain a codified language that can give you information about your Apple device that can’t be found through any other means—everything from the location where it was manufactured, the date it was manufactured, and much more. This information can be valuable for troubleshooting issues with your device or just for curiosity’s sake. So let’s get to breaking down their impenetrable code.”

Bookmarks, 02019-03

stephen@blueslugs.com (Stephen Hahn) — Mon, 01 Apr 2019 18:21:19 +0000

Last month’s bookmarks from pinboard.in:

How I’m able to take notes in mathematics lectures using LaTeX and Vim | Gilles Castel Uses context-aware snippets (and per-course special snippets).
Endlessh: an SSH Tarpit « null program “This program opens a socket and pretends to be an SSH server. However, it actually just ties up SSH clients with false promises indefinitely — or at least until the client eventually gives up. After cloning the repository, here’s how you can try it out for yourself (default port 2222):
```
 $ make
$ ./endlessh &
$ ssh -p2222 localhost
```
Your SSH client will hang there and wait for at least several days before finally giving up. Like a mammoth in the La Brea Tar Pits, it got itself stuck and can’t get itself out. As I write, my Internet-facing SSH tarpit currently has 27 clients trapped in it. A few of these have been connected for weeks. In one particular spike it had 1,378 clients trapped at once, lasting about 20 hours.”

Bookmarks, 02019-02

stephen@blueslugs.com (Stephen Hahn) — Mon, 01 Apr 2019 18:21:10 +0000

February’s bookmarks from pinboard.in:

Modern Alternatives to PGP A set of recommendations for (mostly) Golang modules for various cryptographic use cases, instead of using PGP/GPG.
The Datasaurus Dozen - Same Stats, Different Graphs: Generating Datasets with Varied Appearance and Identical Statistics through Simulated Annealing | Autodesk Research Used annealing to morph a dataset to 12 other shapes, while preserving summary statistics.
More Noise, Less Signal - The Big Picture “[As] a thought experiment, I wondered what the process might look like if went another way: What would happen if you purposefully tried to assemble a “How-to” list to pursue the exact opposite goal — how to get more noise, and less signal?” In other words, what are the exactly wrong things to do as an investor?

Bookmarks, 02019-01

stephen@blueslugs.com (Stephen Hahn) — Mon, 01 Apr 2019 18:20:53 +0000

January’s bookmarks from pinboard.in:

Objective-See BlockBlock “BlockBlock continually monitors common persistence locations and displays an alert whenever a persistent component is added to the OS.” In beta.
UCR Matrix Profile Page “The Matrix Profile (and the algorithms to compute it: …), has the potential to revolutionize time series data mining because of its generality, versatility, simplicity and scalability.”
The Stanford Encyclopedia of Philosophy has achieved what Wikipedia can only dream of — Quartz “[The encyclopedia’s] creators have solved one of the internet’s fundamental problems: How to provide authoritative, rigorously accurate knowledge, at no cost to readers. It’s something the encyclopedia, or SEP, has managed to do for two decades.”

Flocking and out of range

stephen@blueslugs.com (Stephen Hahn) — Sat, 05 Jan 2019 17:28:49 +0000

Seen in Lihue, HI.

Bookmarks, 02018-12

stephen@blueslugs.com (Stephen Hahn) — Sat, 05 Jan 2019 08:00:22 +0000

Last month’s bookmarks from pinboard.in:

Optician Sans – Free font based on historical optotypes “Optician Sans is based on the same visual principles as the LogMAR chart, adjusted to be used as a fully functional display typeface.”
MATRIX VISION - mvBlueLYNX Documentation: Programming own applications | IPK how-tos A good, short overview of ipk package tools and construction.

Bookmarks, 02018-11

stephen@blueslugs.com (Stephen Hahn) — Sat, 05 Jan 2019 08:00:18 +0000

Last month’s bookmarks from pinboard.in:

precurse/ansible-smartos_provision Repository for SmartOS Ansible provisioning role.
vmadm - Manage SmartOS virtual machines and zones. — Ansible Documentation Ansible module for VM provisioning on SmartOS. Requires Python 2 in global zone.
MITRE ATT&CK™ MITRE ATT&CK™ is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. (Referred to as the “attack matrix”.)
Germany pushes router security rules, OpenWRT and CCC push back • The Register German requirements for routers.
7 command-line tools for writers | Opensource.com [No summary.]

Bookmarks, 02018-10

stephen@blueslugs.com (Stephen Hahn) — Sat, 05 Jan 2019 08:00:16 +0000

Last month’s bookmarks from pinboard.in:

Kubernetes: The Surprisingly Affordable Platform for Personal Projects | doxsey.net “Kubernetes makes sense for small projects and you can have your own Kubernetes cluster today for as little as $5 a month.” Small tutorial on setting up a cluster to run a web application.

Bookmarks, 02018-08

stephen@blueslugs.com (Stephen Hahn) — Mon, 01 Oct 2018 18:00:46 +0000

Last month’s bookmarks from pinboard.in:

SMT-LIB The Satisfiability Modulo Theories Library “SMT-LIB is an international initiative aimed at facilitating research and development in Satisfiability Modulo Theories (SMT).” In particular, a language for representing SMT problems that multiple solvers understand.
GitHub - AlDanial/cloc “cloc counts blank lines, comment lines, and physical lines of source code in many programming languages.”
Why Automated Feature Engineering Will Change the Way You Do Machine Learning “Automated feature engineering is a relatively new technique, but, after using it to solve a number of data science problems using real-world data sets, I’m convinced it should be a standard part of any machine learning workflow.”
Oztekin, Dennis Ekin, “The Lattice Boltzmann Methods and Their Applications to Fluid Flows” (2014). Theses and Dissertations. Paper 1581. “The history of the Lattice Boltzmann Method and its application to fluid mechanics are investigated here. Detailed formulations are provided to form a basis for the Lattice Boltzmann Method and its many variations. These variations are designed to overcome shortcomings in the standard single relaxation time Lattice Boltzmann model. Presented here are: a model that utilizes the non-equilibrium parts of the stress tensor, the Regularized Lattice Boltzmann model; a model that converts over to momentum space, the Multi-Relaxation Time Lattice Boltzmann model; and a model that corrects itself using the entropy equation, the entropic Lattice Boltzmann model. Extensions for the Lattice Boltzmann method are derived that include: external forces, multiphase flows, and thermal flows. Various types of boundary conditions are modeled using different approaches. A detailed explanation on extracting common macroscopic flow properties in physical units is provided. These extracted properties can be used to check temporal and spatial convergence. A two dimensional, nine velocity model and a three dimensional, fifteen velocity model are used to provide examples of a number of the approaches mentioned. A two dimensional and three dimensional lid-driven cavity flow is used to illustrate these methods.”
Lattice Boltzmann methods - Wikipedia “Lattice Boltzmann methods (LBM) … [are] a class of computational fluid dynamics (CFD) methods for fluid simulation. " Successor to lattice gas automaton methods.

Bookmarks, 02018-06

stephen@blueslugs.com (Stephen Hahn) — Thu, 19 Jul 2018 07:39:28 +0000

Last month’s bookmarks from pinboard.in:

US Coastal Property at Risk from Rising Seas “This national analysis identifies the number of US homes at risk from chronic flooding over the coming decades due to sea level rise.”

Bookmarks, 02018-05

stephen@blueslugs.com (Stephen Hahn) — Sat, 09 Jun 2018 16:56:12 +0000

Last month’s bookmarks from pinboard.in:

Plumbum: Shell Combinators and More — Plumbum: Shell Combinators “Plumbum… is a small yet feature-rich library for shell script-like programs in Python. The motto of the library is ‘Never write shell scripts again’, and thus it attempts to mimic the shell syntax (shell combinators) where it makes sense, while keeping it all Pythonic and cross-platform.”
GitHub - RelaxedJS/ReLaXed: Create PDF documents using web technologies “ReLaXed creates PDF documents interactively using HTML or Pug (a shorthand for HTML). It allows complex layouts to be defined with CSS and JavaScript, while writing the content in a friendly, minimal syntax close to Markdown or LaTeX.”
MacDown: The open source Markdown editor for macOS “MacDown is an open source Markdown editor for macOS, released under the MIT License. "
WeasyPrint converts HTML/CSS documents to PDF - WeasyPrint “WeasyPrint is a visual rendering engine for HTML and CSS that can export to PDF. It aims to support web standards for printing. WeasyPrint is free software made available under a BSD license.”

Bookmarks, 02018-04

stephen@blueslugs.com (Stephen Hahn) — Thu, 17 May 2018 17:38:21 +0000

Last month’s bookmarks from pinboard.in:

A more privacy-friendly blog | Vincent Bernat “When I started this blog, I embraced some free services, like Disqus or Google Analytics. These services are quite invasive for users’ privacy. Over the years, I have tried to correct this to reach a point where I do not rely on any “privacy-hostile” services.”
Spread syntax - JavaScript | MDN “Spread syntax allows an iterable such as an array expression or string to be expanded in places where zero or more arguments (for function calls) or elements (for array literals) are expected, or an object expression to be expanded in places where zero or more key-value pairs (for object literals) are expected.” The spread operator is “…”.

Bookmarks, 02018-03

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Apr 2018 09:43:06 +0000

Last month’s bookmarks from pinboard.in:

Always start with a stupid model, no exceptions. – Insight Data Advocates starting with simple models, like linear or logistic regression, and then understanding where those initial models fail.
What Is a “Policy Imagination”? - Lawyers, Guns & Money Interesting for citation of mid-20th century “National Balance Sheet” model by Baxter.
Rough.js “Rough.js is a light weight (~8k), Canvas based library that lets you draw in a sketchy, hand-drawn-like, style. The library defines primitives to draw lines, curves, arcs, polygons, circles, and ellipses. It also supports drawing SVG paths.”
GitHub - mpetazzoni/ptftpd: A TFTP/PXE tool suite written in Python “pTFTPd is a collection of tools related to TFTP. It includes a TFTP server, a TFTP client, and a complete PXE solution based on this TFTP server and a micro-DHCP or BOOTP server. All these tools are written in Python and designed to be fast, RFC compliant and easy to use.”
Input: Fonts for Code “Input is a flexible system of fonts designed specifically for code by David Jonathan Ross. It offers both monospaced and proportional fonts, all with a large range of widths, weights, and styles for richer code formatting.”

Bookmarks, 02018-02

stephen@blueslugs.com (Stephen Hahn) — Tue, 06 Mar 2018 18:02:44 +0000

Last month’s bookmarks from pinboard.in:

GitHub - Submanifold/latex-mimosis: A minimal & modern LaTeX template for your (bachelor’s | master’s | doctoral) thesis “This repository contains a minimal & modern LaTeX template for dissertations and other university documents.” Very nice; A4 by default.
SfePy: Simple Finite Elements in Python “SfePy is a software for solving systems of coupled partial differential equations (PDEs) by the finite element method in 1D, 2D and 3D. It can be viewed both as black-box PDE solver, and as a Python package which can be used for building custom applications. The word “simple” means that complex FEM problems can be coded very easily and rapidly.”
Modern CSS Explained For Dinosaurs – Actualize – Medium Survey from CSS basics to methodologies used today.

Bookmarks, 02018-01

stephen@blueslugs.com (Stephen Hahn) — Tue, 06 Mar 2018 18:02:42 +0000

Last month’s bookmarks from pinboard.in:

Not Another Free Image Click-Bait Article | Create “High-quality, free images that you can legally modify for commercial projects do exist. Here’s where to find digital files you can use in any [graphical] application.”
Podcast analytics and hosting - Backtracks Current pricing for a hobbyist is ~480$/a.
Tim O’Reilly - Frank Herbert - O’Reilly Media O’Reilly wrote a short book–now out of print–on Frank Herbert.
55 [In]visible Cities Project — KARINA PUENTE Drawings of the cities described in Calvino’s Invisible Cities.

Bookmarks, 02017-12

stephen@blueslugs.com (Stephen Hahn) — Tue, 23 Jan 2018 17:52:25 +0000

Last month’s bookmarks from pinboard.in:

Let your code type-hint itself: introducing open source MonkeyType “Today we are excited to announce we’re open-sourcing MonkeyType, our tool for automatically adding type annotations to your Python 3 code via runtime tracing of types seen.” Trace Python 3 scripts, then output captured type assignments (or apply to source).
gka/schnack: 🗣️ Simple node app for Disqus-like drop-in commenting on static websites “schnack.js is a simple node app for Disqus-like drop-in commenting on static websites.”
GitHub - jessfraz/weather: Weather via the command line. “Weather via the command line. Uses the darksky.net API so it’s super accurate. Also includes any current weather alerts in the output.”
12 trail runs under 10 miles in SF and the East Bay | Bzarg “… I offer below a catalog of twelve routes we’ve found over the last year. Some of these have alternate routes, so be sure to use the Google Maps layer view to selectively view the one you’re interested in, so the two routes don’t clutter each other.”
Python API Checklist “Useful checklist for building good Python library APIs.” Many of the design principles apply to any small library interface.
HeavyThing x86_64 assembler library | 2 Ton Digital An library for application programming in x86 assembly.

Bookmarks, 02017-11

stephen@blueslugs.com (Stephen Hahn) — Mon, 18 Dec 2017 15:08:42 +0000

Last month’s bookmarks from pinboard.in:

Linux containers in 500 lines of code A very strong article on the code necessary to construct a container from scratch on Linux. Careful commentary and citations on capabilities and seccomp restrictions to limit processes within the container.

Bookmarks, 02017-10

stephen@blueslugs.com (Stephen Hahn) — Sun, 05 Nov 2017 15:30:16 +0000

Last month’s bookmarks from pinboard.in:

Twitter RT @ashleymloftin: RT if you’re an adult who uses the library.
Vim After 15 Years | Ian Langworth’s Things of Variable Interest Vim 8 added a lot of much-needed functionality, and new community sites like VimAwesome have made plugin discovery and evaluation easier. I’ve been doing a lot more work with Vim lately and have spent some time configuring my workflow for peak efficiency, so here’s a snapshot of my current state.
Storyboarder - The best and easiest way to storyboard. | Wonder Unit “Storyboarder makes it easy to visualize a story as fast you can draw stick figures. Quickly draw to test if a story idea works. Create and show animatics to others. Express your story idea without making a movie.”
Errata Security: “Responsible encryption” fallacies RT @ErrataRob: I wrote a thing on “responsible disclosure”. I’m not happy with it, but some ideas need to be addressed.
No brogrammers: Practical tips for writing inclusive job ads “By paying attention to the language we use to describe ourselves, our ideal candidates, and the job responsibilities, we can broaden the net of candidates that might apply and help in some small way to tackle the tech diversity problem.”
Page Visibility API - Web APIs | MDN “The Page Visibility API lets you know when a webpage is visible or in focus. With tabbed browsing, there is a reasonable chance that any given webpage is in the background and thus not visible to the user. When the user minimizes the webpage or moves to another tab, the API sends a visibilitychange event regarding the visibility of the page. You can detect the event and perform some actions or behave differently.” Occurs regularly on mobile browsers.
milochristiansen/lua: A Lua 5.3 VM and compiler written in Go. “A Lua 5.3 VM and compiler written in Go.” Has some language/library pruning, in support of security goals.

Bookmarks, 02017-09

stephen@blueslugs.com (Stephen Hahn) — Sun, 01 Oct 2017 15:32:31 +0000

Last month’s bookmarks from pinboard.in:

OmniOS build - asciinema RT @OmniOSce: Want to build #OmniOS yourself? It’s easier than you think! #illumos
GitHub - tomazk/envcfg: Un-marshaling environment variables to Go structs Interesting golang module that manages marshalling of application-relevant environment variables to and from a struct. Includes a convenient .ClearEnvVars() method that unsets the variable set, which would be used prior to spawning a child process.

Bookmarks, 02017-08

stephen@blueslugs.com (Stephen Hahn) — Wed, 20 Sep 2017 23:32:46 +0000

Last month’s bookmarks from pinboard.in:

Tgres 0.10.0b - Time Series with Go and PostgreSQL - Grisha Trubetskoy “Tgres is a Go program which can receive time series data via Graphite, Statsd protocols or an http pixel, store it in PostgreSQL, and provide Graphite-like access to the data in a way that is compatible with tools such as Grafana. You could think of it as a drop-in Graphite/Statsd replacement, though I’d rather avoid direct comparison, because the key feature of Tgres is that data is stored in PostgreSQL.”
Offline-Friendly Forms | Max Böck - Frontend Web Developer “Forms on the web don’t usually play nice with bad connections. If you try to submit a form while offline, you’ll most likely just lose your input…. With the introduction of Service Workers, developers are now able to supply experiences on the web that will work even without an internet connection. While it’s relatively easy to cache static resources, things like forms that require server interaction are harder to optimize. It is possible to provide a somewhat useful offline fallback though.”
Ferret Programmer’s Manual “Ferret is a free software Clojure implementation, it compiles a restricted subset of the Clojure language to self contained ISO C++11 which allows for the use of Clojure in real time embedded control systems.”
GitHub - hslatman/awesome-threat-intelligence: A curated list of Awesome Threat Intelligence resources “A curated list of awesome Threat Intelligence resources.”

Bookmarks, 02017-07

stephen@blueslugs.com (Stephen Hahn) — Wed, 02 Aug 2017 21:21:03 +0000

Last month’s bookmarks from pinboard.in:

A Massive Guide to Building a RESTful API for Your Mobile App - Savvy Apps “In our experience, we’ve found that establishing guidelines on how to build better APIs for mobile apps saves time and effort during development and reduces headache later on in the process…. We wrote this guide to outline the best practices for building out APIs, web services, and databases for mobile apps and mobile clients. In this post, we go in-depth on creating a RESTful API specifically for mobile apps.”
Calendar Versioning — CalVer “CalVer is a software versioning convention that is based on your project’s release calendar, instead of arbitrary numbers.”
How to turn your website into a PWA | Max Böck - Frontend Web Developer “A Progressive Web App, or PWA, uses modern web capabilities to deliver an app-like user experience. Any website can be a PWA - here’s how to do it.”
Fountain Pen Buying Guide: 7 Options Under $30 | WIRED “We’ve rounded up seven inexpensive fountain pens (all are under $30; one is only $3) that either are disposable or employ swappable cartridges to make refilling simple and tidy.”
Wildcard Certificates Coming January 2018 - Let’s Encrypt - Free SSL/TLS Certificates Wildcard certificates will be offered free of charge via our upcoming ACME v2 API endpoint. We will initially only support base domain validation via DNS for wildcard certificates, but may explore additional validation options over time.

Quick tournament link, 7/22

stephen@blueslugs.com (Stephen Hahn) — Sat, 22 Jul 2017 12:00:00 +0000

Nathaniel is debuting with MVLA this weekend in his first tournament of the summer :

2017 Juventus Tournament of Champions U14B

Glossy

stephen@blueslugs.com (Stephen Hahn) — Sun, 23 Apr 2017 10:40:00 +0000

Even the ceramic cone (then in use) has a sheen.

Building `make-guile` on MacOS

stephen@blueslugs.com (Stephen Hahn) — Sun, 09 Apr 2017 13:10:00 -0700

I’ve been experimenting with GNU Make with the optional Guile support. On Debian, make-guile is a package, but on MacOS I had to build a local copy, tying in the Homebrew build of Guile.

First, to verify that your installation of GNU Make doesn’t already have Guile support, use this small Makefile to test the .FEATURES variable.

$ cat > ~/Makefile.test-guile
has_guile:
@if echo $(.FEATURES) | grep -vq guile; then \
echo "no \c"; \
fi; echo "guile support"
^D

Now we have a test we can use with various GNU Make deployments:

$ make -f ~/Makefile.test-guile
no guile support

To build make, we will need a C compilation environment, the source archive for GNU Make, and a working libguile. For the latter, I chose to reuse the guile built by Homebrew.

$ brew install guile

Then, configuration should look something like

$ ./configure $(autoargs) --with-guile \
GUILE_CFLAGS=-I/usr/local/include/guile/2.2 \
GUILE_LIBS="-L/usr/local/lib \
-lguile-2.2.0"
....

(autoargs is a utility that sets PREFIX and other autoconf variables for installation into my home directory on the local system.)

After a make and make install, we should have a version of make we can invoke:

$ make -f ~/Makefile.test-guile
guile support

Now we have a means to experiment with Scheme in the context of the Make/Guile platform.

Updating CPPFunction in old readline consumers

stephen@blueslugs.com (Stephen Hahn) — Sun, 23 Oct 2016 13:19:10 -0700

One of my favourite open source utilities is tdl, a command-line to-do list manager. I was recently updating a SmartOS VM for development work, and noticed that I lacked a working tdl binary. An attempt to build it resulted in a failure like

gcc -O2 -Wall -DUSE_DOTLOCK=1 -DUSE_READLINE=1 -DBARE_READLINE_H=0 -DUSE_RL_COMPLETION_MATCHES=1 -c inter.c
inter.c: In function 'interactive_text_readline':
inter.c:559:3: error: unknown type name 'Function'
inter.c:562:25: warning: assignment from incompatible pointer type [enabled by default]
inter.c:568:21: warning: assignment from incompatible pointer type [enabled by default]
inter.c: In function 'interactive_text':
inter.c:636:41: error: 'CPPFunction' undeclared (first use in this function)
inter.c:636:41: note: each undeclared identifier is reported only once for each function it appears in
inter.c:636:54: error: expected expression before ')' token
inter.c:638:54: error: expected expression before ')' token
inter.c: In function 'interactive':
inter.c:656:41: error: 'CPPFunction' undeclared (first use in this function)
inter.c:656:54: error: expected expression before ')' token
inter.c: At top level:
inter.c:320:15: warning: 'tdl_completion' defined but not used [-Wunused-function]
inter.c:343:15: warning: 'null_tdl_completion' defined but not used [-Wunused-function]
inter.c: In function 'get_line_stdio':
inter.c:605:1: warning: control reaches end of non-void function [-Wreturn-type]
Makefile:61: recipe for target 'inter.o' failed
make: *** [inter.o] Error 1

Version 1.5.2 of tdl was released on 02004-02-03. The contemporary version of GNU readline would have been 4.3; today, pkgsrc on SmartOS delivers readline 6.3, which was released 02014-02-26. I know I’ve built tdl much more recently than 02004, so I decided to spend some time to determine what’s changed between these two components.

With 6.3, the compatibility type definitions for Function, CPFunction, CPPFunction, and VFunction were finally removed. Each of these “old style” readline function types can typically be replaced with the “new style” types as follows

Function becomes rl_hook_func_t in hook assignments. Function may map to other function types in rltypedefs.h if a non-void argument list is given.
VFunction becomes rl_voidfunc_t.
VCPFunction becomes rl_vcpfunc_t.
CPFunction becomes rl_compentry_func_t.
CPPFunction becomes rl_completion_func_t.

The patch for tdl-1.5.2:inter.c resolves all but the final warning above, and should lead to happy task tracking with tdl.

(If you are on MacOS X (and potentially on other systems that deliver editline), then the editline include files that provide readline.h declarations still provide the old style definitions.)

Panorama near Levi's Stadium

stephen@blueslugs.com (Stephen Hahn) — Sat, 15 Oct 2016 23:30:00 +0000

Last year, I decided to combine and upload our fourteen years of digital photos to Google Photos. An enjoyable aspect of Photos is that the application will notify you of new products it has created from your incoming photos, such as flipbook-style animations and filtered/enhanced pictures.

Nathaniel was playing a first round State Cup game at the Santa Clara soccer complex Saturday evening. (We were here a year ago. I suppose there was a Super Bowl in between these visits.) The weather this weekend was dominated by the remnants of Typhoon Songda, which became an atmospheric river storm along the west coast. It was too wet to bring out the DSLR, so I took quick pictures with my phone.

The next day, Google Photos sent me the above panorama, which it must have stitched together from a couple of pictures I had taken early in the game. Cool.

(Palo Alto lost, 1-2.)

Quick tournament link, 8/5

stephen@blueslugs.com (Stephen Hahn) — Fri, 05 Aug 2016 14:38:17 +0000

Nathaniel is active this weekend in his first tournament of the summer :

Santa Clara Sporting Invitational U14B

Warmup by Toyota Gate

stephen@blueslugs.com (Stephen Hahn) — Wed, 25 Nov 2015 16:51:49 +0000

Nathaniel’s soccer team played a night game in Santa Clara a couple of weeks ago, at the field complex right beside Levi’s Stadium. (We lost, 0-3. It’s been a tough season.)

Quick tournament link, 11/25

stephen@blueslugs.com (Stephen Hahn) — Wed, 25 Nov 2015 15:56:14 +0000

Nathaniel is active in a tournament this coming weekend. Bracket link:

Surf Cup Thanksgiving, U12B, North San Diego

Regional cross-country championships are on Sunday in Fresno. We’re waiting to see which of our teams qualified.

Quick tournament link, 5/23

stephen@blueslugs.com (Stephen Hahn) — Fri, 15 May 2015 16:58:44 +0000

Nathaniel is active in a tournament next weekend. Bracket link:

Cerritos Memorial, U11B Nike, Long Beach and vicinity

Ben’s running in the 3M meet this weekend.

Quick tournament link, 3/7

stephen@blueslugs.com (Stephen Hahn) — Sat, 07 Mar 2015 00:30:49 +0000

Nathaniel is active in a tournament this weekend. Convenient link to his bracket:

Spring Showcase, U12B Gold, Redwood City

Ben is waiting for the youth track season to start.

Using Astral from a known location

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Jan 2015 06:18:16 +0000

I’ve been working out a minor idea involving the control of some household actions based on local time, but relative to sunrise and sunset rather than a naïve time of day. Simon Kennedy’s Astral is a Python module that can compute these times, but its examples focus on retrieval of locations from major cities. Most places aren’t major cities in the module’s list, so I spent a little time to read the source to determine what other entry points were enabled.

It turns out that the Location class is perfect for those times when you can’t associate a site with a major city, but know the latitude and longitude (from GPS or a mapping service, for example). An example usage follows:

#!/usr/bin/python
"""Example of using astral with a location not in the module's
built-in catalog."""
import astral
import datetime
# From A. Mariano, MacOS units(1), 1993.
FT_PER_METRE = 3.2808399
# Construct our location. Longitude west and latitude south are
# negative.
los_altos = astral.Location(info=("Los Altos", "USA", 37.3681,
-122.0975, "US/Pacific",
157/FT_PER_METRE))
# "civil", which means 6 degrees below the horizon, is the default
# value for computing dawn and dusk. But this usage shows how to
# set it before calculation.
los_altos.solar_depression = "civil"
tomorrow = datetime.date.today() + datetime.timedelta(1)
result = los_altos.sun(date=tomorrow)
for k in ["dawn", "sunrise", "noon", "sunset", "dusk"]:
print "%7s %s" % (k, result[k])

node.js v10.03 on oi_151a7

stephen@blueslugs.com (Stephen Hahn) — Sun, 03 Aug 2014 04:21:57 +0000

My 2012 instructions for building node.js on OpenIndiana still work on recent bits. Now to write some software.

piCorePlayer

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 Feb 2014 00:41:15 +0000

I learned that Logitech has brought the Squeezebox product line to end to life. I’m not prepared to replace (or research what might replace) our current audio infrastructure, so I’m experimenting with a piCorePlayer install to see if a low cost, Raspberry Pi solution can solve this problem, much like OpenSprinkler has addressed home irrigation.

Lunch: Fletch's, San Mateo

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 Feb 2014 00:36:29 +0000

Hot Dog and Slider, Fletch’s, San Mateo, CA, a photo by schahn on Flickr.

We went on a drive today, to try Fletch’s Chicago-style hot dogs and sliders. They make their own dipping sauce, which has a touch of spice and went well with the fries. I couldn’t muster the appetite, but there is a Beard Papa on the very same block of 2nd Ave.

Instrumented soccer, fall 2013

stephen@blueslugs.com (Stephen Hahn) — Tue, 14 Jan 2014 07:12:20 +0000

I just started the spring season in my recreational soccer league last weekend. And I’m still using Adidas’s miCoach system to record statistics in each game. Earlier I showed a couple of individual sessions, with image captures from the miCoach site. Here I’m going to summarize the season, and compare it to the previous season (Summer 2013). Top line summary of the season

11 games, 9/8 – 12/29
8 goals
- Compared to 1 in Summer 2013
- All on turf

For the quantities tracked by the Speed Cell, we have a group of measurements, summarized as follows

Distance: 4.0 ± 0.6 miles per game, with a best of 5.1 miles on 12/1. (Improved over Summer 2013, 3.9 ± 0.5 mpg.)
Sprints: 20.5 ± 4.5 sprints per game, with a best of 30 on 12/1. (Improved over Summer 2013, 18.5 ± 4.7 spg.)
Maximum speed: 15.0 ± 0.4 mph, with a best of 15.52 mph on 9/15 and 10/6. (Improved over Summer, 2013 14.5 ± 1.1 mph.)

miCoach offers badges (“achievements”) for various performance levels: on 12/1, 5.1 miles of total distance and 0.55 miles of “hi intensity distance” resulted in the Pro achievements for those two categories. Still haven’t reached the Pro mark for Maximum Speed, which I suspect will require a burst over 16 mph. (On 6/9, 3.9 miles of distance and 15.17 mph maximum speed earned the respective Club achievements. On 7/14, 0.38 miles of high intensity distance earned the corresponding Club achievement.)

I was pretty happy with my showing on December 1, but Sean Ingle’s Sunday article in The Guardian, on Chelsea and England defender Ashley Cole, shows how low on the performance scale these numbers are:

The additional physical demands are clear from Prozone’s data. In 2003-04 Premier League full-backs made an average of 29.5 sprints – any movement greater than seven metres a second – over a game. This season that figure is exactly 50. A decade ago the average recovery time for a full-back between high-intensity activities – any movement greater than 5.5m/s, or a three-quarters speed run – was 56.4sec. Now it is 40.4sec.

— Ashley Cole faces life in the slow lane as demands of the job intensify

(Of course, I’m ten years older than Ashley Cole, so I am pleased just to be on the field, uninjured, and possessing a little bit of pace.)

It’s going to be hard to compare the spring season with this one, as the new team only has part-time goalkeepers, and I’ll have to take many more shifts in net. (The two times I played keeper during the fall were 13 sprint games.) Other than working on my metrics, my only goal for this season is to finally score one on the grass fields.

United States Drought Monitor

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Jan 2014 17:10:00 +0000

As I mentioned in the sprinkler repair/upgrade post, Northern California continues its now multi-year drought. The United States Drought Monitor currently labels the conditions as D2 – severe drought.

Sprinkler valve break and repair

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Jan 2014 07:17:57 +0000

Early last summer, Nathaniel was in the back yard, practicing his kicks into our KixKube. One particularly strong kick missed the cube, and smashed into the exposed sprinkler valves, breaking through the ¾” PVC pipe below one of the valves. The break led immediately to that portion of the garden and lawn being covered in water, so I rushed to close the water main at the street and then scurried to Home Depot to get a PVC cap and fresh (fast setting) PVC cement.

Water supply capped, broken valve turned, a photo by schahn on Flickr.

You can see the cap at the right of the above photo; it’s the piece below the blue PVC cement-stained pipe. The fragments of plastic are the remains of the storage container I placed over the valves to protect them from further misses. (Not successfully.)

I decided to make two improvements when I rebuilt this complex: to add a ball valve before the sprinkler valves, so that I can make repairs without turning off water to the entire house, and to add a third sprinkler valve for a drip system. You can see the ball valve underneath the rightmost valve in the picture below; the new valve is the leftmost.

Sprinklers repaired and robustly reboxed, a photo by schahn on Flickr.

It took a bit of tweaking to get the joints and valves sealed tight, with one or two connections having to be redone. Once I had the valves watertight and the wiring reconnected, my system testing showed that our Rain8 WLM, which was an X10 wireless-based irrigation control, had suffered a power blow-out as well.

OpenSprinkler Pi, a photo by schahn on Flickr.

Since the Rain8 WLM had been discontinued, I decided to replace the controller with an OpenSprinkler Pi, using WiFi for wireless access. For the X10-based controller, I had built my own set of irrigation programs for cooler, which had been moving from shell scripts invoking flipit driven by cron(1M) to using Redis for state and eventually to a web server model, but I dropped all of that—for now—in favour of Sprinklers Pi running directly on sprinkles (the Pi). With the combination of a regular and a seeding schedule, I have equivalent (or better, with the Weather Underground integration) capability to the previous implementation. The new system has been working well.

Overseeding

With the watering infrastructure restored, I did our fall overseeding, three months late, using a mix of low-water grass and white clover, topped with new soil and compost. These guys—mature and immature white-crested swallows (Zonotrichia leucophrys nuttalli)—showed up once the grass seed hit the ground (and walkway). Let’s hope they leave enough seed to grow. (The mature one is the one sporting the white crest with black stripes on his head.)

White-crested swallows feeding on grass seed, a photo by schahn on Flickr.

This is the second California animal species or subspecies we’ve mentioned here that is named in honour of Thomas Nuttall. (Earlier in the summer, we saw a Nuttall’s woodpecker.)

Weeding by grid

Now that we’ve begun repairing the lawn with new grass, the final phase will be plucking out the crabgrass that dominated one corner. I’m going to work systematically through the lawn using a grid system, plucking out each crabgrass shoot.

Grid, a photo by schahn on Flickr.

There’s about ½” of new soil spread over top to provide new nutrients, but there are also some pretty beaten down bare patches. We’ll see what the spring growth brings.

Next steps

Assuming I establish some level of control over the crabgrass, I’ll connect a small drip irrigation system to that unused valve and introduce a corresponding third schedule to the Sprinklers Pi configuration. If that goes well, I’ll add a second valve to the front lawn for drip—unless we decide that even a small front lawn is an indulgence during a severe drought. (In which case, maybe I’ll transplant the sod from the front to the back and go drip-only in the front to the citrus trees and oleander.)

Blog traffic summary, 2013

stephen@blueslugs.com (Stephen Hahn) — Tue, 31 Dec 2013 09:29:46 +0000

wordpress.com annual report: This year’s most popular post was my short treatment of youth soccer formations. It didn’t have much competition, however, as I wasn’t able to write much here in 2013.

Lunch: Freewheel Brewing Co., Redwood City

stephen@blueslugs.com (Stephen Hahn) — Sun, 29 Dec 2013 00:30:59 +0000

Hot Dog, Freewheel Brewing Co., Redwood City, CA, a photo by schahn on Flickr.

Not too far from the office, FreeWheel Brewing has good beer and hearty food. The sausage board is tasty, if one has the appetite, but the hot dog is reliable and well accompanied by a pint. (Half pints available for those meeting-rich afternoons.)

Weekend tournament quick links, 8/31

stephen@blueslugs.com (Stephen Hahn) — Thu, 29 Aug 2013 04:36:58 +0000

The boys are both active in tournaments this weekend. Convenient links to their brackets:

Copa Madera, U12B, Redwood City
NHB Cup 2013, U10B, Huntington Beach

Lunch: New England Lobster Market & Eatery, Burlingame

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Aug 2013 21:51:49 +0000

Seasonal Lobster Roll, New England Lobster Market & Eatery, originally uploaded by schahn.

We drove farther than average to visit the New England Lobster Market & Eatery [Yelp], but the pleasant outdoor space they’ve created a hundred yards or so from the shore of San Francisco Bay combined with a very credible lobster roll made it worthwhile. The potato chips are also excellent.

Figuring out 8v8 formations by reduction

stephen@blueslugs.com (Stephen Hahn) — Tue, 30 Jul 2013 06:00:38 +0000

For the younger ages of the California competitive leagues the boys have played in, sides are eight players each, and don’t move to the adult 11-a-side game until the spring season of U11. Ben is in his second season of 11-a-side, while Nathaniel has a few seasons of 8-a-side still to go. Since we try to be informed spectators of each, I thought it would be interesting to comment on the 8v8 formations as they relate to the adult formations.

Wikipedia’s coverage of soccer (“association football”) topics is extensive—covering players, club and national teams, tournament histories, and aspects of the game. The article on soccer formations gives an overview of both the historical development of formations and a summary of the current common formations. We’ll use it as a basis for identifying key adult formations; the article covers many more.

For formations with three rows, defenders–midfielders–forwards, an 8 player formation can be derived from an 11 player formation by subtracting one player from each row. Since the four row formations are refinements of the midfielders into a pair of sub ranks, we treat formations like the 4–2–3–1 and 4–3–1–2, as variations of the 4–5–1 and 4–4–2, respectively.

4–4–2 → 3–3–1

You’ll see a lot of teams playing the 3–3–1, as it’s the recommended 8v8 formation in the US Soccer curriculum. Although your striker has to work hard to apply pressure on his or her own, he or she benefits by being the focus of attacks up the middle. This formation has natural width and a well-populated midfield, although I haven’t seen many teams that encourage their outside defenders to make runs forward. (So we see many conservative 3–3–1 teams.) When your team moves up, you’ll find the 4–4–2 is one of the recommended 11v11 formations for U12 in the US Soccer curriculum. (Your striker may find the transition challenging, as they will have to pair with a peer, rather than be the single focus.)

4–3–3 → 3–2–2

The 4–3–3 is the other recommended 11v11 formation for U12 in the US Soccer curriculum, so it’s natural to hope that it’s 8v8 reduced form would be easy to use as well. However, the midfield and forward part of the resulting formation is narrow, meaning that it’s easy for opponents to find space to work through; against a 3–3–1, the 3–2–2 is at a numerical disadvantage in the midfield. I’m not sure I’ve seen any team line up in a 3–2–2.

4–5–1 → 3–4–0

Ben’s team in 2012 ended up playing in a 3–4–0 for much of their spring season. I believe what the coach was trying to address was the forwards’ tendencies to not come back on defense once possession was lost. The drawback is that, without a single striker occupying the defense (to maintain length), the opposing defenders can come forward and capture cleared balls easily. It’s also unclear which of the midfield is responsible for pressuring the opponent, although teams with good individual defensive instincts or training might enjoy the flexibility of having the closest player apply the pressure.

3–4–3 → 2–3–2

Wigan Athletic under Martinez in recent years stood out in the Premier League for playing a 3–4–3. Its reduction, the 2–3–2, might work as a tactic for a 3–3–1 team trying to bring more offense to bear in the final portion of a game. The two defender line will have to be very strong, or your keeper will be very busy.

3–5–2 → 2–4–1

Nathaniel’s team often plays in a 2–4–1 since they have a number of outside midfielders willing to run from goal line to goal line and cover the opposing outside forward when on defense. Risky like the 2–3–2 if the defenders aren’t strong and coordinated, although the extra player in midfield should stop the opponent from finding overloads as easily as they might against a 2–3–2.

5–m–n → 4–(m-1)–(n-1)

I haven’t seen anyone playing four defenders in 8v8, because we haven’t seen anyone willing to give up most of their offensive chances by clamping down defensively. (Most youth tournaments award points for goals scored, in addition to the game result, so losing 2–3 can be better than tying 0–0. In other areas, you might see youth teams happy to draw…)

Translating per-position comments

If you are trying to describe each position’s responsibilities in an 8v8, the key difference between the responsibilities in the adult formation and in the reduction comes from the change in each row from an odd to an even number of players. That can be made even simpler: the even-numbered rows need to be very clear about how to divide their coverage of the field (left-right or left-center left-center right-right), while the odd-numbered rows should find this allocation clear (solo or left-center-right). For some formations, positions are otherwise almost unchanged: the wide players in the 3–3–1 have the same responsibility to go forward as their counterparts in the 4–4–2. But, much like our striker in the 3–3–1 must learn to coordinate with a second striker in the 4–4–2, so will the central defender in a 3–m–n mesh with a partner in a 4–(m+1)–(n+1).

Picoides nuttalli foraging

stephen@blueslugs.com (Stephen Hahn) — Mon, 29 Jul 2013 04:25:00 +0000

Nuttall’s woodpecker, originally uploaded by schahn.

While retrieving a few items from the car I parked out front, I heard the burst of a woodpecker tapping, and grabbed my camera (with an 80 – 320 mm lens for soccer) to see if I could get a good enough picture for identification. The camera rig was much more suitable than the camcorder I used to snap a merlin in 2005.

Ben looked at the photo and worked through his options in the AMNH Birds of North America. Based on the range, the barred back, and red crown, he thinks it’s an example of Nuttall’s Woodpecker (but it could be a Ladder-backed or a hybrid of the two).

This serene moment of observation ended when the squirrel in that same tree, directly above me, dropped a walnut on my head.

Instrumented soccer, 6/30

stephen@blueslugs.com (Stephen Hahn) — Sun, 30 Jun 2013 23:07:17 +0000

I managed to get back out on the soccer field, after the two field sessions of the Class E clinic and a weekend in Reno shuttling the boys around. Even though I had worked in some accelerations into my runs, I tweaked my right hamstring at the clinic and so I knew today wasn’t going to be great for sprints. (The hardest part about the clinic was participating in all of the other students’ demonstration drills—about 13 hours of drills.) So, today was about playing smart.

The summary was

miCoach summary, 2013-06-30

The duration suggests a combination of a brief warmup, at least compared to 9 June, and 40 minute halves instead of full 45s. (It’s a 90F weekend.) Since I couldn’t go all out in the sprints, I made sure to cruise a bit more to get into good position, which is reflected in the high intensity distance being about the same as the previous game.

Another useful graph from the miCoach soccer report is the timeline:

miCoach timeline, 2013-06-30

Looks like I sat out for a long substitution in the first half; I’ll have to call for a sub sooner. This week’s goal: further hamstring recovery.

Instrumented soccer

stephen@blueslugs.com (Stephen Hahn) — Fri, 14 Jun 2013 05:10:45 +0000

After talking with one of the dads on Nathaniel’s team, I decided to start playing soccer again, this time in the PAASL Men’s B division. (I played a single season of AYSO Adult League in fall of 2011, but work got a bit too busy. Before that there’s an 18 year hiatus.) For running and lifting, various Android apps—CardioTrainer and Strong Log—have given me enough feedback data to adjust training plans, but for soccer, it’s a bit awkward to carry a phone on your shirt sleeve. Since I didn’t want to play without some form of instrumentation, I began to research.

The only single player system for soccer I’ve found is Adidas miCoach, which consists of a pod that slips into a recess under the insole of your left boot, a receiver, a synchronization program, and a web application which provides the reporting. I chose the Mac/PC package, where the receiver is a USB device, to use with my laptop. Since I hadn’t played in a while, I was curious to see what the summary was. When I got home, I ran the sync app, and logged into the miCoach site. The summary panel from my first game, which includes the warmup, is below:

miCoach summary, 2013-06-09

Even these five numbers were illuminating: distance travelled, at just under 4 miles, was comfortable, given my regular runs. But I don’t do a lot of speed work, and the demonstrations and touches I perform with Ben’s team are low intensity, and the second half—except for one or two runs—was much more of a challenge in terms of summoning speed. (It feels like a brief incantation is required at my age. Twenty years ago speed was always available…) I’ll make sure to add more accelerations when I’m out on the roads, or just suck it up and run some wind sprints.

miCoach gives more detailed results than that summary; I’ll show a few of these in future posts. No measurements this weekend, as I’m attending the Class E coaching clinic.

Blog traffic summary, 2012

stephen@blueslugs.com (Stephen Hahn) — Fri, 11 Jan 2013 22:33:04 +0000

wordpress.com annual report: I turned WordPress.com’s Jetpack on in the fall; this morning I received their generated report for the past few months. #1 this year? The sauce spoon—not a technical post.

node.js for OpenIndiana via SFE

stephen@blueslugs.com (Stephen Hahn) — Sun, 30 Dec 2012 22:17:22 +0000

@tomww points out that the Spec Files Extra (SFE) repository is packaging recent node.js versions, built with their GCC 4.6.3 compilers. That means installation is as simple as a pair of pkg(5) invocations

# pkg set-publisher -p http://pkg.openindiana.org/sfe
# pkg install runtime/javascript/nodejs

The current version is 0.8.16, built less than 24 hours ago. Cool.

Bookmarks from Tue Aug 21 2012 to Thu Dec 27 2012

stephen@blueslugs.com (Stephen Hahn) — Sun, 30 Dec 2012 06:00:34 +0000

These are recent links Stephen has saved on http://pinboard.in.

Building node.js on OpenIndiana | blueslugs.com Building node.js on OpenIndiana
dilshod/xlsx2csv · GitHub “Converts xslx xml excel files to csv format. It is fast, and works for huge xlsx files.” And works well enough to stop me from custom-coding a handler for each spreadsheet I get handed.
CoffeeLint – Lint your CoffeeScript A style checker for CoffeeScript.
Typeset.css – a no-nonsense CSS typography reset for styling user-generated content like blog posts, comments, and forum content “A no-nonsense CSS typography reset for styling user-generated content like blog posts, comments, and forum content.” Reset with good stylistic bias.
Replacing Grub With Syslinux – jasonwryan.com Example of syslinux with arch/pacman.
The Perfected Self – Atlantic Mobile
Generating Poisson random values — The Endeavour “Below is an algorithm that has expected run time independent of the argument λ. The algorithm is fairly simple, though it takes a moderate amount of theoretical justification to explain. It goes back to 1979 and so may not the most efficient algorithm available. It is definitely not the most efficient algorithm if you need to generate a large number of samples using the same parameter λ.”

Building node.js on OpenIndiana

stephen@blueslugs.com (Stephen Hahn) — Fri, 28 Dec 2012 07:58:17 +0000

More specifically, these instructions should let you build node 0.8.16, the current stable version, on oi_151a5:

$ uname -a
SunOS cooler 5.11 oi_151a5 i86pc i386 i86pc Solaris

(cooler is in its seventh year of service, having run many builds of Solaris, OpenSolaris, and, now, OpenIndiana.) First, you’ll need a GCC 4.x compiler. If you attempt to use the 3.4.3 gcc compiler, you’ll get

cc1: error: unrecognized command line option "-fno-tree-vrp"
cc1: error: unrecognized command line option "-fno-tree-sink"

in the output from your failed build. So, use the Illumos GCC 4.4.4 build, which you can install via

$ sudo pkg install developer/illumos-gcc developer/gnu-binutils

which led to the installation on my system, of 3 packages, and a total of 56.9MiB of content downloaded. Include these new tools in your path for the build:

$ export PATH=/opt/gcc/4.4.4/bin:/usr/gnu/bin:$PATH

To help the node build find the appropriate Standard C++ library for linking, we set the linker run path, via the environment. (By having a correct run path, our node binary won’t need LD_LIBRARY_PATH to be set to pick up libstdc++.so.6.) We can then configure, and issue the (GNU) make to start a build:

$ export LD_RUN_PATH=/opt/gcc/4.4.4/lib
$ CC=gcc ./configure --prefix=$HOME
$ CC=gcc gmake

You can test the resulting binary

$ ./node
> process.version;
'v0.8.16'
> ^D

and install the node platform to the configured location.

$ CC=gcc gmake install

And now you have a working node.js for your OpenIndiana system. npm is installed as well, so you can begin downloading the modules needed for your development. (If you’re running OmniOS, it looks like the “managed services” repository includes a pkg(5)-installable node.js package, so you can install that interpreter directly. Maybe that’s what cooler should run next.)

Lunch: Rainbow Pizza, San Mateo

stephen@blueslugs.com (Stephen Hahn) — Tue, 02 Oct 2012 22:21:35 +0000

Gyro, Rainbow Pizza, originally uploaded by schahn.

A morning craving for a gyro sandwich led to a search session. The reviews suggested we try Rainbow Pizza, a Greek-Italian restaurant, tucked away in a valley among the San Mateo hills. Good gyro, physically enormous menus. I think we’ll go back to try the pizza.

Taxi banner

stephen@blueslugs.com (Stephen Hahn) — Tue, 25 Sep 2012 18:16:25 +0000

I haven’t thought about that steaming cup logo lately. (Or Sun Sans…)

post-review returns HTTP 500 with cribbed repository configuration

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 Sep 2012 23:12:40 +0000

We run ReviewBoard at work; we set up each hosted Git repository as new projects are started. I made a silly error a few weeks ago: when I created a new repository, I filled in the Mirror Path setting with that of a previous repository. This mistake leads to client output like

$ post-review 7fdd345e22783b289be128205cc0c47935057e20
Error creating review request: HTTP 500

Your administrator mail address should receive a message with a body like

Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/Django-1.3.3-py2.7.egg/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python2.7/site-packages/Django-1.3.3-py2.7.egg/django/views/decorators/cache.py", line 79, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/Django-1.3.3-py2.7.egg/django/views/decorators/vary.py", line 22, in inner_func
response = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/Djblets-0.6.22-py2.7.egg/djblets/webapi/resources.py", line 397, in __call__
result = view(request, api_format=api_format, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/Djblets-0.6.22-py2.7.egg/djblets/webapi/resources.py", line 581, in post
return self.create(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ReviewBoard-1.6.11-py2.7.egg/reviewboard/webapi/decorators.py", line 127, in _check
return view_func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/Djblets-0.6.22-py2.7.egg/djblets/webapi/decorators.py", line 88, in _checklogin
return view_func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/Djblets-0.6.22-py2.7.egg/djblets/webapi/decorators.py", line 62, in _call
return view_func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/Djblets-0.6.22-py2.7.egg/djblets/webapi/decorators.py", line 231, in _validate
return view_func(*args, **new_kwargs)
File "/usr/lib/python2.7/site-packages/ReviewBoard-1.6.11-py2.7.egg/reviewboard/webapi/resources.py", line 5984, in create
Q(local_site=local_site))
File "/usr/lib/python2.7/site-packages/Django-1.3.3-py2.7.egg/django/db/models/manager.py", line 132, in get
return self.get_query_set().get(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/Django-1.3.3-py2.7.egg/django/db/models/query.py", line 351, in get
% (self.model._meta.object_name, num, kwargs))
MultipleObjectsReturned: get() returned more than one Repository -- it returned 2! Lookup parameters were {}

each time a review is submitted against one of these repositories.

You can fix this condition by correcting the Mirror Path of the new repository; it’s not a ReviewBoard issue.

Bookmarks for June 25th through August 20th

stephen@blueslugs.com (Stephen Hahn) — Mon, 20 Aug 2012 22:01:07 +0000

These are my links for June 25th through August 20th:

How to Build a Plyometric Box – Plan for a pylometric box, rectangular
Crossfit journal – Pylo box with slanted sides – Plan for a slanted side pylometric jumping box.
113 Design Guidelines for Homepage Usability (Nielsen Norman Group) – A laundry list for basic site quality/usability.
Treating JavaScript like a 30 year old language – Jeremy Kahn’s Dev Blog – Some style and tools recommendations for programming JavaScript.
Juice Labs – Chart Chooser – HTML5 tool to select appropriate chart based on (simple) requirements.

Bookmarks for March 10th through May 24th

stephen@blueslugs.com (Stephen Hahn) — Thu, 24 May 2012 17:02:12 +0000

These are my links for March 10th through May 24th:

Google+ Photographer’s Conference | May 22-23, San Francisco – Interesting list of presenters and topics. Some videos are available on the conference’s G+ page.
How P2 Changed Automattic — Matt Mullenweg – P2 is a conversation focused WP blog theme. Simon Phipps is experimenting with P2. Filled a communications gap at WordPress.
Untitled (http://twitter.com/schahn/status/186503261086429186/photo/1) – Never seen ads on corner flags before.
Clone Digger – “Clone Digger aimed to detect similar code in Python and Java programs. The synonyms for the term “similar code” are “clone” and “duplicate code”.”
SwiftKey – smarter typing for everyone – SwiftKey X has saved me 10,000 keystrokes! Check it out at

Lunch: Classic 101 Burgers and Shakes, Belmont

stephen@blueslugs.com (Stephen Hahn) — Sat, 24 Mar 2012 06:40:10 +0000

Mushroom burger, from Classic 101 Burgers and Shakes, originally uploaded by schahn.

Classic 101 Burgers and Shakes [Yelp] makes a reliable, basic burger. It’s one of the few hamburger-serving restaurants from the tour I return to without hesitation. (When I’m not up for a new location, Classic 101, with its generally predictable serving latency, is the current default.) They make an excellent malted milkshake, too.

Lunch: Saruno Burger [Truck], Menlo Park

stephen@blueslugs.com (Stephen Hahn) — Fri, 16 Mar 2012 06:39:58 +0000

Menchi burger, by Saruno Burger, originally uploaded by schahn.

The Saruno Burger truck [Yelp], which is based out of Brisbane, made a stop in our neighborhood, allowing us to try its Japanese interpretation of the hamburger without our usual drive up or down US101. The Menchi burger is a hamburger and onion patty, coated in breadcrumbs, deep fried, and served on a brioche bun. Tasty.

Lunch: Rave Burger, San Mateo

stephen@blueslugs.com (Stephen Hahn) — Thu, 08 Mar 2012 18:44:31 +0000

Greek burger, from Rave Burger, originally uploaded by schahn.

A colleague and I are working through the better burger places on the Peninsula. We’ve been juicily making our way north; Tuesday we hit Rave Burger [Yelp], where I enjoyed the tasty Greek burger shown.

Bookmarks for March 5th through March 7th

stephen@blueslugs.com (Stephen Hahn) — Wed, 07 Mar 2012 18:02:39 +0000

These are my links for March 5th through March 7th:

Colllor – Turn one color into many alternative ones – “Create consistent color schemes: With Colllor it is much easier to generate a consistent color palette with just a few clicks. You should use colors consistently, so you have a common look and feel throughout your design.”
Colorbrewer: Color Advice for Maps – A colour choice advisor, targetted for cartography.
Data Wrangler – “Wrangler is an interactive tool for data cleaning and transformation.”
google-refine – Google Refine, a power tool for working with messy data (formerly Freebase Gridworks) – Google Project Hosting – “Google Refine is a power tool for working with messy data, cleaning it up, transforming it from one format into another, extending it with web services, and linking it to databases like Freebase.”
Induction ⚡ A Polyglot Database Client For Mac OS X – Can talk to (and visually explore on) a variety of SQL/NoSQL datastores.

Bookmarks for January 19th through March 1st

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Mar 2012 07:03:05 +0000

These are my links for January 19th through March 1st:

How to Build a Speech-Jamming Gun – Technology Review – Waiting for the inevitable headphones that auto-respond with “stop copying me!”…
Why I’m ditching the Arduino software platform – Alan’s Ramblings – Alan’s ditching Arduino’s stack and going to the metal; write() is reason enough.
Brendan’s blog » The USE Method: Solaris Performance Checklist – RT @brendangregg: new blog post: the USE Method: Solaris performance checklist
brianfrankcooper/YCSB – GitHub – “The goal of the YCSB project is to develop a framework and common set of workloads for evaluating the performance of different “key-value” and “cloud” serving stores.”
Regular Expression Matching with a Trigram Index – “… Code Search is no longer online. To mark the occasion, I thought it would be appropriate to write a little about how Code Search worked. The actual Code Search was built on top of Google’s world-class document indexing and retrieval tools; this article is accompanied by an implementation that works well enough to index and search large code bases on a single computer.”
The Trello Tech Stack – Fog Creek Blog – “We have consistently opted for promising (and often troublesome) new technologies that would deliver an awesome experience over more mature alternatives. We’re about a year in, and it’s been a lot of fun.”

Bookmarks for October 18th through December 20th

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 Dec 2011 23:01:43 +0000

These are my links for October 18th through December 20th:

Brendan’s blog » Visualizing Device Utilization – Brendan’s overview of visualizing utilization is a must read.
You knew the old Mozilla, meet the new Mozilla – david ascher – RT @davidascher: New blog post about my take on Mozilla in 2012:
Untitled (http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-083-ips-basics-523756.html) – I like @glynnfoster’s pkg(5) summary: a good introduction.
(500) http://ur1.ca/6hwco – Made my annual #Wikipedia donation today.
How to Become a Kickass Plumber – with PEX | Mr. Money Mustache – “The technological revolution that made this activity drastically more easy and fun was the replacement of soldered copper pipes with flexible reinforced polyethylene pipes. These are commonly referred to as PEX (short for Poly-Ethylene-Crosslinked)…. PEX is so much better than copper in every way, you should immediately roll your eyes and sing a circus clown song at anyone who tells you they still do their plumbing with copper.”
SwiftKey X Keyboard Free – Android Market – I’ve just personalized @SwiftKey X for Android with my Twitter posts! Get it free at
splinter – “Splinter is an open source tool for testing web applications using Python. It lets you automate browser actions, such as visiting URLs and interacting with their items.”

Bookmarks for October 10th through October 17th

stephen@blueslugs.com (Stephen Hahn) — Mon, 17 Oct 2011 18:01:41 +0000

These are my links for October 10th through October 17th:

validate.js – “Lightweight JavaScript form validation library inspired by CodeIgniter. No dependencies…”
Official Google Blog: A fall sweep – Disappointed at Code Search closing. It’s an odd man out in the mostly focus-on-G+ fall sweep.
Microcaching: Speed your app up 250x with no new code – Fenn’s Thoughts – “Microcaching is like an insulation layer for your app – Let’s say your wordpress install (or rails app) can handle 20 requests/sec fairly happily. This is fine, up until the point where you get on HN and Reddit at the same time (greatest day of your life) and right at the critical time, your site collapses spectacularly amidst the deafening snarky jeers of your peers. The idea behind microcaching is to cap the amount of requests that can make it through to your app by letting nginx bear the brunt of your pageviews by caching content for a very small amounts of time (ie: 1 second or less).” Interesting technique; example nginx configuration given and explained.
Android-x86 – Porting Android to x86 – “This is a project to port Android open source project to x86 platform…” Works in VirtualBox, at least.
A List Apart: Articles: CSS Positioning 101 – A very clear introduction to CSS positioning.

Bookmarks for September 7th through October 5th

stephen@blueslugs.com (Stephen Hahn) — Thu, 06 Oct 2011 21:02:35 +0000

These are my links for September 7th through October 5th:

Untitled (http://www.slideshare.net/kohsuke/jenkins-user-conference-2011) – RT @kohsukekawa: My #jenkinsconf key note slides posted at
Historians Politely Remind Nation To Check What’s Happened In Past Before Making Any Big Decisions | The Onion – America’s Finest News Source – RT @TheOnion: Historians Politely Remind Nation To Check What’s Happened In Past Before Making Any Big Decisions
Primarily Pandoc: Writing in Markdown instead of LaTeX « PhilTeX – Nice article on using pandoc for composition, rather than LaTeX. Includes example ePUB.
Simple Quickcheck implementation for Python – Also links to a couple of other QuickCheck Python implementations. (Found based on bos’s Strange Loop talk.)
deck.js » Modern HTML Presentations – HTML/JS/CSS for in-browser presentations.

Bespoke services: site/redis

stephen@blueslugs.com (Stephen Hahn) — Fri, 30 Sep 2011 06:35:53 +0000

For prototyping web applications, I have recently come to rely on having Redis handy. In various sketches or early versions, I’ve used it to store event logs, to persist a collection of simple objects, or to conveniently manage a particularly large dictionary.

To make it easy to have a redis-server running on an OpenSolaris-derived system, I’ve written an smf(5) service manifest:

redis.xml

The default configuration of Redis is good enough for most prototyping scenarios, so this manifest assumes (a) that you’ve built and installed Redis to /usr/local, its default install location, and (b) are happy with the default configuration. In its default configuration, redis-server does not daemonize, and writes a log message every 5 seconds—you’ll very much want to change the latter if you move to production.

Exercises

Add a property group and property to store a configuration location, and modify the start method appropriately. This enhancement should be on the service, such that it can be easily overridden on each instance. (*)

Bookmarks for July 21st through August 13th

stephen@blueslugs.com (Stephen Hahn) — Mon, 22 Aug 2011 00:00:07 +0000

These are my links for July 21st through August 13th:

Requests: HTTP for Humans — Requests v0.5.1 documentation – “Requests is an ISC Licensed HTTP library, written in Python, for human beings. Most existing Python modules for sending HTTP requests are extremely verbose and cumbersome. Python’s builtin urllib2 module provides most of the HTTP capabilities you should need, but the api is thoroughly broken. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks. Things shouldn’t be this way. Not in Python.”
Requests: HTTP for Humans — Requests v0.5.1 documentation – “Requests is an ISC Licensed HTTP library, written in Python, for human beings. Most existing Python modules for sending HTTP requests are extremely verbose and cumbersome. Python’s builtin urllib2 module provides most of the HTTP capabilities you should need, but the api is thoroughly broken. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks. Things shouldn’t be this way. Not in Python.”
assaf/zombie – GitHub – “Zombie.js is a lightweight framework for testing client-side JavaScript code in a simulated environment. No browser required.”
aptosid Manuals – Upgrade BIOS FreeDOS – “You may want, or have a need, to update the BIOS of your PC, when the manufacturer of the motherboard announces some improvement of BIOS software. The installer program usually offered is an application to run MS-DOS. This is a way to update BIOS from a USB in linux. This will work with USB keys, USB sticks and with micro/mini/SD cards (with a suitable adapter).”
head.js – “Load scripts like images. Use HTML5 and CSS3 safely. Target CSS for different screens, paths, states and browsers. Make it the only script in your HEAD. A concise solution to universal issues.” Multi-function JS lib, with parallel loading.

Bookmarks for June 22nd through July 19th

stephen@blueslugs.com (Stephen Hahn) — Thu, 21 Jul 2011 07:00:00 +0000

These are my links for June 22nd through July 19th:

CoffeeTable – Code – Alec Perkins – A drop-in workbench for experimentation, CoffeeTable provides a CoffeeScript-fluent console on a page, with persistent history and auto-suggest.
Waterloo – Jobs – Google – Google opened an office a mile from my old high school. That neighborhood was all closed shoe and tire factories.
The Observation Deck » In defense of intrapreneurialism – RT @bcantrill: In defense of intrapreneurialism: And curse you @trevoro for having taunted me with that red cape!
matt blags – hivemind devops alert: nginx sucks at ssl – Benchmark of SSL handling by nginx, stunnel, stud, and the author’s patched stud.
smoke.js – “A framework-agnostic styled alert system for javascript. Lightweight, flexible, css3 animation, blah blah blah easy to use…”
Eli Bendersky’s website » Blog Archive » Parsing C++ in Python with Clang – RT @tuan_kuranes_rs: Parsing C++ in Python with Clang:
Lunch: The Patty Shack, Redwood City, CA | blueslugs.com – Bacon-wrapped hot dogs arrive in #redwoodcity.
Jonas Galvez: HCSS – hcss is a CSS compiler that that allows you to use HTML element hierarchy to define CSS rules. hcss employs simple conventions for defining nested rules and minimalist class inheritance.
Adam Leventhal’s blog » Flash news I wish I could read – RT @ahl: new blog post » Flash news I wish I could read
Traffic Light Protocol – Wikipedia, the free encyclopedia – “The Traffic Light Protocol (TLP) was created[1][2] to encourage greater sharing of sensitive information. The originator signals how widely they want their information to be circulated beyond the immediate recipient.” A simple document classification scheme.
How to take advantage of Redis just adding it to your stack – “Redis is different than other database solutions in many ways: it uses memory as main storage support and disk only for persistence, the data model is pretty unique, it is single threaded and so forth. I think that another big difference is that in order to take advantage of Redis in your production environment you don’t need to switch to Redis. You can just use it in order to do new things that were not possible before, or in order to fix old problems.” Redis’s author illustrates some problems Redis makes easy.
Gephi, an open source graph visualization and manipulation software – “Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs.” Java-based, uses OpenGL, cross-platform.
Dive Into Dojo GFX | Facebook – SitePen’s note on dojox.gfx has slightly more detailed examples than the base documentation.

Menu: grilled salmon, anise, tomato, scallion, plus broiled goat cheese salad

stephen@blueslugs.com (Stephen Hahn) — Sun, 17 Jul 2011 20:45:19 +0000

Cooked because we had new propane, originally uploaded by schahn.

Dina took the boys out for the afternoon, so I decided to fire up the grill for dinner. The recipes are from Carroll (1999) and Brennan (2002). (The salad is not pictured; I didn’t broil the goat cheese pucks long enough, so plating was a challenge.)

Vinum California Rose for preparation and dinner.

References

Brennan, Georgeanne. Salad: William Sonoma Collection. (Free Press, 2002).

Carroll, John Phillip. Grilling. Williams-Sonoma Kitchen Library. (Time Life Medical, 1992).

Irritating retrograde

stephen@blueslugs.com (Stephen Hahn) — Wed, 06 Jul 2011 23:28:39 +0000

A government agency I interact with has updated their web-based client software. The original application was a basic sequence of web forms. Its replacement? An approximately ~50MiB Silverlight-based application. In the process of the update, they discarded my original web account and password. The backend service that the application must communicate with is still slow, operating costs now include the bandwidth to update cached copies (for performance reasons), and the application itself has new usability issues. Because of the switch from standardized Web technologies to Silverlight, the majority of their customers can’t run the application on their phone or tablet. (If it were Flash, iPads would still be excluded.) How was this change an upgrade, again?

Menu: Gazpacho, Grilled Steak with Chimichurri, Taco with Potato-Egg-Chevre-Salse Verde

stephen@blueslugs.com (Stephen Hahn) — Tue, 05 Jul 2011 06:17:02 +0000

Cooked for Independence Day 2011, originally uploaded by schahn.

We served a light meal for Fourth of July this year–we had only a single dinner guest, and Benjamin is still at camp. The gazpacho and steak recipes were based on those in this month’s Cook’s Illustrated, while the taco filling comes from Bayless’s Authentic Mexican. (The filling is becoming a regular offering at our house.)

Red Stripe and Pacifico served during the preparation phase; a Malbec with dinner. Sorbet for dessert.

Lunch: The Patty Shack, Redwood City, CA

stephen@blueslugs.com (Stephen Hahn) — Sat, 02 Jul 2011 20:34:12 +0000

Nathaniel and I, after reading most of the menus on Broadway in search of a hot dog, stopped at The Patty Shack [Yelp] on Main St. Having enjoyed Tijuana Dogs at Fremont’s now-defunct Juan More Taco, I was pretty excited to have a local source for bacon-wrapped hot dogs. Nathaniel enjoyed his corn dog; he’s torn between the Shack’s better product versus Ben Franks’s location beside the train tracks at Whipple.

Bookmarks for May 24th through June 18th

stephen@blueslugs.com (Stephen Hahn) — Sun, 19 Jun 2011 03:00:21 +0000

These are my links for May 24th through June 18th:

Twitpic – Share photos and videos on Twitter – Bike friendliness: free air by rack
Asciiflow – ASCII Flow Diagram Tool – Browser-based drawing tool for creating block comment-friendly diagrams.
Language Log » Translated phrase-list jokes – “The recently-posted Anglo-EU Translation Guide shares 3 phrases with the 2004 Economist column (some expressive details aside), lacking 2 others and adding 12 more. So a combined Anglospeak phrase book would have 17 entries, as compiled below…”
Android.com – Android File Transfer is an application for Macintosh computers (running Mac OS X 10.5 or later) you can use to view and transfer files between your Mac and an Android device (running Android 3.0 or later).
Saab Zero Emissions fleet under construction | Inside Saab by Steven Wade – Prototypes being assembled; possibly our next car. #saab

Never answer surveys!

stephen@blueslugs.com (Stephen Hahn) — Sat, 21 May 2011 08:42:23 +0000

The number of sites asking me to take a brief survey has exploded in the past six months. I’m also seeing email and paper mail requests to participate in surveys creep up too. If this increase is an indicator of an improving economy—as we interpret the worsening traffic on US101—that’s great, but it doesn’t affect my goal, which is to convince you that you should never answer surveys.

There are three reasons I’m trying to persuade you to avoid surveys. First, after almost a year at a startup, I’m very sensitive to time wasting activities. Second, I don’t believe survey originators are valuing that time fairly. Finally, since the survey terms rarely limit use of the submitted information, I want you to recognize that, in addition to being undercompensated for the information you shared, you have no control over how or for what end the information will be used.

We’ll visit each of these in turn, but before that, let me identify two exceptions where you might choose to submit a survey. Much of my arguments are going to depend on viewing one or both of the time and information that the survey involves as having some value. So, if the survey giver has no ability to compensate you for your time, you might choose to view your completion of the survey as a donation. That would allow you to return surveys to a selection of charities. Similarly, if you support the survey giving organization’s goals—like a Swedish car manufacturer with a fine product, but struggling in the market—you could answer their surveys in the hope that the value of the information you shared incrementally helps them to reach their goals. (Acting on a survey request being substantially less investment than purchasing a car.) A shareholder of a company that is surveyed by that same company might choose to answer in the hopes of seeing their investment succeed.

If the survey organization doesn’t fit into those categories, it’s very likely that you shouldn’t respond. Let’s see why.

“I recommend to you to take care of minutes: for hours will take care of themselves.” – Earl of Chesterfield

The first concern is that surveys require time, and return no information. Just the request, as a rejected dialogue on a web page or an email to be deleted, has already cost a minute. Typical surveys run from 3 – 5 minutes to 30 minutes or more, plus that initial minute of interruption. What do we receive for that minute? Presumably the survey giver now knows more about us, and can adjust their products or pricing to be more appealing to those people who answered the survey identically. We receive nothing—not even information shared by other respondents, so that we might get some comparative demographics about our fellow customers.

I think we would all get sensitized to a stream of requests for work (the surveys) for which we got nothing back. Eventually, even the most accommodating personality would just refuse. They might even slam down the telephone, or press the delete key really hard. That’s why surveys often offer to enter submitters into a draw for one or more prizes; this tactic is meant to suggest that you are being compensated for your effort.

Before we analyze a specific survey request, let’s establish a useful baseline for comparison. Minimum wage in California since 2008 is $8/hr, which corresponds to $16,000 annually. The survey requests all claim to need only minutes of our time, so a more natural unit is a minute of compensation. For the California, the minimum per-minute wage is $0.13.

I just received a survey offer today. It’s from a for-profit company, whose goals aren’t of particular interest to me. That means it’s a convenient example to evaluate. The survey claims to require 3 minutes for a chance at $2500. (We’ll omit the minute we lost reading the offer; it makes the numbers worse.) To determine whether or not the survey’s compensation is worthwhile, we will compare the minimum wage for 3 minutes:

$$ 3\thinspace\rm{minutes} \times $0.13\thinspace/\thinspace\rm{minutes} = $0.39 $$

against the expectation value of the lottery draw. The expectation value, written ⟨value⟩, is defined as the sum over all of the outcome values multiplied by the probability that each particular outcome occurs. Since, when we lose, the outcome value is 0, the only contributing term is when we win:

$$ \langle \rm{value} \rangle = \rm{value} \times P [\rm{we\ win\ lottery}] $$

The value of the prize is $2500. To determine the probability, we assume each respondent has an equal chance to win. Then we are left to estimate the likely number of respondents to the survey. If each respondent has an equal chance, then the probability is

$$ P[\rm{we\ win\ lottery}] = {1 \over \rm{number\ of\ respondents}} $$

The company in question, after a brief search, had 3 – 4 million customers in a recent report. If we assume they sampled 1% of their customers, that’s a pool of 30,000 – 40,000 potential respondents. If 1000 people respond, our expectation value is $2.50; if 10,000 people respond, our expectation value is $0.25, less than our minimal compensation. (The expectation value is equal when 6,410 people respond.)

So, for a reasonable assumption about number of respondents, we see that the expectation value of the prize is in the vicinity of the minimum wage for the time contributed. If our example company sampled a larger set, then it’s very likely the expectation value will be less than minimum wage; if you value your time substantially higher than minimum wage, the expectation value of the prize will not be compensating. Thus, in most cases, when you submitted your survey, you received neither useful information nor compensating value.

(Before moving on, we should flip our perspective: how did the surveying company value the information submitted and the impact on customer time? There’s the amount of the prize. We could divide that amount by the number of respondents, and see the valuation above; we might divide by the number of invitations, which is substantially lower. If data acquisition costs only a few cents per customer, is it likely that the data will be valued more highly in a strategic discussion?)

The last reason to avoid submitting surveys is that you are sharing information about yourself, or your household, with an organization into which you likely have limited insight. Beyond whether or not the data you submitted in your response has a meaningful impact on the company’s decisions, there are questions about the use of that data after its initial analysis. Some surveys say they won’t share your email or contact information, or explicitly ask for permissions around contact; in the bulk of these, though, no limitations are placed on the use of the submitted data. That’s worrying, but abstract.

The more concrete concern is that your information will be used by the company for goals other than the product improvement ones we’ve assumed above. Your information might be used to assess your ability to contribute to a political campaign, or to determine the compensation rates of your employer. Your disclosure of a preference for Scandinavian automobiles to your bank might be eventually shared with a domestic or foreign car manufacturer, who would rather you didn’t hold such a preference. Since there is no agreement on how the information can be used, you should expect that some of the survey data you submitted has been shared, or sold, and has circulated wildly. Why continue to refresh it?

Responding to surveys is a flawed activity in three ways: you receive no information in return for the information you submitted, you receive inadequate compensation for the time taken, and you have no assurance or control over the use of the information over any period of time. I suppose, as we look at those three concerns, that an idealized survey can be described as

a survey that includes a simple summary of the responses to each question, returned within a reasonable time following the survey period,
a survey with guaranteed payment in some useful form, in an amount that reflects the value of the responder’s time and the giver’s use of the information, and
a survey that covers the use and retention of the submitted information with a actual, easily understood license.

If you receive a survey with those characteristics, I suppose you might consider submitting it. Otherwise, you should stop.

Bookmarks for May 9th through May 20th

stephen@blueslugs.com (Stephen Hahn) — Fri, 20 May 2011 16:00:47 +0000

These are my links for May 9th through May 20th:

Cool, but obscure unix tools :: KKovacs – “Just a list of 20 (now 28) little-known tools for the command line.” Nice presentation; a few are new to me.
[1105.1383] Topological Considerations for Tuning and Fingering Stringed Instruments – We present a formal language for assigning pitches to strings for fingered multi-string instruments, particularly the six-string guitar. Given the instrument’s tuning (the strings’ open pitches) and the compass of the fingers of the hand stopping the strings, the formalism yields a framework for simultaneously optimizing three things: the mapping of pitches to strings, the choice of instrument tuning, and the key of the composition. Final optimization relies on heuristics idiomatic to the tuning, the particular musical style, and the performer’s proficiency.
Tweetable Sparkblocks – The Data Collective – Form to generate a sparkline bar graph that’s capable of being Tweeted (and presumable pasted many places).
Blogofile – Blogofile is a static website compiler, primarily (though not exclusively) designed to be a simple blogging engine. It requires no database and no special hosting environment. You customize a set of templates with Mako, create posts in a markup language of your choice (see Post Content) and Blogofile renders your entire website as static HTML and Atom/RSS feeds which you can then upload to any old web server you like.
Twitpic – Share photos and videos on Twitter – Why I am changing a flat tire. #nail

Bookmarks for April 25th through May 5th

stephen@blueslugs.com (Stephen Hahn) — Thu, 05 May 2011 19:02:43 +0000

These are my links for April 25th through May 5th:

Innovative Techniques To Simplify Sign-Ups and Log-Ins – Smashing Magazine – Good survey of best practices around login/signup for websites.
Eureqa | Cornell Creative Machines Lab – Eureqa (pronounced “eureka”) is a software tool for detecting equations and hidden mathematical relationships in your data. Its goal is to identify the simplest mathematical formulas which could describe the underlying mechanisms that produced the data.
jonashaag/bjoern – GitHub – A screamingly fast, ultra-lightweight asynchronous WSGI server for CPython, written in C using Marc Lehmann’s high performance libev event loop and Ryan Dahl’s http-parser_.
NodeBox | Graph – “The NodeBox Graph library includes algorithms from NetworkX for betweenness centrality and eigenvector centrality, Connelly Barnes’ implementation of Dijksta shortest paths (here) and the spring layout for JavaScript by Aslak Hellesoy and Dave Hoover (here). The goal of this library is visualization of small graphs (<200 elements)…”
Cyberwarriors on the Eastern Front: In the line of fire packet floods [printer-friendly] • The Register – Interview on 2007 cyberattack with Estonia MoD undersec’y: “… banning the use of cyberweapons is not realistic.”

Lunch: Zero Zero, San Francisco, CA

stephen@blueslugs.com (Stephen Hahn) — Fri, 29 Apr 2011 22:57:50 +0000

Pizza stand at Zero Zero, originally uploaded by schahn.

Our considerate server brought us tastes of two distinct wines to assist our selection. Note that Zero Zero doesn’t open for lunch until 12 p.m. If you arrive before your tablemates, LuLu next door has a bar; in spring, their special cocktails use fresh basil.

Lunch: Birch Street, Palo Alto, CA

stephen@blueslugs.com (Stephen Hahn) — Fri, 22 Apr 2011 23:33:09 +0000

Had a nice lunch off California Ave today. [Yelp: Birch St]

First 'quats

stephen@blueslugs.com (Stephen Hahn) — Sun, 09 Jan 2011 21:30:25 +0000

Kumquats, originally uploaded by schahn.

When Dina and I moved to California, the first plant we bought was a dwarf kumquat. We planted it in an oak half-barrel, which, as an alleged byproduct of the wine industry, was another novelty. It survived years on our apartment’s porch, the move to our small house, and years more on the driveway. I couldn’t, however, resuscitate it from a summer of neglect caused by a renovation project.

So, at the end of the past summer, we bought and planted a new dwarf kumquat. And waited.

This morning, Benjamin and I went for a walk downtown. (7.36 km total, according to the phone.) On our way out, we noticed these two fruits on a limb of the tree. It’s a small first crop, but it won’t be long before I’m making kumquat-lemon marmalade again.

Bookmarks for 29 October through 4 November

stephen@blueslugs.com (Stephen Hahn) — Fri, 05 Nov 2010 04:00:35 +0000

Fuzzy Mathematics with FuzzPy (Part 1) | Mad Python – Python module to work with fuzzy numbers, sets, and graphs. Integrated with GraphViz for visualization.
HTTPS Everywhere | Electronic Frontier Foundation – “HTTPS Everywhere is a Firefox extension …. It encrypts your communications with a number of major websites.”

Jack-o'-lanterns, 2010

stephen@blueslugs.com (Stephen Hahn) — Mon, 01 Nov 2010 01:57:12 +0000

In which Dina patiently uses our pumpkin carving saws on two large specimens.

Bookmarks for 13 October through 23 October

stephen@blueslugs.com (Stephen Hahn) — Sun, 24 Oct 2010 04:00:18 +0000

Bujagali: Incredibly Fast JavaScript Templating – Interesting approach to templating: render on either client or server as part of your application.
py-multicast – Python IP multicast library – “py-multicast provides an easy to use interface to receive multicast streams”.

Bookmarks for 1 October through 7 October

stephen@blueslugs.com (Stephen Hahn) — Thu, 07 Oct 2010 17:07:54 +0000

pycco – A port of docco for Python: produces very attractive literate programming-like HTML output, using Markdown and Pygments.
Software Evolution Storylines – From Michael Ogawa, developer of code swarms, a more linear visualization of code integrations and correlations.
[1009.5718] Monitoring wild animal communities with arrays of motion sensitive camera traps – “Motion sensitive camera traps offer a visual sensor to record the presence of a broad range of species providing location -specific information on movement and behavior. Modern digital camera traps that record video present new analytical opportunities, but also new data management challenges. This paper describes our experience with a terrestrial animal monitoring system at Barro Colorado Island, Panama.” Includes a short discussion of camera failure cases, effective lifetimes, and initial calibration.
[1009.0626] Approximate results for a generalized secretary problem – “A version of the classical secretary problem is studied, in which one is interested in selecting one of the b best out of a group of n differently ranked persons who are presented one by one in a random order. It is assumed that b is a preassigned (natural) number. It is known, already for a long time, that for the optimal policy one needs to compute b position thresholds, for instance via backwards induction. In this paper we study approximate policies, that use just a single or a double position threshold, albeit in conjunction with a level rank. We give exact and asymptotic (as n tends to infinity) results, which show that the double-level policy is an extremely accurate approximation.”
PLoS ONE: Iridovirus and Microsporidian Linked to Honey Bee Colony Decline – “We used Mass spectrometry-based proteomics (MSP) to identify and quantify thousands of proteins from healthy and collapsing bee colonies. MSP revealed two unreported RNA viruses in North American honey bees, Varroa destructor-1 virus and Kakugo virus, and identified an invertebrate iridescent virus (IIV) (Iridoviridae) associated with CCD colonies. Prevalence of IIV significantly discriminated among strong, failing, and collapsed colonies. In addition, bees in failing colonies contained not only IIV, but also Nosema.”

Bookmarks for 14 September through 28 September

stephen@blueslugs.com (Stephen Hahn) — Tue, 28 Sep 2010 21:00:37 +0000

Typographic Marks Unknown – Retinart – A brief history of the pilcrow, ampersand, section sign, asterisk, and hedera, with comments on usage.
H&FJ News – Pilcrow & Capitulum – A brief history of the pilcrow–that “reversed ‘P’ for paragraph” character–with some samples.
MeetBot – Debian Wiki – “MeetBot is designed to assist in running meetings, taking notes, and so on. It is in pure python, as a plugin to supybot.”
Zint Open Source Barcode Generator – “Zint is an Open Source barcode encoding and image generating library written entirely in C and released under GPLv3 or later.” In particular, offers QR code output in SVG format.

Bookmarks for 7 September through 14 September

stephen@blueslugs.com (Stephen Hahn) — Tue, 14 Sep 2010 10:00:52 +0000

Zint Open Source Barcode Generator – “Zint is an Open Source barcode encoding and image generating library written entirely in C and released under GPLv3 or later.” In particular, offers QR code output in SVG format.
[1009.0550] Optimizing Selective Search in Chess – Fun.
[0907.3135] Fast Searching in Packed Strings – “On a standard unit-cost word-RAM with logarithmic word size we present an algorithm using time $$ O\left(\frac{n}{\log_\sigma n} + m + \occ\right). $$ Here $\occ$ is the number of occurrences of $P$ in $Q$. For $m = o(n)$ this improves the $O(n)$ bound of the Knuth-Morris-Pratt algorithm.”
ngIRCd: Next Generation IRC Daemon – “ngIRCd is a free open source server for Internet Relay Chat (IRC), which is being developed under the GNU General Public License (GPL). It has been written from scratch…”
[1008.1770] A complex network approach to robustness and vulnerability of spatially organized water distribution networks – Examines water distribution networks using network and graph theory techinques.

Bookmarks for 3 August through 7 September

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Sep 2010 22:00:23 +0000

ngIRCd: Next Generation IRC Daemon – “ngIRCd is a free open source server for Internet Relay Chat (IRC), which is being developed under the GNU General Public License (GPL). It has been written from scratch…”
[1008.1770] A complex network approach to robustness and vulnerability of spatially organized water distribution networks – Examines water distribution networks using network and graph theory techinques.
[1008.2345] Trident, a new pseudo random number generator based on coupled chaotic maps – Generates random numbers using chaotic mappings. Passes NIST and other random number quality tests.
Less Framework 2 – Lightweight HTML5 friendly layout CSS.
The KLEE Symbolic Virtual Machine – Can explore state space of C programs and auto-generate appropriate test cases to maximize coverage.

Bespoke services: site/supervisord

stephen@blueslugs.com (Stephen Hahn) — Sat, 04 Sep 2010 23:47:24 +0000

Recently, I’ve been experimenting with supervisor, which is a Python-based process restarter for Unix/Linux. Lincoln Loop recently offered instructions on running supervisor under upstart, which is applicable to some of the current Linux distributions. On OpenSolaris and related systems, the service management facility, smf(5), can be used to ensure your supervisors stay online. Below is a simple manifest that starts (and restarts) supervisord after a small set of services becomes available.

supervisord.xml

If you don’t provide a supervisor.conf in one of the standard locations, enabling this service instance will result in it heading immediately to the maintenance state, as the start method will fail repeatedly. You can use svcs -x to perform this diagnosis:

$ svcs -x ~
svc:/site/supervisord:default (supervisor process control system)
State: maintenance since Sat Sep 04 16:34:14 2010
Reason: Start method failed repeatedly, last exited with status 2.
See: http://sun.com/msg/SMF-8000-KS
See: utmpd(1M)
See: utmpx(4)
See: /var/svc/log/site-supervisord:default.log
Impact: This service is not running.

The log file will contain a message, with some amount of repetition, like

[ Sep 4 16:34:13 Enabled. ]
[ Sep 4 16:34:13 Rereading configuration. ]
[ Sep 4 16:34:13 Executing start method ("/usr/bin/supervisord"). ]
Error: No config file found at default paths (/usr/etc/supervisord.conf, /usr/supervisord.conf, supervisord.conf, etc/supervisord.conf, /etc/supervisord.conf); use the -c option to specify a config file at a different path
For help, use /usr/bin/supervisord -h
[ Sep 4 16:34:13 Method "start" exited with status 2. ]

It’s worth noting that all of the programs run by a single instance of supervisord will be in the same process contract. If you know the fault characteristics of your programs, you may wish to use multiple instances of supervisord to keep programs with “sympathetic” failure modes and frequencies. You may also need to ignore core dumps and external signals, depending on the programs you are running; on recent systems, you can see /var/svc/manifest/network/http-apache22.xml for an example of a startd property group that does so. Alternatively, you could modify your configuration to run each of the programs to be started in independent contracts using ctrun(1).

Exercises

We should really provide a property group that contains the key invocation settings as properties. I’ve omitted it here, particularly for the configuration file, because the method token expansion outlined in smf_method(5) lacks handling for unset property values. (*)
Extend supervisord to understand process contracts. This exercise would include constructing a Python module to interact with the contract filesystem. (***)

Bookmarks for 1 July through 19 July

stephen@blueslugs.com (Stephen Hahn) — Mon, 19 Jul 2010 19:01:14 +0000

3 shell scripts: Kill weasel words, avoid the passive, eliminate duplicates – “So, I’ve decided to replace myself with a shell script. In particular, I’ve created shell scripts for catching three problems: the passive voice, weasel words, and
lexical illusions.”
[0811.0113] A Bayesian Framework for Opinion Updates – “Opinion Dynamics lacks a theoretical basis. In this article, I propose to use a decision-theoretic framework, based on the updating of subjective probabilities, as that basis. We will see we get a basic tool for a better understanding of the interaction between the agents in Opinion Dynamics problems and for creating new models.”
TexoTela jQuery – newsticker – A fade-in/fade-out newsticker for jQuery.
jQuery liScroll – a jQuery News Ticker – “liScroll is a jQuery plugin that transforms any given unordered list into a scrolling News Ticker.”
Taguchi methods – Wikipedia, the free encyclopedia – “Taguchi methods are statistical methods developed by Genichi Taguchi to improve the quality of manufactured goods, and more recently also applied to, engineering, biotechnology, marketing and advertising.”

Bookmarks for 8 June through 29 June

stephen@blueslugs.com (Stephen Hahn) — Wed, 30 Jun 2010 04:02:31 +0000

[1006.5169] Hyperbolic Geometry of Complex Networks – Interesting.
Learning about Network Theory – A brief introduction, with links to resources.
NetworkX v1.1 – “NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.”
[1006.1786] Measuring Meaning on the World-Wide Web – A brief paper exploring word-word correlations on the Web, with some interesting specific examples.
[1006.1231] On the Insertion Time of Cuckoo Hashing – “Cuckoo hashing is an efficient technique for creating large hash tables with high space utilization and guaranteed constant access times. There, each item can be placed in a location given by any one out of k different hash functions.”

adirent.[ch]: Adding d_type to struct dirent on OpenSolaris

stephen@blueslugs.com (Stephen Hahn) — Mon, 14 Jun 2010 02:47:47 +0000

An occasional porting problem you may encounter when compiling programs for OpenSolaris is the absence of d_type in the directory entry structure returned by readdir(3C). I hit this issue when experimenting with mu as a search solution for my accumulated email.

A trivial example of the failure you might see would be caused by the following program:

#include <sys/types.h>
#include <dirent.h>
#include <err.h>
#include <stdio.h>
int
main(int argc, char *argv[])
{
DIR *d;
struct dirent *e;
if ((d = opendir("/")) == NULL)
err(1, "opendir failed");
for (e = readdir(d); e != NULL; e = readdir(d)) {
if (e->d_type != DT_UNKNOWN)
(void) printf("recognized filetype for '%s'\n",
e->d_name);
}
(void) closedir(d);
return (0);
}

When we attempt to compile this program with gcc, we get something like

$ gcc a.c
a.c: In function `main':
a.c:16: error: structure has no member named `d_type'
a.c:16: error: `DT_UNKNOWN' undeclared (first use in this function)
a.c:16: error: (Each undeclared identifier is reported only once
a.c:16: error: for each function it appears in.)

Studio cc will give similar output:

$ /opt/SunStudioExpress/bin/cc a.c
"a.c", line 16: undefined struct/union member: d_type
"a.c", line 16: undefined symbol: DT_UNKNOWN
cc: acomp failed for a.c

The addition of d_type to struct dirent came first for the BSD Unixes and was later added to Linux. Because it’s not easy to add members to well-known structures and preserve binary compatibility, OpenSolaris and Solaris lack this field, as well as the DT_* constant definitions. (If d_type were to become part of the Unix standards, Solaris would likely have to introduce a second family of opendir()/readdir()/closedir() functions and a second version of the structure, similar to how large files were introduced for 32-bit programs.)

Because we fail at compilation time, our workaround has to modify either the program’s source code or its build environment. (Preloading is too late.) It’s probably possible to combine a few definitions and a shared object that we include via LD_PRELOAD but it seems easier to just provide a C wrapper around readdir(3C) and an alternate struct dirent. We develop this approach in the next section.

`DIRENT` and `READDIR`

The approach we take is

Introduce DIRENT and READDIR via adirent.h.
Change the source program such that each call to readdir() is replaced by READDIR() and each use of struct dirent is replaced by DIRENT. In each file so modified, add a #include <adirent.h>.
Compile adirent.c via gcc -I. -O2 -c adirent.c or equivalent.
Add adirent.o to the link line for each binary that includes one of the files modified in step 2.

If we apply these steps to our example above, we get

#include <sys/types.h>
#include <adirent.h>
#include <err.h>
#include <stdio.h>
int
main(int argc, char *argv[])
{
DIR *d;
DIRENT *e;
if ((d = opendir("/")) == NULL)
err(1, "opendir failed");
for (e = READDIR(d); e != NULL; e = READDIR(d)) {
if (e->d_type != DT_UNKNOWN)
(void) printf("recognized filetype for '%s'\n",
e->d_name);
}
(void) closedir(d);
return (0);
}

with the result that compilation and execution now work

$ gcc -O2 -I. -c adirent.c
$ gcc -I. a.c adirent.o
$ ./a.out

This shim function and definitions should be sufficient for most ports around this incompatibility, but there are some additional comments worth making.

Performance. Because many programs expect d_type to be one of DT_REG or DT_DIR to save on a stat(2) call, this shim will force those programs into an alleged “slow” path. The actual impact of returning DT_UNKNOWN on every call will be program- and situation-dependent; it didn’t seem to affect my mail indexing.

Multithreaded programs. The current implementation does not protect the static structure defined in adirent.c. Programs with multiple threads performing readdir(3C) calls through READDIR() will get unexpected results. It should be relatively straightforward to dynamically allocate one struct adirent for each thread coming through READDIR() for the first time.

Downloads

I suppose these should be in a repository on Bitbucket or GitHub. For now, they’re just simple downloads:

Acknowledgments

I discussed this problem with Dan, who in particular noted that DT_UNKNOWN was always a legal return value for d_type. Bart looked over my shoulder and spied at least one error during the debugging phase.

A cruise for World Oceans Day

stephen@blueslugs.com (Stephen Hahn) — Sun, 13 Jun 2010 05:44:15 +0000

We went on the Marine Science Institute‘s cruise today, which was in part to celebrate World Oceans Day 2010. Part of the cruise involved sampling the bay floor and a short trawl with a net. (All animals caught are returned to the bay later in the day.) Ben got to hold a juvenile leopard shark.

Bookmarks for 18 May through 2 June

stephen@blueslugs.com (Stephen Hahn) — Wed, 02 Jun 2010 16:03:57 +0000

[1006.0200] The 1958 Pekeris-Accad-WEIZAC Ground-Breaking Collaboration that computed Ground States of Two-Electron Atoms (and its 2010 Redux) – Koutschan and Zeilberger examine and “re-enact” Pekeris’s 1958 calculation of the ground state of the two-electron atom.
Dictionary of Algorithms and Data Structures – “This is a dictionary of algorithms, algorithmic techniques, data structures, archetypal problems, and related definitions.” Hosted at NIST.
pkg(5) on ohloh.net – pkg(5)’s repository is now mirrored at genunix.org, so we have ohloh’s code and commit analysis. I’ll never catch any of those guys.
Towards Enterprise-Class Compatibility for Apache Hadoop (Hadoop and Distributed Computing at Yahoo!) – The Yahoo! Hadoop team has classified the Hadoop interfaces, using a taxonomy derived from the OpenSolaris ARC interface taxonomy.
Eric Schrock, Debugging AJAX in production – ACM Queue – I missed this article last year, but Eric’s written a nice essay with some example code for improving the debuggability of AJAX applications.

Pipes arrive

stephen@blueslugs.com (Stephen Hahn) — Tue, 18 May 2010 05:08:40 +0000

This morning, the pipe segments began to arrive for the section of Bay Division Pipeline No. 5 nearest us. The pipes are a distinct dark blue-green, with the ends capped with paper.

Bookmarks for 3 May through 13 May

stephen@blueslugs.com (Stephen Hahn) — Thu, 13 May 2010 21:00:21 +0000

Social Page Evaluator – “Our solution is designed to provide an assessment of a brand’s Facebook Page value and the effectiveness of its social media efforts.”
SFWATER.ORG : BDPL Reliability Upgrade – Pipeline Peninsula – Project status page for the BDPL5 Peninsula project.
Water System Improvement Program BDPL5Peninsula – Blog for the Bay Division Pipeline No. 5 Peninsula construction project.
Shoot, edit, and manage photos in Android – PC World – A good intro to the Camera and Gallery applications for Android.
A Professional Photo Workflow for OpenSolaris « Unwinding the Stack – Comments on Argyll, LensFun, and UFRaw, all available on OpenSolaris and all available via pkg(5) from http://pkg.thestaticvoid.com:10000

Large machine, trench work

stephen@blueslugs.com (Stephen Hahn) — Sat, 08 May 2010 19:28:35 +0000

They assembled this machine this week, but haven’t fired it up. The pipeline runs right behind our property, so I expect we’ll hear the crunching of treads and a large motor (or more than one) fire up early in the morning soon.

When you’re digging a miles-long trench, there’s a clear axis of work. And it looks like they’ll be removing the road surface soon.

Reasonable photos from 2009

stephen@blueslugs.com (Stephen Hahn) — Thu, 06 May 2010 05:56:30 +0000

I’ve started uploading a subset of my photos to Flickr. Here are four from 2009 I’m reasonably happy with, with some commentary.

We took a short trip north to the coast in the spring, camping a few nights at the Bodega Dunes site in Sonoma Coast SP. This bird was interested in our activities, although not as aggressive as some distant cousins at San Simeon.

NASA Ames had an event last year, with much model rocketry on the airfield. This model Saturn V was one of the most impressive rockets fired, and I was happy to catch it as it took off.

The dock at the cottage. Weathered, if not gnarly.

I was amused by the modification to this sign. (O.P.P. is short for Ontario Provincial Police; C.P.O. for Community Policing Office; C-3PO for a particular protocol droid.)

Tent interior at Peter Pan 360

stephen@blueslugs.com (Stephen Hahn) — Thu, 06 May 2010 02:02:58 +0000

We went up to San Francisco to see the Peter Pan show at Ferry Park today. The kids were very excited by the human-powered ticking crocodile. There’s a large concession tent for use before the show and at intermission; this shot is of one of its peaks.

El Camino beautification scoop

stephen@blueslugs.com (Stephen Hahn) — Thu, 06 May 2010 01:57:59 +0000

There are a couple of major construction projects in Redwood City. On El Camino Real, the block between Broadway and Brewster is being updated, as part of the multi-county Grand Boulevard project. This excavator was standing sentry on the Broadway end. Closer to our house, a new high-pressure water line–part of the Hetch Hetchy system that supplies San Francisco–is being installed, which requires its own host of excavators of all sizes.

Bookmarks for 17 March through 22 April

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 Apr 2010 17:00:04 +0000

MIT Center for Collective Intelligence – “Our basic research question is: How can people and computers be connected so that—collectively—they act more intelligently than any individuals, groups, or computers have ever done before?'”
Online Reputation Systems: How to Design One That Does What You Need – The Magazine – MIT Sloan Management Review – Specific questions to consider when designing a reputation system.
The stunning pictures of sleeping insects covered in early morning dew | Mail Online – “These remarkable photographs were taken by physiotherapist Miroslaw Swietek at around 3am in the forest next to his home. / Using a torch, the 37-year-old amateur photographer hunts out the motionless bugs in the darkness before setting up his camera and flash just millimetres from them.”
Life on the OutSide: Step-by-Step guide to Install OpenSolaris with Automated Installer (2009.06) – Andrew provides a worked example of installing a customized system using AI.
[0902.2920] A phenomenological model of protein folding – “We construct a phenomenological effective field theory model that describes the universality class of biologically active single-strand proteins. The model allows both for an explicit construction of native state protein conformations, and a dynamical description of protein folding and unfolding processes.”

Bookmarks for 8 February through 4 March

stephen@blueslugs.com (Stephen Hahn) — Thu, 04 Mar 2010 10:01:27 +0000

Home – genetify – GitHub – A/B (vector) testing framework that uses genetic programming to optimize the test outcome.
The Elements of Typographic Style Applied to the Web – a practical guide to web typography – “Robert Bringhurst’s book The Elements of Typographic Style is on many a designer’s bookshelf and is considered to be a classic in the field…. In order to allay some of the myths surrounding typography on the web, I have structured this website to step through Bringhurst’s working principles, explaining how to accomplish each using techniques available in HTML and CSS.”
USGS Multi-Hazards Demonstration Project ARkStorm: West Coast Storm Scenario – “The USGS Multi Hazards Demonstration Project (MHDP) is preparing a new emergency-preparedness scenario, called ARkStorm, to address massive U.S. West Coast storms analogous to those that devastated California in 1861–62. Storms of this magnitude are projected to become more frequent and intense as a result of climate change.” The 1861-1862 storm lasted 45 days and flooded the Sacramento Valley.
Why Menlo Park spoke out against Saltworks project – San Jose Mercury News – Menlo Park’s City Council passed a resolution opposing the 50/50 plan for Saltworks. Interesting fact: Saltworks would be 17 times larger than the last Bay fill project–Oakland International Airport, 1968.
End-to-end… and everything in between: Do a “pkg image-update” with multiple zones!!! – Dan shows how to update multiple zones using pkg(5)’s integrated handling for arbitrarily located images and beadm(1M). Dan’s multiple zone configuration is an example of how to use zones to secure an Internet-facing system.

Bookmarks for 8 January through 25 January

stephen@blueslugs.com (Stephen Hahn) — Mon, 25 Jan 2010 20:00:46 +0000

Sphinx Python Document Generator – “Sphinx is a tool that makes it easy to create intelligent and beautiful documentation…”
stanlemon.net : jgrowl – In-browser notification messages.
TipTip jQuery Plugin | Drewwilson.com – A clever tooltip plugin for jQuery.
Proof that just about anything fits in a cup. – Amusing art
MyWikkaSite: OpenSolaris (List of package repositories) – Bill Hathaway’s tracking pkg(5) package repositories. If you know of more, comment on his wiki.

Bookmarks for 29 November through 2 January

stephen@blueslugs.com (Stephen Hahn) — Wed, 06 Jan 2010 05:53:22 +0000

Create PDF thumbnails using ImageMagick on Linux | Prashanth Ellina – Useful tip if you want to render one or more PDF pages to another format.
Python – Tao of Mac – A commented Python link collection, with a few I’ve not seen before.
How to Go Winter Camping and Backpacking : Expert Advice from REI – Someday, when the kids won’t freeze.
Istanbul (Not Constantinople) – Wikipedia, the free encyclopedia – Didn’t know that the TMBG version was a cover. A gold record for The Four Lads in 1953.
dygraphs JavaScript Visualization Library – Cool interactive JavaScript charting: zoom, error bands, relatively easy to use.
Mark Joshi – Associate professor, actuarial studies, Melbourne U.

Bookmarks for 26 October through 10 November

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Nov 2009 20:01:02 +0000

Minimal Solaris: Installing OpenSolaris on a Sparc using LiveCD – Alexander’s extended his earlier SPARC LiveCD to have a basic installer.
Minimal Solaris: OpenSolaris Sparc snv125 LiveCD – Alexander, creator of MiLaX, has made a SPARC LiveCD, based on Build 125.
SXCE OpenSolaris Migration Sparc – Steven Acres’s notes on a SPARC migration from SXCE to OpenSolaris.
miguelventura / after_the_deadline / wiki / Home — bitbucket.org – A Python wrapper for After the Deadline’s spelling/grammar/style checking API.

Bookmarks for 21 October through 24 October

stephen@blueslugs.com (Stephen Hahn) — Sat, 24 Oct 2009 23:00:04 +0000

What to Read on North Korean Politics | Foreign Affairs – Reading list from Nicholas Eberstadt (AEI and National Bureau of Asian Research).
Start – Bottle Web Framework – “Bottle is a fast and simple WSGI-framework for the Python Programming Language. It offers request dispatching with url parameter support (routes), templates, key/value databases, a build-in HTTP server and adapters for many third party WSGI/HTTP-server and template engines. All in a single file and with no dependencies other than the Python standard library.”
The Blog of Ben Rockwood: Solaris Extended Accounting in the Real World – Ben expands his coverage of Solaris extended accounting. In particular, try out “exacttly”, his new process accounting reporting tool.
Code Style: Font stack builder – A useful web application to build CSS font stacks. As you select fonts, coverage for Windows, MacOS, and Linux is calculated based on recent surveys.
Upgrading from Nevada to OpenSolaris the easy way – Alan’s worked out a simple way to add OpenSolaris to a system running recent Nevada bits. Useful as we approach the upcoming transition.

Bookmarks for 11 October through 20 October

stephen@blueslugs.com (Stephen Hahn) — Tue, 20 Oct 2009 18:00:56 +0000

Bug 6186 – OpenSolaris package renaming – Rich has been working through every package in OpenSolaris, proposing friendlier and guessable names.
OGB 2009/009 ReleaseNotes – Genunix – OGB 2009/009 are the release notes for the proposed OpenSolaris constitution. The OGB’s using co-ment.net for interactive comment handling on the draft as well.
manifold – Project Hosting on Google Code – Manifold, an smf(5) manifest creation assistant, has a project page on Google Gode.
chrismiles: Quickly create SMF service manifests for Solaris using Manifold – Chris has written a Python-based tool that guides one through the creation of a new service manifest for smf(5): answer the questions, get a manifest.
building an ON IPS repository – Liane Praza’s Weblog – Liane comments on her work with Mark to convert the OS/Net consolidation (“ON”) to build image packages natively, rather than through our importer magic.

Bookmarks for 2 October through 5 October

stephen@blueslugs.com (Stephen Hahn) — Mon, 05 Oct 2009 20:01:25 +0000

jQuery: The Write Less, Do More, JavaScript Library – “jQuery is a fast and concise JavaScript Library that simplifies HTML document traversing, event handling, animating, and Ajax interactions for rapid web development.”
An Introduction to Processing.js | Design Intellection – First article in a planned six part tutorial in Processing.js.
Processing 1.0 – “Processing is an open source programming language and environment for people who want to program images, animation, and interactions.” The original language.
Processing.js – “Processing.js is an open programming language for people who want to program images, animation, and interactions for the web without using Flash or Java applets.”
An Engineer’s Guide to Bandwidth (Yahoo! Developer Network Blog) – Useful article, with outbound links, about the underlying behaviors behind “bandwidth”. Example of throttling, packet loss using ipfw on BSD/MacOS X.

Bookmarks for 25 September

stephen@blueslugs.com (Stephen Hahn) — Sat, 26 Sep 2009 09:23:57 +0000

Piwik # Open source web analytics – “Piwik is a downloadable, open source (GPL licensed) web analytics software program. It provides you with detailed reports on your website visitors: the search engines and keywords they used, the language they speak, your popular pages… and so much more.”
Scribe, by Facebook – “Scribe is a server for aggregating log data streamed in real time from a large number of servers. It is designed to be scalable, extensible without client-side modification, and robust to failure of the network or any specific machine.”

2010.next: a better netbook for meetings

stephen@blueslugs.com (Stephen Hahn) — Thu, 24 Sep 2009 23:31:25 +0000

I recently borrowed one of the 10″ netbooks that came out this year, and tested Build 123 on it. Here’s a shot of Toshiba’s sleek black NB205 displaying a clean Device Driver Utility run.

The NB205 has a keyboard on par with the r500 I use regularly—large for a netbook—and has a superior trackpad. Suspend-resume is reliable; compiz runs well; the extra vertical inch means fewer application compromises over last year’s netbook crop. (For instance, rather than rearranging panel to the sides, use of gnome-panel‘s auto-hide feature is enough to let applications run well when maximized.)

If I don’t have to give it back, it’ll be my new “netbook for meetings”.

Netbook styling credit to Dave Powell.

[ T: OpenSolaris netbook Atom Toshiba ]

verexec(1): A simple execute-most-recent-version utility

stephen@blueslugs.com (Stephen Hahn) — Sat, 22 Aug 2009 00:01:14 +0000

We’ve published three of the approximately-six-monthly distribution releases now, and we’re starting to refine pkg(5)’s capabilities to handle automatically an ever larger set of configuration scenarios. (Read: “edge cases”.) Bart implemented actuators in 2008.11, and that let us handle any resource that could be configured or reconfigured via an smf(5) service instance. In future releases, we’ll build out the set of instances that handle standard configuration.

In this post, I’d like to talk about a tool—verexec(1)—to assist with executable version selection. One of the common questions that maintainers of the various language platform packages is “how can I make sure that /usr/bin/[language] points to the latest version of the package?”. Attempts to satisfy this goal have led to some of the largest preremove and postinstall script pairs in historical Solaris.

Largest, and most difficult to get completely correct.

We can make this problem much simpler—and scripting free—if we accept the minimal performance penalty of a runtime-based decision.

Basic function

verexec(1) works in a similar manner to isaexec(1), in that dispatch is based on the target command being hardlinked to verexec(1), which then, based on the name of the target command and additional information, will in turn invoke the correct version of the target command.

Like most pkg(5)-driven modifications, we want to make the identification of the newest version based on the presence or absence of a filesystem object delivered into a predetermined location. For verexec(1), one delivers a symbolic link in /etc/verexec.d/[target], pointing to the appropriate executable. For example, for Perl 5, /usr/bin/perl is delivered as a hardlink to /usr/bin/verexec, in SUNWpl5. [TK new name?] The Perl 5.8.4 package would deliver the symbolic link, /etc/verexec.d/perl/5.8.4, pointing to /usr/perl5/5.8.4/bin/perl. (The version ordering matches that of pkg(5).)

When /usr/bin/perl is invoked via exec(2), verexec(1) would open—subject to various override mechanisms—/etc/verexec.d/perl and search through the links, looking for the maximal version. The link is then read, and that program is executed, with the arguments passed to the original invocation.

This approach handles directly the delivery of multiple versions, via separate packages. If we were delivering two versions, then removal or installation of either link would change the default version. For more than two versions, the default version only changes if the currently highest version is changed. If we deliver no versions, then verexec(1) displays an informative error message.

Refinements and commentary

Overrides. Since, at some sites, verexec(1) may be useful to provide access to a variety of local command versions, we provide a number of environment variable-based override mechanisms. First, we allow identification of an alternative symbolic link directory to /etc/verexec.d via the VEREXEC_ALTDIR variable. Second, we allow per-command interception via the VEREXEC_[command] pattern. Thus, in our Perl example, I could make /usr/bin/perl point to my local Perl installation via:

export VEREXEC_perl=/home/sch/bin/$(uname -p)/perl

instead of the system installation.

Native ISA selection. We could combine the function of isaexec(1) with verexec(1), by allowing links to have the pattern [version]-[bits], as returned by isainfo -n.

Search impact. Using a technique like this impacts the search functionality, in that a search request for /usr/bin/perl in our example, will actually return the package delivering the verexec(1) link, and not an actual Perl interpreter. We can address this somewhat by ensuring that (a) the actual Perl interpreter is delivered in a package with "perl" as a basename, and (b) the link-delivering package is named and documented as a supporting package.

Delivery, while delaying change in default. Since this mechanism is so much simpler than the scripting previously required, it means that we should reexamine the transition to a new version of a significant language platform. In the past, because correct packaging was a significant amount of the work involved in integrating a new version, we’ve generally introduced a new version and made that version the default in a single integration. Since we don’t want to introduce unnecessary risk, we have generally lagged integrating the latest version of these interpreters. With verexec(1) and pkg(5) in place, we might pursue an alternative, of introducing the new version early, and then changing the default in a subsequent, small integration later in the release.

Performance. If we find that the overhead of the additional exec(2)—one for verexec(1) and one for the link target—is too high, there are a number of performance optimizations. One approach would be to use verexec(1) to send a message to a privileged daemon, which would loopback mount the appropriately versioned executable at the verexec(1) location. Any loopback mount-based technique makes a trade-off: more efficient execution for more complex handling of updates and deletions, which would require the daemon (or privileged command or service) to inspect the existing loopback mounts and evaluate them for their continuing correctness.

I’ve a refinement-lacking prototype of verexec(1) available for review; we’re discussing it further at pkg-discuss@opensolaris.org.

[ T: OpenSolaris SFW pkg ]

MPK17: Outcome of a long discussion?

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Jul 2009 18:15:12 +0000

The coffee stations in MPK17 are popular choices for technical discussions. Some go longer than others, and occasionally result in byproducts. I wouldn’t want to hazard a guess about how long a discussion resulted in this byproduct, which popped up sometime at the end of last week:

I wonder if the original discussion ever resolved…

[ T: OpenSolaris MPK17 ]

A few hours at Maker Faire 2009

stephen@blueslugs.com (Stephen Hahn) — Sun, 31 May 2009 04:08:23 +0000

We got to this year’s Maker Faire fairly early, and saw the many cool things people have built. (It was our first Maker Faire, and we underestimated the event’s popularity.)

Benjamin, who’s recently become an enthusiastic chess player, got to face off against the chess playing robot:

Ben attempted a Scholar’s Mate, was stymied, and then had to go on defense; I suspect he’s been winning at school with that one.

Nathaniel and I were pressed against the barriers to see the lightning demonstration:

Cool.

At an event where most exhibitors have made or built something physical, I must comment that a purely virtual or computational exhibit underwhelms.

Statistical Data Mining Tutorials

stephen@blueslugs.com (Stephen Hahn) — Thu, 14 May 2009 16:41:15 +0000

Statistical Data Mining Tutorials

Andrew Moore's "set of tutorials on many aspects of statistical data mining, including the foundations of probability, the foundations of statistical data analysis, and most of the classic machine learning and data mining algorithms."

Swish-e :: Home Page

stephen@blueslugs.com (Stephen Hahn) — Tue, 12 May 2009 17:40:18 +0000

Swish-e :: Home Page

"Simple Web Indexing System for Humans – Enhanced". Comparable to htdig.

Hadoop Sorts a Petabyte in 16.25 Hours and a Terabyte in 62 Seconds

stephen@blueslugs.com (Stephen Hahn) — Tue, 12 May 2009 04:08:14 +0000

Hadoop Sorts a Petabyte in 16.25 Hours and a Terabyte in 62 Seconds

Recent results from Yahoo!'s sorting benchmark run.

Running Hadoop On Ubuntu Linux (Multi-Node Cluster) - Michael G. Noll

stephen@blueslugs.com (Stephen Hahn) — Wed, 06 May 2009 04:31:55 +0000

Running Hadoop On Ubuntu Linux (Multi-Node Cluster) – Michael G. Noll

A useful set of instructions on Hadoop setup and issues.

jspdf - Google Code

stephen@blueslugs.com (Stephen Hahn) — Thu, 23 Apr 2009 03:59:34 +0000

jspdf – Google Code

"jsPDF generates PDF documents using nothing but Javascript." Neat.

2009.06: A netbook for meetings

stephen@blueslugs.com (Stephen Hahn) — Wed, 22 Apr 2009 01:20:16 +0000

It’s been a busy few months, but things are clearing up now.

The past few weeks, I’ve been using an Acer Aspire One for coverage when my trusty Toshiba r500 is recharging. It’s my computer for meetings, I suppose. The Aspire One is a great little netbook, and 2009.06 should recognize every device on the system. Here’s an action shot of this Atom-based system on my file cabinet:

The window shown in the display is the Device Detection Utility—the absence of red bars in that list indicates that every hardware component on the system is known to OpenSolaris. The main limitations of a standard single core, dual thread ~~(but dual-core)~~ Atom netbook like this are vertical screen real estate and memory usage. In the screen shot, I’ve moved the GNOME panels to the left and right sides of the display; I’ve also disabled unneeded services by using svcadm(1M).

If you’ve an Aspire One, or a similar netbook, you might want to give 2009.06 a spin. The usbcopy utility, from the SUNWdistro-const package, can be used to prepare a bootable USB stick to use for the installation.

[ T: OpenSolaris netbook Atom ]

OpenSolaris Live USB Creator (Windows/.NET) ?(PID0.ORG DevZone)?

stephen@blueslugs.com (Stephen Hahn) — Fri, 10 Apr 2009 18:06:15 +0000

OpenSolaris Live USB Creator (Windows/.NET) ?(PID0.ORG DevZone)?

Hiroshi's made a Windows .NET utility that lets you burn the OpenSolaris USB image to a stick without needing to run OpenSolaris.

Simplicity is the ultimate sophistication.: Michal Pryc's Weblog

stephen@blueslugs.com (Stephen Hahn) — Tue, 31 Mar 2009 17:10:10 +0000

“Simplicity is the ultimate sophistication.” : Michal Pryc’s Weblog

Michal gives another one-click example: he's made a p5i file with the base dependencies for building OpenSolaris's GNOME consolidation.

PackageManager getting Flash : John Rice's Weblog

stephen@blueslugs.com (Stephen Hahn) — Tue, 31 Mar 2009 17:07:23 +0000

PackageManager getting Flash : John Rice’s Weblog

John gives some screenshots from our one-click install/add publisher support. Most of the sun.com properties now support .p5i files and their application/vnd.pkg5.info MIME type now, so if you want to share packages–or give an initial configuration for some examples–your users are only a click away.

DIY: Home NAS Box with OpenSolaris and ZFS (English version) - Zhong's weblog

stephen@blueslugs.com (Stephen Hahn) — Mon, 23 Mar 2009 16:48:00 +0000

DIY: Home NAS Box with OpenSolaris and ZFS (English version) – Zhong’s weblog

Zhong builds an attractive home NAS box with 2008.11 and a dual-core Atom–the 330.

What's in a name? That which we call an Authority... - Transitive'

stephen@blueslugs.com (Stephen Hahn) — Wed, 11 Mar 2009 16:32:33 +0000

What’s in a name? That which we call an Authority… – Transitive

Shawn explains why we prefer publisher to authority as a term for a producer and distributor of packages.

Kindle Formatting

stephen@blueslugs.com (Stephen Hahn) — Sun, 01 Mar 2009 22:02:33 +0000

Kindle Formatting

Information and blog on formatting books for the Amazon Kindle. The author runs a consulting service; site contains his tips, etc.

Whoosh

stephen@blueslugs.com (Stephen Hahn) — Thu, 12 Feb 2009 17:59:58 +0000

Whoosh

Whoosh: a fast pure-Python search engine. "Whoosh is as fast or faster at indexing as other Python search libraries that wrap C or Java libraries (e.g. PyLucene?, Xappy), and not that much slower at searching."

Continuous Deployment « Timothy Fitz

stephen@blueslugs.com (Stephen Hahn) — Wed, 11 Feb 2009 07:55:35 +0000

Continuous Deployment « Timothy Fitz

The original post proposing continuous deployment; comments interesting, too.

Continuous Deployment at IMVU: Doing the impossible fifty times a day. « Timothy Fitz

stephen@blueslugs.com (Stephen Hahn) — Wed, 11 Feb 2009 05:58:18 +0000

Continuous Deployment at IMVU: Doing the impossible fifty times a day. « Timothy Fitz

IMVU deploys to production after passing its continuous integration tests.

Project Voldemort

stephen@blueslugs.com (Stephen Hahn) — Thu, 15 Jan 2009 20:53:42 +0000

Project Voldemort

"Voldemort is a distributed key-value storage system." Deeper down, we find: "Replication within the cluster handles the case of data loss on a single server, but bugs can still lead to global data corruption or data pollution. To handle this scenario the best option is ZFS filesystem snapshots."

Mounting Amazon S3 buckets as a file system on OpenSolaris : Angelo's Soapbox

stephen@blueslugs.com (Stephen Hahn) — Wed, 14 Jan 2009 18:24:28 +0000

Mounting Amazon S3 buckets as a file system on OpenSolaris : Angelo’s Soapbox

Using FUSE and s3fs on 2008.11 to access Amazon S3 storage with filesystem operations.

Build your own Drobo-Replacement based on ZFS « Agile Developer, Berlin, Germany

stephen@blueslugs.com (Stephen Hahn) — Wed, 14 Jan 2009 17:47:58 +0000

Build your own Drobo-Replacement based on ZFS « Agile Developer, Berlin, Germany

Exploring ZFS on 2008.11 as a "dream backup service". Uses VirtualBox to test various configuration options.

Linux.com :: How to configure a scanner's buttons on Linux

stephen@blueslugs.com (Stephen Hahn) — Thu, 25 Dec 2008 22:39:38 +0000

Linux.com :: How to configure a scanner's buttons on Linux

The secret is the libusb-based application, scanbuttond. Useful invocations of SANE utilities without xsane. scanbuttond works fine on OpenSolaris 2008.11.

2008.11: terminator is my favourite application

stephen@blueslugs.com (Stephen Hahn) — Tue, 23 Dec 2008 19:49:29 +0000

There’s a lot of new applications in 2008.11—cheese is useful for testing your USB webcam and making self-portraits for avatars and profile pictures and I’m very intrigued by gobby—but I already have my favourite: terminator. Here’s a scaled screenshot:

I used to use ion as my window manager, so getting some tiling back is great. A side effect of those days is that I expect F2 to start a terminal, which is easy with a little gconf editing:

$ gconftool-2 -s /apps/metacity/global_keybindings/run_command_1 \
-t string 'F2'
$ gconftool-2 -s /apps/metacity/keybinding_commands/command_1 \
-t string '/usr/bin/terminator'

There’s a little bug in handling the title bar I’d like to see fixed, but I can ignore that in return for greater terminal density.

Update: Col likes it, too.

[ T: OpenSolaris GNOME terminator ]

OpenDX Home Page

stephen@blueslugs.com (Stephen Hahn) — Sun, 21 Dec 2008 17:49:01 +0000

OpenDX Home Page

IBM's Data Explorer (which I used on an SGI workstation years ago) is now an open source project.

Bin-Blog: Subscribing a feed in Google Reader using Firefox

stephen@blueslugs.com (Stephen Hahn) — Tue, 16 Dec 2008 05:18:26 +0000

Bin-Blog: Subscribing a feed in Google Reader using Firefox

Explains the configuration for content handling, such that you can adjust Firefox to subscribe a feed in Google Reader through a direct link.

On packaging

stephen@blueslugs.com (Stephen Hahn) — Sun, 14 Dec 2008 18:29:09 +0000

On packaging

Essay on the impacts of some of setuptools on distributions. I've been worrying about some of these aspects, too; eggs don't seem right for distributors, even if they're convenient for users.

We'll see

stephen@blueslugs.com (Stephen Hahn) — Fri, 12 Dec 2008 03:57:02 +0000

Two days' worth? Here?

I think we’ve had two evenings with short snow—without accumulation—and one significant hailstorm—with accumulation—in eleven years here. Intriguing.

2008.11: A roundup

stephen@blueslugs.com (Stephen Hahn) — Thu, 11 Dec 2008 06:18:19 +0000

With the launch over, I took a few minutes to read through today’s blog entries, and pulled out a list of the either technical or personally significant entries. If you’re looking for the highlights, Calum’s overview seems to be a great place to start. He’s got links to the best summaries and screencasts.

First, if you’re running 2008.05 and considering upgrading—recommended strongly—please watch Markus’s screencast. There are one or two tricky steps to pay attention to. (We got caught offguard by some misinteractions between GRUB, ZFS, and early startup; won’t happen again.)
Erwann’s screencast is a great introduction to Time Slider, and I’m pretty confident no turkeys were harmed during filming.

Peter has screen captures of top(1) on a couple of older laptops, including one with 447 MiB available to the OS. Dave and the Install team worked hard to keep 512 MiB systems booting on the release; here’s evidence.

Doug’s pkgfactory/roboporter discussion and Nico’s explanation of the repositories show an extremely promising way to make sure there’s a wide variety of software available for OpenSolaris—make the computers do the grunt work. Doug and Nico, and Jim Walker, have been pushing hard on improving porting efficiencies.
Knut explains using JavaDB as a managed service on 2008.11. Since Knut’s post combines pkg(5) and smf(5) invocations, it easily wins a spot in any summary I might write. (And so it appears in this one.)
Jerry describes the foundational work for integrating zones, Snap Upgrade, and pkg(5). The changes to filesystem layout to support that integration place limitations on zones across the 2008.05 to 2008.11 transition, so be sure to review Jerry’s post if you’ve been experimenting with ipkg zones on the release.
John reviews 2008.11 as a suitable paravirtualized guest on xVM, with some careful discussion of how to handle the graphical console.

Shawn’s the sole member of the image packaging team to get a technical entry up. Shawn talks about the depot refacing work he’s done and planning on doing for 2009.next. (Shawn modestly omits the fact that he also implemented “pkg history” and operational intent in the client.)

Finally, I’m giving Fintan the nod for best screenshot, even if I prefer to see this release on the metal.

There are a lot of entries under tagged with opensolaris; you’ll find coverage of many other aspects of the release.

[ T: OpenSolaris 2008.11 ]

2008.11: Live chat action

stephen@blueslugs.com (Stephen Hahn) — Wed, 10 Dec 2008 19:26:12 +0000

We’re just wrapping up the live chat for the launch, with interested folks moving to freenode in #os200811. Here’s a quick photo of Danek, Dave, and Bart responding intently to the questions and comments earlier this morning.

As you can see, every question is taken seriously…

[ T: OpenSolaris IRC ]

2008.11: Everyday projectors

stephen@blueslugs.com (Stephen Hahn) — Wed, 10 Dec 2008 09:01:42 +0000

I regularly use my for presentations and demonstrations, which in the past has involved some configuration gymnastics. If your laptop has NVIDIA graphics, the “NVIDIA X Server Settings” Preferences panel lets you configure for most projectors, but if you didn’t have an NVIDIA graphics adaptor, configuration was arduous and fragile. With 2008.11, we get GNOME 2.24; the 2.24 Screen Resolution panel, reachable from the desktop background menu, makes configuration accessible to a wider range of adaptors, including the Intel graphics in my Toshiba r500.

So I made another little film:

Again, you can retrieve this video directly from channelsun.

[ T: OpenSolaris GNOME Toshiba projector ]

Update 2020-05-02: channelsun.sun.com is long gone, but I found the original video and uploaded it to YouTube.

2008.11: More ways to get it, and more ways to share it

stephen@blueslugs.com (Stephen Hahn) — Wed, 10 Dec 2008 08:01:09 +0000

As I did for 2008.05, I’ll collect links to mirror sites here. These will also get links on the various download pages out there.

Construction of 2008.11 was faster, more efficient, and generally more predictable than 2008.05. Image packaging, snap upgrade, and the distribution constructor—and the notorious distro importer—saw many fixes and features that resulted in the ability to rapidly turn out corrected images and packages. As problems were found during testing, we could spin up new test variants in hours or less. (Thus, the existance of RC1.5, which we were able to inject into the schedule because of this newfound speed. Dave’s written more about the constructor.) You can get the ISO image, suitable for burning to a 700 MiB CD or immediate use in virtual environment, directly from the following locations:

via HTTP from dlc.sun.com,
via HTTP from genunix.org.
via HTTP from mirror.switch.ch.
via HTTP from iso.zagbot.com,
Thanks to Lyndsay Roger of zagbot.com.
via HTTP from opensolaris.linuxfreedom.com,
Thanks to linuxfreedom.com’s Jonathan.
via HTTP from fastbull.org,

Thanks to Rodolfo Boraso for this site (and for working on the NLUUG sites below).

via HTTP from mirrors.cat.pdx.edu,
via rsync from mirrors.cat.pdx.edu
Thanks to Matt Whiteley at Portland State University for these two sites.
via HTTP from lifewithsolaris.jp,
Thanks to Shunsuke from lifewithsolaris.jp.
via HTTP from os.elemental.org,
Thanks to Dale from elemental.org.
Thanks to the Dutch Unix Users Group (NLUUG) for these sites.
and via BitTorrent.

As Glynn notes, we’ve set up anonymous rsync to make acquiring and mirroring the ISO images. In addition, I’ve posted draft instructions on how to be a content mirror for pkg.opensolaris.org/dev. We’ve tested the pkg(5) content mirroring, as well as the rsync service, but would be happy to get feedback on pkg-discuss@opensolaris.org.

These links either take you to the site’s page or directly to the osol-0811.iso CD image, which contains 11 “primary languages”. It installs quite a bit faster, particularly on systems with slower CPUs. There is also an LZMA-compressed image, which has localization support for 42 languages, including those primary ones. It’s available from dlc.sun.com, genunix.org, and as a

torrent. (Consult the language lists for specifics.)

[ T: OpenSolaris indiana ]

OpenSolaris Support Services Training and Help – Docs Documentation White Papers FAQs Subscriptions OpenSolaris guides – Features

stephen@blueslugs.com (Stephen Hahn) — Mon, 01 Dec 2008 01:50:28 +0000

OpenSolaris Support Services Training and Help – Docs Documentation White Papers FAQs Subscriptions OpenSolaris guides – Features

Glynn's written up a very comprehensive What's New for 2008.11.

CS 294-5, Spring 2006 Great Algorithms

stephen@blueslugs.com (Stephen Hahn) — Wed, 19 Nov 2008 18:58:31 +0000

CS 294-5, Spring 2006 Great Algorithms

Interesting course from Berkeley's Richard Karp.

OpenSolaris System Administration Topics List – wikis.sun.com

stephen@blueslugs.com (Stephen Hahn) — Mon, 17 Nov 2008 19:01:12 +0000

OpenSolaris System Administration Topics List – wikis.sun.com

A companion page to the tasks list, with useful links to troubleshooting articles as well. Forget how to make a change to an OpenSolaris system? Read this page.

OpenSolaris System Administration Tasks List – wikis.sun.com

stephen@blueslugs.com (Stephen Hahn) — Mon, 17 Nov 2008 19:00:05 +0000

OpenSolaris System Administration Tasks List – wikis.sun.com

A very useful list of administrative tasks–and the "how to" instructions to make them happen. New to OpenSolaris? This page is for you.

Simple Recipes in Python

stephen@blueslugs.com (Stephen Hahn) — Tue, 11 Nov 2008 18:17:05 +0000

Simple Recipes in Python

Some simple implementations of numerical algorithms in Python.

2008.next: demonstrating suspend-resume

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 Nov 2008 18:11:51 +0000

Randy has been blogging on the suspend-resume feature and how it’s been improving on a variety of x86-based systems. A few weeks ago, I was sitting on the USGS Menlo Park campus, and decided to record a demonstration of my own laptop being suspended and resumed multiple times:

(You can also access the video on channelsun.)

The demonstration is using Build 99, so there are a few differences and improvements in the release candidates soon to be published. With careful timing of your battery discharging and recharging, you can shoot for uptimes in days to weeks.

I’m also demonstrating my ignorance of all video-making rules, in that I used a Panasonic Lumix LX-2 in movie mode, outdoors, near traffic, and then edited the video using OpenSolaris and open source tools. But I’ll note that, even in real life, you’ll have to peer at your r600 closely to see the orange suspension indicator.

[ T: OpenSolaris Toshiba laptop ]

Update 2020-05-02: channelsun.sun.com is long gone, but I found the original video and uploaded it to YouTube.

ImageMagick Tutorial

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 Nov 2008 17:56:51 +0000

ImageMagick Tutorial

Some basic operations with the ImageMagick convert utility.

ffmpeg audio/video manipulation

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 Nov 2008 17:55:53 +0000

ffmpeg audio/video manipulation

A well-constructed introduction to using ffmpeg for video and audio manipulation.

N.E.R.D.: ffmpeg command line quickies

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 Nov 2008 17:55:01 +0000

N.E.R.D.: ffmpeg command line quickies

Useful one-line invocations of ffmpeg.

FFmpeg Documentation

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 Nov 2008 17:54:06 +0000

FFmpeg Documentation

Options and operands for ffmpeg. Combine with "ffmpeg -formats" to figure out what you build of ffmpeg can do.

A gentle evening, in contrast

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 Nov 2008 06:39:56 +0000

Ready for any recounts

Dina and I were, despite all polls, anticipating election nights like the preceding two, where any hope of resolution was delayed, and tension distributed to all election watchers. This year, I was determined to enjoy the evening, and poured something light for the both of us to sip while listening to analyses. Historians will write page upon page about tonight, what it signifies, and so on; my toast is to the many states, Secretaries of State, and election workers who ran their polling and counting with great predictability and competence—the best of attributes. Thanks.

We roused the kids to listen to both of tonight’s speeches. Congratulations to the candidates, the President-elect, and the country.

FFmpeg FAQ

stephen@blueslugs.com (Stephen Hahn) — Mon, 03 Nov 2008 18:29:31 +0000

FFmpeg FAQ

Note 3.2 "How do I encode single pictures into movies?" and 3.19 "How can I join video files?" Engage brain, produce movie.

2008.next: Recovering a dropped data pool

stephen@blueslugs.com (Stephen Hahn) — Sun, 02 Nov 2008 21:20:06 +0000

As you update your pkg(5)-based system to build 100a, you might see a message on console during startup. It will look something like

WARNING: pool 'zeta1' could not be loaded as it was last accessed by
another system (host: rosseau hostid: 0x581d45e). See:
http://www.sun.com/msg/ZFS-8000-EY

and in /var/adm/messages, you’ll find it again:

Oct 27 14:55:21 rosseau zfs: [ID 427000 kern.warning] WARNING: pool
'zeta1' could not be loaded as it was last accessed by another system
(host: rosseau hostid: 0x581d45e). See:
http://www.sun.com/msg/ZFS-8000-EY

zeta1 is a data pool on my system—the boot pool, rpool, upgraded normally—and recent changes in the hostid implementation mean that we have to import (or reimport) any non-boot pools on the system. That is,

$ pfexec zpool import -f zeta1

and our zeta1-based filesystems are again online.

[ T: OpenSolaris zfs pkg ]

OpenSolaris Indiana on a Sony UX Micro PC – Brad's weblog

stephen@blueslugs.com (Stephen Hahn) — Sun, 02 Nov 2008 20:58:09 +0000

OpenSolaris Indiana on a Sony UX Micro PC – Brad’s weblog

Brad, with limited acrobatics, gets 2008.05 installed and running on the teeny-tiny UX.

IRS Withholding Calculator

stephen@blueslugs.com (Stephen Hahn) — Sat, 18 Oct 2008 00:25:20 +0000

IRS Withholding Calculator

"The purpose of this application is to help employees to ensure that they do not have too much or too little income tax withheld from their pay. It is not a replacement for Form W-4, but most people will find it more accurate and easier to use than the worksheets that accompany Form W-4. You may use the results of this program to help you complete a new Form W-4, which you will submit to your employer." If you can parse this paragraph, you may find this tool useful.

Shooting the Moon w/K10D and Cheap Glass: Pentax Forum

stephen@blueslugs.com (Stephen Hahn) — Fri, 17 Oct 2008 04:46:52 +0000

Shooting the Moon w/K10D and Cheap Glass: Pentax Forum

An article and followup comments describing various moonshots.

The eyeballing game

stephen@blueslugs.com (Stephen Hahn) — Mon, 06 Oct 2008 20:30:09 +0000

The eyeballing game

Correct geometric diagrams by eye.

2008.next: Easy install for docutils

stephen@blueslugs.com (Stephen Hahn) — Thu, 18 Sep 2008 23:33:34 +0000

It’s been a busy summer, mostly trying to manage conflicting schedules at home while staying loyal to the six month release schedule for OpenSolaris at work. Schedules everywhere, it seems.

I’ve been probing deeper into the ecosystem around Python, and have been trying to determine whether restructured text might have some value in the workflow around OpenSolaris development: it can output to a variety of formats, including XML, LaTeX, and HTML, and doesn’t have an onerous list of dependencies.

I’ve scars from building AsciiDoc 8.

docutils isn’t available via image packaging yet, but it’s easily retrievable using easy_install. That means the question is: how do I get easy_install? I’m running Build 98, so I’ll restrict my search to packages on the development branch for that build:

$ pkg search -r easy_install | grep 0.98
basename file usr/bin/easy_install pkg:/SUNWpython-setuptools@0.5.11-0.98

Okay, easy_install is part of setuptools—you probably knew that already. Let’s install the appropriate package using pkg(1):

$ pfexec pkg install SUNWpython-setuptools
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 81/81 0.30/0.30
PHASE ACTIONS
Install Phase 93/93
PHASE ITEMS
Reading Existing Index 9/9
Indexing Packages 1/1
$ which easy_install
/usr/bin/easy_install

Having setuptools in place, we can now attempt installs of any Python module available in the Python Package Index (pypi), via the easy_install command. (easy_install --help will display a usage message for easy_install.) Let’s install the docutils package:

$ pfexec easy_install docutils
Searching for docutils
Reading http://pypi.python.org/simple/docutils/
Reading http://docutils.sourceforge.net/
Best match: docutils 0.5
Downloading http://prdownloads.sourceforge.net/docutils/docutils-0.5.tar.gz?download
Processing docutils-0.5.tar.gz
Running docutils-0.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-57yD32/docutils-0.5/egg-dist-tmp-Ip1xgp
"optparse" module already present; ignoring extras/optparse.py.
"textwrap" module already present; ignoring extras/textwrap.py.
zip_safe flag not set; analyzing archive contents...
docutils.parsers.rst.directives.misc: module references file
docutils.writers.html4css1.init: module references file
docutils.writers.pep_html.init: module references file
docutils.writers.s5_html.init: module references file
docutils.writers.newlatex2e.init: module references file
Adding docutils 0.5 to easy-install.pth file
Installing rst2xml.py script to /usr/bin
Installing rst2pseudoxml.py script to /usr/bin
Installing rst2html.py script to /usr/bin
Installing rst2s5.py script to /usr/bin
Installing rst2latex.py script to /usr/bin
Installing rstpep2html.py script to /usr/bin
Installing rst2newlatex.py script to /usr/bin
Installed /usr/lib/python2.4/site-packages/docutils-0.5-py2.4.egg
Processing dependencies for docutils
Finished processing dependencies for docutils
$ which rst2html.py
/usr/bin/rst2html.py

And now I can start experimenting with restructured text.

[ T: OpenSolaris Python pkg docutils setuptools ]

2008 Winning Contest Entries at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Thu, 18 Sep 2008 22:11:17 +0000

2008 Winning Contest Entries at OpenSolaris.org

Winning entries for the community innovation awards. If you're no longer (or don't plan to be) an undergraduate, or you don't want to call your work "research", maybe this award program's more suitable.

2008 Winning Entries at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Thu, 18 Sep 2008 22:11:17 +0000

2008 Winning Entries at OpenSolaris.org

2008 Winning Grant Proposals at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Thu, 18 Sep 2008 22:00:42 +0000

2008 Winning Grant Proposals at OpenSolaris.org

The first group of grant recipients for undergraduate research on OpenSolaris.

Solar Water Heating Systems – Welcome to VELUX America

stephen@blueslugs.com (Stephen Hahn) — Sun, 14 Sep 2008 17:32:44 +0000

Solar Water Heating Systems – Welcome to VELUX America

Skylight-style solar water heaters.

Chromium – Google Code

stephen@blueslugs.com (Stephen Hahn) — Wed, 03 Sep 2008 05:08:36 +0000

Chromium – Google Code

Development home for Chromium, the open source components for Google's Chrome browser.

[0808.3310] Inferring basic parameters of the geodynamo from sequences of polarity reversals

stephen@blueslugs.com (Stephen Hahn) — Sun, 31 Aug 2008 08:45:58 +0000

[0808.3310] Inferring basic parameters of the geodynamo from sequences of polarity reversals

The asymmetric time dependence and various statistical properties of polarity reversals of the Earth's magnetic field are utilized to infer some of the most essential parameters of the geodynamo, among them the effective (turbulent) magnetic diffusivity, the degree of supercriticality, and the relative strength of the periodic forcing which is believed to result from the Milankovic cycle of the Earth's orbit eccentricity. A time-stepped spherically symmetric alpha^2-dynamo model is used as the kernel of an inverse problem solver in form of a downhill simplex method which converges to solutions that yield a stunning correspondence with paleomagnetic data.

[0808.3278] Thump, ring: the sound of a bouncing ball

stephen@blueslugs.com (Stephen Hahn) — Sun, 31 Aug 2008 08:37:26 +0000

[0808.3278] Thump, ring: the sound of a bouncing ball

Analysis of a simplified model basketball.

[0808.2624] Mark J Reid, Is there a Supermassive Black Hole at the Center of the Milky Way?

stephen@blueslugs.com (Stephen Hahn) — Thu, 21 Aug 2008 17:52:29 +0000

[0808.2624] Mark J Reid, Is there a Supermassive Black Hole at the Center of the Milky Way?

"This review outlines the observations that now provide an overwhelming scientific case that the center of our Milky Way Galaxy harbors a supermassive black hole."

Mathematics and Computation " Intuitionistic mathematics for physics

stephen@blueslugs.com (Stephen Hahn) — Wed, 20 Aug 2008 18:58:06 +0000

Mathematics and Computation ” Intuitionistic mathematics for physics

Package List – Life with Solaris

stephen@blueslugs.com (Stephen Hahn) — Fri, 15 Aug 2008 19:51:38 +0000

Package List – Life with Solaris

Life with Solaris has begun publishing some popular multimedia formats using image packaging. Current packages include BMPx, Goggles/Ogle, VLC, and Xine.

Open Problem Garden

stephen@blueslugs.com (Stephen Hahn) — Sat, 09 Aug 2008 19:19:46 +0000

Open Problem Garden

"Welcome to the Open Problem Garden, a collection of unsolved problems in mathematics."

Life is so good, it gets better every day » Blog Archive » Playing with IPS and SFE

stephen@blueslugs.com (Stephen Hahn) — Mon, 04 Aug 2008 16:43:25 +0000

Life is so good, it gets better every day » Blog Archive » Playing with IPS and SFE

Glynn works through how to build a "spec files extra" (SFE) package and convert it for publication in a pkg(5) repository.

moving from nevada and live upgrade to opensolaris : Edward Pilatowicz's Weblog

stephen@blueslugs.com (Stephen Hahn) — Mon, 28 Jul 2008 17:59:25 +0000

moving from nevada and live upgrade to opensolaris : Edward Pilatowicz’s Weblog

Ed has a procedure to construct a 2008.05 system alongside an existing Nevada system. Slick.

[0807.2097] Testing a priority-based queue model with Linux command histories

stephen@blueslugs.com (Stephen Hahn) — Thu, 17 Jul 2008 15:23:47 +0000

[0807.2097] Testing a priority-based queue model with Linux command histories

"We study human dynamics by analyzing Linux history files." Attempt to verify one of the power law behaviours, but with a very small sample.

MIT OpenCourseWare – 6.831 User Interface Design and Implementation, Fall 2004 – Readings

stephen@blueslugs.com (Stephen Hahn) — Mon, 16 Jun 2008 19:25:29 +0000

MIT OpenCourseWare – 6.831 User Interface Design and Implementation, Fall 2004 – Readings

The reading list for MIT's user interface design course.

Metakit embedded database library

stephen@blueslugs.com (Stephen Hahn) — Mon, 16 Jun 2008 00:29:39 +0000

Metakit embedded database library

Metakit is an efficient embedded database library with a small footprint. Python binding, as well as C++ and Tcl. MIT license.

Logic Supply – Leaders in Mini-ITX & Small Form Factor Solutions

stephen@blueslugs.com (Stephen Hahn) — Thu, 12 Jun 2008 05:55:21 +0000

Logic Supply – Leaders in Mini-ITX & Small Form Factor Solutions

Currently offering the Intel Little Valley system, among other options

Logic Supply – Leaders in Mini-ITX and Small Form Factor Solutions

stephen@blueslugs.com (Stephen Hahn) — Thu, 12 Jun 2008 05:55:21 +0000

Logic Supply – Leaders in Mini-ITX and Small Form Factor Solutions

Currently offering the Intel Little Valley system, among other options

The View from the Moon – ii_bitmap=1; ii_bitmap=1; ii_bitmap=1; ii_bitmap=1; …

stephen@blueslugs.com (Stephen Hahn) — Wed, 04 Jun 2008 23:54:24 +0000

The View from the Moon – ii_bitmap=1; ii_bitmap=1; ii_bitmap=1; ii_bitmap=1; …

Why image packaging doesn't have scripting, exhibit 731.

2008.05: Who's got it

stephen@blueslugs.com (Stephen Hahn) — Sat, 24 May 2008 00:28:57 +0000

Well, I don’t know exactly who’s running a copy of 2008.05, but I do know what IP addresses have interacted with pkg.opensolaris.org from the server logs. I used MaxMind’s GeoIP Lite to map these back to countries, and Google’s chart API to make a little map:

If 2008.05 is your first experience with the technology developed at opensolaris.org, welcome. (You’re not alone—the map suggests that, in many places, there might even be a fellow user nearby.) Don’t have it? Try a convenient mirror site.

Maybe we need a TechDays in Greenland next year?

[ T: OpenSolaris pkg nationalism ]

suds

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 May 2008 04:13:21 +0000

suds

"Suds is a lightweight SOAP python client that provides a service proxy for Web Services."

CherryPy

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 May 2008 04:09:33 +0000

CherryPy

"CherryPy is a pythonic, object-oriented HTTP framework." Shawn's almost done refactoring pkg.depotd to use CherryPy rather than BaseHTTPServer.

Python Web Services

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 May 2008 03:54:47 +0000

Python Web Services

Home of SOAPpy and Zolera SOAP Infrastructure (ZSI)

Mail droppage

stephen@blueslugs.com (Stephen Hahn) — Tue, 13 May 2008 19:57:05 +0000

It looks like I missed making a configuration change last update, and opensolaris.org has been dropping my mail messages for the past week as a result. Sorry—if you’re waiting for mail from me, you may need to ping me again. Otherwise, I’ll try to reinsert myself in various threads…

CommunityOne: here and there

stephen@blueslugs.com (Stephen Hahn) — Mon, 05 May 2008 08:38:30 +0000

Sunday I spent at Moscone, teaching laptops and projectors to get along. Saturday, Nathaniel, Benjamin, and I dropped in for lunch at the Developer Summit. I managed to talk with a few people before the boys found large whacking sticks, and then it seemed best to drive to Pescadero for some beach time.

I’ll be busy for the morning of CommunityOne. For Rich‘s keynote, I’ll be running some of the less violent demos. Almost immediately after that, lead modernizer David Comay and I will be going into more detail in our session

S297399 Getting Started with OpenSolaris™; New Features & Building OpenSolaris™ Packages; 11:00 a.m., Moscone South/Esplanade 300. We’ve got some additional demonstrations, including a worked example of package publication using some pre-release tools, which could be exciting.

I’m hoping to have some time for questions during the session but, if not, I’ll be circulating during the afternoon and happy to talk to people about 2008.05, image packaging, or whatever. And, of course, there will be time to talk at the party after the day’s sessions. In any case, I should be easy to spot: I have a new tie.

I’m told you can still register on-site—it’s not too late.

[ T: OpenSolaris pkg CommunityOne ]

2008.05: More ways to get it

stephen@blueslugs.com (Stephen Hahn) — Mon, 05 May 2008 08:11:39 +0000

As I did for the preview releases, I’ll collect links to mirror sites here. These will also get links on the various download pages out there.

Bart and I just finished updating the package repository with the new packages we’d received the past few weeks, and that means 2008.05 is out the door. (Thanks to everybody who tried the release candidate, filed a bug, shuffled a package, wrote or proofread a document, or just spent energy anticipating the bits.) You can get the ISO image, suitable for burning to a 700 MiB CD or immediate use in virtual environment, directly from the following locations:

via HTTP from dlc.sun.com, and
via HTTP from genunix.org,
via HTTP from ftp.df.lth.se,
via FTP from ftp.df.lth.se,

via HTTP from mirrors.xservers.ro,

via HTTP from gulus.usherbrooke.ca,

via FTP from ftp.nluug.nl,

via HTTP from ftp.nluug.nl,

via rsync from ftp.nluug.nl,

via FTP from ftp.surfnet.nl,

via HTTP from ftp.surfnet.nl,

via rsync from ftp.surfnet.nl, or

 * <a href="http://dlc.sun.com/torrents/info/os200805.iso.torrent" class="broken_link" rel="nofollow"><img class="icon-inline" src="/icon/dr-theoretical-blue-32.png" border="0" title="" alt="" /> via BitTorrent</a>.</ul> </p>
Reading the logs, and talking with some of our mirror sites, we know we all served out a lot of downloads for the previews; if you&#8217;re interested in being a mirror, please let me know. (2008.05 remains completely redistributable.) We&#8217;re using a bigger download complex this time, but every mirror helps.
If you already downloaded and installed Preview 2, it&#8217;s more complete, easier, and probably faster to update directly using image packaging: see the update guidelines. These instructions involve a small script to safely update a clone of your installed system, and then switch to that on a system reboot. (If you were running Preview 1, you should update that to Preview 2, and then go on to 2008.05.)
**Update 1**: My thanks to Tobias Lundquist, who&#8217;s once again mirrored via FTP and HTTP (Internet 2) in Sweden.
**Update 2**: My thanks to Luca, who&#8217;s put up an HTTP mirror in Romania.
**Update 3**: Simon Valiquette has put up an HTTP mirror (Internet 2) at the Universit&eacute; de Sherbrooke in Qu&eacute;bec, Canada. Merci, Simon.
**Update 4**: <a href="http://blogs.sun.com/bartm" >Bart Muijzer</a> has arranged for mirrors by the Netherlands Unix Users Group (NLUUG) and `surfnet.nl`. Thanks, Bart.
<!--

Update 2: My thanks to trisk, who's once again put up an HTTP mirror (Internet 2) on the East Coast of the USA.

-->

 <p class="note">
These links are for the gzipped CD image, which contains the 12 &#8220;primary languages&#8221;. It installs quite a bit faster, particularly on systems with slower CPUs. There is also an LZMA-compressed image, which has localization support for 42 languages, including those primary ones. It&#8217;s available from <a href="http://dlc.sun.com/osol/opensolaris/2008/05/os200805g.iso" class="broken_link" rel="nofollow">dlc.sun.com</a>, <a href="http://www.genunix.org/distributions/indiana/os200805g.iso" >genunix.org</a>, <a href="http://ftp.df.lth.se/pub/opensolaris/current/os200805g.iso" class="broken_link" rel="nofollow">ftp.df.lth.se</a>, <a href="http://mirrors.xservers.ro/opensolaris/200805/os200805g.iso" class="broken_link" rel="nofollow">mirrors.xservers.ro</a>, and as a <a href="http://dlc.sun.com/torrents/info/os200805g.iso.torrent" class="broken_link" rel="nofollow">torrent</a>. (Consult the <a href="http://www.opensolaris.com/get/" >language lists</a> for specifics.)
</p>
<p class="tag">
[ <i>T</i>: <a href="http://www.technorati.com/tag/OpenSolaris" rel="tag nofollow" class="broken_link">OpenSolaris</a> <a href="http://www.technorati.com/tag/indiana" rel="tag nofollow" class="broken_link">indiana</a> ]
</p>

Live chat on #os200805

stephen@blueslugs.com (Stephen Hahn) — Thu, 01 May 2008 21:00:50 +0000

We’re talking about OpenSolaris 2008.05 over on IRC, using the #os200805, from now until 2 p.m. Pacific Time. If you haven’t got an IRC client, you can use freenode’s Java applet, which worked great (until Danek kickbanned me). Enter a nickname, and then ask some questions.

Apparently the load from 60 000 users blew out the chat system, so we moved to IRC.

~4100MiB

stephen@blueslugs.com (Stephen Hahn) — Sat, 26 Apr 2008 19:30:12 +0000

I seeded the 2008.05 release candidate for about 45 hours, ultimately shipping a little over 4100 megabytes. I’m going to take a break, because I want to update my DP2-based workstation and get some work done, but, once we have new bits, I’ll getting seeding again.

(I found the actual result: 4237MiB sent up, so almost 4GiB.)

OpenSolaris: Bug dependencies and release management

stephen@blueslugs.com (Stephen Hahn) — Fri, 18 Apr 2008 01:38:20 +0000

Right now, if you’re subscribed to any of the Installation and Packaging community group or project lists, you’ll see a lot of commit notifications as the various teams attempt to fix various bugs noted since the second Developer Preview release. We’ve been using the trial Bugzilla instance—which is becoming the default defect tracker for opensolaris.org it appears—and trying out various features.

For tracking release completeness, we’re using “blocker bugs” or “tracker bugs”, which are synthetic bug entries that we mark various important bugs (“stoppers”) as blocking. That means that we end up creating a little dependency graph that shows what unfixed bugs are stopping us from reaching some initial set of release criteria. We have two tracker bugs

571 Blocker bug for Indiana 3
1190 Packaging bugs to fix for May (tracker bug) that we’re monitoring to make sure we’ve got a handle on things.

Bugzilla has two nice summaries for showing this information, in addition to the default bug status page. I’ll use 571 as the example tracker bug, since we’ve made 1190 block it—which leads to a more nteresting graph. The tree view is a useful and succinct representation, where indentation shows dependencies. The graph view is a bit unwieldy for this bug, but might be useful if the tree view became too long.

A useful technique if you’re trying to bring a release together.

[ T: OpenSolaris Bugzilla pkg ]

It Must Be Time for Tea: Converting Projects to Mercurial

stephen@blueslugs.com (Stephen Hahn) — Thu, 10 Apr 2008 16:49:17 +0000

It Must Be Time for Tea: Converting Projects to Mercurial

Mike talks about helping projects move from TeamWare to Mercurial. Large projects may have been running a project gate ("branch") for a year or more, and with a clear schedule in mind. Minimizing the cost of the main gate switching source code managemen

Intel® Software Network Blogs » Making access to memory faster in OpenSolaris (and Core2)

stephen@blueslugs.com (Stephen Hahn) — Sat, 05 Apr 2008 03:22:28 +0000

Intel® Software Network Blogs » Making access to memory faster in OpenSolaris (and Core2)

Dave reports on performance improvements in OpenSolaris for memcpy(), memmove(), and memset() on Core2-based systems.

The View from the Moon: Finding bugs in python code, using DTrace

stephen@blueslugs.com (Stephen Hahn) — Sat, 05 Apr 2008 02:59:13 +0000

The View from the Moon: Finding bugs in python code, using DTrace

We've been using DTrace to check performance as we develop image packaging; here Dan reviews one of his recent wins. Because DTrace makes it extremely cheap to check for algorithmic mistakes, and the yields of correcting such mistakes are so high, it's c

The View from the Moon: cr.opensolaris.org gets an ATOM feed

stephen@blueslugs.com (Stephen Hahn) — Mon, 24 Mar 2008 21:59:22 +0000

The View from the Moon: cr.opensolaris.org gets an ATOM feed

Dan extends code review to provide updates via syndication, which is another step to a virtual hallway for OpenSolaris development. Sweet.

Fiscally Conservative | Musings

stephen@blueslugs.com (Stephen Hahn) — Mon, 17 Mar 2008 03:22:59 +0000

Fiscally Conservative | Musings

Budget deficit and GDP growth compared by party across administrations, since 1930.

School Rankings for Writing OpenCourseWare

stephen@blueslugs.com (Stephen Hahn) — Wed, 12 Mar 2008 03:16:41 +0000

School Rankings for Writing OpenCourseWare

Article listing a bunch of online courseware on writing topics.

Election Candidates at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Wed, 12 Mar 2008 00:10:31 +0000

Election Candidates at OpenSolaris.org

Currently displaying the candidates for the 2008 election. Affiliations, bios, position statements, and podcasts.

The Open Solaris Governing Board 2.0 – Elections and Interviews – Barton's Blog

stephen@blueslugs.com (Stephen Hahn) — Wed, 12 Mar 2008 00:02:07 +0000

The Open Solaris Governing Board 2.0 – Elections and Interviews – Barton’s Blog

Barton intends to interview as many of this year’s OGB candidates as he can.

OpenSolaris Polling Open [on Simon Phipps, SunMink]

stephen@blueslugs.com (Stephen Hahn) — Tue, 11 Mar 2008 18:17:25 +0000

OpenSolaris Polling Open [on Simon Phipps, SunMink]

Simon explains the two proposed amendments, his candidacy, and a list of candidates "committed to asking Sun’s management to communicate regularly and clearly with the OpenSolaris community".

pkg(5): Reverse proxying your depot with Apache HTTPD

stephen@blueslugs.com (Stephen Hahn) — Thu, 21 Feb 2008 23:52:11 +0000

As part of the changes to get Developer Preview 2 ready, we decided to rejigger the HTTP handling on pkg.opensolaris.org so that we could have more options as more people attempt to use the early versions of image packaging. Previously, we ran pkg.depotd directly on port 80, in its read-only mode; now we use Apache HTTPD to listen on port 80, and use mod_proxy to proxy those incoming requests to a pkg.depotd instance listening on a separate port. With a couple of different approaches

Proxying and rewriting is one of those endlessly fun activities that somehow actually ends up being productive. Last time, proxying fun led Steve and I to fiddling around such that we ended up with proxy and rewrite patterns to enable the country portals for opensolaris.org.

If you want to share the top-level component of your URL space, you’ll need to watch pkg(5) developments, as you have to map the list of operations one-by-one—and I know there are some new operations forthcoming. That would involve adding something like the following to a VirtualHost directive in your Apache configuration.

ProxyRequests On
Redirect /index.html http://pkg.opensolaris.org/status
ProxyPass /abandon http://pkg.opensolaris.org:10000/abandon
ProxyPass /add http://pkg.opensolaris.org:10000/add
ProxyPass /catalog http://pkg.opensolaris.org:10000/catalog
ProxyPass /close http://pkg.opensolaris.org:10000/close
ProxyPass /feed http://pkg.opensolaris.org:10000/feed
ProxyPass /file http://pkg.opensolaris.org:10000/file
ProxyPass /filelist http://pkg.opensolaris.org:10000/filelist
ProxyPass /manifest http://pkg.opensolaris.org:10000/manifest
ProxyPass /open http://pkg.opensolaris.org:10000/open
ProxyPass /search http://pkg.opensolaris.org:10000/search
ProxyPass /versions http://pkg.opensolaris.org:10000/versions
ProxyPass /css http://pkg.opensolaris.org:10000/css
ProxyPass /logo http://pkg.opensolaris.org:10000/logo
ProxyPass /icon http://pkg.opensolaris.org:10000/icon
ProxyPass /status http://pkg.opensolaris.org:10000/

Configuring your server in this fashion allows you to mix an image packaging server in with your other site content. You can easily deliver static content alongside your depot, for example.

If you don’t mind pushing your package repository down one level in your URL space, then the above simplifies to

ProxyRequests On
ProxyPass /pkg/ http://pkg.opensolaris.org:10000/

(which should be a hint on how to create a repository farm under a single URL). To use the latter, you would use pkg(1)’s image-create subcommand

$ pkg image-create -F -a mypkgs.com=http://www.myserver.com/pkg /path/to/image

to connect your image to your reverse-proxied packaging depot.

In the two examples above, you should of course replace machine names like pkg.opensolaris.org and port numbers like 10000 with values appropriate to your own installation.

Happy proxied package serving!

Feel free to share your alternative configurations or approaches with other HTTP servers here, or on pkg-discuss@opensolaris.org.

[ T: OpenSolaris Solaris pkg httpd ]

Indiana: More ways to get to Preview 2

stephen@blueslugs.com (Stephen Hahn) — Wed, 13 Feb 2008 01:51:46 +0000

As I did for Preview 1, I’ll collect links to mirror sites here, as well as on the project page.

So we just released the second Indiana preview ISO, which you can get from the following locations:

We had a lot of downloads for Preview 1; if you’re interested in being a mirror, please let me know. (Indiana remains completely redistributable.) We’re using a bigger download complex this time, but every mirror helps.

If you already downloaded and installed Preview 1, it’s more complete, easier, and probably faster to update directly using image packaging: see the update guidelines on opensolaris.org. These instructions use ZFS and pkg(5) to safely update a clone of your installed system, and then switch to that on a system reboot.

Update 1: My thanks to Tobias Lundquist, who’s once again mirrored via FTP (Internet 2) in Sweden.

Update 2: My thanks to trisk, who’s once again put up an HTTP mirror (Internet 2) on the East Coast of the USA.

[ T: OpenSolaris Solaris indiana ]

all200.gif (GIF Image, 1278x8425 pixels)

stephen@blueslugs.com (Stephen Hahn) — Thu, 31 Jan 2008 17:46:24 +0000

all200.gif (GIF Image, 1278×8425 pixels)

Graph of distances from the centre of the Earth, outwards.

OpenSolaris: Finding work(ers)

stephen@blueslugs.com (Stephen Hahn) — Wed, 23 Jan 2008 20:36:17 +0000

A relatively new forum on opensolaris.org is opensolaris-jobs, a community-moderated list where OpenSolaris-related positions can be posted and which should give access to the people most passionate, most expert, and most interested in making OpenSolaris work their job. In the past few weeks, there’s been two postings you might find interesting:

Have a look, or—if you’re hiring for OpenSolaris knowledge—get in touch with great candidates.

[ T: OpenSolaris jobs ]

Create your own OpenSolaris IPS repository in a Weekend! : Michal Pryc's Weblog

stephen@blueslugs.com (Stephen Hahn) — Tue, 22 Jan 2008 23:29:32 +0000

Create your own OpenSolaris IPS repository in a Weekend! : Michal Pryc’s Weblog

Michal collects his image packaging screencasts, which constitute a six-part series on how to get image packaging up and running and ready to publish your own software packages.

mozdev.org – vimperator: index

stephen@blueslugs.com (Stephen Hahn) — Fri, 18 Jan 2008 00:18:02 +0000

mozdev.org – vimperator: index

A Firefox extension which wraps the browser in a vi/vim-like interface. Intriguing.

10 Steps to Caiman Development on Indiana : Fore!

stephen@blueslugs.com (Stephen Hahn) — Fri, 11 Jan 2008 21:28:22 +0000

10 Steps to Caiman Development on Indiana : Fore!

With a few easy pkg(5) operations, you can add the tools needed to use the Developer Preview as a development system. Dave gives the step-by-step instructions.

[sfwnv-discuss] draft webrev for star integration into SFWNV

stephen@blueslugs.com (Stephen Hahn) — Tue, 18 Dec 2007 17:18:37 +0000

[sfwnv-discuss] draft webrev for star integration into SFWNV

Darren offers up a webrev showing how straightforward the "technical" part of integrating a component into SFW is.

Stopping Firefox's restore session dialog

stephen@blueslugs.com (Stephen Hahn) — Thu, 06 Dec 2007 17:47:43 +0000

I’m sure that there are Firefox users out there who want to restore their previous session; I never do, and so deactivating that dialog is a big timesaver. If you search, you’ll find a few writeups on how to adjust the configuration, but I want to document the minimum steps to suppress the restore session dialog.

Enter about:config into the URL field.
Enter sessionstore into the displayed filter field and press Enter.
Double-click the value cell of the browser.sessionstore.enabled row, so that the value field reads False.
Restart Firefox. One click saved.

[ T: Firefox ]

Busted Tees – Indiana Pwns

stephen@blueslugs.com (Stephen Hahn) — Wed, 28 Nov 2007 07:38:43 +0000

Busted Tees – Indiana Pwns

Tempting…

White-tailed Kite

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Nov 2007 19:31:22 +0000

White-tailed Kite

Maybe what I saw at Ano Nuevo SR last week.

Tokyo Cabinet: a modern implementation of DBM

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Nov 2007 16:30:33 +0000

Tokyo Cabinet: a modern implementation of DBM

"Tokyo Cabinet is a library of routines for managing a database. The database is a simple data file containing records, each is a pair of a key and a value. Every key and value is serial bytes with variable length. Both binary data and character string ca

Coastal Thanksgiving

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Nov 2007 09:01:02 +0000

For the Thanksgiving holiday this year, we decided to explore San Mateo County’s section of the Pacific Coast. Dina settled on Butano State Park as our base, which is mostly made of a redwoods-forested canyon, south of Pescadero and on the western slopes of the Santa Cruz Mountains. We arrived in the late afternoon, having taken CA-84 through Woodside to San Gregario, and then heading south on CA-1.

We arrived late enough that, after setting up camp, it soon became too dark for even a short hike. It’s been cool on the Peninsula this fall, and it felt cool in the morning. We were all grateful when, with temperatures between 30 – 40 °F (0 – 5°C), Dina made oatmeal for breakfast.

One of the first things I noticed—and probably evident from the use of the flash in the above picture—is how dimmed the light on the forest floor is. The cool, dim environment appears to let some pretty sizeable mushrooms thrive; this specimen’s cap is about 2.5″ (63 mm) in diameter, although apparently there are bigger species around. The ribbon-like growths show a bit of élan.

Ben captured an interesting diffracted view of the forest around us:

Out of the dimness, we drove down to Año Neuvo State Reserve, and worked off a little energy in our walk out to the dunes and possible elephant seals.

As we walked, I watched a raptor maintain a position for seconds at a time, presumably as part of its hunting method. Here are three frames, over a total of 8 seconds, from a fair distance away.

The mating season for elephant seals doesn’t begin for a few weeks yet, so that apparently makes these early arrivals juvenile females that won’t actually participate directly.

We explored Pescadero State Beach and the Pigeon Point Lighthouse, but the formation revealed at Pebble Beach between the two was striking. Apparently, the ocean (and already pulled pebbles) knock out other pebbles and slowly widen the sockets.

We went back for another crisp evening and chilly morning at Butano, revisited a few of the sites on Friday, and headed back—with a detour through Gilroy to sample a bit of Black Friday—in time for a dinner at home.

Photos taken with a Panasonic Lumix LX-2 and with a Canon PowerShot S2 IS.

NOAA Still Sees Above Average Temperatures for Most of the U.S. and Below Normal Precipitation Across the South

stephen@blueslugs.com (Stephen Hahn) — Sun, 25 Nov 2007 06:21:22 +0000

NOAA Still Sees Above Average Temperatures for Most of the U.S. and Below Normal Precipitation Across the South

The 2007 winter outlook: a La Nina year, with typical temperatures, drier than average along the California coast.

number 9 : Dvorak on Solaris

stephen@blueslugs.com (Stephen Hahn) — Tue, 20 Nov 2007 16:48:19 +0000

number 9 : Dvorak on Solaris

Rasputnik uses xmodmap to remap to a Dvorak layout.

Don't forget the little people

stephen@blueslugs.com (Stephen Hahn) — Tue, 20 Nov 2007 16:15:21 +0000

I just want to mention that I knew Alan before his current fifteen minutes.

pkg(5): Fueling the next steps

stephen@blueslugs.com (Stephen Hahn) — Mon, 05 Nov 2007 22:11:03 +0000

Most Fridays, I spend the two hours before lunch assisting at our co-operative daycare. The hour and a half before are pretty good thinking hours, most of which lately have been spent on packaging. As the year passes, and we move into fall here in Northern California, the thinking’s been best assisted by having a brief, and warming, snack.

The notes being written are about some of the points raised about the use of hashing, but then I buckled down and wrote an outline for our ARC inception materials, which will probably take more than one or two Friday sessions.

Location: Café Borrone, Menlo Park, California.

[ T: OpenSolaris pkg ]

Indiana: More ways to get it

stephen@blueslugs.com (Stephen Hahn) — Sat, 03 Nov 2007 01:55:39 +0000

So there are a few ways to get a copy of the Indiana preview ISO:

Both our mirrors are in Northern California, so—since it’s a completely redistributable set of software—if you’re interested in being a mirror (even in California), please let me know. If you’re a torrent fan, come join me and seed.

Update 1: My thanks to trisk, who’s put up a fast mirror (Internet 2) on the East Coast of the USA.

Update 2: My thanks to Tobias Lundquist, who’s put up an FTP mirror (Internet 2) in Sweden.

Icon Design by IconBuffet (from “Durango Office” and “Durango Research”).

[ T: OpenSolaris Solaris indiana ]

mozilla.org – Code Review FAQ

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Nov 2007 22:34:40 +0000

mozilla.org – Code Review FAQ

Mozilla FAQ around their code review process, which has a super reviewer much like the RTI advocate model we use.

RFC 3852 Cryptographic Message Syntax (CMS)

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Nov 2007 03:56:28 +0000

RFC 3852 Cryptographic Message Syntax (CMS)

"This document describes the Cryptographic Message Syntax (CMS). This syntax is used to digitally sign, digest, authenticate, or encrypt arbitrary message content."

Indiana: VESA if you need it

stephen@blueslugs.com (Stephen Hahn) — Thu, 01 Nov 2007 05:26:36 +0000

Over the past week, as we kept reassembling the distro constructor, image packaging, and slim install, we tested installs on a bunch of laptops. This manual operation let me rehearse how to get things working if the Preview LiveCD doesn’t have a graphics driver that will work. So: here’s the VESA workaround.

Boot text mode. One of the non-default lines in the GRUB menu should say “text”—pick that one.
Login as jack. User is “jack”, password is “jack”.
Generate a representative xorg.conf. Become root and have Xorg generate an initial configuration. The root password is “opensolaris”. (Enjoy that `root`’s default shell is ‘bash’ for a bit.)
```
$ su
Password:
```

/usr/X11/bin/Xorg -configure

… This procedure should create

`/jack/xorg.conf.new`.

Change the driver. As root, you’ll continue by editing that xorg.conf.new file in vi(1). Search for the Device section, and modify the Driver line to
```
Driver “vesa”
```

Make it go. Stay root and hand-launch GNOME.

# /usr/X11/bin/xinit /usr/bin/dbus-launch gnome-session – 

/usr/X11/bin/Xorg -config /jack/xorg.conf.new :0

Launch the installer. Bring up a terminal, and invoke the installer, by typing install-lan &.

If your install fails, please file a bug report at defect.opensolaris.org. If it succeeds, but graphics doesn’t work, you can follow the same steps, but you’ll have to edit the GRUB entry. (Roughly, type ‘e’, add “-s” to the end of the line with kernel/unix, and log in with the new root password you set during install. I haven’t tested this portion—let me know if it fails.)

Last night, I had to do this for my trusty VAIO T370P, but tonight it’s fine:

Please let us know how it goes.

[ T: OpenSolaris Solaris indiana pkg ]

Dario Taraborelli: The beauty of LaTeX

stephen@blueslugs.com (Stephen Hahn) — Fri, 26 Oct 2007 03:13:06 +0000

Dario Taraborelli: The beauty of LaTeX

Nice summary and short collection of links.

Open Source Unleashed: A pot for Sun's open source potpourri?

stephen@blueslugs.com (Stephen Hahn) — Fri, 19 Oct 2007 21:17:44 +0000

Open Source Unleashed: A pot for Sun’s open source potpourri?

I believe Alex is looking for something more unified or coherent across Sun’s open source communities, but I’m not sure exactly what. (Reread later, maybe?)

pkg(5): Talking in the redwoods, talking on the beach, …

stephen@blueslugs.com (Stephen Hahn) — Sat, 13 Oct 2007 20:55:46 +0000

Danek and I are at the 2007 OpenSolaris Developer Summit at UC Santa Cruz, talking about and, I hope, later demonstrating the image packaging prototype so far. I gave a brief overview of the project’s goals and status, and mentioning some of the unmentionables we’ve encountered—some particularly undisciplined configuration files, some apparently important but encumbered drivers, and so on.

View the slides

[ T: OpenSolaris pkg UCSC opensolaris_summit_2007 ]

Predictable – pkg(5): Talking in the redwoods, talking on the beach, …

stephen@blueslugs.com (Stephen Hahn) — Sat, 13 Oct 2007 19:59:42 +0000

Predictable – pkg(5): Talking in the redwoods, talking on the beach, …

I’ve been at the 2007 OpenSolaris Developer Summit, presenting and discussing where we are in the current image packaging prototype.

"Why Programmers suck at Picking Colors" (Stefano's Linotype)

stephen@blueslugs.com (Stephen Hahn) — Thu, 04 Oct 2007 05:25:22 +0000

“Why Programmers suck at Picking Colors” (Stefano’s Linotype)

A review leading to a NASA colour picker implementing selection based on the Munsell colour system.

Indispensible?

stephen@blueslugs.com (Stephen Hahn) — Tue, 02 Oct 2007 21:47:08 +0000

Last October, the New York Times was considering extreme yachts:

â€œToday, a megayacht is indispensable,â€ said Olivier Milliex, head of yacht finance at the Dutch bank ING. â€œItâ€™s not like 15 years ago, when a yacht was a luxury item.â€ Hmm. I don’t have one, so I think “indispensible” isn’t the right word. Perhaps “megayacht ownership is assumed”?

OpenSolaris New User FAQ – Genunix

stephen@blueslugs.com (Stephen Hahn) — Tue, 02 Oct 2007 16:56:25 +0000

OpenSolaris New User FAQ – Genunix

Brian’s led an effort to produce a FAQ targetted at new users of OpenSolaris. Peruse, contribute, etc.

Brian Marick – Writings

stephen@blueslugs.com (Stephen Hahn) — Mon, 01 Oct 2007 18:58:41 +0000

Brian Marick – Writings

Brian writes on a variety of software development topics. His paper, "How to misuse code coverage", provides a good explanation of what you do and don’t know after a code coverage assessment.

pkg(5): project opens, development continues

stephen@blueslugs.com (Stephen Hahn) — Fri, 28 Sep 2007 00:21:22 +0000

I asked my teammates to take a brief break from prototyping, and we jumped from a collection of systems inside Sun to the clean project hosting at opensolaris.org.

As the migration registers, you should be able to access

the project web site at http://opensolaris.org/os/project/pkg/,

the Mercurial repository via

$ hg clone ssh://anon@hg.opensolaris.org/hg/pkg/gate pkg-gate

the same source via OpenGrok (using the ‘pkg’ project as a filter) at http://src.opensolaris.org/, and
have a discussion on pkg-discuss@opensolaris.org.

If you’re interested, please come check it out.

[ T: OpenSolaris Solaris pkg ]

Image Packaging System at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Fri, 28 Sep 2007 00:13:08 +0000

Image Packaging System at OpenSolaris.org

Project page and links to related resources for development.

Predictable – pkg(1): project opens, development continues

stephen@blueslugs.com (Stephen Hahn) — Fri, 28 Sep 2007 00:12:09 +0000

Predictable – pkg(1): project opens, development continues

Join the development. Or just kibitz.

What's new in Solaris Install? : Sarah's Excellent Adventures in Solaris

stephen@blueslugs.com (Stephen Hahn) — Mon, 24 Sep 2007 18:08:27 +0000

What’s new in Solaris Install? : Sarah’s Excellent Adventures in Solaris

Sarah has the latest screenshots from the Caiman installer project for OpenSolaris.

The Engine Room – Rethinking Patching

stephen@blueslugs.com (Stephen Hahn) — Tue, 11 Sep 2007 18:09:18 +0000

The Engine Room – Rethinking Patching

Bart on the complexity imposed by separating patching from packaging.

Yahoo! UI Library: Rich Text Editor

stephen@blueslugs.com (Stephen Hahn) — Sun, 09 Sep 2007 17:29:05 +0000

Yahoo! UI Library: Rich Text Editor

YUI delivers a JavaScript-based editor suitable for reuse.

pkg(5): a no scripting zone

stephen@blueslugs.com (Stephen Hahn) — Sat, 08 Sep 2007 00:28:45 +0000

In my previous two posts, we examined two packaging system options—installer-specific knowledge and integrated build system—that I believe present costs that exceed their benefits. Here, we will again examine a design choice from a negative perspective: package-associated scripting.

System V packaging is rich with scripting hooks; scripts named checkinstall, preinstall, postinstall, preremove, request, and the class action scripts. Each of these scripts can do anything they like. Scripting, even in a relatively primitive shell, is an open-ended program—opaque to the invoking framework. It’s difficult to catch an incorrect script prior to package publication time, which blocks our intent to prevent propagation of bad package versions. With a more limited set of actions—potentially with that limit enforced or marked—a class of incompletely known resource handling mechanisms can be kept off the most conservative systems.

One goal we have is to preserve or improve the hands-off behaviour associated with package operations. Legacy packaging allows hands-off by imposing a series of tasks on the deploying administrator. The pkgask(1M) tool can enable the deployer to develop a response to the request script; coming up with an appropriate admin(4) can restrict the framework’s built-in interactive queries. (Interaction with signed packages also requires the deployer to modify their pkgadd invocation.) Removing the scripting degree of freedom means that obstacles to hands-off behaviour come solely from an interactive installer or from interactive services acting during system startup.

There's some amusingly egregious violations of the hands-off principle across the space of known packages. Less fun is that these set a bad example for later package developers.

A particularly error-prone aspect of the scripting interface in packaging comes from the variety of contexts the package developer must understand (and test within). It is legitimate to install packages on live systems, in alternate filesystem hierarchies of the same or different architecture, and in whole-root and filesystem inheriting zones; in fact, you have multiple choices about how your package should install in a zone.

We can expect the proliferation of virtualized systems, via the various mechanisms like LDOMs and xVM, to keep all of these contexts relevant as degrees of sharing make virtualization even more appealing. Making sure that the package system operates safely in these shared contexts is critical—another of our goals.

Returning to the zones case, the example pseudo-script in pkginfo(4)—a series of nested shell if ; then blocks to navigate some of these contexts—is helpful, but misleading. There is much more variable state a package developer needs to consider to reach correctness. In fact, if you aren’t required to rediscover or reinvent a set of resource-handling cases for each components your package delivers, it becomes substantially simpler to make the package and return to improving the software it contains. Reducing the set of steps reduces developer burdens associated with packaging.

Two particular resources stand out: device drivers and smf(5) services. Although some limited amount of awareness—or at least easily duplicated code—makes these resources somewhat well-behaved during package operations, there are still problems that scripting presents: the addition of new contexts, the provision of multiple genealogies of copied code, and the failure to discover an associated best practice for any particular kind of resource.

There are other resources, of course; as a start, you could duplicate our survey of the ON postinstall and class action scripts.

I believe the key counterargument supporting scripting is that the set of configuration patterns on Unix-like systems is large, and that the easiest means of upgrading each of these potential patterns is to allow a complete programming environment to the package developer. Probably true, but if we look at service and application configuration with respect to when a correct configuration state is required, the update step appears to separate into three classes:

Correct at system startup, no runtime context needed. These are the configuration settings that the various low-level boot components, the kernel, and the drivers need to bring the system to its running state. This class of configuration is generally limited to a specific set of resources, potentially established by a packaging system via corresponding resource-handling actions—or by an installer.
Correct at system startup, requiring runtime context. These are settings where the manipulating agent might be influenced by policy or require some form of interprocess communication to effect configuration changes. smf(5) is an example of the latter, and handles its configuration evolution via the manifest-import service. Manipulation of the various local name service tables, like passwd or the RBAC configuration is another example, since data about potential principals must be correct for a group of affected services. Since such configuration can be required on the system as a result of package operations, these resources must also be handled via packaging, or require the use of an appropriate installer.
Correct prior to service startup. Most service and application configuration falls into this class. It’s not necessary, for instance, to bring a web server’s configuration up to date if the service has no enabled instances. There seem to be a number of avenues for handling this kind of configuration: leaving it to the service or application, providing assistance via a configuration mechanism, or giving a hook where such updates can be made as needed. But the packaging system needn’t provide this hook—there are a number of possible facilities, of varying suitability.

I should point out that David is making the smf(5) configuration update scenarios much more capable and precise with the Enhanced Profiles project. So, at least, a “configuration mechanism with assistance” is likely to be present soon.

Since the first and second classes and how their configuration manipulations vary in the various operating contexts are generally known, elimination of the third class makes precise, no-scripting packages a viable design choice.

That’s a long series of arguments in favour of a scripting-free package system. It would be reasonable to ask: “can you actually do it?” So, as a check on our prototype, we used the branded zone capability to let us create a pkg(5)-based whole root zone. Here’s a transcript

# zonecfg -z pkg_test
pkg_test: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:pkg_test> create -t SUNWipkg
zonecfg:pkg_test> set zonepath=/export/pkg_test
zonecfg:pkg_test> commit
zonecfg:pkg_test> ^D
# zoneadm -z pkg_test install
Preparing image
Retrieving catalog
Installing SUNWcs SUNWesu SUNWadmr SUNWts SUNWipkg
Setting up SMF profile links
Copying SMF seed repository
Done (115s)

There's dependency following, but no constraint handling; there's no filtering or snapshotting, but also none of the obvious performance optimizations has been implemented (for our 211MB resultant image). But the main point is: it works—installs, boots, upgrades, and still boots—with no scripting. Time for a project proposal.

[ T: OpenSolaris Solaris zone Indiana pkg ]

Predictable – pkg(1): a no scripting zone

stephen@blueslugs.com (Stephen Hahn) — Fri, 07 Sep 2007 23:30:46 +0000

Predictable – pkg(5): a no scripting zone

In which we propose the elimination of open scripting hooks from packaging, and use branded zones to show that it indeed works.

Intel® Software Network Blogs » Blog Archive » Making Xeon the best platform for Solaris

stephen@blueslugs.com (Stephen Hahn) — Thu, 06 Sep 2007 16:25:55 +0000

Intel® Software Network Blogs » Blog Archive » Making Xeon the best platform for Solaris

David Stewart on the real results from Intel’s OpenSolaris participation. Once we finish the SCM migration work, contribution will be even easier.

Predictable – pkg: Leaving the build system "out"

stephen@blueslugs.com (Stephen Hahn) — Fri, 24 Aug 2007 23:13:39 +0000

Predictable – pkg: Leaving the build system “out”

You can layer a build system on top of a packaging system, but you needn’t include it. Having one can make you less nimble.

pkg(5): Leaving the build system "out"

stephen@blueslugs.com (Stephen Hahn) — Fri, 24 Aug 2007 21:37:12 +0000

I’ve been busy the past weeks with school transitions and with getting the community defect tracking system requirements into a shape where we can start evaluating candidates.

Identifying the boundaries of a system during the design phase affects feasibility critically, perhaps more than any other choice. This choice reduces to “know what you’re trying to create”. As with my previous post, I’m going to describe something we’re not doing—and explain why. I envision one more negative post, and then we’ll get onto more positive expressions.

So: this packaging system does not contain a build or compilation component in its core architecture. There are pragmatic reasons for this choice, as well as technical ones.

One way to look at a packaging system is, similar to how we saw it as the connecting layer between the installer and lower-level OS services, as a way to collect and organize the set of components, binary and otherwise, into an always “bootable” flow of change. That is, various groups of people are, using their systems, emitting binaries, documents, images, and so on, that other groups of people combine into releases (or atoms of a release, like an updated package). The packaging system has to assist the latter group in that combination, by providing visibility into the completeness of the outputs of the former groups. It’s less clear that the latter group needs to have the ability to construct the objects, as long as they can assess that the environment the objects need to execute can be realized.

Just as an aside, we’re treating as critical design input exactly who touches and how they touch a software component as it proceeds through development, localization, release engineering, sustaining, and so forth. These touches might be reduced as the development becomes more open, but at present, they’re a useful constraint on any tendency to make the system overly rigid (or overly monolithic).

The OpenSolaris consolidations—and related open source outputs like OpenJDK or OpenOffice, among others—don’t presently share a common build system. In fact, one of the real impacts of Sun’s progression to an open software development culture is that we’re moving from being the originator or primary maintainer of 90% of the software we deliver, to a much more modest percentage—let’s say 30 – 50%. Forcing a unified build system upon all of these disparate products is asserting the need for a long series of difficult conversations over many months.

Instead, let’s defer those discussions, and see if we can get the same benefits while only managing the end outputs of each of the participating (or aggregated) publishers. (A byproduct is that injecting received binaries into the system is the common case, rather than a strange or special workaround.) As we noted in the earlier post, we have goals about safety, developer burden, and stopping “bad stuff” as early as possible.

Static analysis can get most of the dependency cases correct. Binaries, whether they are user applications, libraries, or kernel modules, contain a significant amount of dependency information. It turns out that many scripting languages can be roughly interpreted to determine their module requirements. Similarly, Java class files and JAR files—and even smf(5) manifests—contain a great deal of information that lets us determine the self-consistency of a system.

Of course, a program can evade these dependencies: beyond use of dlopen(3C), it can implement its own linking or overlay mechanism, or simply be implemented in a language unknown to the packaging system. The point is we can drive out most of the inconsistencies that a purely manual dependency statement allows. In fact, we can warn a developer about the incompleteness of their dependency statements, potentially correcting them: adjusting version requests, inserting omissions, even asking about possibly superfluous dependency correctness.

That said, it might be that an ideal build system is lurking out there to be layered atop this system; we’ll leave room for expressions, like stating a build dependency on a certain tool, like lex(1) and yacc(1), say, so that a build system (or systems, if folks can’t close those difficult conversations I mentioned) can benefit from the metadata discovered about each component. (If you’re interested in constructing such a system, we thought a little about requirements for the SFW consolidation.)

[ T: OpenSolaris Solaris Indiana pkg smf ]

Problem Management Tools Summary

stephen@blueslugs.com (Stephen Hahn) — Wed, 22 Aug 2007 22:04:32 +0000

Problem Management Tools Summary

The Software Problem Management summary from comp.software.config-mgmt’s FAQ.

david ascher – » A Facebook Story

stephen@blueslugs.com (Stephen Hahn) — Mon, 20 Aug 2007 18:00:13 +0000

david ascher – » A Facebook Story

ActiveState releases “Up4”, a social availability “anti-calendar” for Facebook. Some notes on F8 API and feedback received.

Liane Praza's Weblog – zone out and speed up your development cycles

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Aug 2007 17:55:35 +0000

Liane Praza’s Weblog – zone out and speed up your development cycles

Modifying userland system components? Use whole root zones.

Joyent (TextDrive) Accelerator Review « Postfully Yours

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Aug 2007 17:52:30 +0000

Joyent (TextDrive) Accelerator Review « Postfully Yours

“Far superior process management and restart support. Rather than working with init scripts, you [write] SMF manifests with start/stop/restart directives …. This system doesn?t just run at startup but also monitors processes during operation. If somet

pkg(5): No more installer magic

stephen@blueslugs.com (Stephen Hahn) — Mon, 06 Aug 2007 23:51:31 +0000

I thought I would continue probing some of the problems that present themselves to any packaging system that might follow the System V packaging. For the next few topics, I’ll phrase the problem in terms of an outcome I believe we want to avoid. Here we discuss aspects of eliminating special metadata from installers.

If you’re familiar with the packaging and installation components involved with Solaris Express, Developer Edition, or any of the OpenSolaris distributions that Sun produces, then it won’t surprise you that there’s a large amount of upgrade specific knowledge in the installation layer. For instance, to upgrade a package installed with Solaris 10 to its currently delivered version in a recent build, there’s the set of files associated with the package—and then there’s the package history, which collects additional information, like files no longer delivered, or the responsibility of another package. The presence of this file, and the absence of a true package upgrade operation in System V packaging, mean that any kind of upgrade requires some form of installer: the operating system installer for upgrading the entire installation; an application-specific installer for upgrading a group of related application packages.

There are a number of other metadata files trapped in the installation layer, but the most important are the metacluster files, which group the package clusters into large installation profiles, and the package clusters themselves, which group sets of packages along feature boundaries, approximately. It seems evident that these groupings are merely another kind of dependency, much like how a System V package can, naïvely state a dependency on any other System V package.

The System V dependencies also show that another critical piece of metadata—the versioning vector, or “arrow of time”—is encoded in the installer.

All of this information, if we are to allow packages to upgrade from one version to another in some linear fashion, needs to be pulled out of the installers and moved into some aspect of the packaging system. This change in responsibility means that the role of the installer becomes more precise: it must prepare a location for software installation, optionally lay down some initial, and possibly stale image, and collect any required configuration information. Subsequent upgrade operations are driven primarily by the packaging system, which can utilize the version history and dependencies in a manner at least equivalent to what the historical metadata allows.

The other reason that we wish to push that historical metadata into the packaging system is so that it becomes accessible to a new class of application: the distro construction toolkit, which needs the dependency and versioning information to simplify the construction of self-consistent installable images. That leads us to an architectural diagram like

where we’ve suggested some internal structure to an updated packaging system. This separation shouldn’t be that surprising: in the current system, lumake(1M) and most of LiveUpgrade as well as zoneadm(1M) for Zones are both performing image operations on either side of any packaging operations they might invoke. Designing a packaging system that makes the constructions of distributions and their installers substantially simpler will require such an API layer.

There are some open questions that come to mind:

How much variation is there in the initial preparation an installer offers for the image location? Is space management a policy owned exclusively by the installer, or should the image operations layer have the ability to influence that allocation?
Is there a marshallable image description, beyond the package dependencies? Is there a marshallable image format, or are we restricted to end-use formats, like ISO images or archived zones?
What OS services should we consider modifying, beyond taking advantage of ZFS’s current capabilities?

If you’re interested in these questions, or the potential architecture of a packaging system, we’ll be discussing these topics further in the Install community group.

[ T: OpenSolaris Solaris Indiana pkg ]

Predictable – pkg: No more installer magic

stephen@blueslugs.com (Stephen Hahn) — Mon, 06 Aug 2007 22:53:08 +0000

Predictable – pkg: No more installer magic

On moving magic knowledge out of the installer layer, and back into the packaging system.

Predictable – Observations on packaging

stephen@blueslugs.com (Stephen Hahn) — Tue, 24 Jul 2007 23:30:09 +0000

Predictable – Observations on packaging

Trying to summarize what a new packaging system (for OpenSolaris, I hope) should aspire to.

BBC NEWS | Special Reports | Floods: At-a-glance

stephen@blueslugs.com (Stephen Hahn) — Tue, 24 Jul 2007 16:21:53 +0000

BBC NEWS | Special Reports | Floods: At-a-glance

Large chunks of England are underwater.

Observations on packaging

stephen@blueslugs.com (Stephen Hahn) — Thu, 19 Jul 2007 20:58:19 +0000

Over the past few months, a bunch of us have been exploring various options for packaging. (Actually, I suppose I’ve been pursuing this on a part-time basis for a year now, but it’s only recently that it’s made it to the top of the stack.) I’ve looked at a bunch of packaging systems, ported a few to OpenSolaris, run a bunch on their native platforms, and read a slew of manual pages, FAQs, blogs, and the like. In parallel with those efforts, Bart, Sanjay, and others have been analyzing the complexity around patching in the Solaris 10 patch stream, and improving the toolset to address the forms of risk that the larger features in the update releases presented.

In the course of those investigations, we’ve come up with a number of different approaches to understanding requirements and possibilities; I’ll probably write those out more fully in a proper design document, but I thought it would be helpful to outline some of those constraints here. For instance, one way to look at how we might improve packaging is to separate the list of “design inputs” for a packaging system into “old” and “new”.

When I make a list of this kind, I know it's bound to offend. (I already know it's more scattershot than the argument we'll present in a design document.) Feel free to send me the important inputs I've omitted, as I have a few follow-up posts on requirements—lower level, more specific intentions—and architectural thoughts where I can cover any issues not mentioned here.

“Old” inputs

The “old” inputs are those that are derived from current parts of the feature set around software packaging and installation. Many of these inputs will really be satisfied by the new installer work (“Caiman”) Dave‘s leading, but the capabilities of the packaging system will change some from difficult and fragile to straightforward and robust. With effort, some might even achieve some elegance.

Hands off. For a long time, via the JumpStart facility, it’s been easy to provision systems via scripted installed. This technology is still sound, particularly when we consider that the Flash archives allow a mix of image-based provisioning with the customizations JumpStart offered.
Virtualized systems. As long or longer, we’ve supported the notion of diskless systems, where the installed image is shared out, in portions, between multiple running environments. Zones was a direct evolutionary successor of the diskless environments, and shows that this approach can lead to some very economical deployment models. Lower-level virtualization methods can also benefit from the administrative precision that comes out of sharing known portions of the installation image.
Availability and liveness. With Live Upgrade, it’s been possible for a while now to update a second image on a running system while the first image is active—a command to switch images and a reboot is all that’s required to be running the new software. This approach requires too much feature-specific knowledge at present, but provides a very safe approach to installing an upgraded kernel or larger software stack, as reverting to the previous version is just a matter of switching back to the previous image. So, a package installation option that doesn’t ruin a working system is a must.
Change control. In principle, with the current set of package and patch operations, it is possible to create a very specific configuration—that may never have been tested at any other site, mind you—from the issued stream of updates. From a service perspective, the variety of possible configuration combinations is too broad, but the ability to precisely know your configuration and make it a fixed point remains important.
Migration and compatibility. There’s a very large library of software packaged in the System V format that will run on most OpenSolaris-based systems. Providing well-defined compatibility and migration paths is an obvious constraint, given the goals of the development process.
Networked behaviour. Believe it or not, the current installer and packaging support a large set of network-based operations—you can install specific packages and systems over the network, if you know where everything is. That is, the current components need to be assembled into some locally relevant architecture by each site to be useful—any replacement needs to make this assembly unnecessary, potentially via a number of mechanisms, but definitely via a list of known (trusted?) public package repositories.

“New” inputs

But, as you might expect, the efforts around Approachability, Modernization, and Indiana have brought to light some new qualities a packaging system must possess.

Safety. One of the real complications from virtualized systems, at least in current packaging, is that a developer has to understand each of the image types his or her package might reach, and make sure that the set of pre- and post-install scripts, class actions scripts, and the like are correct for each of these cases. When that doesn’t happen, the package is at a minimum incorrectly installed; in certain real cases, this class of failures compromises other data on the system. Restrictions in this space, particularly during package operations on an inert image, seem like a promising trade-off to achieve greater safety.
Developer burden. Current packaging requires the developer provide a description of the package, its dependencies, and contents across a minimum of three files, in addition to knowing a collection of rules and guidelines around files, directories, and package boundaries. Most of these conventions should be encoded and enforced by the package creation and installation tools themselves, and it should be possible to construct a package with only a single file—or from an existing package or archive.
smf(5) aware. OpenSolaris-based systems have an understanding of services and service interdependencies, derived from the Service Management Facility’s service descriptions. smf(5) service management during packaging operations is awkward under current packaging and very much needs to be improved but, more importantly, the service graph provides a rich set of relationships among larger components that should lead to a better understanding of the rules around consistent system assembly.
Minimization. For a while, the current system has had some very coarse package “blocks”, from which one could construct a system—the various metaclusters. These, with the exception of the minimally required metacluster and the entire metacluster, split the system along boundaries of interest only to workstation installs. Any suitable packaging system must provide query and image management tools to make image construction more flexible and less error-prone (and eliminate the need for things like a “developer” metacluster, for that matter). It’s also pretty clear that the package boundaries aren’t optimized in any fashion, as evidenced by the differing rates of change of the binaries they currently enclose—in the form of the issued patches against Solaris 10.
Multiple streams of change, of a single kind. Although we noted the continued need to control change above, it’s also important to be able to subscribe to a stream of change consistent with one’s expectations around system stability. The package system needs to allow one to bind to one or more streams of change, and limit how the interfaces the aggregate binaries from those streams evolve. That is, it should be possible to subscribe to a stream of only the most important updates, like security and data corruption fixes, or to the development stream, so that one’s system changes in a way consistent with one’s expectations. Conversely, the tradeoff between complexity and space optimization in current patches—which introduce a separate and difficult-to-calculate dependency graph and distinct namespace entries for each platform, among other issues—has slid much too far, given the increase in system richness and the increases in disk space and bandwidth. There seems to be little long-term benefit in preserving the current patch mechanism, particularly since Sun never offered it in a form useful outside of Sun’s own software.
ZFS aligned. ZFS offers the administrator access to so many sophisticated options around deployment and data management that it would be foolish for a packaging system to not explore how to take advantage of them—zfs snapshot and zfs clone are the most obvious capabilities that allow the packaging system to limit the risk of its modifications to an image, without excessive consumption of storage space or system time.
Prevent bad stuff early. Another classic OpenSolaris-style requirement is that the set of available packages be known to be self-consistent and correct at all times, to spare the propagation of incomplete configurations. In a packaging system, this input expands on our intent to reduce the developer burden to assist the developer in writing as correct a package as possible, and to enable the repository operator to block incomplete packages from being made available to clients. There’s a rich set of data here, as we noted for smf(5) service descriptions above.
Friendly user deployment. Direct from Indiana, but sensible and apparent to all is that packaging systems have advanced to a point where the usability of the client is expected, and not an innovation. I haven’t got the complete timeline of packaging developments—the literature survey continues—but it’s clear that Debian’s apt system marks the current expectations about the client-side capability and ease-of-use.

In the course of the Indiana discussions, Bryan raised one point, which I’ll paraphrase as “it’s not an idea until there’s code”. That’s a sound position, which I also happen to hold—Danek and I (and Bart and Daniel, I hope) and have been quietly prototyping for a little while now, to see if we had a handle on this collection of inputs. I’d like to give a bit more background, in the form of requirements and assertions, in an upcoming post or two. Then we’re hoping to start a project on opensolaris.org to take the idea all the way from notional fancy to functioning code.

[ T: OpenSolaris Solaris smf Indiana pkg ]

Sebastien Pierre’s Mercurial Quick Reference Card

stephen@blueslugs.com (Stephen Hahn) — Fri, 06 Jul 2007 17:47:15 +0000

Sebastien Pierre’s Mercurial Quick Reference Card

Nicely designed quick references to Mercurial’s command set and workflow.

Weekday Caltrain Schedule for j2me java-enabled cell phone.

stephen@blueslugs.com (Stephen Hahn) — Sat, 23 Jun 2007 20:30:53 +0000

Weekday Caltrain Schedule for j2me java-enabled cell phone.

A midlet with a basic query interface to the Caltrain weekday schedule. Simple, but sufficient.

[0706.2748] A Survey of Unix Init Schemes

stephen@blueslugs.com (Stephen Hahn) — Wed, 20 Jun 2007 14:12:01 +0000

[0706.2748] A Survey of Unix Init Schemes

Capsule descriptions of various *ix system startup approaches. Some inaccuracies and curious assertions. (SMF has been available as open source since June of 2005.)

:: Pardus :: TÜB?TAK/UEKAE ::

stephen@blueslugs.com (Stephen Hahn) — Wed, 20 Jun 2007 00:20:23 +0000

:: Pardus :: TÜB?TAK/UEKAE ::

A history of the construction of a Linux distribution, with configuration and package management tools written in Python.

Fore! – Live Media installer

stephen@blueslugs.com (Stephen Hahn) — Tue, 19 Jun 2007 17:53:17 +0000

Fore! – Live Media installer

Dave gives us an update on the LiveMedia work in OpenSolaris. Summary is 15 minutes for a 2GB install to ZFS.

Nature’s guide for mentors

stephen@blueslugs.com (Stephen Hahn) — Mon, 18 Jun 2007 18:55:03 +0000

Nature’s guide for mentors.

Nice article on mentor characteristics and practices.

OpenSolaris: Picnic, California Edition

stephen@blueslugs.com (Stephen Hahn) — Fri, 15 Jun 2007 21:55:54 +0000

OpenSolaris just turned two—enjoy some good food and good company in the Northern California sun. Tomorrow’s first ever OpenSolaris picnic [Upcoming] will be at <a href=http://sunnyvale.ca.gov/Departments/Parks+and+Recreation/Parks/Baylands/">Baylands Park in Sunnyvale, starting at 11:00 a.m..

Tamara and Ben will, of course, be superlative hosts. See you there!

[ T: OpenSolaris picnic ]

Patch Check Advanced

stephen@blueslugs.com (Stephen Hahn) — Fri, 15 Jun 2007 20:22:58 +0000

Patch Check Advanced

Martin Paul’s utility for analyzing the Solaris patch stream.

Alan Coopersmith's Weblog: OpenSolaris Second Anniversary

stephen@blueslugs.com (Stephen Hahn) — Fri, 15 Jun 2007 20:21:12 +0000

Alan Coopersmith’s Weblog: OpenSolaris Second Anniversary

Alan produces the cat.

Tim Foster's Weblog: LOL ANNIVERSARY!

stephen@blueslugs.com (Stephen Hahn) — Thu, 14 Jun 2007 22:31:31 +0000

Tim Foster’s Weblog: LOL ANNIVERSARY!

Tim says happy birthday in a lolcat dialect. (Where is ceiling cat?)

Jim Grisanzio: OpenSolaris at Two

stephen@blueslugs.com (Stephen Hahn) — Thu, 14 Jun 2007 22:30:01 +0000

Jim Grisanzio: OpenSolaris at Two

Jim summarizes the past year, with an extensive set of notable events and links. If you’re new to the OpenSolaris community, this post is a great way to get caught up quickly.

Paludis, the Other Package Mangler

stephen@blueslugs.com (Stephen Hahn) — Wed, 13 Jun 2007 00:41:54 +0000

Paludis, the Other Package Mangler

A new package manager for use with Gentoo-style distributions. Maintains Portage compatibility.

Deputy

stephen@blueslugs.com (Stephen Hahn) — Sat, 09 Jun 2007 05:44:34 +0000

Deputy

C compiler with safety analysis via annotations.

Half a week

stephen@blueslugs.com (Stephen Hahn) — Sat, 09 Jun 2007 01:39:25 +0000

I was out half this week, helping my younger son deal with a nasty sinus infection—and then an allergic reaction to the prescribed antibiotic. So, between finishing various deliverables, I’ve been trying to catch up on my opensolaris.org mailing list traffic. This afternoon, I got to ogb-discuss, the Governing Board alias, which has been exploring the next steps for the Architecture Review process for open development.

A few minutes ago, I replied to one message in the thread, and then reread the message was from Tuesday. Given that I was out three days, this makes quasi-arithmetical sense, as

Friday – ~2.5 days = Tuesday Maybe in an hour or two, I’ll be able to prove “Monday = Monday”, give or take.

[ T: OpenSolaris OGB ARC ]

Jargon: accelerant versus catalyst

stephen@blueslugs.com (Stephen Hahn) — Fri, 08 Jun 2007 20:42:39 +0000

One of the features of our home library is a sizeable collection of texts on English diction, grammar, usage, and so forth. (The collection was banished to the back office, but bursts out regularly.) Most of these are focussed on general use, and shy away from the changes in meaning that might accompany use in a technical field. Completely untouched is the smaller subset of technical words that could be useful more generally.

The “wished for general use” word-pair I came up with today is catalyst and accelerant, both of which are precise descriptive words for agents of change. The problem is that a catalyst remains unchanged across the transformation—meaning that there was only unilateral adjustment—while an accelerant, very much a part of the transformation, seems to be exclusively connoted with arson.

Know any example usages of accelerant that attempt to stifle a sense of ignition?

OpenSolaris: Five updates conservative developers should make

stephen@blueslugs.com (Stephen Hahn) — Thu, 07 Jun 2007 01:17:06 +0000

It’s been almost two-and-a-half years since Solaris 10 was released, and if we look at Nevada (via Developer Edition or one of the other distributions), we can see that many of the technologies introduced in S10 are becoming still more capable. At this point, even the most conservative software developer can assume that certain features are always present. So, for the conservative OpenSolaris application developer, here are the five low-risk, high-reward updates you should make to your application:

Provide x86 and SPARC versions. OpenSolaris has two primary instruction set targets, i386 and sparc. Each of these has both a 32- and a 64-bit variant. The metrics on Solaris 10 and SX:CE/DE downloads tell us that the Solaris volume is substantial on both targets so, for maximum uptake, you should attempt to offer software on both.

On x86, you should consider delivering both 32- and 64-bit versions, if your application can take advantage of a 64-bit address space. But there is a large contingent of 32-bit only users, so don&#8217;t stop delivering appropriate binaries prematurely.
Of course, if you&#8217;re writing at a hardware-independent level, like on a Java language platform, then you get x86 and SPARC (and presumably others) for free.

Make packages that deliver into sparse zones. The primary software delivery mechanism is still System V packages—but your software’s already packaged properly, so that’s not an issue. (Right?) With Solaris 10, the Zones feature offers a sparse variant that requires package support. Roughly, this support means that the package author shouldn’t deliver into /usr and should add the three properties needed to the pkginfo file.

There are some fairly serious Zones deployers out there; <a href="http://www.joyent.com" >Joyent</a> is probably the most public example, but there are plenty of corporate datacentres using Zones to their advantage. If you want your software run by them or their customers, providing a Zones-compatible package seems like the easiest way to get it into their hands and onto their Zones.

Replace your init.d scripts with smf(5) manifests. The Service Management Facility (smf(5)) provides a collection of capabilities that make service administration easier, while also reducing the development burden on application authors. Converting your service from the rc*.d script to a service description and methods means that administrators get automatic restart (and higher service availability), an easy on/off switch, and a place to make site-specific annotations (using the various template properties). There’s a free comptetive advantage here, if your service runs under smf(5) and a rival’s doesn’t.

Of course, you can do more: placing key configuration values in the service repository means that various administrative utilities can be taught to make manipulating your application&#8217;s feature set easy to the deploying administrator. But that won&#8217;t happen without an initial service conversion.
(Once you write a manifest for your service, you&#8217;ll also probably want to write a rights profile, so that administrative authority for your service and its instances can be easily delegated.)

Understand needed privileges. One of the more interesting features in Solaris 10 and later is the work Casper did to split out the absolute privilege owned by root into a specific collection of privileges. That means that you can take away a process’s ability to fork or exec, change file ownership, or manipulate or utilize various subsystems of the operating system. If your application runs with the minimal set of privileges it needs to function, then the set of actions a hypothetical exploit against your application can invoke becomes limited, which reduces the impact of an intrusion. You can reduce your privileges via the smf(5) manifest you wrote for #3, via the role-based access control (RBAC) configuration, or via the privileges API.
Don’t unnecessarily ship duplicate components. The various OpenSolaris distributions include a lot of software; most of these offer one or more update mechanisms for the components they include. Whether or not you prefer minimal patches to wholesale package replacements, if you ship a duplicate component, it’s your responsiblity to update it if a defect or security hole is found. Sometimes you have to ship a component—the distros don’t update it often enough—but private libraries (or private copies of the Java runtime) have a collection of costs, many of which are imposed on your customer.

For specific kinds of software, there’s more to investigate. Language interpreters and byte-code virtual machines (and probably complex daemons) should have DTrace providers. Network device drivers should write to the latest version of the generic LAN device (gld) interface. Archival programs should be ZFS-compatible—there’s going to be a lot of data on ZFS. Daemons should investigate using libumem for memory allocation (and event ports in place of poll(2) or select(3C)). And so on.

There are OpenSolaris communities for each of these topics but, if you’re having trouble getting started, I would suggest an email to opensolaris-code, that reads something like: “I have a [daemon/driver/app] that does [practical purpose/amazing feat/curious entertainment]. Are there any OpenSolaris features I can use to make it better?”

Looking forward to your mail.

Thanks to Dave for #5. Dave also confesses to being keen on #3.

[ T: OpenSolaris Solaris smf privilege RBAC zones ]

SFW: Integrating coreutils and which variants

stephen@blueslugs.com (Stephen Hahn) — Tue, 05 Jun 2007 23:47:41 +0000

Last night, I finished up another task, in an attempt to reduce my current multitasking factor: I integrated initial versions of coreutils and which from the GNU Project into the Freeware consolidation. As a lower priority task, it took longer than a more dedicated developer might have managed, but it’s reasonably pleasing to look back:

initial idea, in case form, proposed and repeatedly revised on companion-discuss,

draft fast track circulated on sfwnv-discuss,

fast tracks for /usr/gnu, coreutils, and which submitted as open PSARC fast track cases,
preliminary, intermediate, and final code reviews on sfwnv-discuss, and,
finally, an integration message to sfwnv-discuss.

There’s still a bunch of process associated with SFW that requires redesign—legal review and Section 508 compliance, in particular—but I think, barring the latent intervals, this sequence was a reasonable consensus-driven open development experience.

If you look at other opensolaris.org mailing lists during June – November and February – April, you’ll be able to verify that I was indeed working—just on other things, and not just surfing the Web…

Of course, now that I know that these commands will start to show up more widely when Build 67 is released, I can update my dotfiles, so I get the versions I prefer:

$ svn diff

Index: sh-functions


--- sh-functions (revision 91)
+++ sh-functions (working copy)
@@ -36,6 +36,10 @@
PATH=$HOME/bin:$HOME/bin/$(/usr/bin/uname -p):$PATH
MANPATH=$HOME/man:$MANPATH
;;
+ gnu) # PREPEND: Bundled GNU command variants
+ PATH=/usr/gnu/bin:$PATH
+ MANPATH=/usr/gnu/share/man:$PATH
+ ;;


Index: bashrc


--- bashrc (revision 91)
+++ bashrc (working copy)
@@ -49,8 +49,12 @@
path clear home sfw csw
fi
-if hash gls > /dev/null 2>&1; then
- alias ls="gls --color -CF"
+if [ -x /usr/gnu/bin/ls ]; then
+ alias ls="/usr/gnu/bin/ls --color -CF"
fi
+if [ -x /usr/gnu/bin/which ]; then
+ alias which="/usr/gnu/bin/which"
+fi
+
export CDPATH=$MACHINE_CDPATH_PRE:$HOME/projects:$HOME:$MACHINE_CDPATH_POST

If you’re using a distribution that offers SUNWgnu-coreutils, SUNWgnu-which, and the other /usr/gnu packages, do share your feedback with the maintainers on sfwnv-discuss—or become one and pick your favourite package.

[ T: OpenSolaris Solaris SFW coreutils which ]

Download Opera Web Browser [Solaris x86]

stephen@blueslugs.com (Stephen Hahn) — Sat, 02 Jun 2007 05:02:50 +0000

Download Opera Web Browser [Solaris x86]

Opera 9.x now available for OpenSolaris on x86 processors.

How to make a good ID in Atom [dive into mark]

stephen@blueslugs.com (Stephen Hahn) — Wed, 30 May 2007 00:34:03 +0000

How to make a good ID in Atom [dive into mark]

Useful guidance on how to construct the unique, unchanging IDs the Atom protocol requires for entries.

maxq.tigris.org

stephen@blueslugs.com (Stephen Hahn) — Fri, 25 May 2007 04:07:50 +0000

maxq.tigris.org

A recording-playback system for web functional testing. Emits a Python script.

Staring At The C – Python and DTrace in build 65

stephen@blueslugs.com (Stephen Hahn) — Thu, 24 May 2007 20:30:50 +0000

Staring At The C – Python and DTrace in build 65

John’s integrated DTrace support for Python in OpenSolaris, build 65. The python provider has function-entry, function-return, and userland stack support. (Examples are from executing various Mercurial commands.)

OpenSolaris: Defect tracking relationships

stephen@blueslugs.com (Stephen Hahn) — Thu, 24 May 2007 05:42:30 +0000

Just prior to the Board election, we ran a poll of the core contributors to get some sense of what one active subset felt were the five most pressing obstacles to open development. Dan just issued an initial Beta of a webrev-based approach, derived from his earlier experiment on http://cr.grommit.com, so that’s a starting point for Priority 3. The Board is tackling, in full public view on ogb-discuss, Priority 4: there’s OGB/2007/001, Project creation enhancements and OGB/2007/002, Community and Project Reorganization as two significant chunks of a stablized reorganization. Priority 2, the deployment of a “request to integrate” system, is somewhat gated on ON and sister consolidations’ switch to Mercurial, being pursued in the scm-migration project—it’s an aspect of workflow that isn’t required by all of the hosted consolidations. Priority 5, the deployment of an opensolaris.org-hosted wiki, is in a requirements gathering phase over on website-discuss.

That leaves Priority 1, the deployment of a public bug tracking system. Bug tracking has loomed over the OpenSolaris effort for pretty much its entire implementation phase; we’ve known that aspects of the current bug tracking methodologies impact many parts of Sun’s business, and that any solution will require the identification of which entanglements are strategic—meaning that there’s a requirement for any new system—and which are accidental—meaning that there’s only some transition cost, as the entangled system can be adjusted to consume information in some designed fashion. So, as part of getting a set of draft requirements together for discussion, I thought I would work through some of the issues facing defect tracking as we move to open development. Most of these points are about the developer–distribution (or upstream–downstream) interface that a defect tracking system (DTS) or defect tracking architecture represents.

The primary requirement that a distribution has on their DTS is some ability to maintain confidential data associated with each tracked defect. Let’s call that database a proprietary annotation system (PAS)—the data within it capture the customer histories associated with various defects or collections of defects (“products”?) represented in the system. The DTS, meanwhile, is meant to be neutral across all participants, developers and distribution assemblers, and unconcerned with non-technical characteristics of the defect.

This contrast allows us to postulate a set of relationships among various active DTSs and PASs for an open development community.

The association of confidential information with an existing defect looks something like

Of course, in the OpenSolaris case, SMI’s annotations become just one potential PAS; other distributions may also choose to annotate publicly known (“community tracked”) defects:

One requirement that we’ve heard during preliminary discussions with the various teams is there must be some ability to search the entirety of the product-relevant portion of the DTS. One possibility is that each PAS operator builds a search corpus that combines the upstream DTS with the PAS content. A potentially more economic alternative would be to allow the the associations to be bidirectional (so that an indexer with authorizations allowing it to access one or more proprietary annotation systems can present a complete defect corpus). Making the existence of the annotations public does not seem like a significant leak of proprietary information, while the existence of annotations might be a useful measure of defect significance. (It is probably worth explicitly stating that having a complete defect corpus for searching does not imply that use of a single DTS is the only means of obtaining such a corpus.)

These associations are more complex objects than the current See Also links in typical monolithic DTSes, in that they carry one or more mappings of status and responsibility between the DTS and each PAS. Potentially, this capability could lead to more precise handling of release readiness, in that a query involving a group of PAS-tracked defects could indicate that one or more stoppers are blocking the release of a certain version of the tracked product. Of course, prior to open development, the Solaris organization has historically managed that fairly well for its products, so why is it worth discussing?

Well, for OpenSolaris today, the set of defects in our own DTS isn’t the complete set of relevant defects for product release. In fact, as examples, Solaris tracks the defects, branches, and patches for GNOME, Mozilla Firefox, and Perl, among others. That is, Sun’s current combined DTS/PAS, Bugster, is in the following relationships with an upstream source

when the product is affected by an upstream-sourced component, and

when a customer is affected by an upstream-sourced component. There’s also an awkward local cost since upstream state is usually tracked manually (or possibly via custom scripting).

We can choose to apply this tracking need to our second defect/annotation figure above, in two ways: we can track the OpenSolaris consolidation defects as a peer of other upstream DTSs

or, more interestingly, we can allow community management of the relevance of upstream defects, like the following

(A distribution may not choose to annotate the public DTS record on an upstream bug, but may instead choose to annotate on the upstream DTS record directly. Ignoring the community signal when it has a mismatch with distribution priorities seems likely; using it as a guide when no conflicting principle exists seems like a safe course as well.)

I think that introducing these kinds of associations allows us to solve a large class of defect tracking problems that are currently impacting the OpenSolaris space. (Obviously there are issues, too–for instance, one could introduce association cycles (that clients must detect).) I would expect that the confidentiality issue impacts all of the distributions pulling from OpenSolaris consolidations (and probably, more generally, all groups looking at open development): I believe every distribution has customers of some kind. The importance of upstream relationships may presently be operating environment-specific, although I know other groups are also dependent on some number of OSS components.

Real-World Haskell

stephen@blueslugs.com (Stephen Hahn) — Wed, 23 May 2007 16:08:52 +0000

Real-World Haskell

Bryan and co-authors begin a book for O’Reilly.

The Blog of Ben Rockwood – IO Benchmarking: How, Why and With What

stephen@blueslugs.com (Stephen Hahn) — Wed, 23 May 2007 16:07:35 +0000

The Blog of Ben Rockwood – IO Benchmarking: How, Why and With What

Ben surveys a set of tools for evaluating aspects of I/O performance.

Particletree » Subdomains + Development = Sucks

stephen@blueslugs.com (Stephen Hahn) — Wed, 23 May 2007 16:06:23 +0000

Particletree » Subdomains + Development = Sucks

Wufoo’s per-user domain experience shows the costs and complexity that accompany vanity subdomains.

InotifyExtension – Mercurial

stephen@blueslugs.com (Stephen Hahn) — Thu, 17 May 2007 03:28:34 +0000

InotifyExtension – Mercurial

Interesting application of Linux inotify API to SCM status.

RFC1297 NOC Internal Integrated Trouble Ticket System Functional Specification Wishlist ("NOC TT REQUIREMENTS")

stephen@blueslugs.com (Stephen Hahn) — Sat, 05 May 2007 03:33:50 +0000

RFC1297 NOC Internal Integrated Trouble Ticket System Functional Specification Wishlist ("NOC TT REQUIREMENTS")

An informal set of requirements for a network operations organization’s ticketing system.

Joe Gregorio | BitWorking | How to create a REST Protocol

stephen@blueslugs.com (Stephen Hahn) — Wed, 02 May 2007 17:17:11 +0000

Joe Gregorio | BitWorking | How to create a REST Protocol

One procedure for designing a REST-style protocol; reposted from a 2004 article for XML.com.

Modules — Software Environment Management

stephen@blueslugs.com (Stephen Hahn) — Mon, 30 Apr 2007 18:11:50 +0000

Modules — Software Environment Management

Manages $PATH and other environment variables to allow different "setting stacks" to be constructed.

WOVOdat Design Document: The Schema, Table Descriptions, and Create Table Statements for the Database of Worldwide Volcanic Unrest (WOVOdat Version 1.0)

stephen@blueslugs.com (Stephen Hahn) — Thu, 26 Apr 2007 21:22:18 +0000

WOVOdat Design Document: The Schema, Table Descriptions, and Create Table Statements for the Database of Worldwide Volcanic Unrest (WOVOdat Version 1.0)

Dina’s schema for storing volcanic event history, as recorded by observers and sensors. (USGS Open File Report 2007-1117.)

[Phoronix] Phoronix To Support Solaris OS

stephen@blueslugs.com (Stephen Hahn) — Tue, 24 Apr 2007 01:28:50 +0000

[Phoronix] Phoronix To Support Solaris OS

Phoronix, an OSS hardware review site, begins covering Solaris as one of its standard review platforms.

Universal Feed Parser

stephen@blueslugs.com (Stephen Hahn) — Sat, 14 Apr 2007 00:07:54 +0000

Universal Feed Parser

"Parse RSS and Atom feeds in Python. 3000 unit tests. Open source."

baetle – Bug And Enhancement Tracking LanguagE

stephen@blueslugs.com (Stephen Hahn) — Fri, 13 Apr 2007 23:58:23 +0000

baetle – Bug And Enhancement Tracking LanguagE

Henry Story and collaborators are developing a description of the common attributes of bugs, as mapped in RDF

DamageControl – Continuous Integration Server Feature Matrix

stephen@blueslugs.com (Stephen Hahn) — Fri, 13 Apr 2007 23:54:22 +0000

DamageControl – Continuous Integration Server Feature Matrix

A comparison of a set of continuous integration systems. (DamageControl itself is not under active development.)

hudson: Hudson: an extensible continuous integration engine

stephen@blueslugs.com (Stephen Hahn) — Fri, 13 Apr 2007 23:52:23 +0000

hudson: Hudson: an extensible continuous integration engine

"Hudson monitors executions of repeated jobs, such as building a software project or jobs run by cron." Provided as a WAR; just drop in servlet container (or run directly with java(1)).

CruiseControl Home

stephen@blueslugs.com (Stephen Hahn) — Fri, 13 Apr 2007 23:50:22 +0000

CruiseControl Home

"A ramework for a continuous build process." Plugins for Ant, various SCMs.

Apache Maven Project

stephen@blueslugs.com (Stephen Hahn) — Fri, 13 Apr 2007 23:49:28 +0000

Apache Maven Project

"A software project management and comprehension tool." Can manage builds, reporting, and documentation.

Panoramic Feedback®: Open Source Software – Bugs Everywhere

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Apr 2007 17:48:04 +0000

Panoramic Feedback®: Open Source Software – Bugs Everywhere

Bug tracking tied to distributed SCM: bug data is stored beside source code.

JTrac – open source and customizable issue-tracking in Java

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Apr 2007 17:44:42 +0000

JTrac – open source and customizable issue-tracking in Java

Implemented in Java, atop the Wicket framework.

Malone Bug Tracker in Launchpad

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Apr 2007 17:43:49 +0000

Malone Bug Tracker in Launchpad

The bug tracking component for the Launchpad suite. Likely implemented in Python; no source downloads?

Roundup Issue Tracker

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Apr 2007 17:37:14 +0000

Roundup Issue Tracker

Based on the "Track" category winner from the Software Carpentry contest. Implemented in Python.

Comparison of issue tracking systems – Wikipedia, the free encyclopedia

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Apr 2007 17:35:26 +0000

Comparison of issue tracking systems – Wikipedia, the free encyclopedia

A matrix comparing issue and bug tracking systems. Labelled as requiring an update.

Home :: Bugzilla :: bugzilla.org

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Apr 2007 17:33:02 +0000

Home :: Bugzilla :: bugzilla.org

Mozilla’s bugtracking system. Implemented in Perl.

Mantis Bug Tracker

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Apr 2007 17:32:09 +0000

Mantis Bug Tracker

A web-based bug tracking system. Written for PHP/MySQL, can work with PostgreSQL.

scarab.tigris.org

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Apr 2007 17:31:15 +0000

scarab.tigris.org

An "artifact tracking system", capable of tracking defects and RFEs. Uses Java Servlets.

Pacman – a simple package manager

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Apr 2007 02:32:42 +0000

Pacman – a simple package manager

Arch Linux distribution’s packaging system.

libMicro – portable microbenchmarks at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Mon, 09 Apr 2007 20:45:39 +0000

libMicro – portable microbenchmarks at OpenSolaris.org

Dan has made Bart and Phil’s microbenchmark collection available via Mercurial. Consider porting to your favourite POSIX-like platform.

SMF Documentation at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Mon, 09 Apr 2007 20:44:37 +0000

SMF Documentation at OpenSolaris.org

Liane opens a Mercurial repository to start collaboration on smf(5)-related documentation.

GoboLinux – the alternative Linux distribution

stephen@blueslugs.com (Stephen Hahn) — Fri, 06 Apr 2007 02:45:53 +0000

GoboLinux – the alternative Linux distribution

"You don’t a package manager because the filesystem is the package manager: each program resides in its own directory…"

Nix

stephen@blueslugs.com (Stephen Hahn) — Mon, 02 Apr 2007 01:10:22 +0000

Nix

A purely functional (in the PL sense) package manager.

Comparing Linux/UNIX Binary Package Formats

stephen@blueslugs.com (Stephen Hahn) — Sat, 31 Mar 2007 05:31:09 +0000

Comparing Linux/UNIX Binary Package Formats

A comparative feature matrix among various package formats.

Jim Grisanzio: The OpenSolaris Election

stephen@blueslugs.com (Stephen Hahn) — Thu, 29 Mar 2007 18:55:57 +0000

Jim Grisanzio: The OpenSolaris Election

Jim’s congratulations to the incoming Board and to the Community for voting to ratify.

Glynn Foster's Blog: OpenSolaris – Where Next?

stephen@blueslugs.com (Stephen Hahn) — Wed, 28 Mar 2007 00:28:58 +0000

Glynn Foster’s Blog: OpenSolaris – Where Next?

Glynn starts to rev up for his coming term on the Governing Board.

Simon Phipps, SunMink: Ratified and Elected

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Mar 2007 19:29:02 +0000

Simon Phipps, SunMink: Ratified and Elected

Outgoing Board member, Simon, announces the successful ratification of the Constitution, and cheers on its evolution.

/dev/dump: Congratulations new OGB

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Mar 2007 19:27:36 +0000

/dev/dump: Congratulations new OGB

Garrett analyzes the 2007 Board Election results, with commentary on some aspects he’d like to see changed. Article VI of the Constitution does place term limits on the Board Members.

Rich Teer's Blog: OGB Election results are in!

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Mar 2007 19:25:14 +0000

Rich Teer’s Blog: OGB Election results are in!

Rich summarizes the ratification/2007 board election poll. (Rich is the only non-Sun member elected to the 2007 board.)

that?s whacked » we have consensus!

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Mar 2007 19:23:15 +0000

that?s whacked » we have consensus!

Steve Lau accepts the responsibility not to screw up on the OpenSolaris Governing Board.

poll.opensolaris.org: Board Election 2007/Ratify Constitution

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Mar 2007 14:45:21 +0000

poll.opensolaris.org: Board Election 2007/Ratify Constitution

The OpenSolaris Community completes ratification of its Constitution and elects its Board for 2007 – 2008. The Board election was resolved by Single Transferable Vote, using the Meek algorithm.

Elections Canada On-line

stephen@blueslugs.com (Stephen Hahn) — Mon, 26 Mar 2007 17:51:12 +0000

Elections Canada On-line

Site for the Chief Electoral Officer of Canada, and his/her organization.

LinkChecker – check websites and HTML documents for broken links

stephen@blueslugs.com (Stephen Hahn) — Sat, 24 Mar 2007 05:27:02 +0000

LinkChecker – check websites and HTML documents for broken links

A multithreaded link evaluator, in Python.

Open Your Mind Today

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 Mar 2007 17:10:27 +0000

Open Your Mind Today

Sun will give you a free T-shirt if you answer a skill-testing question.

Niath: Command Not Found

stephen@blueslugs.com (Stephen Hahn) — Wed, 21 Mar 2007 16:23:32 +0000

Niath: Command Not Found

Ubuntu has a utility, command-not-found, which with shell integration, recommends the packaging command to invoke to obtain a missing command.

poll.opensolaris.org: Community Election Management

stephen@blueslugs.com (Stephen Hahn) — Mon, 19 Mar 2007 23:06:50 +0000

poll.opensolaris.org: Community Election Management

OpenSolaris’s community polling/voting/consensus mechanism site. Includes status for current and future elections, plus results for past ones.

Vela Canopies

stephen@blueslugs.com (Stephen Hahn) — Sat, 17 Mar 2007 17:59:35 +0000

Vela Canopies

Residential shade structures and canopy awnings. May help me reclaim summer use of the deck.

Busy, stopping email

stephen@blueslugs.com (Stephen Hahn) — Fri, 16 Mar 2007 23:43:12 +0000

The past couple of months have been busy: I’ve been working on different deliverables for OpenSolaris, SMF, and SFW. I’ll try to give summaries on each of these items as they complete in the coming weeks, but I thought I should mention the consequences of some anti-spam provisions I implemented on opensolaris.org in February. Since these restrictions can cause mail to OpenSolaris lists to be dropped, you may wish to read on.

At the time, we had essentially no anti-spam on opensolaris.org‘s mail pipeline, and we relying on Mailman (which handles our 120+ lists) to reject spam directly, or for list moderators to do so manually. We knew that some lists were promptly moderated, and others less so, but figured things were working, for the most part. This state of affairs then changed.

It was public news that there was a jump in spam volume the previous fall. On opensolaris.org, spam hit all of our addresses, including administrative aliases for list subscription and removal. Mailman requires time to process each request, and with the rate of inflow, finally built to a backlog of almost 35 000 unprocessed Mailman administrative queries in early February. For a potential OpenSolaris list subscriber, this translated to a multi-hour wait for subcription requests—no way to make it easy to get in on a conversation.

To stop the -subscribe and -unsubscribe addresses from getting pounded, I started activating the basic anti-spam features of Postfix (which we have run on opensolaris.org since the initial setup). Presently, we expect a host to have legitimate DNS entries and to not be on various blacklists, and we expect envelopes to be well-formed. The backlog started to clear as these controls were in place, and was completed with no dropped requests a day or two later. (We had one hiccough: I had initially written the configuration so that Postfix didn’t trust the Mailman outbound mail, which meant that we did both anti-spam on the inbound submission and also on the dozens or hundreds of outbound messages to the list subscribers. Once we straightened that out, things have been much better.)

Since we imposed these requirements after initial subscriptions, it is possible that a valid subscriber to a Mailman list can no longer send mail acceptable for opensolaris.org. This problem is particularly likely to be related to a host lacking a valid MX record. If you’ve sent mail to opensolaris.org that hasn’t appeared in the lists, please check that you have a valid MX record. You can use one of the web forms offering MX lookups, but on an OpenSolaris-based system it’s easy to do this using nslookup(1M):

$ nslookup

set querytype=mx rosseau.eng.sun.com Server: 192.168.1.2 Address: 192.168.1.2#53 ** server can't find rosseau.eng.sun.com: NXDOMAIN rosseau.sfbay.sun.com Server: 192.168.1.2 Address: 192.168.1.2#53 Non-authoritative answer: rosseau.sfbay.sun.com mail exchanger = 10 btmx1.sun.com. rosseau.sfbay.sun.com mail exchanger = 10 btmx3.sun.com. rosseau.sfbay.sun.com mail exchanger = 10 btmx4.sun.com. rosseau.sfbay.sun.com mail exchanger = 20 mx3.sun.com. rosseau.sfbay.sun.com mail exchanger = 20 mx4.sun.com. Authoritative answers can be found from: sun.com nameserver = ns1.sun.com. ....

(In fact, rosseau lost its eng.sun.com MX record in the past couple of weeks, so I, too, got caught by the new policy.)

Both our Mailman instance and our many moderators are much happier now, processing many fewer, mostly legitimate list postings. If you’ve a valid MX record, aren’t blacklisted, and are still having problems, website-discuss is the place to ask.

[ T: OpenSolaris Postfix Mailman spam ]

governance.lopsa.org: GovernanceWiki

stephen@blueslugs.com (Stephen Hahn) — Thu, 15 Feb 2007 01:44:15 +0000

governance.lopsa.org: GovernanceWiki

LOPSA’s governance wiki, with a well-written FAQ and results of their 2005 election using Meek STV.

that?s whacked » OpenSolaris/Solaris Relationships

stephen@blueslugs.com (Stephen Hahn) — Wed, 14 Feb 2007 02:42:17 +0000

that?s whacked » OpenSolaris/Solaris Relationships

Steve explains how source become binaries, consolidations, and finally distributions from the OpenSolaris code base. With diagram!

SMF FAQ at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Fri, 09 Feb 2007 15:13:18 +0000

SMF FAQ at OpenSolaris.org

The long-awaited FAQ for smf(5).

Glynn Foster: OpenSolaris Membership

stephen@blueslugs.com (Stephen Hahn) — Mon, 29 Jan 2007 23:53:25 +0000

Glynn Foster: OpenSolaris Membership

Glynn comments on the proposed governance for OpenSolaris; he’s read the document, lived an alternative via participation in GNOME, and makes a good point about how the differences in the two approaches might matter.

The Blog of Ben Rockwood: OpenSolaris Governance: It's time to start caring

stephen@blueslugs.com (Stephen Hahn) — Wed, 24 Jan 2007 22:14:13 +0000

The Blog of Ben Rockwood: OpenSolaris Governance: It’s time to start caring

Ben on governance in OpenSolaris (with a capsule history of what’s happened so far).

Simon Phipps, SunMink: OpenSolaris Governance Ready to Boot

stephen@blueslugs.com (Stephen Hahn) — Wed, 24 Jan 2007 22:12:48 +0000

Simon Phipps, SunMink: OpenSolaris Governance Ready to Boot

Simon on where we are in the OpenSolaris governance work.

Scalla: Scalable Cluster Architecture for Low Latency Access Using xrootd and olbd Servers

stephen@blueslugs.com (Stephen Hahn) — Thu, 18 Jan 2007 15:16:35 +0000

Scalla: Scalable Cluster Architecture for Low Latency Access Using xrootd and olbd Servers

A clusterable low-latency, high-bandwidth data access framework. Hierarchical clustering model. Used in HEP community.

It's official – Sun is the #1 contributor to Open Source – by a long way

stephen@blueslugs.com (Stephen Hahn) — Wed, 17 Jan 2007 16:42:30 +0000

It’s official – Sun is the #1 contributor to Open Source – by a long way

Alan finds the EU has valued the contributions of various software and systems companies to open source.

CSRF Guard – OWASP

stephen@blueslugs.com (Stephen Hahn) — Thu, 11 Jan 2007 06:34:27 +0000

CSRF Guard – OWASP

A Java EE filter proof of concept implementation of a unique token defense (or “guard”) against Cross-Site Request forgery.

Finger Trees: A Simple General-purpose Data Structure

stephen@blueslugs.com (Stephen Hahn) — Tue, 09 Jan 2007 18:47:58 +0000

Finger Trees: A Simple General-purpose Data Structure

“We present 2-3 finger trees, a functional representation of persistent sequences supporting access to the ends in amortized constant time, and concatenation and splitting in time logarithmic in the size of the smaller piece. Representations achieving the

Brown CS: Tech Report CS-97-12: MW Shapiro, RDB: A System for Incremental Replay Debugging

stephen@blueslugs.com (Stephen Hahn) — Mon, 08 Jan 2007 19:56:41 +0000

Brown CS: Tech Report CS-97-12: MW Shapiro, RDB: A System for Incremental Replay Debugging

Replay debugging seems to have a resurgence every few years. Here’s Mike’s take (and description of his implementation) from a few years ago.

A Periodic Table of Visualization Methods

stephen@blueslugs.com (Stephen Hahn) — Mon, 08 Jan 2007 19:45:37 +0000

A Periodic Table of Visualization Methods

Interesting catalogue.

DWR – Easy AJAX for JAVA | Getahead

stephen@blueslugs.com (Stephen Hahn) — Sat, 06 Jan 2007 02:02:54 +0000

DWR – Easy AJAX for JAVA | Getahead

“DWR allows Javascript in a browser to interact with Java on a server and helps you manipulate web pages with the results.” Interesting.

Locationbar² | Firefox Add-ons | Mozilla Corporation

stephen@blueslugs.com (Stephen Hahn) — Thu, 04 Jan 2007 18:35:58 +0000

Locationbar² | Firefox Add-ons | Mozilla Corporation

It’s simple–merely changing some of the URL to bold–but I like it.

OpenSolaris: Restricted builds through the ages

stephen@blueslugs.com (Stephen Hahn) — Thu, 04 Jan 2007 00:47:12 +0000

OpenSolaris: Restricted builds through the ages

On the history of restricted integration builds for ON, the recent ON restrictions, and the need to develop a community policy on frequency and publicity of future integration restrictions.

OpenSolaris: Restricted builds through the ages

stephen@blueslugs.com (Stephen Hahn) — Thu, 04 Jan 2007 00:45:18 +0000

Last month, Stephen Harpster announced that there would be a two-build sequence with restricted integration conditions. This sort of restriction isn’t new for Solaris development, as I’ll show below, but the means of arriving at the restriction and the manner in which we shared it shows we are still learning what an “open process” must do—and when we miss, to make sure we learn from those mistakes.

So Stephen’s announcement noted that Build 54 and 55 would be “only bug fixes”. That’s not new; we can look at the build-by-build restrictions on ON integrations for Solaris 8

Solaris 9,

and Solaris 10,

Here, each box represents a cycle of the two-week build clock. Solid boxes indicate a strong “bugfix only” integration policy (like “stopper only” or integrations requiring the consolidation’s technical lead’s approval); outlined boxes indicate a weak “bugfix only” integration policy (meaning no specially approved projects). I’ve worked out these integration policy digests from the ON schedules of past releases; policies will vary for other consolidations, but should match up roughly with ON’s history.

Solaris 8 and 9 show the classic “(Development Complete–)Beta–Beta Refresh–Ship” pattern that characterized Solaris releases. In Solaris 10, the Solaris Express program gets reflected in that, although there were identified Beta and Beta Refresh builds, the shift from early release to late release really takes place with the Beta Refresh build, which was the first in a series of restricted builds, until Ship.

Thus, for current Nevada, ON looks a lot like it did for the release that became Solaris 10:

Inferences you might make are at your own risk.

Of course, the freezes we took in the past were easily achieved as there was only one distribution derived from the ON sources. Now that we are a community with a collection of active distributions, the notion of periodic and public restricted builds may still be attractive, so that distributions could choose to issue their latest version from a known stabilized build. Certainly current distributions (or emerging ones) might consider using the stabilized build 55—however we got here—as an opportune version to base their next release upon.

Of course, there is a tradeoff between letting each distribution stabilize and a consolidation-wide restriction. Perhaps a topic for a future post?

We, as a community, will need to have a position on restricted builds. Please think about it, and then feel free to comment here, and mail me privately. Public discussions, I suspect, are more valuable: opensolaris-code would be a good forum for the technical tradeoffs, possible frequency of stabilized builds, and so on, while cab-discuss would be the best place for inter-distribution and coordination issues.

[ T: OpenSolaris Solaris distribution consolidation ]

Project Euler

stephen@blueslugs.com (Stephen Hahn) — Tue, 02 Jan 2007 06:30:35 +0000

Project Euler

Collection of mathematical problems that require computation.

TypeMatrix EZR 2030 Keyboard Skins

stephen@blueslugs.com (Stephen Hahn) — Fri, 22 Dec 2006 00:19:42 +0000

TypeMatrix EZR 2030 Keyboard Skins

Cool silicone skins for TypeMatrix keyboards. Need to get a black Dvorak or two.

Amateur Astronomy Magazine

stephen@blueslugs.com (Stephen Hahn) — Sun, 17 Dec 2006 21:45:59 +0000

Amateur Astronomy Magazine

clever, witty blog title » Blog Archive » Collaborative filtering made easy

stephen@blueslugs.com (Stephen Hahn) — Wed, 13 Dec 2006 22:35:42 +0000

clever, witty blog title » Blog Archive » Collaborative filtering made easy

Bryan explains the use of “slope one predictors” for collaborative filtering, with an example in 40 lines of Python.

opensolaris.org: Prevent accidental robotomies

stephen@blueslugs.com (Stephen Hahn) — Mon, 11 Dec 2006 23:36:36 +0000

We’ve been working through bits and pieces of <a href="http://opensolaris.org" >opensolaris.org</a>, looking for the troubling portions where regular human touches are currently required.

These are troubling because (a) someone forgets to touch when someone else is on vacation, sick, or whatever, and then (b) someone depending on that resource being touched gets annoyed/loses time/sends us Kafkaeque email about transforming into an automaton.

Recently, Steve pulled a little Python script together and now we can smoothly offer direct DVD downloads of Solaris Express Community Release across each incremental release. In honour of his script, a new download button:

<a href="http://opensolaris.org/sxcr_dvd" >http://opensolaris.org/sxcr_dvd</a> should work, without problem, for the duration of Nevada. But pipe up on website-discuss if it fails.

[ T: OpenSolaris Solaris DVD Downloads ]

Hypertext Markup Language – 2.0 – The HTML Coded Character Set

stephen@blueslugs.com (Stephen Hahn) — Mon, 11 Dec 2006 23:16:24 +0000

Hypertext Markup Language – 2.0 – The HTML Coded Character Set

I keep using this page, so best simply to bookmark it.

Slightly better nfs v4 support for nomadic systems

stephen@blueslugs.com (Stephen Hahn) — Mon, 11 Dec 2006 20:35:17 +0000

Slightly better nfs v4 support for nomadic systems

Chris offers patches to modify nfsmapid to allow more convenient assignments between IDs from different NFSv4 domains. Useful on a laptop transiting between two NFS environments.

Infinite Knots: Ideas On Issue Tracking in the FLOSS World

stephen@blueslugs.com (Stephen Hahn) — Wed, 06 Dec 2006 19:46:21 +0000

Infinite Knots: Ideas On Issue Tracking in the FLOSS World

Some thoughts on inter-product issue tracking, mostly from a user perspective.

Interactive Mathematics Miscellany and Puzzles, Index

stephen@blueslugs.com (Stephen Hahn) — Wed, 06 Dec 2006 18:23:40 +0000

Interactive Mathematics Miscellany and Puzzles, Index

Many applets illustrating various numerical, logic, and visual puzzles.

Untuning for 53

stephen@blueslugs.com (Stephen Hahn) — Mon, 04 Dec 2006 22:08:29 +0000

Since I bumped into a couple of problems after moving to Build 53, I thought I should make a note and save folks some time. My laptop’s a reasonably standard install, but I do turn off a few more daemons, in the hope of squeezing out a few minutes’ more battery life. With the new GNOME bits, you need to make sure that

$ svcs *hal* *fc-cache*
STATE STIME FMRI
online Nov_28 svc:/application/font/fc-cache:default
online 12:27:47 svc:/system/hal:default

shows that both

hal(1M) and fc-cache(1M) are running, as their functionality is required by other components. (If you don’t have application/font/fc-cache enabled, you’ll have trouble getting much of GNOME to run–you’ll see core files, running on which pstack(1) will reveal a stack trace filled with symbols beginning with "FC"…) A couple of svcadm enable invocations will bring things back into spec.

If you haven’t customized your system and you’re still having similar problems, you probably want to visit desktop-discuss.

[ T: OpenSolaris Solaris GNOME HAL fc-cache ]

[cs/0612009] Users' participation to the design process in an Open Source Software online community

stephen@blueslugs.com (Stephen Hahn) — Mon, 04 Dec 2006 21:53:13 +0000

[cs/0612009] Users’ participation to the design process in an Open Source Software online community

A study of the Python Enhancement Proposal, as an OSS design process. Included field interviews, discussion analysis, etc. as part of study.

53 looks good

stephen@blueslugs.com (Stephen Hahn) — Thu, 30 Nov 2006 00:36:55 +0000

About six weeks ago, in anticipation of GNOME 2.16, I switched my desktop from an Ion window manager-based X11 environment to the JDS environment integrated with Solaris. Now, with the delivery of Build 53—not yet on the Download Centre, apparently—the wait is over:

[Formerly, a scaled down screenshot of a system running Build 53.]

A few keyboard shortcuts input, a tweak of the window preferences, and it’s back to normal work. (Or better: using evince and gnome-terminal in their full screen modes are distinct improvements for reading documents onscreen.)

Background image by Mandolux.

[ T: OpenSolaris Solaris JDS GNOME ]

Tools at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Mon, 27 Nov 2006 17:53:54 +0000

Tools at OpenSolaris.org

PlanetDreamcast: Games – Best Dreamcast Games

stephen@blueslugs.com (Stephen Hahn) — Mon, 27 Nov 2006 05:06:13 +0000

PlanetDreamcast: Games – Best Dreamcast Games

MogileFS

stephen@blueslugs.com (Stephen Hahn) — Wed, 22 Nov 2006 22:48:11 +0000

MogileFS

Danga’s userland distributed file service.

[cs/0611108] On the Complexity of Processing Massive, Unordered, Distributed Data

stephen@blueslugs.com (Stephen Hahn) — Wed, 22 Nov 2006 18:50:09 +0000

[cs/0611108] On the Complexity of Processing Massive, Unordered, Distributed Data

Compares complexity of single stream algorithms to multiple stream algorithms on “massive, unordered, distributed” data.

OpenSolaris Forums: Progress with System z Port …

stephen@blueslugs.com (Stephen Hahn) — Wed, 15 Nov 2006 20:12:16 +0000

OpenSolaris Forums: Progress with System z Port …

Neale Ferguson shows some early startup output for a port of OpenSolaris to a System z9 (IBM) mainframe.

Redwood City Center of Palo Alto Medical Foundation, Redwood City, California

stephen@blueslugs.com (Stephen Hahn) — Wed, 15 Nov 2006 20:06:26 +0000

Redwood City Center of Palo Alto Medical Foundation, Redwood City, California

PAMF’s new office is now the closest to our home.

Planet OpenSolaris

stephen@blueslugs.com (Stephen Hahn) — Wed, 15 Nov 2006 00:52:38 +0000

Planet OpenSolaris

Brought to life by Gman and stevel! Now, how do I get a kooky headshot up there…?

chandanlog(3C): Copyrights, Licenses and CDDL Illustrated

stephen@blueslugs.com (Stephen Hahn) — Tue, 14 Nov 2006 18:25:26 +0000

chandanlog(3C): Copyrights, Licenses and CDDL Illustrated

Chandan developed these diagrams a couple of months ago, but since I keep explaining these distinctions over and over, it seems best to make it easier to find.

Tip: Mercurial atop ssh

stephen@blueslugs.com (Stephen Hahn) — Thu, 09 Nov 2006 18:41:49 +0000

On opensolaris.org, we host Mercurial and Subversion via their ability to tunnel through an SSH connection. A slight gotcha with Mercurial is that it turns off compression when tunnelled through SSH, as described in hg(1):

$ man hg
....
- Mercurial doesn't use its own compression via SSH; the right thing
to do is to configure it in your ~/.ssh/ssh_config, e.g.:
Host *.mylocalnetwork.example.com
Compression off
Host *
Compression on
Alternatively specify "ssh -C" as your ssh command in your hgrc or
with the --ssh command line option.
....

For pulls of a large repository, such as ON, compression changes an approximately 51 minute operation to a 17 minute operation. So you’ll want something like

$ cat ~/.ssh/config
....
Host hg.opensolaris.org
Compression yes
...
....

in your SSH configuration.

[ T: OpenSolaris Solaris Mercurial hg ]

Ajaxian » Dojo Charting Engine Released

stephen@blueslugs.com (Stephen Hahn) — Thu, 09 Nov 2006 01:27:04 +0000

Ajaxian » Dojo Charting Engine Released

Another AJAX charting library; pleasant looking. Apparently SVG.

milek – ZFS saved our data

stephen@blueslugs.com (Stephen Hahn) — Wed, 08 Nov 2006 19:29:27 +0000

milek – ZFS saved our data

Robert describes how end-to-end checksum detected and protected against a failing disk controller.

UCSB Kavli Institute for Theoretical Physics: Talks

stephen@blueslugs.com (Stephen Hahn) — Wed, 08 Nov 2006 19:28:07 +0000

UCSB Kavli Institute for Theoretical Physics: Talks

Published talks and public lectures given at KITP.

OpenSolaris: Documentation tutorial buffet

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Nov 2006 19:33:59 +0000

Each of the community groups that make up OpenSolaris is experimenting with different styles of introductory materials (or procedures), so that potential contributors don’t get blocked on the details and can focus on their real intention—contributions. The Documentation community group has been energetically pursuing a complete offering: they have a large document tree, they’re using a “scratchpad” on the Genunix wiki, and they have at least one mailing list. They even have a Flash animated tutorial, illustrating how to use a DocBook XML toolchain. Slick.

[ T: OpenSolaris Solaris documentation DocBook ]

OpenSolaris Day, Buenos Aires

stephen@blueslugs.com (Stephen Hahn) — Mon, 06 Nov 2006 19:05:48 +0000

I’ve been back from Buenos Aires for a week now, and just wanted to comment on my trip, and show the few pictures I managed to take. Like its sister events, Buenos Aires TechDays was a densely packed three day conference, with multiple tracks available on OpenSolaris, Java, NetBeans, SunStudio, Glassfish, and other technologies and platforms. I updated the “Building and Deploying OpenSolaris” talk, to cover the convenient Starter Kit, which has the tools and source (and OS images) and to mention the ONNV Mercurial repository on opensolaris.org.

The talk was simultaneously translated into Spanish, using some slick IR emitter arrays and receivers. Russ, I think, was much more effective, as he gave his presentations in Spanish.

The event was held at the Hilton Buenos Aires, which is located in the recently redeveloped Puerto Madero area. When Teresa and I walked around on the first day, we saw activity in the harbour, and paused to watch (along with a few hundred other pedestrians), as this tugboat

helped this coast guard ship

turn out of the relatively narrow channel.

Back at TechDays, we enjoyed the chance to have further technical discussions, with many resolutions

to study new problems, reengage with old ones, or just return to Argentina.

The Inquirer‘s Fernando Cassia has more on the NetBeans portion of the event.

In Argentina, but missed TechDays? There’s an OpenSolaris user group—check it out.

[ T: Solaris OpenSolaris TechDays Buenos_Aires ]

Jeff Bonwick's Weblog: ZFS Block Allocation

stephen@blueslugs.com (Stephen Hahn) — Mon, 06 Nov 2006 18:28:52 +0000

Jeff Bonwick’s Weblog: ZFS Block Allocation

Jeff describes the steps involved in allocating blocks across devices in ZFS. (Interested in OS work? Note the open areas of investigation, involving optimum handling in the case of failures and in the cases of different kinds of data.)

NOAA News Online (Story 2724)

stephen@blueslugs.com (Stephen Hahn) — Sun, 05 Nov 2006 01:13:33 +0000

NOAA News Online (Story 2724)

NOAA’s winter weather outlook for 2006-2007. It’s apparently a mild El Niño year, but doesn’t look like we’ll see the rains of ’97-’98.

TinyMCE Javascript Content Editor by Moxiecode Systems AB

stephen@blueslugs.com (Stephen Hahn) — Mon, 30 Oct 2006 23:05:22 +0000

TinyMCE Javascript Content Editor by Moxiecode Systems AB

A JavaScript WYSIWYG HTML editor component.

OpenSolaris Forums: ON Mercurial repository mirror …

stephen@blueslugs.com (Stephen Hahn) — Wed, 18 Oct 2006 23:38:20 +0000

OpenSolaris Forums: ON Mercurial repository mirror …

Steve announces the availability of a public live read-only Mercurial repository for ON. It’s bridged from the main ON TeamWare repository, so new code is available to all, as soon as it’s integrated.

tumgreyspf: Greylisting and SPF for Postfix in Python

stephen@blueslugs.com (Stephen Hahn) — Wed, 18 Oct 2006 21:08:26 +0000

tumgreyspf: Greylisting and SPF for Postfix in Python

A different greylisting implementation for Postfix. No database dependencies.

Improving the Availability and Performance of Network-Mediated Services

stephen@blueslugs.com (Stephen Hahn) — Tue, 17 Oct 2006 01:04:05 +0000

Improving the Availability and Performance of Network-Mediated Services

A now-older investigation into load balancing proxy servers, here SOCKS4/5 and HTTP.

Periodic reducation of email interrupt rate

stephen@blueslugs.com (Stephen Hahn) — Wed, 11 Oct 2006 20:23:37 +0000

At work, I’ve ended up on many email aliases; since we moved to pursuing open development on opensolaris.org, my subscriptions and incoming email flow have grown tremendously, to the point where they were interfering with my ability to get work done. My present filter setup involves delivery to 56 inboxes.

I’ve had a small email toolset built around the Mutt email client (or mail user agent (MUA) in email-speak). My .muttrc includes $HOME/.mailboxes, which is a list of the mail inboxes into which my procmail filter delivers. The m and check commands use .mailboxes to determine where new messages have arrived. Happily, this infrastructure can easily give me a few hours of reduced interrupts, so that I can get some thinking and writing done.

The idea is to strongly reduce the number of significant inboxes during what we’ll label as “prime” work hours, between lunch and dinner on weekdays. Given the use of .mailboxes outlined above, the solution is pretty easy:

Move .mailboxes to .mailboxes.offprime.
Take a copy of .mailboxes.offprime as .mailboxes.prime. Prune the prime mailboxes severely.
Install these files in $HOME. (My Makefiles do this, and all the mailbox lists and the Makefiles are version-controlled.)

Add something like the following using crontab -e:

# We want to eliminate mail testing on all mailboxes during prime work

hours. This change means that between 12pm - 6pm, mailboxes is

pointed to a reduced mailboxes file.

0 12 * * 1,2,3,4,5 /usr/bin/ln -sf /home/sch/.mailboxes.prime /home/sch/.mailboxes 0 18 * * 1,2,3,4,5 /usr/bin/ln -sf /home/sch/.mailboxes.offprime /home/sch/.mailboxes

Run whichever of the two ln(1) invocations above correspond to the current time.

You’re now ready to relax and enjoy reduced interrupt rates, leaving you time to focus on real work… or write blog postings about managing your interrupt rates.

[ T: OpenSolaris Solaris mutt procmail ]

Periodic reducation of email interrupt rate

stephen@blueslugs.com (Stephen Hahn) — Wed, 11 Oct 2006 20:22:58 +0000

Move .mailboxes to .mailboxes.offprime.
Take a copy of .mailboxes.offprime as .mailboxes.prime. Prune the prime mailboxes severely.
Install these files in $HOME. (My Makefiles do this, and all the mailbox lists and the Makefiles are version-controlled.)

Add something like the following using crontab -e:

# We want to eliminate mail testing on all mailboxes during prime work

hours. This change means that between 12pm - 6pm, mailboxes is

pointed to a reduced mailboxes file.

0 12 * * 1,2,3,4,5 /usr/bin/ln -sf /home/sch/.mailboxes.prime /home/sch/.mailboxes 0 18 * * 1,2,3,4,5 /usr/bin/ln -sf /home/sch/.mailboxes.offprime /home/sch/.mailboxes

Run whichever of the two ln(1) invocations above correspond to the current time.

You’re now ready to relax and enjoy reduced interrupt rates, leaving you time to focus on real work… or write blog postings about managing your interrupt rates.

[ T: OpenSolaris Solaris mutt procmail ]

OpenSolaris: TechDays, Buenos Aires, 23 – 25 October

stephen@blueslugs.com (Stephen Hahn) — Wed, 11 Oct 2006 19:24:43 +0000

I’ll be visiting Argentina for the first time, to present at the upcoming TechDays in Buenos Aires. As with all TechDays, the event itself is free, but you must register in advance. The event is being held at the Hilton Buenos Aires; OpenSolaris day is 23 October.

As always, if you want to meet up and talk about OpenSolaris, software development in Argentina, or any other topic, leave me a comment or send me some email.

I’m a little tired today, as I made a round-trip to Los Angeles to visit the Argentinian consulate there and obtain a visa. 700+ miles on the road, but the friendly officers were able to do the turnaround in only a few hours.

[ T: Solaris OpenSolaris TechDays Buenos_Aires ]

Parallel BZIP2 (PBZIP2)

stephen@blueslugs.com (Stephen Hahn) — Fri, 06 Oct 2006 18:36:57 +0000

Parallel BZIP2 (PBZIP2)

A parallel implementation of the bzip2 compression implementation, using POSIX threads. C++, wrapping base routines from libbz2.

milek's blog: SMF management for normal users

stephen@blueslugs.com (Stephen Hahn) — Thu, 05 Oct 2006 21:57:24 +0000

milek’s blog: SMF management for normal users

Robert gives an example of authorization assignments so that an unprivileged user can manage a service via smf(5).

Derek Crudgington » Blog Archive » SMF Monitor Alert v1

stephen@blueslugs.com (Stephen Hahn) — Wed, 04 Oct 2006 18:23:54 +0000

Derek Crudgington » Blog Archive » SMF Monitor Alert v1

Derek’s got a preliminary release of a daemon that monitors smf(5) managed services.

Solaris PowerPC Port at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Tue, 03 Oct 2006 22:46:16 +0000

Solaris PowerPC Port at OpenSolaris.org

The effort to bring OpenSolaris back to PowerPC made its first large code drop today.

Mailgraph Homepage

stephen@blueslugs.com (Stephen Hahn) — Tue, 03 Oct 2006 21:28:03 +0000

Mailgraph Homepage

“A very simple mail statistics RRDtool frontend for Postfix and Sendmail”, by the author of Postgrey.

Martin Englund's Weblog: Mercurial for TeamWare users

stephen@blueslugs.com (Stephen Hahn) — Wed, 27 Sep 2006 18:33:58 +0000

Martin Englund’s Weblog: Mercurial for TeamWare users

Martin assembled a quick concept map from TeamWare commands to Mercurial commands.

Martin Englund's Blog: Migrating from TeamWare to a new SCM

stephen@blueslugs.com (Stephen Hahn) — Wed, 27 Sep 2006 18:24:41 +0000

Martin Englund’s Blog: Migrating from TeamWare to a new SCM

And Martin’s been looking at the performance and resource issues anticipated by the SCM switch for Java.

Kelly O'Hair's Blog: Teamware, Mercurial, and SCCS revs that go bump in the night

stephen@blueslugs.com (Stephen Hahn) — Wed, 27 Sep 2006 18:23:34 +0000

Kelly O’Hair’s Blog: Teamware, Mercurial, and SCCS revs that go bump in the night

Kelly, on the technical aspects of converting TeamWare history to Mercurial.

Mark Reinhold?s Other Blog: Source-code management for an open JDK

stephen@blueslugs.com (Stephen Hahn) — Wed, 27 Sep 2006 18:21:29 +0000

Mark Reinhold?s Other Blog: Source-code management for an open JDK

Mark’s the Chief Engineer for Java SE, which is shifting to become an open development project. Here Mark talks about their intent to investigate Mercurial as a preferred SCM candidate, as OpenSolaris did.

Solaris vs. Asterisk running Asterisk PBX

stephen@blueslugs.com (Stephen Hahn) — Wed, 27 Sep 2006 04:25:08 +0000

Solaris vs. Asterisk running Asterisk PBX

A comparison of various systems running the Asterisk software PBX.

Google testing Sun's OpenSolaris, sources say

stephen@blueslugs.com (Stephen Hahn) — Thu, 21 Sep 2006 16:24:44 +0000

Google testing Sun’s OpenSolaris, sources say

Eric Lai finds other organizations are deploying or reevaluating OpenSolaris as well.

OpenSolaris: Gary's in Brussels

stephen@blueslugs.com (Stephen Hahn) — Tue, 19 Sep 2006 19:39:28 +0000

Gary Pennington is attending EuroOSCON, being held this week in Brussels. He’s reporting on the various sessions. Worth a read, particularly if you’ve not been to an open source conference in a while…

[ T: OpenSolaris Solaris EuroOSCON Brussels ]

Reading at work: Rosenfeld and Morville's Information Architecture for the World Wide Web, 2e

stephen@blueslugs.com (Stephen Hahn) — Mon, 18 Sep 2006 20:12:43 +0000

My flight to Seattle’s TechDays a couple of weeks ago ended up being delayed by about an hour, so I had enough reading time to finish the second edition of Louis Rosenfeld and Peter Morville‘s Information Architecture for the World Wide Web.

Because the book’s publication date is 2002, the various dynamic elements of AJAX-style interactions are touched upon very lightly. The book ends up arguing along the lines of “designing for the lowest fidelity is safest”, but I’m not sure that browser expectations can be traded away so quickly today. Certainly some of the example websites cited in the text now use a substantial amount of AJAX enhancements that improve site interaction.

Rosenfeld and Morville’s main point is that information architecture—the organization of information so that that information can be successfully retrieved by the various individuals that interact with a site—is hard. Having spent some time worrying about how to publish the engineering workflow around ON (and, I believe, OpenSolaris more widely) open software development, and to make sure that that publication is navigable and meaningful over time, I can only agree. The book presents a medium-scale overview of information architecture: there are “basic principles” chapters on organization, labelling, navigation, and search, as well as case studies and thoughts about practice. Various classic tradeoffs, simple usability tests, and kinds of site prototypes are concisely introduced and then expanded upon, leading to more advanced considerations.

Beyond the illustrative examples from a wide set of websites, one thing that was of particular value to me was learning about the potential role of thesauruses as a specific form of metadata, and how that metadata can be used to make search-based navigation capture a larger set of related documents. We’re also outgrowing the initial design for the opensolaris.org site, so we’ll use some of the ideas presented in the “Process and Methodology” section. The example strategy document is also a useful case study—we probably need to develop a similar document for opensolaris.org in a collaborative fashion in the website project.

The authors also identify some of the potential career forms an information architectural role might take, as well as how information architecture’s shared boundaries with implementation and graphic design are fuzzy. It seems to me that every substantial open development community needs some kind of information architecture, if only to support the classic user/developer audience split, so some of the credentials they suggest might be necessary for a corporate information architect appear elite, or unusual, for a community effort.

Interesting.

Available via Sun’s subscription to the O’Reilly Safari online library.

[ T: OpenSolaris TechDays ]

OpenSolaris Day, Seattle

stephen@blueslugs.com (Stephen Hahn) — Thu, 07 Sep 2006 21:23:50 +0000

It’s nice to be in Seattle. I spent a couple of weeks on the UW campus a dozen or so years ago; on customer visits, we’re ususally in a suburb. Enjoying the chance to wander in a different downtown, I decided to walk from the hotel to the TechDays site, at Marion Oliver McCaw Hall, in Seattle Center.

This let me do a few errands—like eat a quick breakfast at Specialty Café and replace the broken SD card for my camera—and walk past a well known building

and under the disappointingly-testing-only-service monorail

Has no one learned from North Haverbrook?

We had an audience split roughly between developers and administrators at the first of a global series of OpenSolaris TechDays. After an introduction by Teresa, I gave Frank‘s new presentation, “Building and Deploying OpenSolaris”, which is an overview of what’s out there and how to build some of it. Post-presentation, we got some good questions: on switching easily between the various in-progress development project variants; on various forking scenarios; )and on arriving at a shared development methodology. I think the talk actually has some room for covering one or two more topics, but I’ve posted the presentation in the meanwhile.

Nicolas, Glenn, and Jim gave great technical talks on Virtualization, Security, and Performance for OpenSolaris, respectively. Slides for all of these talks should be at the TechDays developer site.

We finished with an open question-and-answer session.

p>On Thursday’s TechDays session, there is a Solaris track which should provide folks with additional information about Solaris futures—derived, of course, from OpenSolaris—as well as development techniques available atop OpenSolaris systems. I’ll be around until late afternoon, when I’ll take #194 back to the airport.</ p>

Thanks all those who came, plus nods to Clayton for the conversations as well as a fine restaurant recommendation, the fellow at the airport who pointed out the fast bus to downtown, and to the folks who drove 350 miles round trip from Portland for the day.

[ T: Solaris OpenSolaris TechDays Seattle ]

Edgewall Software: Python Sidebar

stephen@blueslugs.com (Stephen Hahn) — Thu, 24 Aug 2006 17:00:07 +0000

Edgewall Software: Python Sidebar

A sidebar for Mozilla–or Firefox–with many categorized links to, and into, various Python references.

The PracTeX Journal – TeX Users Group

stephen@blueslugs.com (Stephen Hahn) — Thu, 24 Aug 2006 04:25:19 +0000

The PracTeX Journal – TeX Users Group

A magazine on TeX/LaTeX, hosted by TuG.

OpenSolaris: TechDays, Seattle, 5 – 7 September

stephen@blueslugs.com (Stephen Hahn) — Tue, 22 Aug 2006 19:47:56 +0000

I’ll be up in the Seattle area for 5 – 7 September—I’ll be presenting on OpenSolaris at the upcoming Tech Days there. If you’re interested in attending, then get over to the registration site; the event is free, but space is limited.

(Undecided? Peruse the Seattle agenda.)

If you’d like to have a beer or a coffee and talk about OpenSolaris, Solaris, software development, or the like, drop me a line. 6 September has the main OpenSolaris track, but I’ll be around the evening before and most of the day following.

I can’t find one here, so I’ll ask: anyone thinking about starting a Seattle OSUG?

[ T: OpenSolaris Solaris Tech_Days Seattle ]

opends: An open source directory server

stephen@blueslugs.com (Stephen Hahn) — Mon, 21 Aug 2006 18:17:04 +0000

opends: An open source directory server

Implemented in Java, licensed under CDDL, initial team composed of Sun directory engineers.

Angry Kitchen Appliances: Drives a Saab 9-5

stephen@blueslugs.com (Stephen Hahn) — Mon, 14 Aug 2006 01:41:44 +0000

Angry Kitchen Appliances: Drives a Saab 9-5

And has written one of the few 9-5 specific FAQs I’ve found.

fabprefab – modernist prefab dwellings

stephen@blueslugs.com (Stephen Hahn) — Sat, 12 Aug 2006 05:11:39 +0000

fabprefab – modernist prefab dwellings

Tracks developments in modern manufactured or prefabricated houses.

YAP Prolog

stephen@blueslugs.com (Stephen Hahn) — Thu, 10 Aug 2006 04:46:28 +0000

YAP Prolog

Another Warren Abstract Machine-based implementation.

Home – tuProlog

stephen@blueslugs.com (Stephen Hahn) — Thu, 10 Aug 2006 04:45:26 +0000

Home – tuProlog

Java-based implementation.

AI::Prolog – Perl extension for logic programming. – search.cpan.org

stephen@blueslugs.com (Stephen Hahn) — Thu, 10 Aug 2006 04:44:50 +0000

AI::Prolog – Perl extension for logic programming. – search.cpan.org

Pure Perl implementation.

The GNU Prolog web site

stephen@blueslugs.com (Stephen Hahn) — Thu, 10 Aug 2006 04:43:55 +0000

The GNU Prolog web site

Compiler. Uses Warren Abstract Machine representation.

A not-quite-isochronous upgrade

stephen@blueslugs.com (Stephen Hahn) — Wed, 09 Aug 2006 17:33:56 +0000

In the midst of many pieces of work, a new machine showed up on my doorstep. It’s pretty sweet, and I wanted to get it running latest Solaris Nevada bits. So I repartitioned the disk to make room for an additional boot environment (BE), updated the Live Upgrade support, upgraded (to Build 44), and got back to work.

But this week I saw something pretty strange. I was working on making some software more portable, updating some Makefiles as well as changing some of the source. On a rebuild, my gcc compilation failed with

collect2: ld terminated with signal 9 [Killed]
ld.so.1: ld: fatal: libld.so.4: version `SUNWprivate_4.2' not found (required by file /usr/ccs/bin/ld)
ld.so.1: ld: fatal: libld.so.4: open failed: No such file or directory
ld.so.1: ld: fatal: relocation error: file /usr/ccs/bin/ld: symbol ld32_main: referenced symbol not found

After checking with a few neighbours about any recent linker fixes, and reviewing package installation times, we finally started looking inside the binaries. The version from the install server showed Build 44:

$ mcs -p reloc/usr/ccs/bin/ld
reloc/usr/ccs/bin/ld:
@(#)SunOS 5.11 snv_44 October 2007

But the version installed on the machine was from the future:

$ mcs -p /usr/ccs/bin/ld
/usr/ccs/bin/ld:
@(#)SunOS 5.11 snv_45 October 2007

What were we running on this box?

$ cat /etc/motd
Sun Microsystems Inc. SunOS 5.11 snv_44 October 2007

I suppose if this were an Encyclopedia Brown mystery, you would have to flip to the end of the book to discover the solution to the mystery. Unfortunately, I haven’t given you quite enough information–you need to know how our install servers share new images.

As you might expect, there is a large set of install servers available across Sun. Some are run by IT operations, some are in development labs—but all pull their images across at different times. It turns out that the most recent build is linked to a directory called “latest“, so that one can reinstall a system every two weeks and have it running the most recently assembled version of Solaris.

Generally, the action to get a copy of the newest image and update the links happens automatically, and outside of business hours. But sometimes in a development lab, someone wants an image early, perhaps to upgrade a collection of test machines for verification purposes. And, it turns out, luupgrade(1M) works just fine across such a rename, if it’s not specifically accessing the filesystem during the image changeover. So my Live Upgrade installed a swathe of packages from 44 followed by a smattering from 45.

One more Live Upgrade (using the numbered path, rather than latest) to Build 45, and it’s back to porting.

This problem isn’t going to be seen very often—particularly if you don’t have a multi-version install server setup like we do—but it seems appropriate to document the underlying issue: Live Upgrade doesn’t detect an image change during the operation. I’m not sure it should, but it’s pretty clear that the underlying package operations are incomplete with respect to versioned dependencies.

[ T: OpenSolaris Solaris install luupgrade ]

Weird UPS package state

stephen@blueslugs.com (Stephen Hahn) — Thu, 27 Jul 2006 05:33:35 +0000

I’ve been waiting for a new electric fan to arrive. I checked in on the shipment today, and the UPS site reports:

07/26/2006 5:30 A.M. THE PACKAGE IS DELAYED DUE TO EMERGENCY CONDITIONS BEYOND UPS' CONTROL

On the retailer’s site, this state is reported as a “Delay in delivery due to external factors”.

Huh. Well, I hope it shows up soon: it’s still taking a while for the house to cool down in the evening.

GNATS

stephen@blueslugs.com (Stephen Hahn) — Wed, 26 Jul 2006 00:01:26 +0000

GNATS

The FSF/GNU problem report management system.

OpenSolaris: Noted on zfs-discuss

stephen@blueslugs.com (Stephen Hahn) — Tue, 25 Jul 2006 19:07:37 +0000

Since the break, I’ve been working with Gary, Steve, Derek and Alan—plus Dermot and Bonnie, neither of whom blog—on the source code repository hosting for opensolaris.org, which I expect will come out of Beta pretty soon. The hosting model we worked out showed its versatility last week, when we added Mercurial support to the test site with only a few hours of work. (We had been doing the initial development using only Subversion repositories.) There are still a few bugs to fix and a few features to add, but adding this capability will take our open development effort a sizeable step forward.

In fact, I need to put the SCM work aside soon and join in the structuring of the final review of the OpenSolaris constitution. If you’re interested, see the Governance category at genunix.org.

But I’m still keeping up with the mail flows. One particularly interesting alias is zfs-discuss, which is a mix of the ZFS developers, early adopters, and potential deployers. There’s been discussions on reliability, Thumper, and a few issues people have had using zfs(1M) and zpool(1M). Occasionally, something more curious pops up, like Jim Connors’s posting, where he reveals

… I have a version of Solaris 10 U2 with ZFS functionality with a disk footprint of about 60MB. Creating a miniroot based upon this image, it can be compressed to under 30MB. Currently, I load this image onto a USB keyring and boot from the USB device running the Solaris miniroot out of RAM.

Neat.

[ T: OpenSolaris Solaris zfs Mercurial Subversion ]

Docutils: Documentation Utilities

stephen@blueslugs.com (Stephen Hahn) — Mon, 24 Jul 2006 18:14:10 +0000

Docutils: Documentation Utilities

A lightweight markup language and utilities. Written in Python, and with related activity in the PEP processes.

Solaris VoIP – VoIP Telephony Software and Hardware for Solaris and Asterisk

stephen@blueslugs.com (Stephen Hahn) — Mon, 17 Jul 2006 18:25:59 +0000

Solaris VoIP – VoIP Telephony Software and Hardware for Solaris and Asterisk

“Solaris Voip holds the responsibility for the development of Zaptel device drivers for Solaris – the most advanced UNIX operating system in the world. Additionally, Solaris Voip maintains Asterisk PBX packages for Solaris.”

[quant-ph/0312174] Quantum Computation, Categorical Semantics and Linear Logic

stephen@blueslugs.com (Stephen Hahn) — Fri, 14 Jul 2006 00:36:59 +0000

[quant-ph/0312174] Quantum Computation, Categorical Semantics and Linear Logic

A follow-on paper. From the abstract: “In our semantics, terms inhabit certain Hilbert bundles, and computations are interpreted as the appropriate inner product preserving maps between Hilbert bundles. These bundles and maps form a symmetric monoidal c

[quant-ph/0307150] A Lambda Calculus for Quantum Computation

stephen@blueslugs.com (Stephen Hahn) — Fri, 14 Jul 2006 00:35:42 +0000

[quant-ph/0307150] A Lambda Calculus for Quantum Computation

Andre’s paper.

Andre van Tonder: A lambda calculus for quantum computation

stephen@blueslugs.com (Stephen Hahn) — Fri, 14 Jul 2006 00:35:08 +0000

Andre van Tonder: A lambda calculus for quantum computation

A simulator for Scheme extended to allow quantum calculation.

Open Source Developers' Conference, 20-22 February 2007, Israel

stephen@blueslugs.com (Stephen Hahn) — Tue, 11 Jul 2006 22:42:40 +0000

Open Source Developers’ Conference, 20-22 February 2007, Israel

In Netanya; paper proposals due September 2006.

GEOPRODUCTS CORP – Advanced Construction Technology – salvaverde

stephen@blueslugs.com (Stephen Hahn) — Mon, 10 Jul 2006 20:19:05 +0000

GEOPRODUCTS CORP – Advanced Construction Technology – salvaverde

A plastic product, composed of hexagonal cells, that allows the construction of a grassy driveway.

Jiniology Articles

stephen@blueslugs.com (Stephen Hahn) — Fri, 30 Jun 2006 04:43:28 +0000

Jiniology Articles

A collection of articles on Jini programming.

How to Write Parallel Programs

stephen@blueslugs.com (Stephen Hahn) — Fri, 30 Jun 2006 04:40:27 +0000

How to Write Parallel Programs

Nicholas Carriero and David Gelernter’s 1992 text. Uses Linda as a model.

Edsger Dijkstra – How do we tell truths that might hurt?

stephen@blueslugs.com (Stephen Hahn) — Thu, 29 Jun 2006 17:14:03 +0000

Edsger Dijkstra – How do we tell truths that might hurt?

It’s not hard to think of a few more truths that could be added to Dijkstra’s list.

How to design programs

stephen@blueslugs.com (Stephen Hahn) — Sat, 24 Jun 2006 03:46:36 +0000

How to design programs

A textbook.

System V IPC at OpenSolaris.org

stephen@blueslugs.com (Stephen Hahn) — Fri, 23 Jun 2006 18:54:36 +0000

System V IPC at OpenSolaris.org

Dave Powell posted detailed information about Solaris 10’s handling of System V IPC (shared memory, message queues, and semaphore) limits via resource controls.

Brooks Brothers | Classic Tie Knots

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 Jun 2006 09:12:17 +0000

Brooks Brothers | Classic Tie Knots

Brooks Brothers reviews (with step-by-step instructions) various tie knots.

A List Apart: Articles: How to Plan Manpower on a Web Team

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 Jun 2006 07:53:51 +0000

A List Apart: Articles: How to Plan Manpower on a Web Team

Shane Diffily proposes metrics relating content generation time, complexity, and activity to website staffing.

Dive Into Python

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 Jun 2006 07:28:48 +0000

Dive Into Python

Mark Pilgrim’s online text on Python.

Still, hot

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 Jun 2006 05:35:44 +0000

It’s past ten o’clock in the evening, but it’s still 88°F—31°C—and there’s little to no wind. That means the back office is a little too hot to be comfortable…

Top 10 Solaris Installation Annoyances at VMUNIX Blues

stephen@blueslugs.com (Stephen Hahn) — Wed, 21 Jun 2006 19:03:58 +0000

Top 10 Solaris Installation Annoyances at VMUNIX Blues

Mark Mayo on Solaris installation annoyances.

The View from the Moon — OpenSolaris birthday festivities

stephen@blueslugs.com (Stephen Hahn) — Thu, 15 Jun 2006 15:15:03 +0000

The View from the Moon — OpenSolaris birthday festivities

Dan Price catalogues many of the interesting posts from the OpenSolaris anniversary blog postings.

It Must Be Time for Tea — .o files: Just say no

stephen@blueslugs.com (Stephen Hahn) — Thu, 15 Jun 2006 15:11:53 +0000

It Must Be Time for Tea — .o files: Just say no

Mike Kupfer explains the difficulties associated with shipping closed object files for the OpenSolaris ON build, and justifies the few exceptions where we do.

Anniversary sunlight

stephen@blueslugs.com (Stephen Hahn) — Thu, 15 Jun 2006 07:20:57 +0000

Midday during the anniversary, a few of us zipped off campus for a short lunch in downtown Redwood City. Obviously, this was part shared celebration, but also I wanted to make sure that Dan and Steve exited #opensolaris in favour of a few minutes outdoors and some refueling.

I’m not enough of a photographer to catch candidly seven faces, from table’s end. But lunch was pleasant—and my presence as a parental unit at home was mandatory this evening, so I will only get stories of the meet-up.

[ T: OpenSolaris anniversary 2006 ]

OpenSolaris: on one year

stephen@blueslugs.com (Stephen Hahn) — Wed, 14 Jun 2006 17:00:00 +0000

Anniversaries provide a convenient excuse to be contemplative—not that I would struggle to justify contemplation. So, I’ll take a moment to look both back and forward. At one year, the OpenSolaris effort has exhibited steady growth across the entire period; Patrick’s community metrics have shown that the interest in participating isn’t waning. The community has passionate evangelists and communicators, and their efforts in blogs, at conferences, and in various forums are unflagging. Just as valuable have been the reasoned critiques of OpenSolaris’s technical gaps (and other challenges)—although there has been some concern about the conflict between advocacy and criticism.

As long as we keep this conflict vigorous and civil, and avoid dogma, it will be constructive. I know that the technical contributors have examined and adjusted for every concrete criticism they’ve read; these realizations I expect will emerge as more projects move to developing on opensolaris.org directly. On the other hand, hearing these critiques allows the evangelist to amend their presentation—to discover workarounds, to make counter-recommendations, and so forth.

I’ve written at length about the internal transformations underway as we move from an open-within-the-institution development model to one that is open to all. It’s been just as gratifying to see the changes in the community-wide conversations, as the formerly-Solaris-only jargon evolves into a wider community idiom that we all share. The dropping of the most awkward or provincial terms is no cause for sorrow; the acquisition of new and useful expressions—like the Apache-style “+1” for “I agree”—has been a pleasant surprise. (Expressions that are unambiguous in email seem to be rare and therefore valuable.)

One of the things I’m looking forward to is unanticipated use. Jason Hoffman’s entertaining presentation on their use of Solaris Express in production probed the boundaries of surprising use; I’ll enjoy getting caught off guard by at least one project in the coming year. Or so I predict.

Don’t get me wrong: obstacle-free anticipated use is good, too. But CDDL is so rational and relaxed compared to previous Solaris source arrangements that the bulk of previously impossible collaborations are now legit—so I want to be surprised by their output, not by their existence.

Whether or not my hoped-for dramatic novelty arrives, the coming year will bring change. Jim’s off to a new stage on the other side of the Pacific; the community will hold elections for a new Governing Board, operating under a new and newly ratified Constitution; new requirements, be they technology-driven or technology-driving, will present new challenges for software systems. We might even get a new lead-in image on opensolaris.org.

There’s no doubt it’s taken much time and work to get here. There’s also no doubt that it’s been worth it. Enjoy the year past and the year to come—there’s much to do.

[ T: OpenSolaris anniversary 2006 ]

OpenSolaris one year on: Success or failure?: ZDNet Australia: News: Software

stephen@blueslugs.com (Stephen Hahn) — Tue, 13 Jun 2006 21:32:02 +0000

OpenSolaris one year on: Success or failure?: ZDNet Australia: News: Software

Renai LeMay (ZDNet Australia) covers the OpenSolaris community’s first anniversary.

OpenSolaris One Year On

stephen@blueslugs.com (Stephen Hahn) — Tue, 13 Jun 2006 21:30:53 +0000

OpenSolaris One Year On

Slashdot discussion of OpenSolaris community, at one year.

smf(5): Recent (and open) developments

stephen@blueslugs.com (Stephen Hahn) — Thu, 08 Jun 2006 22:29:50 +0000

Keeping up with the technical discussions on opensolaris.org probably requires a full time team. Even with Glynn‘s weekly news and Eric‘s list summaries, I’m still not able to track it all—which shouldn’t be a surprise, as I couldn’t track everything when it was all closed (and somewhat smaller). In the various areas close to smf(5), there have been some developments worth mentioning.

David Bustos unveiled the Enhanced SMF Profiles project. There have been some clear shortcomings in our current use of configuration snapshots across upgrade operations, and enhanced profiles will clear these up very nicely. (But, like any well thought out feature, enhanced profiles will make a number of other configuration manipulation problems much more tractable.)

One such consumer will be the Network Auto-Magic project, which John Beck has been pursuing. The NWAM team recently released an initial prototype, which is letting them explore some of the issues—architectural, usability, generality—with making such a broad change.

I need to review the source to the iwi wireless driver to see whether it issues the appropriate DL_NOTE_LINK_UP and DL_NOTE_LINK_DOWN notifications to ip

Finally, we’ve long discussed extending the <template> section of the smf(5) service manifests with various forms of metadata that we believe might be useful to management software. Dave Powell wrote and released a draft discussion of template extensions under the auspices of the Visual Panels project. These properties will be useful to other smf(5) commands as well—svccfg(1M) being one of the most obvious target consumers.

Not all smf(5) work can be done virtually. Last month, Liane had a good trip to SANE 2006. She’s posted the slides from her tutorial as well.

Probably need to get out there talking again soon myself.

[ T: OpenSolaris Solaris smf ]

Little patch for Mediawiki external editor

stephen@blueslugs.com (Stephen Hahn) — Tue, 06 Jun 2006 18:12:26 +0000

For the OpenSolaris governance discussions, the community is using the wiki at genunix.org. The genunix team installed Mediawiki, used most famously for the Wikipedia, to support communal editing.

I’m a creature of Vim, and am quite happy with the recently released Vim 7. Tying the external editor feature of Wikipedia to Vim is straightforward, if you install Erik MÃ¶ller’s ee script—and tweak it a little.

Below you’ll find a modified version that respects the http_proxy setting, handles the case where no matching site is present in your ee.ini file, and adds a configuration option to not append the “using an external editor” string to your article’s change summary.

[ T: OpenSolaris Solaris genunix Mediawiki ]

Reading, 2006Q1

stephen@blueslugs.com (Stephen Hahn) — Tue, 06 Jun 2006 06:59:36 +0000

(I’ve realized I need to deal with a much-too-high interrupt rate at work, in part by ensuring I take out a bit of time for leisure. Here’s an entry I started in April.)

Over the past few years, my reading rate has climbed; perhaps I’ve unwittingly dropped a periodical, or maybe I’m getting back to splitting my reading time across a few books at once. In any case, I thought it would be pleasant to get back to recommending recent reading I’ve enjoyed.

When we were in Long Island at the end of our winter vacation, I secured sufficient late night reading to get through three 20th century classics:

Chesterton’s The man who was Thursday (1907) [Wikipedia] [Gutenberg], which was an entertaining story that appears to have simultaneously pioneered the spy novel, takeoffs of the spy novel, and a number of forms of “postmodern paranoid” storytelling. It would be interesting to contrast with Conrad’s The secret agent (1907), but I won’t have time to work through these contemporary novels in parallel.
Christopher Morley’s [Wikipedia] Parnassus on wheels [Gutenberg] and The haunted bookshop [Gutenberg]. These were light novels (about booksellers); one of the funniest parts was the introduction given in the edition of Parnassus that I read, which suggested that full comprehension of the novel would only be available to readers born in a three to four month period in the early 1920s. I wasn’t, but the books are still fun—although I never worried about highwaymen, however shabby, in any of my traverses of Connecticut.

One of Benjamin or Nathaniel, and sometimes both, would accompany me to the Redwood City library. We’ve been finding some fun books, plus I can try to read science fiction again.

Ben and I have been working our way, planet by planet, through Dav Pilkey‘s Ricky Ricotta and his mighty robot series. The stories are on the corny side for adults, although I admire the determined construction of a monsters-on-planets cosmology. (Plus the cheese surnames on mouse characters are good silliness.)
I blitzed through Cory Doctorow’s Eastern standard tribe, and Bruce Sterling’s Zeitgeist and The zenith angle. I liked the last of these best; the other two were simple. (I like Doctorow’s story ideas initially, but I find that the unfolding is too pat—obvious complications of the hypothesis are ignored.)
Out of some unknown reptilian duty—I started following this series after my undergraduate degree—I read Robert Jordan’s Knife of dreams, which is the eleventh book in his Wheel of time series (not counting prequels). Apparently, the series will end with Book Twelve and, for what seems like forever, some plotlines appear to be coming to their conclusions.
From Ben’s continuing exploration of prehistory, I recommend Alan Turner’s National Geographic prehistoric mammals and Tim Haines’s Walking with prehistoric beasts. The latter is a companion to the Discovery Channel series—narrated by Stockard Channing—and appears to be illustrated with high quality stills from the shows, along with expanded text retellings of each episode. The National Geographic book is more of a complete text about the major prehistoric mammal groups. We enjoyed both of these enormously—suggestions on further reading are welcome, as I fear we’ll be off into college texts otherwise.
My final novel of the quarter was Philip Roth’s The plot against America, which was very finely written. I kept comparing it to the famous science fiction novel by Philip K. Dick, The man in the high castle, which is also an alternate history of World War II. Dick’s novel eventually focuses on the detection of wrongness by the inhabitants of his reality; Roth’s eponymous protagonist on more personal disquiet. Recommended.

Of course, none of us stopped reading in April, but a quarterly summary seems like a reasonable balance.

Ilya Voronin's Blog: Managing instances within SMF

stephen@blueslugs.com (Stephen Hahn) — Tue, 06 Jun 2006 06:02:23 +0000

Ilya Voronin’s Blog: Managing instances within SMF

Ilya constructs service instances dynamically by piping svccfg snapshot output through Perl and back to svccfg (via xargs). Nifty, but we need to make this easy.

Limited reputation sharing in P2P systems

stephen@blueslugs.com (Stephen Hahn) — Wed, 31 May 2006 04:21:26 +0000

Limited reputation sharing in P2P systems

Marti and Garcia-Molina of Stanford, presented at the 5th ACM conference on Electronic commerce.

Tom White's Blog: MapReduce

stephen@blueslugs.com (Stephen Hahn) — Thu, 25 May 2006 17:08:18 +0000

Tom White’s Blog: MapReduce

Review of the Nutch team’s work to implement a variant of MapReduce in Java.

ongoing · On Grids

stephen@blueslugs.com (Stephen Hahn) — Thu, 25 May 2006 17:00:08 +0000

ongoing · On Grids

Tim Bray reviews current developments in grids and grid software.

SourceForge.net: Voting systems toolbox

stephen@blueslugs.com (Stephen Hahn) — Tue, 09 May 2006 20:45:44 +0000

SourceForge.net: Voting systems toolbox

A Java implementation of Single Transferable Voting (Meek).

PeterVG : Software / svclog

stephen@blueslugs.com (Stephen Hahn) — Tue, 09 May 2006 18:10:09 +0000

PeterVG : Software / svclog

Peter van Gemert has written a nice little script that uses the restarter/logfile property to fetch and display the recent log contents for a service instance.

http://www.jini.org/jdp/Constitution.html

stephen@blueslugs.com (Stephen Hahn) — Mon, 01 May 2006 21:33:58 +0000

http://www.jini.org/jdp/Constitution.html

The Jini community’s constitution. One of a collection of documents outlining how their community functions.

How the ASF works – The Apache Software Foundation

stephen@blueslugs.com (Stephen Hahn) — Mon, 01 May 2006 21:32:28 +0000

How the ASF works – The Apache Software Foundation

Apache’s governance document.

Paul McClintock, PRP, Bylaw Articles

stephen@blueslugs.com (Stephen Hahn) — Sat, 29 Apr 2006 04:54:18 +0000

Paul McClintock, PRP, Bylaw Articles

A collection of example bylaws from various parliamentary authorities.

Metadata as a 'filing system' – Lifehacker

stephen@blueslugs.com (Stephen Hahn) — Thu, 27 Apr 2006 21:51:09 +0000

Metadata as a ‘filing system’ – Lifehacker

A little bit about Vista and MacOS X generic features, but mostly about using Quicksilver. There’s another way to do this, of course.

XWiki . Main . WebHome

stephen@blueslugs.com (Stephen Hahn) — Thu, 27 Apr 2006 20:21:41 +0000

XWiki . Main . WebHome

A Java-based Wiki.

Gtags

stephen@blueslugs.com (Stephen Hahn) — Thu, 27 Apr 2006 18:29:22 +0000

Gtags

Client/server tags implementation for large source trees.

Scale with Rails

stephen@blueslugs.com (Stephen Hahn) — Wed, 26 Apr 2006 20:13:10 +0000

Scale with Rails

Jason Hoffman of Joyent on their Solaris/Tx000 datacenter plan.

Progress in open development

stephen@blueslugs.com (Stephen Hahn) — Wed, 26 Apr 2006 07:58:14 +0000

Just prior to this week’s news, Glenn had an all-hands for the Operating Platforms Group. Since the all-hands is performed once for the Pacific through European business day and again later for Asia through Pacific working hours, I thought I would compose my presentation in full in the hopes of being consistent, and knowing that I could share it here as well. I fear the later presentation will become known as the “apple juice” talk, as my not-quite-two-years-old son Nathaniel came to the phone a few times to inform the audience of his preferred remedy to a sudden thirst. Your comments are welcomed—we’ll be expanding and revising the talk and its material as the program and its tools mature. The talk benefitted from criticism and suggestions from Jim and Karyn.

Overview

Thank you for this opportunity to speak with you. Over the next thirty minutes, I thought that we could review how software development for the various consolidations will change into open development, on opensolaris.org. I will attempt to be clear about what you can do today and what’s yet to come. Along the way, I’ll also share what I believe are candid criticisms of what we have so far.

In the interests of lending some structure to this talk, we’ll look at typical tasks an individual developer tends to do in the course of working on an operating system component. On a given day, a developer might

read and comment on a modified bug report,
update a child source tree,
change some code in their source tree,
build the binaries in their source tree,
solicit or provide feedback for a code review, or
make or evaluate a request to integrate reviewed code into a consolidation’s gate,

as actions that involve small group of people. Our developer might also be taking actions that involve a larger number of people, up to the entire OpenSolaris community. For instance, they could be

asking or answering questions in a public forum,
proposing or commenting on a new interface or implementation choice, or
proposing or joining a new area of investigation.

I intend to touch briefly on each of these examples, which I hope will make it clear that we’re committed to making it feasible—even easy—to perform the complete set of development tasks in an open environment.

Bug reporting

The first part of my morning is still spent sipping a hot cup of tea and reading new and update bug reports in the categories I’m interested in. For any of the reports I might wish to comment on, I can use the Java client to connect to the bug database and make appropriate updates, perhaps making connections to related bugs or adding the email of a developer I suspect might be concerned about this issue. Unfortunately, none of these actions is presently convenient for a developer without that access: bug submission and manual querying is available at bugs.opensolaris.org, but ongoing reporting and updates are not.

There are two adjustments to make if you’re using the bug database. First, use the Description field for the bulk of your updates, as all other fields are not published. This means the end of the “See Comments” bug idiom. And also, check Hook 6, which will contain the OpenSolaris ID of the submitter; if you make a significant status change, send the submitter an update.

We’ve given input into one effort to examine the bug database’s role in open development, and we’re funding work to improve the functionality of bugs.opensolaris.org. But if you have Bugster access and can help a contributor who can’t make an update, please do so.

There are additional complications for the present bug database with respect to upstream and downstream relationships—with other communities whose software we rely upon or whose rely upon ours—but those aren’t part of a typical developer’s day.

One aspect of bug reporting we have noticed is that the bite sized bugs, with the keyword “oss-bite-size“, provide a very effective starting point for developers new to OpenSolaris. I encourage you all to start marking introductory bugs with this keyword; because this keyword has been so effective, we’re going to start tracking the marking and fixing rates as a metric for the program. For experienced developers, I appeal to you to respect some kind of rough law of “conservation of bite-sized bugs”; if you fix one, mark another one.

Source code management

Usually my tea is still warm when I’m deciding whether or not my current source tree is sufficiently up-to-date. Generally, the tea is tepid once my bringover is done.

If you’re following the tools-discuss list on opensolaris.org, you’ll know that we recently selected a source code management tool called Mercurial to be the distributed source code management utility for opensolaris consolidations. We will be working with the Mercurial community to get a few features implemented, and I expect that most of you will be pleased with the performance of this very nice utility. I expect in most cases my tea will still be pleasantly warm.

There will be an interim period where we’ll be publishing Mercurial bundles externally, but still using TeamWare inside Sun. This period will mostly be spent on tools updates, but in most cases we should see all developers on an equal footing, whether their primary consolidation uses Subversion or Mercurial.

Editing code

Once I’ve synced up, I’m in position to start making changes. I promise not to recommend a specific editor. Generally, expectations about code have not changed: the C style is expected and enforced, other conventions will be verified by your code reviewers.

Practices across internal and external contributors are common for this stage.

New files written by contributors to ON should contain the CDDL copyright block; other consolidations may have a different default license. An example is given for common types of files in

$SRC/prototypes/prototype.{Makefile, c, csh, h, java, ksh, man, s, sh}

in the ON consolidation. If your work involves integration into a Solaris release prior to 14 June, then the CDDL block should not be carried back into that source tree.

The “wx cddlchk” invocation, which is also part of “wx nits” and “wx pbchk”, will check that the license block is correct and up-to-date.

Building a consolidation

Typically, building your consolidation or a portion thereof takes some time, usually based on your system’s performance or how many people you share that system with. Once you’ve started your build, this is generally a good time to stretch or make a new cup of tea.

The various consolidations will tax build systems differently. With the recent integration of GCC shadow compilation, ON build system administrators need to be aware that there are now potentially two compiler invocations for each changed C file. On aggressively configured systems, this may result in paging.

For full builds, the experience of an internal and external developer should be equivalent.

A key difference at present is the availability of test suites. As you might suspect, there are many test suites that depend on the Sun corporate network environment; the result is that the suites are not presently available. Under the present sponsor process, a code submission is tested by the sponsor, or testing is arranged by the sponsor. There are a number of paths to balancing the test responsibility: the Test Community is exploring a hosted self-service test run, and would probably welcome discussions about what tests a developer should be able to run in their own environment to validate their changes.

Code review

With validated changes in hand, a developer is ready to start down the path towards integration. This commences with getting one or more developers to review your changes to the code. Of course, because you want your reviewers to find actual problems, you already made certain that your changes build cleanly and have no obvious style errors.

Finding a code reviewer often involves checking who’s around, either physically or by various network protocols. The webrev tool allows you to construct a small web tree that contains your changes in a variety of formats so that a reviewer can work through your modifications and make appropriate suggestions. This makes it more convenient to have reviewers who aren’t physically contiguous to you.

Since webrev generates a tree of HTML documents, all you need to publish your code review materials is a place to put them. This is easiest if you have an external site, and is an area where a developer primarily using the Sun corporate environment is at a disadvantage.

In the interim, Dan Price and Steve Lau have provided a “code review swamp” hosted at cr.grommit.com to offer a place to host your webrevs. The swamp is set up to handle publishing current format webrevs today; instructions are present on that site.

Now, I’ve also asked the Tools community to examine enhancing webrev to put the changes into a single document and use browser features to make the review experience equivalent. In this case, publication becomes substantially easier, and even blogs become a possible location for publishing code reviews.

Even with publication eased, finding reviewers can still end up relying on personal networks, which is difficult for a developer new to the community. I also believe that the request for a code review needs to be narrowcasted to those fellow developers who are interested in the area that is changing. So I believe we’ll need to develop code review workflow support on the opensolaris.org site. If you’re interested in how this feature might work, we will be discussing it in the Tools community.

Code integration

With support for your changes from your code reviewers, and after building and testing your possibly modified code once more, you are ready to make your request to integrate—which has long been known by its acronym, RTI.

At present, the “request to integrate” is made via a tool only accessible on the Sun network. That, and the current source code management situation, led to the current sponsor process.

It’s worth noting that each time I mention an aspect where Sun corporate network access is required to access a resource, the sponsor is performing the work on behalf of the contributor. That can add up to a substantial amount of work for some changes. Please be supportive and respectful of both the contributor and the sponsor; if you’ve the time to be a sponsor, or assist a sponsor-contributor interaction by participating in a review or running tests or shepherding a build, please consider helping out.

Once the various consolidations move to external writable consolidations, we’ll see direct integration by all committers, but for new developers, the sponsor process will remain in place. Aspects will become simpler: the new source code tools make passing patches around substantially easier than before. As new capabilities are brought online on opensolaris.org, like code review hosting and self-service testing, we’ll see the sponsor-contributor process distilled down to its core: mentoring on technical participation in operating systems development.

Forum participation

Of course, technical guidance isn’t only provided on a one-on-one basis, and not everyone is intending that their changes target mainline. In the various discussion forums, you’ll see questions on how best to do something, or where it’s presently done, or whether a particular choice has drawbacks.

You should engage in these conversations. Questions, and the context behind them, is valuable to all community members, whether they are planning to develop code or merely consider that feature in their deployment. A considered answer has value beyond the immediate conversation; all accurate technical content, be it a forum response, project documents, or blog entries, will increase the search engine cross section for opensolaris.org. As with any public forum, your participation should be confined to the technical aspects of the discussion; private information concerning your organization is not suitable as a topic for public discussion. That’s not a new restriction—merely common sense.

I’ll talk about forum creation in a moment. But first, I want to talk about a few special forums on opensolaris.org.

opensolaris-discuss is, in an extremely loose metaphor, the town square of opensolaris.org. The various procedures that the community’s Governing Board has developed generally involve making an announcement or request for discussion on opensolaris-discuss. But, like any public place, there are conversations on pretty much any topic; it’s a high traffic alias.

opensolaris-code has in recent days had its role clarified: it’s a purely technical discussion alias on any aspect of the various components of OpenSolaris, architectural, implementation, or otherwise.

Architectural process

The third forum I want to mention is the ARC community and its associated mailing list, because it brings us back to our typical development activities.

Occasionally, in the course of fixing a bug or considering a problem, we realize we need to add a new interface to some OpenSolaris component. The process for doing this is to take the interface through architectural review. We’ve had a couple of contributors take their new interfaces through the fast track process, and reviewed some of the difficulties associated with each of those cases.

We also have found that the site isn’t well structured for the frequent publication required to track the currently active ARC cases, or to publish key historical cases.

So, while we have an awkward way of making the process participatory, the correct course appears to be to take the fast track process for open consolidations and modify it such that it can be hosted on opensolaris.org. If you’re interested in this topic, we’ll be starting this discussion in the ARC community.

Proposals

But sometimes, you realize that the changes you’re envisioning are sufficiently large that they will require a group effort. This effort usually needs a forum of its own, plus a tree of documents, plus a source code repository.

opensolaris.org can handle this with its project hosting support, which will shortly offer the source code hosting feature. To get a project, you just propose the project on opensolaris-discuss. Over time, your project may garner interested individuals and communities, whose association will become a way for community members to identify projects they may find interesting. In the meanwhile, your project is well underway.

New projects and even existing projects for Nevada should use opensolaris.org for hosting their project; please don’t wait until you’re ready to integrate. That’s far too late. As you might suspect, we’re developing a metric to monitor the progression towards external hosting.

The other kind of forum that we can host on opensolaris.org is called a community, which has been confusing in a manifold sense. The term itself is overloaded: we can talk of the OpenSolaris membership as a whole as a community, or of a specific OpenSolaris community. And the role of a community in OpenSolaris governance is still being debated by the governance working group.

But I can tell you a few aspects of communities that might help you determine whether your initiative is a potential community. First, a community is not a program, strategy, or lifestyle. If you haven’t heard these terms in reference to software development, you may wish to plug your ears for a little while. The programs, and so forth, are mechanisms for organizing projects with respect to the business priorities associated with the development of one distribution of OpenSolaris. Second, those distribution groupings are likely to have a finite period of applicability whereas a community, because it will be identifying its contributors to the larger OpenSolaris membership, is expected to be a long lived group.

Finally, a community is expected to appeal to a fairly wide subset of the membership; it attempts to gather combine a collection of interests into a coherent group. That’s fuzzy and it’s made a bit worse by the fact that there are some communities that were grandfathered from being proposed prior to the introduction of a community proposal process or prior to the introduction of project hosting. Reorganizing some of the communities will be an area of effort over the next few months, but the basic guideline is propose your effort as a project.

Summary

I hope this review of typical developer activities, how they are changing as we develop in an open community, and how certain aspects of these activities are simply hard to do if your workstation is not on Sun’s network. None of these deficits is likely new to those Sun developers who work from a remote location or from a campus distant from California. And of course there are other aspects to collaboration that we need to explore, but that would be another talk entirely. It’s my intent that we get to a situation where a developer, without reference to affiliation or location—but possibly dependent on their available bandwidth—can participate equivalently in opensolaris development. That will require ideas, feedback, code, and action from all of us.

Thanks.

Your questions, please.

Links

http://cvs.opensolaris.org/source/xref/on/usr/src/prototypes/

http://selenic.com/mercurial/

http://cr.grommit.com/

http://opensolaris.org/os/projects/
http://opensolaris.org/os/communities/

A short set of slides were made to accompany the talk: PDF, ODP.

This talk and the slides are each licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License.

[ T: OpenSolaris Solaris ]

Google's Secret Weapon

stephen@blueslugs.com (Stephen Hahn) — Fri, 21 Apr 2006 22:56:58 +0000

Google’s Secret Weapon

Amazon staffer compares their recruiting against Google’s (and Microsoft’s), circa 2004.

NotaBene: Solaris 10 — new features

stephen@blueslugs.com (Stephen Hahn) — Fri, 21 Apr 2006 20:09:32 +0000

NotaBene: Solaris 10 — new features

One person’s (negative) take after a Solaris 10 training course. Sounds like the course doesn’t cover “svcs -x”, among other aspects.

My Experiences With Subversion

stephen@blueslugs.com (Stephen Hahn) — Thu, 20 Apr 2006 16:50:46 +0000

My Experiences With Subversion

A detailed article by Simon Tatham about his experiences with Subversion.

ARLIS, Northern California: Bylaws

stephen@blueslugs.com (Stephen Hahn) — Wed, 19 Apr 2006 21:11:54 +0000

ARLIS, Northern California: Bylaws

Bylaws of the Art Libraries Society of Northern California.

Debian Constitution

stephen@blueslugs.com (Stephen Hahn) — Wed, 19 Apr 2006 21:04:50 +0000

Debian Constitution

Constitution for the non-commercial Linux distribution, Debian.

Bylaws of the California Community College Student Affairs

stephen@blueslugs.com (Stephen Hahn) — Wed, 19 Apr 2006 21:02:48 +0000

Bylaws of the California Community College Student Affairs

Example of a California non-profit’s bylaws, with multiple membership classes.

IComplete – A code completion system

stephen@blueslugs.com (Stephen Hahn) — Wed, 19 Apr 2006 16:56:54 +0000

IComplete – A code completion system

Symbol completion utility for C/C++ and potentially Java. Command line based; support for Vim integration.

Your Family Disaster Supplies Kit

stephen@blueslugs.com (Stephen Hahn) — Tue, 18 Apr 2006 23:31:20 +0000

Your Family Disaster Supplies Kit

FEMA’s pamphlet on disaster preparedness kits (via Pueblo).

Largest carnivorous dinosaur (fossil) found in Argentina

stephen@blueslugs.com (Stephen Hahn) — Tue, 18 Apr 2006 17:53:31 +0000

Big news for our household: the New York Times has an article on the announcement of the discovery of Mapusaurus roseae, which is larger than Giganotosaurus (which is in turn felt to be larger than Tyrannosaurus rex). Both Mapusaurus and Giganotosaurus were discovered in Argentina, and perhaps hunted Argentinosaurus, a huge plant-eating sauropod.

Ben’s home sick today, so reading about the find—particularly of a new, giant meat-eater—was a nice break.

Using Solaris SMF

stephen@blueslugs.com (Stephen Hahn) — Tue, 18 Apr 2006 03:26:42 +0000

Using Solaris SMF

Chris Josephes writes a nice summary article about smf(5) for O’Reilly’s SysAdmin channel.

Historical CA DMV VLF online

stephen@blueslugs.com (Stephen Hahn) — Sun, 16 Apr 2006 14:43:29 +0000

I’ve been wrapping up our tax returns, running reviews and reviewing an actual paper copy. One document (well, two) that I couldn’t find were the bills from the California Department of Motor Vehicles for the car registrations. But I was very pleased to find that they have an online retrieval system to help you find your vehicle license fee (VLF) for 2004 – 2006.

I bet I filed the registration bills in a “special” folder… which I’ll find in August.

LighttpdOnSolaris – lighttpd – Trac

stephen@blueslugs.com (Stephen Hahn) — Wed, 12 Apr 2006 22:21:08 +0000

LighttpdOnSolaris – lighttpd – Trac

A service manifest for lighttpd, “a secure, fast, compliant, and very flexible web-server”.

Dimpler

stephen@blueslugs.com (Stephen Hahn) — Wed, 12 Apr 2006 21:30:32 +0000

Dimpler

A tool for creating dimples in necktie knots.

Mailman, the GNU Mailing List Manager

stephen@blueslugs.com (Stephen Hahn) — Wed, 12 Apr 2006 00:00:59 +0000

Mailman, the GNU Mailing List Manager

Main page for Mailman, a mailing list manager written in Python.

subversion.tigris.org

stephen@blueslugs.com (Stephen Hahn) — Tue, 11 Apr 2006 23:59:29 +0000

subversion.tigris.org

Main page for Subversion, a centralized source code management solution, designed to address the shortcomings of CVS.

Program technical status, 11 April

stephen@blueslugs.com (Stephen Hahn) — Tue, 11 Apr 2006 23:49:16 +0000

[Repost from forum thread.]

Since my previous status summary, written in December, a number of technical aspects to the program have made progress. (It’s becoming clear to me that I only write these when we actually deliver stuff.)

Website capabilities.

Basic community and project hosting seems to functioning reasonably well. There are 28 approved projects, some of which are still preparing their initial content, and 41 communities. It’s probably worth noting that we can easily support multiple mailing lists for each project or community, if the project or community leadership requests additional lists.

There is still ongoing development to simultaneously relax and tighten our Mailman moderation settings, so that a registered email address can send to any list and so that an unregistered email address will always bounce. Mailing list moderators will appreciate the latter.

Source code management (SCM).

The software to add repository hosting to projects is getting close to completion. Internal testing with the Subversion-only option suggests that we should be ready to do a beta deployment in the next six weeks. Project leads interested in hosting their source using Subversion as part of this beta should contact me, directly or via website-discuss.

We’re awaiting further comments on the tentative selection of Mercurial as the distributed SCM selection. Extending the hosting support to handle Mercurial repositories is straightforward, and I expect will be ready–with the then-current version of Mercurial–shortly after the deployment of the Subversion hosting beta.

The DSCM selection and related documents about hosting are available in the Tools community at

http://www.opensolaris.org/os/community/tools/scm/

In terms of Mercurial use for the ON consolidation, I would like to discuss releasing Mercurial bundles, with per-integration granularity, as part of the weekly drops Steve Lau has been doing. The drop frequency will hold at weekly, until we are hosting read-only ON on opensolaris.org. On tools-discuss, we’ll be talking about specific tools that need to be updated (or have the opportunity to be improved) because of the switch in the DSCM.

Continued source publication.

I believe that, since December, the Developer Tools, Documentation, Install, Network Storage, Solaris Freeware, and X Windows consolidations have all released portions or the entirety of their source trees. In many of these cases, an active community and/or project is available to discuss how these components can evolve.

The split ON tree has made deliveries easy, and Steve and others have continued to move code that we know to be unencumbered out of usr/closed and into usr/src (which is what you get from opensolaris.org for ON source).

ON GCC readiness.

Keith Wesolowski implemented shadow compilation with GCC as a means to keep the ON consolidation warnings-clean with respect to both the Sun Studio and the GNU C compilers. The choice of shadow compilation means that a typical developer’s test burden is still centered on the binaries produced with the primary compiler for the platform; at present, Sun Studio is the primary compiler for both SPARC and x86. Individuals and teams interested in a pure GCC build for those platforms or using GCC for ports to other platforms will still need to examine and test the binaries produced, but should not have to do “basic GCC cleanup”.

The ON GCC status, and related links, are available in the Tools community at

http://www.opensolaris.org/os/community/tools/gcc/status/

Governance development.

On February 8th, the CAB approved the OpenSolaris Charter. Glenn Weinberg, Vice-President of the Operating Platforms Group signed it, on behalf of Sun, on February 10th. The Charter is linked to by the CAB community at

http://www.opensolaris.org/os/community/cab/charter/

For those of you enthusiastic about historical documents, however trivial, Jim Grisanzio made a virtual exhibit of the charter Glenn signed.

http://blogs.sun.com/roller/page/jimgris?entry=opensolaris_charter_approved

(Whether we do a theatrical all-signatories-signing event is dependent on their schedules permitting them to physically congregate, I expect.)

The now-OGB (OpenSolaris Governing Board) is now working on the Constitution and/or By-Laws for the community. The cab-discuss alias is one forum for discussion; the developing document(s) are being composed in the wiki at genunix.org, specifically:

http://www.genunix.org/wiki/index.php/OpenSolaris_Governance

Input into this drafting process is, I believe, always welcomed.

After we get through the various repository deployments, there are a number of opensolaris.org software components that I would like to host, pretty much immediately: we need to improve many aspects of the site, including voting support for the OGB election, code review notifications, per-user repositories, and community, project, and user file management. Plus we could really use some new pictures for the front page. For all these developments, I expect website-discuss and tools-discuss to be the primary mailing lists. Please join in.

As always, please share your concerns; I am happy to receive them privately or on the list.

[ T: OpenSolaris Solaris ]

Edgewall Software: Trac

stephen@blueslugs.com (Stephen Hahn) — Tue, 11 Apr 2006 00:23:22 +0000

Edgewall Software: Trac

A nice, integrated software project hosting application. If you need to host your own, a good choice. Can use different SCM backends.

Continued indefinite drizzle

stephen@blueslugs.com (Stephen Hahn) — Mon, 10 Apr 2006 20:57:30 +0000

Bloglines again suggests the week is shaping up wetly:

marTux

stephen@blueslugs.com (Stephen Hahn) — Mon, 10 Apr 2006 20:49:56 +0000

marTux

An OpenSolaris distribution for SPARC. Apparently provides X.Org server for various Sun graphics adaptors.

Andrei's blog: CPU caps Preview on OpenSolaris

stephen@blueslugs.com (Stephen Hahn) — Thu, 06 Apr 2006 07:34:21 +0000

Andrei’s blog: CPU caps Preview on OpenSolaris

Andrei developed the fair share scheduling (FSS) class for Solaris 9, and I know he’s been thinking about adding caps for a long time. There were just a long list of other things that needed his attention.

Jython Home Page

stephen@blueslugs.com (Stephen Hahn) — Thu, 06 Apr 2006 04:31:07 +0000

Jython Home Page

Python for the Java VM.

JRuby Home

stephen@blueslugs.com (Stephen Hahn) — Thu, 06 Apr 2006 04:29:05 +0000

JRuby Home

Ruby atop the Java VM.

Fortune Magazine – Secrets of greatness: How I work

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 Apr 2006 04:10:05 +0000

Fortune Magazine – Secrets of greatness: How I work

Interviews with 12 successful individuals. Ideas about managing incoming/outgoing contact, for various media. Sometimes it’s just raw capability, like Judge Richard Posner: “I’m a very fast writer. I can write 20, 30 manuscript pages in an evening.”

Coyote Point Systems: Application Traffic Management Solutions

stephen@blueslugs.com (Stephen Hahn) — Tue, 04 Apr 2006 21:21:26 +0000

Coyote Point Systems: Application Traffic Management Solutions

An application-level load balancer. Can redistribute telnet, SSH, etc. as well as HTTP.

Indefinite drizzle to continue

stephen@blueslugs.com (Stephen Hahn) — Mon, 03 Apr 2006 19:12:33 +0000

Courtesy of bloglines.com:

It doesn’t look like I’ll be packing away the downspout extensions or the portable sump anytime soon…

Predictable – Bespoke services: network/rmi/registry

stephen@blueslugs.com (Stephen Hahn) — Thu, 30 Mar 2006 01:36:59 +0000

Predictable – Bespoke services: network/rmi/registry

An smf(5) service conversion for the Java RMI registry daemon, rmiregistry.

Ajax Link Tracker

stephen@blueslugs.com (Stephen Hahn) — Wed, 29 Mar 2006 21:32:32 +0000

Ajax Link Tracker

An AJAX-based approach to link selection statistics. Overlay of the current statistics (as percentages) for each link available via a keystroke.

Bespoke services: network/rmi/registry

stephen@blueslugs.com (Stephen Hahn) — Tue, 28 Mar 2006 23:26:56 +0000

Gary and I were recently prototyping an application that uses Java RMI, and so I ended up searching around to see if anyone has done a service conversion for rmiregistry(1). (rmiregistry(1) is the daemon that lets RMI clients find the available remote objects being served by various virtual machines on a given system.) Turns out no one has (or no one’s published it), which means it’s time to rev up the convert-o-tron.

Since we’re still developing our application and it’s likely we’ll change a definition or two, and since we need to restart the registry to cause the remote objects, we’re going to make our prototype service restart automatically if we restart the registry. That means our prototype service has a dependency on network/rmi/registry with specific restart_on behaviour, meaning that its service description has a fragment like the following:

<!--
As an RMI server application, we expect to be able to
register our RMI classes with the registry server.
-->
<dependency
name='rmi-registry'
grouping='require_all'
restart_on='restart'
type='service'>
<service_fmri value='svc:/network/rmi/registry' />
</dependency>

Inject that fragment into your various RMI servers’ descriptions (or the equivalent property group into the repository) and you’ll save a bit of time on application reinitializations.

So, if you’re interested, please feel free to take a copy of network/rmi/registry; comments and corrections welcome.

[ T: OpenSolaris Solaris smf RMI ]

My_Little_SMF_FAQ

stephen@blueslugs.com (Stephen Hahn) — Tue, 28 Mar 2006 22:01:42 +0000

My_Little_SMF_FAQ

Bernd Schemmer has collected a bunch of responses and blog posts, and constructed a short FAQ on smf(5)

joey interactive » Blog Archive » Desktop Backgrounds For Designers

stephen@blueslugs.com (Stephen Hahn) — Tue, 28 Mar 2006 18:54:14 +0000

joey interactive » Blog Archive » Desktop Backgrounds For Designers

A good collection of links to designers creating some slick desktop backgrounds.

FiST: Stackable File System Language and Templates

stephen@blueslugs.com (Stephen Hahn) — Mon, 27 Mar 2006 19:56:08 +0000

FiST: Stackable File System Language and Templates

Erez Zadok’s file system development system.

Please excuse the interruptions in the feed

stephen@blueslugs.com (Stephen Hahn) — Thu, 23 Mar 2006 05:27:43 +0000

I was having a bit of trouble with duplicate entries from Word Pre.cio.us (for entries containing '\'' characters in their title, and so was fiddling with transforming the bookmarks at del.icio.us, deleting duplicate posts, finally debugging the Word Pre.cio.us import code, and converting the bookmark titles back.

Sadly, some of the dates on the bookmarks got changed, so the time correlations between certain links have been lost.

This little patch seems to correct the issue for my PHP 5.0.4/MySQL 4/WordPress 1.5.1.3 setup, but shouldn’t be considered authoritative. I’ve sent it to the author for consideration. (The chief issue is whether the extra escaping I’ve removed is the correct excision, given potential backing database differences.)

In any case, the syndication feeds should be less bumpy henceforth.

Peter's Solaris Zone: JKstat

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Mar 2006 18:58:07 +0000

Peter’s Solaris Zone: JKstat

Peter Tribble has a JNI binding for Solaris kstats, and has graphical tools that provide example use of the interface.

Why Bitkeeper Isn't Right For Free Software

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Mar 2006 04:09:11 +0000

Why Bitkeeper Isn’t Right For Free Software

An essay linking choice of SCM to free software development organization. Vague and unjustified (by article) association of Linux integration process with SCM supported topologies. Cited elsewhere.

Robert, Henry M. 1915. Robert's Rules of Order Revised for Deliberative Assemblies

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Mar 2006 04:08:34 +0000

Robert, Henry M. 1915. Robert’s Rules of Order Revised for Deliberative Assemblies

Fourth edition online

Now integrating del.icio.us

stephen@blueslugs.com (Stephen Hahn) — Mon, 20 Mar 2006 15:52:38 +0000

I decided to start pulling my del.icio.us stream in—I’d been debating whether to republish it here or on my work blog for a couple of weeks. The Word Pre.cio.us script, patch, and CSS are a reasonably elegant way to present the new posts, but there appear to be a couple of kinks to work out. (Maybe long grey blocks of bookmark posts will embarrass me into more regular posting.

heisel.org > Projects

stephen@blueslugs.com (Stephen Hahn) — Mon, 20 Mar 2006 15:47:53 +0000

heisel.org > Projects

Home of Word Pre.cio.us, a patch to WordPress 1.5.x to integrate one’s del.icio.us bookmarks stream (or subset) with your blog.

Stroller mileage

stephen@blueslugs.com (Stephen Hahn) — Mon, 20 Mar 2006 06:11:38 +0000

Most Saturdays, Dina and Benjamin head off to swimming lessons. Nathaniel and I are left to our own devices, which means we go for one of two kinds of walk: a walking walk, where Nathaniel walks, and a seeing walk, where Nathaniel gets pushed. This week, Nathaniel expressed that he wanted to see trains on the walk, which meant a seeing walk.

We headed east towards the Redwood City Caltrain station. On the way, we passed the most recent arrival in Redwood City’s gradual approach to Seattle-class latte infrastructure:

We walked through the station, intending to cross the tracks and visit a couple of the construction sites downtown. When you’re nearly two years old and focused on being an expert in trains and trucks, seeing a truck that can ride on the rails is perhaps mind expanding:

The renovation of the Old San Mateo County Courthouse—now home to the San Mateo County History Museum—appears to involve extensive work on the columns:

Finally, we walked back to the station to see the 10:41 Northbound train,

wheeled into Peet’s to grab a cappuccino, and sipped our drinks—Nathaniel is usually prepared with a travel flask—as the Southbound 26 pulled in ten minutes later. Two trains (and a “train truck”) spotted, we headed home.

How?s This for Load Average

stephen@blueslugs.com (Stephen Hahn) — Fri, 17 Mar 2006 00:44:21 +0000

How?s This for Load Average

Colin runs a fork bomb on a zone bound to a resource pool. The zones.max-lwps resource control would further restrict the impact of the attempt to deny service, as he notes.

How's This for Load Average

stephen@blueslugs.com (Stephen Hahn) — Fri, 17 Mar 2006 00:44:21 +0000

How?s This for Load Average

Colin runs a fork bomb on a zone bound to a resource pool. The zones.max-lwps resource control would further restrict the impact of the attempt to deny service, as he notes.

The Blog of Ben Rockwood: RHEL-to-Solaris Transition Reference

stephen@blueslugs.com (Stephen Hahn) — Fri, 10 Mar 2006 22:39:41 +0000

The Blog of Ben Rockwood: RHEL-to-Solaris Transition Reference

Ben gives an admin’s cheatsheet for moving from one operating system to another.

Zones development: You're invited

stephen@blueslugs.com (Stephen Hahn) — Thu, 09 Mar 2006 21:50:29 +0000

Zones development: You’re invited

Dan Price explains why OpenSolaris matters to ISVs delivering on Solaris. Not just relevant to Zones.

Mentawai Web Framework – Mentawai Overview

stephen@blueslugs.com (Stephen Hahn) — Thu, 09 Mar 2006 19:17:03 +0000

Mentawai Web Framework – Mentawai Overview

A lightweight Java web application framework. No use of XML configuration, in contrast to most other Java frameworks.

RIFE : About

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Mar 2006 22:06:02 +0000

RIFE : About

A lightweight Java web applications framework.

Textism: Tools: Textile

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Mar 2006 21:16:33 +0000

Textism: Tools: Textile

Textile’s home page.

SourceForge.net: textile4j

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Mar 2006 21:15:36 +0000

SourceForge.net: textile4j

Another Textile markup implementation in Java.

philwilson.org :: jtextile

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Mar 2006 21:14:41 +0000

philwilson.org :: jtextile

A Java implementation of the Textile markup language.

LiteratePrograms:Welcome – LiteratePrograms

stephen@blueslugs.com (Stephen Hahn) — Thu, 02 Mar 2006 22:32:01 +0000

LiteratePrograms:Welcome – LiteratePrograms

A wiki of literate programs: programs presented in an expository format, that can be reordered into a compilation ready form.

GIT – Tree History Storage Tool

stephen@blueslugs.com (Stephen Hahn) — Wed, 01 Mar 2006 19:30:11 +0000

GIT – Tree History Storage Tool

Software used as the basis for a number of source code managers. Includes “core GIT”, a basic but powerful SCM.

Getopt: Parse command-line arguments from shell scripts

stephen@blueslugs.com (Stephen Hahn) — Wed, 01 Mar 2006 19:20:46 +0000

Getopt: Parse command-line arguments from shell scripts

getopt(1) implementation with long option support.

Predictable – Gates, projects, and developers in ON

stephen@blueslugs.com (Stephen Hahn) — Tue, 28 Feb 2006 21:48:11 +0000

Predictable – Gates, projects, and developers in ON

Aspects of source code management practices in the OpenSolaris ON consolidation.

monotone documentation

stephen@blueslugs.com (Stephen Hahn) — Tue, 28 Feb 2006 05:41:19 +0000

monotone documentation

Monotone’s documentation in a single HTML page. The CVS-Monotone equivalence is an easy way to get started.

Gates, projects, and developers in ON

stephen@blueslugs.com (Stephen Hahn) — Tue, 28 Feb 2006 00:17:31 +0000

If you’re following the tools-discuss alias on opensolaris.org, then you’re also aware that the ON consolidation uses a distributed source code management tool to track changes to the ON source (which includes the kernel as well as many device drivers, libraries, and commands). I thought I would review the current relationships between code repositories in typical development scenarios in ON, in part to practice using tool-neutral terminology in describing how code changes move from a developer into the main repository for the development release, but also to reveal some of the assumptions behind the distributed source code requirements and evaluation.

To stay neutral, the various invocations of a fictional ‘scm’ tool below are hypothetical; different SCMs might treat some of these command sequences as a more monolithic operation or might provide even more primitive operations.

For reference of the relationships we discuss below, here’s a simple diagram, in which the arrows indicate the flow of code changes. The two shaded repositories are the first to receive changes from their respective children, which we’ll call “gates”. (The unshaded repositories receive their changes from a gate or via a child pulling from a gate.)

Let’s consider first an individual developer addressing one or more bugs in ON. He or she will need a copy of the source to work on, which we choose to store in /path/to/fix-0.

$ scm clone /path/to/onnv-clone /path/to/fix-0

For ON, this clone (or initial pull or initial bringover or initial checkout) operation involves the creation of a little over 34 000 files and nearly 7 000 directories.

Although we use file system paths in these examples, most of the candidates can perform their operations over the network, via `ssh`, `http`, or a native protocol.

The developer then makes his or her changes to the files requiring correction. During that period of time—editing, compiling, and testing, as well as devising a fix—the user may make intermediate commits of their current work and pull new changes (made by the other ON committers—let’s say a thousand). This operation looks like

$ scm commit
$ scm pull
$ scm merge

with all operations going to onnv-clone.

Finally the developer is ready for final integration. Their code has been tested, code reviewed, and an integration request has been approved. Their final sequence is

$ scm reparent /path/to/onnv-gate
$ scm sanitize
$ scm commit
$ scm pull
$ scm merge

until they are completely up-to-date with respect to onnv-gate, at which point, they can publish their changes

$ scm push

The “sanitize” operation removes all of the intermediate commits made in the developer’s repository—from merges made along the way—so that the main gate only sees the important portion of the history of their work: what specific defects, via bug IDs, are addressed by this changeset.

`sanitize` isn’t typically implemented in source code management tools; it is a local addition that might be seen as controversial in some circles. ON isn’t a pull- or patch-driven integration process, so if we didn’t sanitize, we would have commit logs with the merge records resulting from 1 000 developers merging at different rates.

At this point, it’s probably worth commenting on the existence of the onnv-clone repository, which is provided as a read-only child of the main development repository, onnv-gate. All of the operations in our scenario would have been sensible if we had used onnv-gate as our parent repository (and omitted our reparent operation). So why have a ‘clone’ at all? There are two reasons, one technical and one social.

The technical reason is to make sure that contention for the gate’s repository locking is limited to developers integrating code. Integrating into ON carries with it a substantial burden of testing, watching for other integrations, rebuilding, and so on: getting blocked by an initial pull of the gate repository taking the various repository locks is a difficult scenario we want to avoid.

The social reason is that the clone represents a steadying rhythm for developers: unless you are preparing to integrate that day, you know that the pull/merge you do from the clone is, on some best effort basis, safe. Small build mistakes have been fixed and large ones undone prior to potentially impacting your development repositories.

So a clone is a useful technique when a DSCM’s locking might be overwhelmed by the number of child repositories needing to be kept up-to-date.

Our second scenario, where a few developers are pursuing a project that involves a large batch of coordinated changes to ON, is an example where a clone repository isn’t necessary, as the three developers are unlikely to contend on the repository lock. But the role of project-gate is similar: to provide a repository which is regularly built and whose binary products are regularly tested (and may even be shared with potential users or act as parents to other projects’ gates), so that the developers can make confident forward progress.

One of the developers (or more) will, on some schedule, do pulls and merges from onnv-clone and integrate them into project-gate. This “resync” can be done from the project gate itself or (and more preferably, but not illustrated) from an additional child. In the latter case, this sequence looks like

$ scm parent
/path/to/project-gate
$ scm pull /path/to/onnv-clone
$ scm merge
$ # test build, corrections, with possible a commit/pull/merge
$ scm push

where we have pulled from onnv-clone to push to our project repository.

When the development team is ready to integrate their changes, their invocations from the project-gate repository are similar to their colleague’s from fix-0 above:

$ scm reparent /path/to/onnv-gate
$ scm sanitize
$ scm commit
$ scm pull
$ scm merge
$ scm push

For a project, this action is usually followed by the consumption of one or more celebratory beverages, while the most nervous member watches for mail from one or more gatekeepers regarding build nits or more serious omissions. We’ll have to work on collaboration tools so that this team experience can be shared across the network.

[ T: OpenSolaris Solaris scm ]

Daemon News 200602 : A Brief History of Apple's Open Source Efforts

stephen@blueslugs.com (Stephen Hahn) — Fri, 24 Feb 2006 06:36:22 +0000

Daemon News 200602 : A Brief History of Apple’s Open Source Efforts

A review of the progressive reductions in Apple’s commitment to open development around Darwin. Notes specific complexity of the internal build infrastructure and how that translated to external developer participation.

unixzone dot dk » JES Directory Server SMF Manifest

stephen@blueslugs.com (Stephen Hahn) — Fri, 24 Feb 2006 06:28:42 +0000

unixzone dot dk » JES Directory Server SMF Manifest

Lasse Østerild has a conversion for the Java ES Directory Server.

New pocket location

stephen@blueslugs.com (Stephen Hahn) — Thu, 23 Feb 2006 18:37:08 +0000

I’m not the most device-laden person around here—by far. But I occasionally like to carry a camera and my Visor, as well as the obligatory tether-phone. An extra pocket would be nice, as wearing a jacket (with its 2 – 6 pockets) is uncomfortable for much of the year in a California climate. I always thought that the Dockers was onto something with their Mobile Pants, but now The Register reports that Thomas Pink has upped the ante by placing a pocket on the back of a necktie.

Seems like a natural place for any digital music player or even a voice-driven phone, given a headset with one or two buttons.

The National Security Archive

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Feb 2006 06:11:22 +0000

The National Security Archive

Research archive at George Washington University.

Library and Archives Canada

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Feb 2006 05:35:53 +0000

Library and Archives Canada

National archives of Canada, English site, online.

The National Archives

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Feb 2006 05:35:03 +0000

The National Archives

National archives of the UK, online.

NARA

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Feb 2006 05:34:18 +0000

NARA

National archives of the USA, online.

Yahoo! UI Library

stephen@blueslugs.com (Stephen Hahn) — Tue, 14 Feb 2006 17:37:35 +0000

Yahoo! UI Library

Yahoo’s library and design patterns, available under a BSD license.

BigAdmin Feature Article: GRUB and the Solaris 10 1/06 OS: The New Bootloader for x86 Platforms

stephen@blueslugs.com (Stephen Hahn) — Mon, 13 Feb 2006 22:50:54 +0000

BigAdmin Feature Article: GRUB and the Solaris 10 1/06 OS: The New Bootloader for x86 Platforms

Technical overview of Solaris/OpenSolaris booting via GRUB.

Cryptically content

stephen@blueslugs.com (Stephen Hahn) — Sat, 11 Feb 2006 00:00:52 +0000

For me, on the whole, this week was a pretty good one. Dave and I wrapped up some interface work on Wednesday (and one of my favourite projects, Visual Panels, went public using the project hosting on opensolaris.org). And we’ve been making progress on many fronts for OpenSolaris: the source code management evaluation is starting to pick up, project proposal rates are increasing, and governance efforts are nearing their next milestone.

I may even have enough time to dust off some unfinished blog entries.

[ T: OpenSolaris Solaris smf vpanels ]

Posix for Java

stephen@blueslugs.com (Stephen Hahn) — Thu, 09 Feb 2006 23:12:56 +0000

Posix for Java

A different implementation. More objects, but less coverage.

Jtux – Java-To-Unix Package

stephen@blueslugs.com (Stephen Hahn) — Thu, 09 Feb 2006 23:12:20 +0000

Jtux – Java-To-Unix Package

Classes and JNI to allow access to POSIX functionality from Unix. Appears to be self-consistent; known to work on BSD, Linux, MacOS X, and Solaris.

TWiki . Linux . LinuxInternalsSystemcallsEtcFromJava

stephen@blueslugs.com (Stephen Hahn) — Thu, 09 Feb 2006 23:07:05 +0000

TWiki . Linux . LinuxInternalsSystemcallsEtcFromJava

A few links to various “access POSIX from Java” projects.

SuperBowl ads wanted: must have science or stunts

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Feb 2006 08:48:14 +0000

The Wall Street Journal has an article, “Clever Gags Score High On Super Bowl Ads”, with best and worst polls to accompany the online edition. We, for a change, actually watched the SuperBowl this year—with appropriate TiVo gymnastics to cover the grossly underestimated duration given in the television listings—and so can comment on the advertisements. The popular FedEx cavemen-and-dinosaurs ad was a hit with Benjamin, as was the offspring-of-monster-and-robot bit from Hummer. My favorite? I think Stunt City [Degree] was underappreciated: the admonishment to the motorcycle deliveryman who has just crashed through the window (“no helmets”) was silent, low-key, and clever.

Oh, and we liked the meteoroid-geologically-becomes-PEBL [Motorola] ad, too.

PublicRadioFan.com – What's on now or soon

stephen@blueslugs.com (Stephen Hahn) — Tue, 07 Feb 2006 05:10:34 +0000

PublicRadioFan.com – What’s on now or soon

Upcoming programming on Internet public radio stations using MP3, Ogg, or Real.

Dense election coverage

stephen@blueslugs.com (Stephen Hahn) — Mon, 30 Jan 2006 06:46:38 +0000

A couple of people have asked me to comment on the Canadian election last week. Although I was up to date with the limited reporting in The Economist, the two to six weeks of latency in my Maclean’s subscription meant I never really connected with the current situation—I was already rather fed up with the random delays, and am switching to The Walrus as an experiment. In any case, I’ve been in the USA for over thirteen years now; my perceptions of Canada are, politely, distorted by nostalgia.

I did get to admire the CBC election night coverage. Check out the density of this snapshot of reporting from a British Columbia riding:

That’s four distinct presentation areas: (1) riding result, (2) national seat summary, (3) provincial changes, and (4) a message area:

Although there isn’t one in the snapshot above, area (4)’s text messages would sweep in, and were mapped on to a curve parallel to the lower side of the elliptical ring.

The screen is reporting 22 numbers—more when more parties or independent candidates were standing—which seems pretty high for a television graphic (plus the party associations, candidate and riding names, and the riding’s location in the country). The beige field is a good neutral background for the presentation of the range of colours needed to handle the multiple parties participating.

Unfortunately, I didn’t take a picture of the confusing artwork behind Peter Mansbridge and his panel, which lacked the elegance of the reporting screen. (The panel’s banter wasn’t that entertaining, but I get the impression Senator Segal is quite an operator.)

State of the domain

stephen@blueslugs.com (Stephen Hahn) — Thu, 26 Jan 2006 07:28:43 +0000

As I did last year, I’m again reviewing the year from the perspective of blueslugs.com.

Overview. Over 2005, we served up a total of over 8.3 GB of data, through our standard static DSL line: almost an order of magnitude higher than last year. 5.6 GB was blueslugs.com, while the remaining 2.7 GB was from the new domain, highmaintenancemom.com, which is Dina’s electronic motherhood site. HMM only came online in April, and has been steadily building in traffic. Log analysis translates to over 930 000 hits and over 45 000 visitors. Over the year, 36 blog posts were written on ``As I did last year, I’m again reviewing the year from the perspective of blueslugs.com.

Content. The most popular page on the site this year was the introduction of tag(1), a Unix-like command for tagging files. I have a long overdue post that analyzes the incoming traffic associated with the post, which hit a few of the “emergent importance” sites as well as getting mentioned on some individual blogs. (I have an equally overdue response regarding some of the technical and quasi-social analysis of the utility of tag(1).) Pages from Benjamin‘s alphabet book are still regularly requested: the alphabet page trailed tagging by only a few hundred hits. (Each had over 16 000.) The grammar page is the top individual page, but the country flags and mathematicians images get pulled in via search terms. Search engine passes and syndication feed pulls are as or more regular than ever.

Beyond the tag(1) release, I ported Audacity to Solaris x86, which helped a few people, based on downloads. The dockapps I wrote a few years ago still get pulled regularly. Most of my Solaris posts are published at my work blog, but I’m still exploring “personal (Unix) computing” posting as a topic to cover here.

My Redwood City writings dropped off; I must get back to this topic, as the downtown theatres are nearly full built, and the commercial changes—new restaurants!—have already started. Or maybe midterm elections will prove interesting…

System. As chronicled, we unexpectedly had to replace the server due to hardware failure. This system is, of course, faster than its predecessor and, even with the higher load of the two sites, is essentially idle. The site data is on a UFS partition with triple mirroring via SVM, but I expect to experiment with ZFS after the next software upgrade.

The system is presently running Solaris 10 03/05 with appropriate patches; the current Web stack was built by hand, as HMM software requirements required somewhat atypical settings. Expect to receive bits served from Solaris Express builds in the next few months.

Future. I doubt we’ll launch another site this year, so I expect no new leaps in traffic. One possibility is a DSL upgrade; I believe the higher speed version approximately twice guaranteed bandwidth, but I haven’t heard anyone talking about what they’re actually getting. Although I like having a camera phone, the camera I bought takes such better pictures that, if I’m going to post pictures, they’ll be taken with a real camera. And I would like to get the backlog of partial posts under control—I started a notebook for post ideas, but I seem to be filling the book, rather than the blog.

SIP.edu Cookbook : Introduction

stephen@blueslugs.com (Stephen Hahn) — Wed, 18 Jan 2006 00:02:46 +0000

SIP.edu Cookbook : Introduction

Internet2 educational SIP deployment practices.

[cs/0601038] Constraint-based automatic verification of abstract models of multithreaded programs

stephen@blueslugs.com (Stephen Hahn) — Thu, 12 Jan 2006 00:31:50 +0000

[cs/0601038] Constraint-based automatic verification of abstract models of multithreaded programs

[cs/0503058] On the Stopping Distance and the Stopping Redundancy of Codes

stephen@blueslugs.com (Stephen Hahn) — Tue, 03 Jan 2006 16:55:44 +0000

[cs/0503058] On the Stopping Distance and the Stopping Redundancy of Codes

[cs/0512065] Tradeoffs in Metaprogramming

stephen@blueslugs.com (Stephen Hahn) — Tue, 03 Jan 2006 16:53:47 +0000

[cs/0512065] Tradeoffs in Metaprogramming

Metaprogramming and safety analyzed via restricted language choices.

CanvasGraph.js : Graphing in Javascript | liquidx

stephen@blueslugs.com (Stephen Hahn) — Fri, 16 Dec 2005 22:23:04 +0000

CanvasGraph.js : Graphing in Javascript | liquidx

Graphing in the browser–needs Firefox 1.5, but very useful once canvas support widespread.

Camera gets a workout

stephen@blueslugs.com (Stephen Hahn) — Thu, 15 Dec 2005 06:46:35 +0000

A while back, I was speculating about getting a new camera. I finally gave up using the S30 with the broken screen and decided to act. Although I showed some recent system-administration-confererence-action shots, indoor snapshots are never as much fun.

So when we decided to go see the visiting reindeers—from central Oregon apparently—at the San Francisco Zoo, the camera went along for the ride. One of the guests of honour:

Note the ground-scraping horn extending well over the left brow.

Nathaniel thought the reindeer were cool and kept trying to pass through the outer fence to get a closer look:

Next up was seeing the bears, but the playground is a mandatory stop on this route:

Since our trip to San Diego, Ben has been very interested in the state of the Alaskan Brown Bear at that zoo. If the bear’s blogging I don’t know about it, but I was able to convince Ben that seeing the young grizzlies get larger was just as interesting—and they are getting bigger:

Down the way, at the spectacled bear grotto, I noticed some writing on a railing:

The bear dens were built by the WPA, so this railing’s Bethlehem Steel parentage is not surprising. (Apparently, these grottos and the other WPA-built exhibits are some of the earliest “barless” zoo exhibits in the world. I’ve noticed that children’s books often render cagebar zoo displays, even when written many years after such displays were common.)

On the way out, we noticed the otters were enjoying a light lunch:

One tip with electronic viewfinder cameras: turn off the review period to make the camera feel faster. (It is often set to about 2 seconds.) For outdoor shots, where the flash isn’t a factor, you’ll be much less likely to miss a good photo.

store.sun.com to Niagara/Solaris 10

stephen@blueslugs.com (Stephen Hahn) — Thu, 15 Dec 2005 05:59:38 +0000

James Dickens spied a new BluePrint on the planned redeployment of store.sun.com [PDF] to SunFire T2000 systems running Solaris 10. Beyond the tremendous reduction in occupied space and the around 90 percent estimated reduction in input power and output heat, the document describes the use of Solaris Containers to consolidate the middle tier servers, complete with invocations of zonecfg(1M) (for the zones the applications run inside of) and poolcfg(1M) (for the resource pools the zones sit upon).

Business applications are complex—maybe some smf(5) service conversions will appear in the next version to make the dependency and failure handling more precise.

[ T: Solaris zones pools ]

Reading at work: Sturgis, Standard code of parliamentary procedure

stephen@blueslugs.com (Stephen Hahn) — Wed, 14 Dec 2005 22:51:31 +0000

[ Alice Sturgis, The Standard Code of Parliamentary Procedure, 4e., Mc Graw-Hill, 2000. ]

As I mentioned, I wanted to make sure the team gave some additional support to the governance work being done for OpenSolaris. When I thought about it for a little bit, I realized that, while we have recent and valuable experiences from the open source communities that have established organizational structures and codes, setting up societies and deliberative bodies and their rules is something that’s been going on for a while. So, on a late night dealing with a disk problem at home, I read most of the transcripts from the U. S. Senate’s Oral History Program’s interviews of Floyd M. Riddick, who was Senate Parliamentarian from 1964 – 1974. (There were a number of reasons I ended up here, but one was to determine what lessons bodies like the Senate might have for a fledgling organization.)

The fact that a word like “parliamentarian” exists means you have a solid angle for searching, as well as a hint that you might not need to reinvent everything. If you search, you’ll find there are two U. S. national organizations for parliamentarians, and that the two primary rulesets are Robert’s Rules of Order, Newly Revised and The Standard Code of Parliamentary Procedure. There’s a whole industry around RONR and much information online; the SCPP is a complete procedure, simplified to remove some of the more unusual parts of RONR, so I thought I would work through that. (So more reading at work.)

One of the first quotations in the book, in Chapter One, is in turn a quotation from Clarence Cannon, who was Parliamentarian of the U. S. House of Representatives (and subsequently a Representative), which reads:

These rules of Parliament and Congress are designed for bicameral bodies, generally with paid memberships, meeting in continuous session, requiring a majority for a quorum, and delegating their duties largely to committees. Their special requirements . . . have produced highly complex and remarkably efficient systems of rules peculiar to their bodies, but which are, as a whole, unsuited to the needs of the ordinary assembly.

— Clarence Cannon, “Rules of Order”, Encyclopædia Britannica, 1964.
I interpret this paragraph as “imitate us at your own risk”.

But SCPP does identify when reasonable alternatives to its formulation might be considered, and how they may or may not lead to distortion of the principles of majority-based decision making or the silencing of minority viewpoints. For instance, the idea of requiring a unanimous result for a particular vote means that a minority of one can block resolution of the motion being voted upon. Similarly, a supermajority voting requirement means that a smaller minority can block that action. (Perhaps I’m just pleased that the reference frame that’s being taken is similar to an engineering frame: “how do we construct an assembly so that it can make progress on difficult issues?”.)

I’m still reading SCPP, mostly with an eye to being ready to design opensolaris.org‘s support for various kinds of voting, but feel free to recommend you have a look at it now.

[ T: OpenSolaris ]

Seasonal OpenSolaris shirt

stephen@blueslugs.com (Stephen Hahn) — Tue, 13 Dec 2005 19:00:16 +0000

Sara’s tree [was once available] on a shirt.

[That was pretty] slick.

Update, 2009-03-26: The swag store and the shirt are no longer available.

[ T: OpenSolaris ]

LISA05 Friday

stephen@blueslugs.com (Stephen Hahn) — Mon, 12 Dec 2005 18:08:35 +0000

After commiserating with my room’s ceramic bunny—his feet are glued to the cabinet’s top and so he has limited opportunities for conversation—I finished packing and headed out for the last few hours of LISA05.

First up was the last papers session, with Liane presenting on smf(5), in addition to papers on a spreadsheet-based scripting environment and a whizzy DNS script for medium sized sites. There was setup before the session, with Luke Kanies outlining the basics of how the sessions run:

The talk was well delivered and Liane answered the audience questions adeptly so, after a group lunch, we headed off to watch Liane, Bryan, Dan, and Bill lead a “guru” session on Solaris 10.

My flight time meant that I wasn’t able to stay for the entire session, so I can’t assess the total guru-osity displayed over the 90 minutes. But I certainly think that this year’s trip to LISA was worthwhile.

[ T: Solaris OpenSolaris LISA05 smf ]

LISA05 Thursday: Reception(s)

stephen@blueslugs.com (Stephen Hahn) — Mon, 12 Dec 2005 16:07:05 +0000

Thursday evening was a series of sponsored events. The LISA reception was a combination of dinner, desserts, drinks, and a miniature casino floor:

A fascinating part of the reception was the balloon assemblies produced by the man pictured; here he’s making a birthday cake hat:

People lucky enough to receive wings, a palm tree hat (complete with balloon monkey), and the like were good enough to wear them around the ballroom so that we might all see what objects can be constructed from inflated rubber.

After the reception, Sara hosted an OpenSolaris hospitality suite, which came furnished with a very generously pouring bartender. The crowd was pretty good:

As the crowd thinned out, I noticed that a complimentary laptop servicing was being handled by a group of engineers:

Ultimately, the laptop was restored and the evening ended happily.

[ T: Solaris OpenSolaris LISA05 ]

Warren Strange: SMF manifest for SlimServer

stephen@blueslugs.com (Stephen Hahn) — Mon, 12 Dec 2005 06:42:08 +0000

Warren Strange: SMF manifest for SlimServer

A service conversion for SlimServer.

LISA05 Thursday: Cost models and wiretaps

stephen@blueslugs.com (Stephen Hahn) — Mon, 12 Dec 2005 05:55:35 +0000

I went to the midday configuration management theory session as well as to Matt Blaze’s plenary. (I missed the invited talks as I lost track of time during a late lunch.)

The talk I enjoyed out of the theory session was by Alva Couch and colleagues from Tufts, who presented steps towards a cost model for system management. They worked through over a year’s worth of RT logs from their site, working out the background effects of holidays and weekends, and then being able to model alternate staffing choices and their impact on response time. Alva also made some provcative statements about best practices; with respect to a time-sharing/interactive user dominated site, these statements likely hold. (Less applicable to server farms, though.) It was pleasant to see some math; I plan to work through this paper carefully to see what might be extracted.

Matt Blaze gave an energetic talk about his team’s recent work on wiretap technology and how the various implementations can be evaded or confused. There were complimentary copies of the issue of IEEE Privacy and Security that contained the full article, which was a nice touch.

[ T: LISA05 ]

LISA05 Thursday: Booth and panels

stephen@blueslugs.com (Stephen Hahn) — Fri, 09 Dec 2005 21:27:17 +0000

While I’ve been listening to various talks, other folks have been answering questions about Solaris, the new systems, and other Sun products in our booth on the vendor floor. We have a double wide, and it was busy whenever I passed by during session breaks:

In addition to the engineers I’ve mentioned were here, Cindy Swearingen, who is one of the lead technical writers for the Solaris System Administration Guide, handled many inquiries. She’s manning the center table here:

There were new Opteron and UltraSPARC T1 systems in the booth; Jonathan has a conversation, with the T2000 (“Niagara”) system in the foreground:

Let’s look closely at the Niagara box:

I do like the aluminum cases, although I’ll always have fond memories of entering datacentres with racks upon racks of wine-coloured systems.

Finally, if you poked around, you might have got a demonstration from Dave or David of some ideas in smf(5) visual administration:

That’s real code and a live graph, by the way.

[ T: Solaris OpenSolaris LISA05 smf Niagara ]

LISA05 Wednesday BoF

stephen@blueslugs.com (Stephen Hahn) — Thu, 08 Dec 2005 19:37:36 +0000

After a rapidly served and consumed dinner at Taste of Thai, the Solaris BoF squad headed back for the Wednesday session at 9 pm. The room filled up fast, with some folks ultimately having to stand in the back:

I decided to spell Jonathan as T-shirt gerbil, and was happy to give out Sara‘s OpenSolaris ringer Ts and the new Inside Jack shirts in exchange for good questions or smart remarks.

Next time we’ll be sure to order XXL sized shirts.

Dan mentioned that his laptop was spallating, or emitting shrapnel, or something like that, but he and Bill managed to get both their presentations to share what little functionality remained.

After Dan did a great job covering the coming work in the near future of Solaris, Bill took the stage for a two hour thorough overview of ZFS.

Even after two hours and many answers dispensed, Bill fielded midnight questions with enthusiasm.

Nice camera awareness on Bill’s part.

[ T: OpenSolaris Solaris LISA05 ZFS ]

LISA05 Wednesday: Autopod and PKI

stephen@blueslugs.com (Stephen Hahn) — Thu, 08 Dec 2005 18:23:52 +0000

While the vendor exhibition opened and inquiries were handled at the Sun booth, I went to the technical session on Configuration Managment Theory and to Radia Perlman‘s invited talk.

The most interesting talk to a kernel developer in the technical session was Shaya Potter presenting on the Autopod work he and Jason Nieh have been pursuing at Columbia. It’s a nice combination of an isolated application environment with process checkpointing, with a number of nice advantages—foremost among them being migration of a running pod (application environment) between systems running different operating system versions.

Shaya mentioned about the port from 2.4 Linux kernel to 2.6, and the loss of simple system call interpositioning in the newer implementation. One interesting area for investigation of similar work on OpenSolaris would be to assess the suitability of the BrandZ technology for creating a checkpointable zone—the preexisting application environment on which to build. There’s a wide variety of potential kernel state to capture, but the appropriateness of the brand architecture may simplify aspects of a prototype.

If you’re interested in thinking about such things, the BrandZ community should open up on OpenSolaris soon; you can already subscribe to the brandz-discuss alias on the Discussions page.

Despite being at Sun for a while, I’ve never heard Radia Perlman speak before, and I can now highly recommend it: she’s engaging and funny, as well as a very clear technical presenter. Radia gave an overview of the ideas and implementation issues around Public Key Infrastructure (PKI). Apparently, she and her co-authors have released a new edition of their text, Network Security: Private Communication in a Public World [Amazon], which I’ll put on my wish list.

[ T: OpenSolaris LISA05 BrandZ PKI ]

LISA05 Tuesday: device errors, iostat, and logging

stephen@blueslugs.com (Stephen Hahn) — Wed, 07 Dec 2005 19:44:15 +0000

One of the questions raised at Tuesday night’s BoF was “why are some of the statistics that iostat -E displays result in a console message and some do not?” I was sitting in the back with a copy of Mike Kupfer’s split ON source tree, and decided to have a look. iostat(1M) is a kstat reader, with some simple processing and formatted output. The output function is show_disk_errors() but requires understanding how iostat groups the disks and statistics in its implementation; the code that acquires the device statistics is located in cmd/stat/common/acquire_iodevs.c. Searching for error show that the critical function is acquire_iodev_errors(), and that two classes of kstats contribute to the error output: device_error and iopath_error.

The most direct way to see these statistics is to invoke kstat(1M) with these classes. On my laptop, the result for device_error is

$ kstat -p -c device_error
sderr:0:sd0,err:Device Not Ready 0
sderr:0:sd0,err:Hard Errors 0
sderr:0:sd0,err:Illegal Request 1
sderr:0:sd0,err:Media Error 0
sderr:0:sd0,err:No Device 0
sderr:0:sd0,err:Predictive Failure Analysis 0
sderr:0:sd0,err:Product UJ-832D Revision
sderr:0:sd0,err:Recoverable 0
sderr:0:sd0,err:Revision 1.50
sderr:0:sd0,err:Serial No
sderr:0:sd0,err:Size 0
sderr:0:sd0,err:Soft Errors 1
sderr:0:sd0,err:Transport Errors 0
sderr:0:sd0,err:Vendor MATSHITA
sderr:0:sd0,err:class device_error
sderr:0:sd0,err:crtime 76.139658104
sderr:0:sd0,err:snaptime 1857.960128997

(The laptop has no kstats of class iopath_error.)

We then look for the creation of the named kstat for each of these strings—invocations of kstat_create() to identify the structure member names associated with each. And then we can look for statements that involve those member names; this leads us to the various SD_UPDATE_ERRSTATS() invocations throughout uts/common/io/scsi/targets/sd.c.

The discrepancy between updates and logging arises because the macro, SD_UPDATE_ERRSTATS(), which bumps the counters and the function which displays the error, sd_print_sense_msg(), are sometimes both invoked, and sometimes are not. I don’t know the details of the SCSI error categories, but the decision to make some of these errors silent and some not appears arbitrary. So, unfortunately, the only answer today to determine “why is this messaged” is to look at the code. (Perhaps an ‘sd’ expert can offer an enlightening comment.)

If you’re trying to anticipate and avoid potential failures, having random messages emitted arbitrarily isn’t very helpful: that’s why Mike and the FMA team developed the fault management architecture to have a framework in which errors are processed in a predictable fashion, resulting in the proper diagnosis of faults. Eric described one possible scenario involving disk errors, FMA, and ZFS a couple of weeks ago, but there’s a smaller step that seems useful involving only sd and FMA: the error increments could be converted to error events, and the decision to issue a notice deferred until a series of errors is diagnosed into a fault except, I suppose, if the error can be immediately diagnosed as a critical fault. Taking this step would result in consistent reporting for all disk consumers, including less sophisticated consumers than ZFS, like older filesystems and raw disk accessors.

In a software engineering sense, the FMA approach, where error issuance and diagnosis are separated, is much more sound: at the initial driver software composition, the field experience with the hardware device is typically limited. Over time, the actual impact of errors on system practice becomes better known, and the diagnosis and the actions associated with it can be refined. fmd(1M) can handle on-the-fly module updates gracefully, and also deal with overlapping event flows so that both primitive and ZFS-specific fault handling policies can be implemented, depending on the use of a particular device.

Now, the community that’s discussing technical issues and directions for fault management is aptly called the Fault Management community; if you are interested in how this work is going to proceed, and ways to contribute, I suggest joining it.

[ T: Solaris OpenSolaris LISA05 FMA zfs iostat kstat ]

LISA05 Tuesday

stephen@blueslugs.com (Stephen Hahn) — Wed, 07 Dec 2005 19:07:48 +0000

After the N1 Grid BoF session led by Eric Nielsen, Liane took the lead on a well attended BoF about Solaris 10 and OpenSolaris. (I missed the N1 BoF, enjoying a nice conversation with Bernd Haug, who has been architecting solutions using Solaris in various scenarios in Austria.)

As with any presentation, it takes some cable jujitsu to achive laptop/projector union:

The audience was definitely interested in Zones, and Dan fielded some of the more involved questions with gusto:

A particularly passionate point about the general performance issues around patching systems with zones was raised; one audience member noted that placing one’s zones explicitly in single user mode would speed zones patching.

Dan and Jan watch the questions come in:

Jan answered some questions regarding PXE and the new boot architecture on x86/x64 systems.

Frank Liborio came down to explain some of the finer points of the patching and update changes and their association with the Solaris support plans:

There were some good questions about the new pricing for software in general; the main point, I thought, was to emphasize that the components are not crippled free variants, but the same enterprise version, just with no support.

At the end of the BoF, people moved purposefully to the room’s front, either to ask questions of the various experts, or perhaps to snag a free Solaris DVD kit:

John Groenveld stopped to say hello as David and I were milling near the doors; it’s nice to talk with other Solaris advocates (other than the bunch I lunch with every day, I mean).

Today, the vendor exhibits open, so there should be some technical discussions and demos at the Sun booth, and Dan will host a BoF of Solaris futures tonight. My plan is to listen to some of the interesting looking talks, and then get to the BoF.

I’m getting used to the behaviour of the camera, particularly at full telephoto. Being polite, I didn’t use the flash much—and I’m doubt it would matter at 60’—so these photos vary in quality. I was fiddling with adjusting for indoors, incandescent lighting, and longer exposures at different times.

[ T: Solaris OpenSolaris LISA05 ]

Recovering from a failed (TiVo) upgrade

stephen@blueslugs.com (Stephen Hahn) — Mon, 05 Dec 2005 06:12:58 +0000

I ordered an additional drive for our long running Series 1 Tivo last week from one of the firms that specializes in such things. The drive “marriage” didn’t go well, and the unit was stuck at the Green Screen of Death—and, since these upgrades are supposed to just work, I didn’t have a backup. Scrambling through various forums searching made it clear that finding a software image was unlikely, but also pointed to a solution: PTVupgrade’s InstantCake.

InstantCake is a CD-based program that, if you reconfigure a PC such that the CD-ROM boots as the primary IDE and the drive or drives for your TiVo are on the secondary IDE, will format the drives and install the necessary software to operate your TiVo. For most PCs now, this procedure is at matter of swapping cables around and waiting for a little bit. You’ll lose all your programs and settings, but it means that you can pick any pair of drives as replacements. (Remember that the Series 1 can’t use more than 137GB of its drives, so going past 160GB is pointless.) And you can restore regardless of whether you have a drive or not; you have a read-only image on stable media.

Highly recommended.

[ T: TiVo ]

Conferring again

stephen@blueslugs.com (Stephen Hahn) — Sun, 04 Dec 2005 22:51:48 +0000

It’s been a few years and a couple of kids since my last conference, but I’m happy to be packing to be ready to attend LISA 2005, which is happening in San Diego, CA this week. My plane lands mid-afternoon Tuesday, and I’ll be there until Friday. Liane‘s giving a paper on smf(5), and Liane and Dan are each hosting BoF sessions on aspects of Solaris and OpenSolaris. Bryan is coming for a technical session, and Jonathan, Dave, and David from the smf(5) team will be there, too. (Ask Dave and David for a demo.) Plus, there’s a booth in the Vendor Exhibition and a sponsored reception, and events I don’t know about and am probably not invited to…

A particular attraction for administrators: I wanted to point out that Bill Moore of the ZFS team will be available at the booth to demonstrate and answer your initial, subsequent, and follow-up questions about ZFS. If you’re curious about ZFS or have been working on your best volume management/file system stumpers, come interrogate Bill.

I’ll be at LISA with two hats on: I’m still trying to think about service management and system approachability (and administrators see a lot of common and worst cases for each of these efforts), but I’ll also be there to hear about issues and concerns you might have about OpenSolaris. The schedule for the conference is busy, but I’d be happy to meet up with folks during the day, as well as at either BoF session; do feel free to send me email. If you’re looking for me physically, I’ll be the one wearing the tie.

UPS willing, I’ll also be trying out a new camera—I hope to get some photos up during the week.

[ T: Solaris OpenSolaris smf zfs LISA05 ]

Fairbanks weather payoff

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Dec 2005 06:24:04 +0000

By the way, Dina went to Fairbanks a few weeks ago, enjoyed the chilly contrast with Bay Area weather, and had a productive and fun visit. And, to keep karmic scales in balance, I caught a nasty cold for reminding her of the low low temperatures she could expect.

Falco columbarius in pole position

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Dec 2005 06:14:11 +0000

Nathaniel has keen vision; we often see mourning doves on the power and telephone lines that pass by the front of our house and he is happy to point them out. And occasionally, a raucous crow will perch in one of the nearby trees or atop a pole shouldering the local lines. But a couple of weekends ago, he was quite insistent about the novelty of the bird upon the pole’s top:

A good catch: this was the first Merlin I had seen in any of the neighbourhoods we’ve lived in. (I once watched a red-tailed hawk eat a dead pigeon on a corner ledge of MPK17 a few years back, while I waited for a printout one Saturday.)

The little raptor was good enough to stay in place long enough for us to take pictures from a couple of points with a couple of cameras. The above, the best of the lot, was taken with our camcorder (Canon Optura 200) in photo mode—the 10 × optical zoom was essential, and it’s still not a great shot.

Audacity for Solaris 10 x86

stephen@blueslugs.com (Stephen Hahn) — Fri, 18 Nov 2005 04:18:48 +0000

So both Dan Price and Ben Rockwood have asked again and again and again for Audacity, a free audio editor, to be ported to Solaris. A reasonable challenge. After a bit of fiddling and some decisions, I have a functional version of Audacity 1.2.3. Whether or not this will result in me issuing podcasts, mixes, or mashups is presently unknown, but decreasingly likely in that order.

This build does not have input support for MP3 or Ogg. You will have to use tools like sox or oggdec to convert those formats to WAV. Outputs to those formats should work, but I’ve mostly been testing with Ogg.

I thought I had notes on the port, but I can’t find them. So, from memory, here’s the rough idea:

Use blastwave.org for pre-packaged libaries where possible.
Use g++ because Audacity seems to use specific features of this C++ implementation.
Build your own Motif wxwidgets library using g++, because there isn’t a common C++ ABI on Solaris.
Drop in the Solaris-specific pieces from a recent version of the PortAudio library, use that as the primary sound option, and make a few edits for header file issues.
Get the run paths right. Sorry for the implicit lameness; depending on feedback, maybe I can retrace my steps more precisely.

Download

Audacity-1.2.3 [5.10 i86pc, 10M]

Please don’t decoralize the link, or I’ll have to take the binary down.

Installation

This build of Audacity 1.2.3 relies on you using blastwave.org for a collection of libraries. I assume you have a working version of pkg-get.

Get required libraries.

# pkg-get -i libvorbis libogg libmad libid3tag gcc3corert gcc3g++rt

You may also want

`lame`, `sox`, and `vorbistools` for audio transcoding.

Download package.
Uncompress and install.
```
# gunzip SLUGaudacity.pkg.gz
```

pkgadd -d SLUGaudacity.pkg

[Interactive questions. The dependencies against the Blastwave packages will be checked.]

Audacity is now installed as /opt/SLUGaudacity/bin/audacity. You will probably want to make a symbolic link from a more usual place in your path.

Have fun!

[ T: OpenSolaris Solaris Audacity Blastwave ]

And pack warm socks

stephen@blueslugs.com (Stephen Hahn) — Fri, 11 Nov 2005 07:50:39 +0000

It’s a balmy 9°F (-13°C) tonight in Fairbanks, although with windchill it feels like -6°F (-21°C). Why one would be naked and outdoors to verify that measurement is beyond me.

Better wear a hat

stephen@blueslugs.com (Stephen Hahn) — Tue, 08 Nov 2005 05:06:46 +0000

The current temperature in Fairbanks, Alaska is again -6Â°F, or -21Â°C.

Program technical status, 7 November

stephen@blueslugs.com (Stephen Hahn) — Mon, 07 Nov 2005 23:24:18 +0000

[Repost from forum thread.]

It’s been two months since my initial status message, which is probably a bit too long. Most of the efforts I mentioned in that update have made progress, so it’s worth mentioning their current status:

Elementary project hosting support.

The web site development for simple project hosting is underway. At present, those enhancements are expected for delivery sometime in December. (You may have noticed that a number of issues with the site were recently fixed.) In parallel, we are attempting to eliminate some of the manual steps in community and project creation.

Source code management (SCM), first phase.

The first phase of source code management involves adding Subversion hosting for individual projects and for the ON consolidation to publish ongoing changes in a read-only Subversion repository. Prototyping for the former has begun, but is dependent on the elementary project hosting work.

The first phase of source code management involves adding For ON, Stephen Lau has been experimenting with publishing a “squelched” SCCS history; you can see the result of Chandan integrating this work in the latest release of the source browser. For instance, here’s a file with some recent revisions and also the squelched history:

http://cvs.opensolaris.org/source/history/on/usr/src/cmd/svc/configd/configd.h

This work, along with some SCCS and TeamWare insights gleaned by Alan Burlison, is expected to be the basis of any SCM migration we pursue, as well as for the short term read-only Subversion publication.

Partitioned ON source tree.

The partitioned ON source tree is undergoing internal code review–internal only due to the encumbered code involved. Mike Kupfer gave much more detail in a recent blog entry

http://blogs.sun.com/roller/page/kupfer?entry=on_the_road_to_nightly

As noted in my previous update, a partitioned tree means that projects can issue public source drops of their development code, and get proper community development going.

ON GCC readiness.

Bug fixes that eliminate GCC warnings have continued to integrate into the Nevada gate. As a result, we’re now able to finish drafting a cleanliness policy for integrations and making the tools changes to support them. This policy will be discussed in the Tools community, once the draft is complete.

Because we’ve been making progress on those initial items, new aspects of the program will receive more attention. The two new areas of focus are discussed below.

Governance development.

The development of the governance for the OpenSolaris community is being led by the CAB. One effort in support of that has been creating a document articulating the engineering values that go into OpenSolaris software, as well as a process that contributors can follow that results in the production of such software. The most recent draft of the development process is available as HTML

http://www.opensolaris.org/os/community/onnv/os_dev_process/

or as PDF

http://www.opensolaris.org/os/community/onnv/os_dev_process/d-devproc-alpha.pdf

If you’re subscribed to cab-discuss, you know that drafts of the charter, which transfers responsibilities for OpenSolaris to the community from Sun, are being vigorously discussed in that forum. We’re attempting to keep the various stakeholders engaged so that the charter document can be completed and ratified promptly.

SCM, second phase.

The evaluation of a distributed SCM solution will be pursued directly now. I’ll be issuing draft requirements and an initial candidate list very shortly; this discussion will take place in the Tools community, although I’ll send a notice to opensolaris-discuss.

In terms of source releases, I thought I should also mention that libm, the C math library, was released in binary form in the past month. Cleaning up the source code for release is in progress. And, as I mentioned in another thread, team members have reviewed the packaging tools source and are working out whether a two step release is needed there as well. The Java Desktop System consolidation successfully published their code last week; other consolidations are getting close to releasing their trees as well.

As always, please share your concerns; I am happy to receive them privately or on the list.

[ T: OpenSolaris ]

Pretty cold up there

stephen@blueslugs.com (Stephen Hahn) — Sat, 05 Nov 2005 05:22:05 +0000

The current temperature in Fairbanks, Alaska is -6°F, or -21°C.

Old school

stephen@blueslugs.com (Stephen Hahn) — Fri, 04 Nov 2005 07:09:48 +0000

I’m still a bit woozy from a late night disk recovery at home which kept me up until about 4am Wednesday morning, but I thought I would point out two items in an old area of interest of mine: theoretical physics.

First, Physics Today has devoted a special issue to Hans Bethe. Bethe’s range and impact on physics is astounding to me: his 1967 Nobel prize for determining the nuclear reactions that result in the burning of stars came after many contributions to atomic and nuclear physics, roles at both the MIT Radiation Lab and Los Alamos during the Second World War, and important calculations for statistical physics. Oh, and he performed the first theoretical calculation of the Lamb shift, which validated the idea of quantum electrodynamics. The series of articles by friends and colleagues is excellent and, if you can grab a copy, definitely worth a read. (The site doesn’t have it, on first glance.)

Reading the practical scenarios where Bethe connected theory to a problem at hand, and then kept going, I wonder if Bethe might have been a better role model than my graduate school choice of Lev Landau, the 1962 Nobel winner in Physics (for the theory of superfluid helium, among other contributions). Of course, Landau’s preferred working style involved reclining upon his couch, which is a fundamental contribution I still hope to emulate.

The other item is that an old graduate school friend of mine, Mark Trodden, is, with a group of other—I’ll say young—theoretical physicists, writing a focussed-on-science-but-still-popular blog entitled Cosmic Variance. These are smart folks and, since we all can’t be research scientists, it’s a treat to get a chance to see some distinct intellectual viewpoints. Check it out.

(Maybe if I end up with another involuntary late night of repairs and resyncing, I’ll have a look around and see who else from school is blogging actively. Or just melancholily ruminate on lapsing and relapsing between engineering and science (which is why, when I confessed to dilettantism, I was being quite honest).)

Updated 2010-09-05: Fixed link to reflect Mark being at UPenn.

opensolaris.org: Java Desktop source release, and scfdot, too

stephen@blueslugs.com (Stephen Hahn) — Sat, 29 Oct 2005 01:16:13 +0000

As Glynn just blogged about it, I’d like to congratulate the JDS team (and the Desktop Community) on releasing the source to the Java Desktop System on opensolaris.org. In particular, I want to mention the build instructions, which provide a very clear starting point for those interested in exploring the source and how to build it.

I also wanted to point out that not only is a newer graph of the smf(5) services available:

but that, courtesy of

David and Dan, you can now generate your own, using scfdot. Visit the smf(5) community if you’re interested.

[ T: OpenSolaris Solaris smf JDS ]

Utility of a notched spoon

stephen@blueslugs.com (Stephen Hahn) — Wed, 19 Oct 2005 07:50:42 +0000

I went to a nice dinner a couple of weeks ago, to celebrate a couple friends’ visit to the Bay Area. The restaurant offered some tasting menus, so the more adventurous indulged in the six course offering, with a variety of tastes and textures. In line with current trends, flavoured foams accented between a quarter and a third of the different dishes served. It turns out some people really aren’t interested in eating foam.

At one point in the meal, a waiter positioned a peculiar spoon at my side, which looked something like this:

Peculiar because of the pronounced notch, clearly indicated by the red arrow above. Since the next course had “lobster” as a named ingredient, I speculated that the spoon might be tuned in some way for shellfish, as the utensil engineers have never really nailed down the best toolset for eating crab, lobster, and relatives. (Clawed lobsters have been around for 140 million years or so, and the fork for over a thousand.) Perhaps the notch was for trapping an errant leg.

The lobster turned out to be in the form of a purée or a bisque (but not a foam), so the bowl of the spoon was needed and the notch only useful for artful dribbling. Notched spoons were also provided for the dessert course, but no other mysterious utensils were manifested that night.

Update. Dan’s link to the L. A. Times has broken with time, so I’ll link to a reasonable Yahoo! answer. Since that may also someday fail, the answer is the term “sauce spoon”.

Forecast for coming late nights

stephen@blueslugs.com (Stephen Hahn) — Tue, 18 Oct 2005 04:57:10 +0000

I can see late nights ahead: there’s a lot of writing to do for work, and enough daytime interruptions and meetings to stop me from proceeding swiftly. Happily for those idle enough to read this blog, this new phase of sleep deprivation means the completion of assorted partial entries—and the release of the various pieces of code I’ve written/salvaged/jury-rigged over the summer. (For instance, I’ve got an unfinished book review of an overview of professional poker started over two years ago: since then, poker became hot, went mainstream, and then sold out (or maybe that was Bravo). Better to finish, or to have a running joke about the “poker book review”?)

smf(5): Stepping through an rc.d conversion

stephen@blueslugs.com (Stephen Hahn) — Mon, 10 Oct 2005 19:45:23 +0000

Over on mediacast.sun.com, I see Bob Netherton has posted a nice tutorial from Solaris Boot Camp, entitled “Migrating a legacy RC service”. The presentation covers the hiccoughs you might run into during your first conversion, and its step-by-step approach is very soothing.

[ T: OpenSolaris Solaris smf ]

Arrogance and approachability

stephen@blueslugs.com (Stephen Hahn) — Thu, 29 Sep 2005 07:30:10 +0000

Although I have been writing a lot lately—slides, draft policies and processes, and slews of email—none of it has managed to reach my blog.

I’ve been a Wall Street Journal subscriber since graduate school—skimming the paper in the morning is a habit I’ve maintained despite the hazard of having milk-sodden cereal hurled in my face by an unconciously ignored child. Last week, Walter Mossberg, in his Personal Technology column entitled “Yahoo Email Delivers That Desktop Feel Most Users Expect” [$$], reviewed Yahoo’s forthcoming web mail implementation, which is currently in beta test. The review is positive, but at the end Mr Mossberg shares what, for him, are some key constraints on software improvements. These constraints—which I think are overly severe—are very relevant to our work in Approachability, in which we’re reexamining some of the steps required to configure and operate a Solaris system.

The major aspect of the Yahoo Mail Beta appears to be the use of an increased amount of “AJAX” technologies in the implementation, such that the application pushes more logic local to an advanced client (like Mozilla Firefox or any of the “big” browsers) and thus appears to be more responsive. However, the final paragraphs in the column instead concern how Google Mail differs from Mossberg’s assessment of typical mail interactions:

…. On several key issues, Google’s engineers have decreed that familiar email practices are no longer useful, and have substituted approaches they prefer, arrogantly denying users any choice.

Gmail doesn’t allow folders, only color-coded labels, as an organizing technique. It forces you to view all of your email in groups of related messages called “conversations,” instead of viewing them individually as they arrive. Other email programs also allow such grouped views, but they permit users to choose. Not Gmail, where “option” is a term too rarely employed, except in reference to employee compensation.

This language is strong (and the closing sentence of the passage veers off into an ad hominem cornfield, since there was surely an internal user interface discussion behind this particular choice). The underlying request I believe is being expressed here is “justify your differences from convention”. Now, technically speaking, the labelling mechanism (or “tagging”) is a legitimate categorization offering that some users find superior to the implicit single category provided by the folder mechanism. If you’ve ever wanted to file an message in two folders, you know what I mean. (It’s also pretty reasonable to say we’re living in the Golden Age of Tagging and perhaps should try to enjoy it…) And, if you use one label per mail, you’ve reproduced the folder model via convention. So, while different, there’s no doubt that this interface offers a capability that some users will prefer.

The second objection, to the absence of an individual message view, I can’t get excited about, but I suspect that the UI discussion about this aspect concerned the deliberate elimination of various views. That discussion might have involved consistency of presentation (frame and window counts constant) or increased responsiveness (document model-based inbox with dynamic population). Software engineering rationale, like the maintenance costs of additional features may also have entered the discussion.

Mr Mossberg concludes by emphasizing his concern that Google will fail to meet certain users’ needs for cultural reasons:

I’m sure Gmail will get better and better, and will eventually adopt the new programming techniques that allow desktop-like ease of use. But I’m not sure Google’s arrogance will ever make room for user preferences on things like folders or ads, or how emails are grouped.

Yahoo’s new email program would blow Gmail away if it were widely released today. That’s partly due to its features, but also to its respect for user choice.

Again, strong words.

In Solaris Approachability, we’re wrestling with similar issues: should we offer all envisionable configuration choices for each feature? Where’s the sweet spot between too few choices and too many? We’ve been operating with a philosophy that the system should adjust itself dynamically, eliminating former tunables (or refusing to introduce new ones) in the interest of reduced management costs. The tradeoff is potentially a tax on performance or a loss in some notion of absolute flexibility.

The concern I have when I read a review like Mr Mossberg’s is that he presents a critique of a software artifact without even a cursory mention of his applicable criteria, and sweeps all failures to meet those criteria to Google’s engineering approach. What are those criteria? Under what circumstances would those criteria be invalid?

We’re planning to make changes to Solaris: typically, we will solicit feedback on the proposal itself, as well as through the development cycle. (Feedback is much easier to solicit with the growing OpenSolaris community.) If those changes risk violating your implicit criteria, let us know; if we make questioned changes even after that notice, it’s due to our drawing different conclusions from the available data, and not the result of doctrinal arrogance.

[ T: OpenSolaris Solaris approachability ]

Introducing JSON

stephen@blueslugs.com (Stephen Hahn) — Wed, 28 Sep 2005 17:53:20 +0000

Introducing JSON

JavaScript Object Notation site.

The Grove – SMF Description for Sun Java System Application Server

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Sep 2005 16:53:20 +0000

The Grove – SMF Description for Sun Java System Application Server

Jeff Bounds provides a service conversion for the Java ES App Server.

Cornell Notetaking Method Custom PDF Generator

stephen@blueslugs.com (Stephen Hahn) — Thu, 15 Sep 2005 04:53:20 +0000

Cornell Notetaking Method Custom PDF Generator

Make your own note paper.

Are you prepared

stephen@blueslugs.com (Stephen Hahn) — Wed, 14 Sep 2005 06:40:00 +0000

Are you prepared

San Francisco’s emergency prep site.

DTrace Troubleshooting

stephen@blueslugs.com (Stephen Hahn) — Wed, 14 Sep 2005 03:53:20 +0000

DTrace Troubleshooting

Brendan Gregg compares DTrace with other Solaris system monitoring tools.

High Hat, another Flash game

stephen@blueslugs.com (Stephen Hahn) — Mon, 12 Sep 2005 00:06:01 +0000

Like RSVP, the CBC‘s High Hat is another well-executed card game implemented in Flash. What’s interesting about High Hat is that it looks like a good candidate for playing with a physical deck as well. Maybe a good alternative to hours of War with Ben during the coming rainy winter here.

Recommended.

Pens At JOON New York – All The Best Pens In The World

stephen@blueslugs.com (Stephen Hahn) — Sun, 11 Sep 2005 17:33:20 +0000

Pens At JOON New York – All The Best Pens In The World

A fountain pen retailer.

Program technical status, 9 September

stephen@blueslugs.com (Stephen Hahn) — Fri, 09 Sep 2005 20:00:00 +0000

[Reposted from forum thread.]

I’ve had a few weeks to get oriented and thought I would bring folks up to date on the work we intend to pursue in the short term. My primary concern after an initial survey is creating additional infrastructure to get currently internal-to-Sun projects and conversations out in the open, on the opensolaris.org site, before the development phase of Nevada closes, and we move into a more limited Beta phase, where project work starts to focus exclusively on integration readiness.

That is, even though we are working to define and transition to a community development process and governance model, I don’t want the absence of the documents describing such things to be an excuse for not moving conversations outside. Accordingly, we are working on a number of items that eliminate that excuse. I’ll start with the general items, and then some specific to the ON consolidation.

Elementary project hosting support.

Typically, in terms of communication, projects inside Sun are very similar to projects we see in open source efforts: they have web pages, mailing lists, and download areas. (Some may also have IRC channels, Wikis, and blogs.) Much useful conversation and experimentation can take place using only these media, and so I want to get basic project hosting functionality up on opensolaris.org.

Projects that prefer to host elsewhere can use this facility as a pointer to their primary site, so that they appear in lists of OpenSolaris-related work.

Source code management, first phase.

The Code Manager (“TeamWare”) distributed source code manager (SCM) has been in use at Sun for over twelve years; its predecessors were also distributed SCM solutions. It is difficult to envision how we might move the current practices of the consolidations using TeamWare to an SCM that doesn’t match up well with the features and extensions that have been in use for so long. However, TeamWare itself has deficits when we consider its use on the open Internet (and even within Sun’s wide area network).

In order to make progress, and in order to support new and current projects and consolidations that are not tied to TeamWare, I believe that we must offer a centralized SCM facility while the current set of open source distributed SCM solutions are evaluated against criteria based on TeamWare’s use within Sun and on suitability for use on an Internet-hosted site. Luckily, recent developments in the SCM space suggest that one or more SCMs may meet many of these criteria already. A draft set of criteria will be published shortly, after which candidate SCMs will be evaluated against them.

The proposed centralized SCM solution is Subversion, based on features, ease of integration, and community vigor. Information on Subversion may be found at

http://subversion.tigris.org/

Tools to make the source drops of TeamWare-based consolidations available via a read-only repository will also be found/refined/developed. We will publish a representation of ON via a read-only repository during this phase.

Partitioned ON source tree.

A number of ON-based projects are ready to share their development versions, in code form, on the site. However, the current source publication process for ON is too arduous to expect individual projects to use team member time to handle their own publication. We are thus partitioning the source tree into open (“usr/src”) and encumbered (“usr/closed”) subtrees, so that projects can easily publish and independently buildable open tree containing their changes. This work is already underway.

ON GCC compiler readiness.

Steady progress is being made to make ON builds GCC warnings free. As we are now closing in on a warnings-free build, we will be examining the current build tools to make GCC clean (or, if you prefer, alternate compilation clean) an ongoing requirement for integration, without being an undue burden on contributors. There are numerous possibilities that are opened up by a GCC build, but the primary justification for the present work is that it finds bugs prior to integration.

There are, of course, many other activities going on: Karyn’s list of consolidation priorities hints at what additional code is nearing openness, creation of additional website content, the already mentioned development process and governance and charter work, and the publication of ON source drops. (Plus work in community building and support, marketing efforts, …) I thought I should call out that we have added additional sponsors for ON and that John Beck has been working to close some gaps in sponsor technical coverage, so that code fixes submitted during this interim phase don’t get chilled waiting for a sponsor.

Feedback welcome. I’ll read it all, but I can’t promise replies to all.

[ T: OpenSolaris ]

memcached: a distributed memory object caching system

stephen@blueslugs.com (Stephen Hahn) — Thu, 08 Sep 2005 17:20:00 +0000

memcached: a distributed memory object caching system

Use to speed up your dynamic Web applications.

Mystery Bulge in Oregon Still Growing

stephen@blueslugs.com (Stephen Hahn) — Thu, 08 Sep 2005 03:26:40 +0000

Mystery Bulge in Oregon Still Growing

Swelling near southernmost of the Three Sisters.

Reexamining OpenSolaris technical concerns

stephen@blueslugs.com (Stephen Hahn) — Wed, 07 Sep 2005 00:23:15 +0000

I’ve recently rejoined the team working directly on the OpenSolaris effort. The past few weeks I’ve been getting caught up on where Open is with respect to a number of areas, both technical and process. During my survey, I’ve been pleased to see how good the current technical team (“I-team”, or implementation team, in local software development framework-speak) is—they’ve got a handle on the complexities of delivering the nearly 36 000 files in the ON consolidation in a partitioned, cleansed form, and are spending a good chunk of time so that the other major source trees that make up Solaris (“consolidations”) don’t have to reinvent these procedures themselves.

There are a number of technical topics—involving how we develop software—that I want to visit over the next few weeks in this blog, because it seems like a reasonable way to describe some of the key aspects to what we’re doing now and lay out the set of issues that need to balanced in any effort to make a principles-driven community development process. For instance, a short list of topics would include at least

multiple compiler support implications,
“shrink to fit” architectural processes,
source code management requirements and options,
programmatic facilities at opensolaris.org (“web services versus web pages”), and
workflow tools and support, on and off the site. And probably some other things that are worrying me (like how to keep up with a few hundred additional email messages per day, above the few hundred I was already receiving).

That said, I am happy to hear your concerns, hopes, fears, and speculations about OpenSolaris, and email is, of course, fine.

[ T: OpenSolaris Solaris ]

Citizen Corps

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Sep 2005 22:26:40 +0000

Citizen Corps

Another citizen-oriented effort from Homeland Security.

Jakarta Slide

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Sep 2005 16:53:20 +0000

Jakarta Slide

Apache project for a Java-based WebDAV server.

darcs

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Sep 2005 05:46:40 +0000

darcs

A distributed revision control system, written in Haskell.

monotone: distributed version control

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Sep 2005 05:46:40 +0000

monotone: distributed version control

A distributed version control system with no external dependencies.

California State Parks

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Sep 2005 03:00:00 +0000

California State Parks

Listings, access to reservations for California state recreation areas, reserves, and parks.

Mercurial Distributed SCM

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Sep 2005 00:13:20 +0000

Mercurial Distributed SCM

Python-based source code management, used by Xen.

Advanced Building Technologies

stephen@blueslugs.com (Stephen Hahn) — Mon, 29 Aug 2005 07:20:00 +0000

Advanced Building Technologies

Varous energy/resource efficiency techniques for buildings.

Rainwater Harvesting

stephen@blueslugs.com (Stephen Hahn) — Mon, 29 Aug 2005 04:33:20 +0000

Rainwater Harvesting

Barrel-based system; nice parallel storage plumbing example.

MBA Toolbox : Hello world

stephen@blueslugs.com (Stephen Hahn) — Sun, 28 Aug 2005 03:33:20 +0000

MBA Toolbox : Hello world

Some chapters from an business professor’s courses.

Smatch the Source Matcher

stephen@blueslugs.com (Stephen Hahn) — Sat, 27 Aug 2005 16:26:40 +0000

Smatch the Source Matcher

Open source metacompilation, applied to Linux kernel.

OpenOffice.org: Home

stephen@blueslugs.com (Stephen Hahn) — Thu, 25 Aug 2005 06:06:40 +0000

OpenOffice.org: Home

Open source office suite; basis for Sun’s StarOffice

Version Control with Subversion

stephen@blueslugs.com (Stephen Hahn) — Tue, 23 Aug 2005 20:46:40 +0000

Version Control with Subversion

The O’Reilly text, available online.

Doug McIlroy

stephen@blueslugs.com (Stephen Hahn) — Wed, 17 Aug 2005 06:26:40 +0000

Doug McIlroy

M. Douglas McIlroy’s home page

DesktopBSD 1.0-RC1 screenshots

stephen@blueslugs.com (Stephen Hahn) — Fri, 12 Aug 2005 18:06:40 +0000

DesktopBSD 1.0-RC1 screenshots

Installation and first boot configuration for Desktop BSD

smf(5) Sybase conversion

stephen@blueslugs.com (Stephen Hahn) — Fri, 12 Aug 2005 18:06:40 +0000

smf(5) Sybase conversion

Sjaak Westdijk has an smf(5) manifest and method for Sybase.

The Opening of Solaris

stephen@blueslugs.com (Stephen Hahn) — Fri, 12 Aug 2005 04:13:20 +0000

The Opening of Solaris

Peter Baer Galvin gives an overview of the OpenSolaris program.

thttpd on Solaris 10

stephen@blueslugs.com (Stephen Hahn) — Thu, 11 Aug 2005 17:06:40 +0000

thttpd on Solaris 10

Todd Carson offers patches and an smf(5) conversion for thttpd.

BigAdmin Feature Article: Configuring Sun Java System Identity Manager as a Service for the Solaris 10 OS

stephen@blueslugs.com (Stephen Hahn) — Thu, 11 Aug 2005 06:00:00 +0000

BigAdmin Feature Article: Configuring Sun Java System Identity Manager as a Service for the Solaris 10 OS

An smf(5) conversion of a Java ES component.

Concrete Abstractions

stephen@blueslugs.com (Stephen Hahn) — Thu, 11 Aug 2005 06:00:00 +0000

Concrete Abstractions

“An introduction to computer science using Scheme.”

smf(5): Identity Manager conversion

stephen@blueslugs.com (Stephen Hahn) — Wed, 10 Aug 2005 21:39:34 +0000

Scott Fehrman‘s written a detailed conversion of the Sun Java System Identity Manager to use smf(5). In Scott’s scenario, the Identity Manager runs inside Apache/Tomcat and uses MySQL as the backing database; Scott discusses both service and their dependencies as he constructs the manifests.

See BigAdmin for Scott’s article.

[ T: OpenSolaris Solaris smf ]

Caltrain Schedule for PalmOS

stephen@blueslugs.com (Stephen Hahn) — Tue, 09 Aug 2005 17:53:20 +0000

Caltrain Schedule for PalmOS

My most useful Palm application, without exceptions.

Transom: A Showcase for New Public Radio

stephen@blueslugs.com (Stephen Hahn) — Sun, 07 Aug 2005 02:00:00 +0000

Transom: A Showcase for New Public Radio

Programming, instruction, and perspective.

Transom Tools: Recording/Interviewing: What Microphone Do I Get

stephen@blueslugs.com (Stephen Hahn) — Sat, 06 Aug 2005 17:40:00 +0000

Transom Tools: Recording/Interviewing: What Microphone Do I Get

A summary of microphones for interviewing.

Yet another quiet week

stephen@blueslugs.com (Stephen Hahn) — Wed, 03 Aug 2005 19:26:40 +0000

Once again, we’ve sent a delegation of engineers to Portland for OSCON. The absence of that group—Bryan, Keith, and Liane are there —and of the many other folks involved with OpenSolaris, plus the usual August vacations, is leaving the halls here in Menlo Park rather quiet.

Of course, while I may be enjoying an audible quiet, the OpenSolaris lists are hopping: the vigorous discussions on compatibility as a community value, packaging systems, new communities, as well as technical questions, are showing how much energy there is across the larger (Open)Solaris community. If only I could sleep less…

[ T: OpenSolaris Solaris ]

smf(5): Manifest destiny, Round One

stephen@blueslugs.com (Stephen Hahn) — Fri, 15 Jul 2005 19:56:37 +0000

I received polite mail this morning reminding me that I have mugs to ship. I got distracted during break, but we’ll be boxing up the mugs and getting them to the authors. So, for Round One, here is the set of submitted service manifests submitted to date:

<td>
Informix Dynamic database server
</td>
<td>
Prasad Jampala
</td>
<td>
USA
</td>

<td>
MySQL database server
</td>
<td>
Keith Lawson
</td>
<td>
Canada
</td>

<td>
Oracle database control
</td>
<td>
Joost Mulders
</td>
<td>
Netherlands [Sun]
</td>

<td>
Oracle listener
</td>
<td>
Joost Mulders
</td>
<td>
Netherlands [Sun]
</td>

<td>
[<a href="http://popfile.sf.net/" >POPFile</a> mail classifier]
</td>
<td>
Iouri Goussev
</td>
<td>
Canada
</td>

<td>
Common Unix Printing system
</td>
<td>
Boyd Adamson
</td>
<td>
Australia
</td>

<td>
X Window System Print server
</td>
<td>
Peter Eriksson
</td>
<td>
Sweden
</td>

<td>
pident IDENT daemon
</td></td>
<td>
Gary Mills
</td>
<td>
Canada
</td>

<td>
INN NNTP server
</td></td>
<td>
Gary Mills
</td>
<td>
Canada
</td>

<td>
OpenNTPD daemon
</td>
<td>
Todd Carson
</td>
<td>
USA
</td>

<td>
postfix SMTP MTA*
</td>
<td>
Ben Rockwood
</td>
<td>
USA
</td>

<td>
qmail SMTP MTA
</td>
<td>
Iouri Goussev
</td>
<td>
Canada
</td>

<td>
TXPI TCP
</td>
<td>
Hans van Maaren
</td>
<td>
The Netherlands
</td>

<td>
CA-XCOM data transport
</td>
<td>
Hans van Maaren
</td>
<td>
The Netherlands
</td>

application/informix

application/mysql

application/oracle/[database]

application/oracle/[listener]

[application]/popfile

[application/print]/cups

[application/print]/xprint

[network/ident:pident]

[network/nntp:inn]

network/ntp:openntp

network/[smtp:postfix]

network/[smtp:qmail]

network/txpi:tcp]

[network/xcom:default]

* Ben’s Postfix conversion is very simple, and works if postfix is in root‘s default path; in contrast, Peter Tribble built a full stack of mail processing atop Postfix in smf(5) a few months back. Plus they both have mugs already.

We’re now in the process of assembling all of the manifests we know about at the OpenSolaris smf(5) community. If you sent in a manifest without a URL, it would be preferable to give us a link to it, and to give it a copyright and (OSI-approved) licence. (Send me mail if this appears confusing or is difficult.)

Suggestions for Round Two (beyond PostgreSQL and the other database management systems) are welcomed!

[ T: OpenSolaris Solaris smf ]

What I did during my summer vacation: hack hack hack

stephen@blueslugs.com (Stephen Hahn) — Tue, 12 Jul 2005 19:10:33 +0000

The rest of the family didn’t have vacation time, beyond the Fourth of July weekend, so I got a chance to do some reading and to write some code.

I am particularly excited to use tag(1) at work: my home directory is hosted on the main zfs(7FS) server, which means practically unlimited metadata. Tagging—I hope—will let me coalesce my projects/, projects/old-projects, and play/ into a single projects directory with current and obsolete tags. I’m also toying with ideas around HTML files involving redirects as a private link collection and tagging file fragments via their offsets.

[ T: OpenSolaris Solaris zfs ]

Making secureimage.php strict

stephen@blueslugs.com (Stephen Hahn) — Tue, 12 Jul 2005 15:20:51 +0000

One of the aspects that kept me up late writing the tag(1) entry was getting MathML to work. Getting itex2MLL to compile was easy, but correcting the various WordPress templates took longer.

This morning I found that the SecureImage plugin needs a very small change to make its output strict XHTML:

``One of the aspects that kept me up late writing the tag(1) entry was getting MathML to work. Getting itex2MLL to compile was easy, but correcting the various WordPress templates took longer.

This morning I found that the SecureImage plugin needs a very small change to make its output strict XHTML:

And now the permalinked page works. If you wanted to comment on tag(1), please try again.

tag(1): del.icio.us-style file tagging

stephen@blueslugs.com (Stephen Hahn) — Tue, 12 Jul 2005 09:50:15 +0000

I’ve been using the social bookmarking service, del.icio.us, for a while now, and have watched similar tagging features move from site to site as a flexible way to lightly categorize various resources: bookmarks, blog posting, photos, etc. It’s therefore a little strange that no one has written a simple utility to give you similar category construction capabilities on your typical Unix-like file system. In this post, I propose a simple, portable design, contrast my design choices against some other possibilities, and then provide an initial implementation of that design in Perl.

Tagging

If you’ve not used del.icio.us, Technorati’s tag system, or any of the Web tagging systems, then a brief introduction is worthwhile. If you’re already familiar with tagging, then this section will alert you to my personal terminology around tagging operations and the resulting metadata.

By tagging, we mean the association of one or more keywords, called tags, with an entity. The tag set is the set of tags associated with the entity, which for the purposes of this post, is always a file.

Once you’ve accumulated a set of tagged files, the next step is to examine sets of files with common tags: each such set is a query. The link set is the set of all possible queries against the set of tags. That is, the set of files tagged with the same set of tags is a member of the link set. A particular member of the link set can be calculated dynamically, or the entirety of the set can be calculated in advance.

Querying tags

del.icio.us appears to calculate the per-tag queries on the fly, which is acceptable as the primary interface is the web page presented to the reader. (Caching is probably involved.) We instead have a choice: we can use our command, tag(1), or we can use the filesystem itself. Our choice is accompanied by a tradeoff between time (to compute queries) and space (inodes consumed). Although, as we will see, the space consumption is significant, the filesystem approach presents an easy interface for exploration.

By default, the link set is constructed in $HOME/tag:

$ ls ~/tag/
code/ doc/ linux/ rm/ thread/ utility/
conference/ greenline/ note/ solaris/ umem/

If we look in ~/tag/code, we see

$ ls ~/tag/code/
gumemd.pl@ pause.c@ tag.pl@ tagging/ umem/ utility/

symbolic links to files with that tag, and directories representing additional tags. This directory hierarchy is the link set: all possible queries are precalculated:

$ ls ~/tag/code/umem/
gumemd.pl@

If we go to the original file, tag(1) will tell us the tags on the file:

$ tag gumemd.pl
code umem

We can also query using the tag command, but it is merely walking the directory structure we constructed.

$ tag -q code
/home/sch/old-home/gumemd.pl
/home/sch/old-home/pause.c
/home/sch/play/tag/tag.pl

(This operation is relatively fast, as we would expect:

$ time tag -q code
/home/sch/old-home/gumemd.pl
/home/sch/old-home/pause.c
/home/sch/play/tag/tag.pl
real 0m0.068s
user 0m0.057s
sys 0m0.009s

Timing run performed on a 1.8 GHz Athlon64 running Solaris 10 and Perl 5.8.4.)

Adding tags is simple:

$ tag -a perl gumemd.pl
$ tag gumemd.pl
code umem perl
$ ls ~/tag/perl
code/ gumemd.pl@ umem/

As is deleting them:

$ tag -d perl gumemd.pl
$ ls ~/tag/perl
/home/sch/tag/perl: No such file or directory

So we have a design for basic tagging of files.

Tags and file system metadata consumption

It’s worth noting that precalculating the entire link set and storing it in a file system consumes file system metadata combinatorically, but that this consumption is capped by the (typical) limit on path length. Generally, we make M symbolic links for the n distinct tags on a single file, with M given by

$$ M(n) = \sum_{r = 1}^{n} \frac{n!}{(n – r)!} $$

That’s 325 links for 5 tags, plus as many directories again, for a total of 650 metadata units. On file systems that preallocate a fixed size pool of inodes, this kind of feature could lead to metadata exhaustion.

Tagging and file system operations

The key file system operation in our tagging system is renaming: how do we update the link and tag sets when a particular file is relocated within the filesystem? Deletion is a subset of this case; copying can be treated similarly to a set of tag-add operations on a previously untagged file.

If we just use mv(1), and have the link set and tag set stored externally to the file, then we end up with a depressing result:

The link set points to and the tag set is associated with the old location, and operations on these objects will fail. The file in its new location has lost all of its tags-derived metadata, and so has dropped out of our tagspace.

On some Unix-like OSes, certain filesystems support file attributes, such as Solaris’s UFS and Linux’s ext3fs. With attributes, data associated with the file is associated with the file in such a way that filesystem operations leave the association invariant (unless they involve the attribute data specifically). This modifies our operation to something more acceptable:

Although the link set is now stale, and we would have to rebuild the entire link set to remove the stale links to the old location, and to generate the correct links to the new one.

It is clear that what we want to occur is simultaneous updating of the link and tag sets with each operation: we want transactional operations.

One way to achieve a transactional operation is to get beneath the rename(2) system call, and to update both the link set and the tag set within an underlying tagging file system. It would be straightforward to implement a translucent file system that offered a namespace for the link set, and used an underlying on-disk filesystem to store the appropriate data. But, like our declining to use attributes earlier, we pass on an in-kernel implementation because of the associated portability costs. Instead, we have to encapsulate the various steps into an apparent transaction, by offering rename and remove operations in the command itself.

We implement encapsulated operations in the prototype version by wrapping the standard Unix-like commands: cp(1), mv(1), and rm(1). The syntax is

$ tag --mv srcfile dstfile

and similarly for the other commands.

Implementation

As I noted earlier, the code is a draft implementation and is grossly undertested. There are numerous improvements that could be made to the implementation—the tying and untying for multiple tag operations is an obvious performance sink. Architecturally, the script should be separated into a module providing tagging functionality and a command that presents the module’s functionality as an elegant command line interface. (It would be a reasonable position to move tagging into a small C library, such that a simple common tagging infrastructure could be made available. We could alternatively commit to a specific DBM file format and offer access from multiple symbolic link-capable languages.)

Implementation improvements would include: recognizing when an operation (tag retrievals in particular) is on an in-the-link-set symbolic link, resynchronizing/repairing databases and link hierarchies, better documentation (although the script does embed minimal documentation using POD), a test suite, and use of ExtUtils::MakeMaker or Module::Installer.

As always, I’m interested in your comments, suggestions, and references to similar work.

[T: Solaris ]

Manufacturing week reading

stephen@blueslugs.com (Stephen Hahn) — Wed, 06 Jul 2005 17:00:19 +0000

It’s Sun’s U.S. manufacturing shutdown, so I’m getting caught up on reading and household repairs. In addition to last week’s work reading, I read two other business/investing books:

Jeffrey Zygmont’s The VC Way, which was interesting (but didn’t live up to its subtitle). Mostly structured stories about venture capital, and its role in the startup process. Ok.
Jeremy J. Siegel’s The Future for Investors, which was simply an excellent book for investors. (Particularly those with a 30 – 50 year window ahead.) Recommended.

Since I also installed all the new doorknobs, perhaps I’ll spend today writing some code for fun.

Unexpected new server

stephen@blueslugs.com (Stephen Hahn) — Wed, 06 Jul 2005 06:41:21 +0000

Now that the Shuttle system appears unsalvagable—no activity, even with a nice fresh BIOS chip—I suppose I should record the emergency server rebuild from a few weeks ago. On Saturday, Dina noticed she wasn’t getting any email. We had houseguests, so I didn’t slip out to Central Computer until Sunday, with a plan to wrap a new system around the old disks. I ultimately ended up with

Asus A8V Deluxe
AMD Athlon 64 3000+
1 GB
Intel PRO/1000 GT [32-bit drivers from Intel for now]
ATI PCI graphics card
2 × Western Digital 120GB drives

all in a CoolerMaster <a href=“http://www.coolermaster.com/index.php?LT=english&Language_s=2&url_place=product&p_serial=CAC-T05&other_title=0"

Centurion 5 case, and which is now running Solaris 10. The motherboard works well, although I’m only using the IDE controllers and none of the SATA or RAID functionality. (I also couldn’t match a driver to the onboard Gigabit Marvell Ethernet.)

Like any project at our place now, willing helpers materialize, tools in hand, even for a no-tools case:

I’ve built enough systems now that we shortly were ready for action, and blueslugs.com and highmaintenancemom.com were back:

cooler lives in the office closet, which it shares with the UPS which selflessly filters PG&E’s rot gut, leaving only nicely distilled power:

(Without the Shuttle as a new desktop, I’m deciding between building another A8V-based system in a CoolerMaster Cavalier 4, or buying an Ultra 20.)

[T: Solaris]

First bike commute

stephen@blueslugs.com (Stephen Hahn) — Sat, 02 Jul 2005 18:22:14 +0000

Ben and I biked to GeoKids and then I carried on to the Menlo Park campus yesterday, perhaps in honour of Canada Day. Ben was pretty happy for the first few miles, but I suspect was bored by the time we made it to downtown Menlo. (Google Maps has the distance at about 5 1/2 miles, although our route was longer—but on bike-friendlier streets.)

Dina took a photo of us setting forth from the driveway:

(For Peninsula readers, the main roads we used were Virginia-to-Woodside-to-Alameda de las Pulgas-to-Avy-to Santa Cruz. There appear to be some neighbourhood roads that we can use to skip Woodside—California Highway 84—and make the route safer still.)

Reading at work: Lencioni's Death by Meeting

stephen@blueslugs.com (Stephen Hahn) — Sat, 02 Jul 2005 00:32:20 +0000

One of the nice aspects of working at Sun is that the company subscribes to a number of online libraries. Usually I consult these for overviews and introductions to technologies I use rarely or need to put into context. But occassionally, I’ll read a business book.

This past week, I took two chunks of time out—an evening at home, and a half hour this afternoon—to read Patrick M. Lencioni’s Death by Meeting : A Leadership Fable… About Solving the Most Painful Problem in Business. The title is melodramatic from every possible angle, and the main argument is targeted much more to executives and general management teams, but I can see a few aspects that will help in our own technical meetings. The point about the hidden cost of “sneaker networking” as opposed to having an effective meeting hits close to home.

This text is in Sun’s available collection at NetLibrary.

OpenSolaris BoF at JavaOne

stephen@blueslugs.com (Stephen Hahn) — Fri, 24 Jun 2005 19:09:16 +0000

On Monday evening, a few of us—Bryan, Mike, Adam, and myself (plus others, I hope)—will be hosting an OpenSolaris birds-of-a-feather session during JavaOne. The BoF, Java Technology on the OpenSolaris Platform”, will be from 1930 – 2020 in Golden Gate B1 at the Marriott Hotel.

With Team DTrace there, the BoF should be an additional opportunity to ask questions about some of the preliminary Java–DTrace instrumentation techniques and how that might apply to your programming problems. (Of course, you should go to Adam’s session for the full story first.) I’m there for a different reason. As Dave mentioned, we’re planning a set of classes to make the smf(5) functionality available from the Java platform. It turns out that these might be the first public Solaris-specific Java interfaces we publish, which means we’ve reached a small unexplored architectural area. To start surveying this terrain, I need to ask the community a couple of questions:

What else can we do to make Java a first class systems programming language for Solaris?
For instance, what other classes, defaults, or conventions should we offer or modify to make Solaris-based Java development easier/more capable/more fabulous?

Of course, you don’t have to go to JavaOne to make suggestions—feel free to make them here. But if you’re at the conference, please drop by and make your points in person. The BoF details once more:

Java™ Technology on the OpenSolaris™ Platform
Monday 6/27
7:30-8:20pm
Golden Gate B1
Marriott Hotel

Hope to see you there.

[ T: OpenSolaris Solaris smf JavaOne2005 ]

Prosumption vs. consumption

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Jun 2005 18:33:44 +0000

The Digital Photography Review just posted its review of Canon’s PowerShot S2 IS, a 5 megapixel camera with 12 × zoom and image stabilization.

Hard to resist; I’m leery of going the full prosumer (or even professional) SLR route, and this camera has almost everything I would want.

Manifests so far

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Jun 2005 05:48:25 +0000

To ward off unintentional duplication of effort, I thought I would list the service manifests submitted to date:

<td>
MySQL database server
</td>
<td>
Keith Lawson
</td>
<td>
Canada
</td>

<td>
Oracle database control
</td>
<td>
Joost Mulders
</td>
<td>
Netherlands [Sun]
</td>

<td>
Oracle listener
</td>
<td>
Joost Mulders
</td>
<td>
Netherlands [Sun]
</td>

<td>
[<a href="http://popfile.sf.net" >POPFile</a> mail classifier]
</td>
<td>
Iouri Goussev
</td>
<td>
Canada
</td>

<td>
X Window System Print server
</td>
<td>
Peter Eriksson
</td>
<td>
Sweden
</td>

<td>
OpenNTPD daemon
</td>
<td>
Todd Carson
</td>
<td>
USA
</td>

<td>
qmail SMTP MTA
</td>
<td>
Iouri Goussev
</td>
<td>
Canada
</td>

application/mysql

application/oracle/[database]

application/oracle/[listener]

[application]/popfile

[application]/xprint

network/ntp:openntpd

network/[smtp:qmail]

I hadn’t heard of POPFile before, but a multiple bucket mail classifier might be the only way for me to keep up with opensolaris-discuss@opensolaris.org.

I’ve read each of the above manifests quickly, so I’m confident we’ll be sending out mugs to each author—yes, even to you, Joost. We’ll do a detailed read, send some suggestions (like my first thoughts on service names), and publish links at the contest end. Keep the manifests coming.

[ T: OpenSolaris Solaris smf nationalism ]

Contest progress; smf(5) discussion

stephen@blueslugs.com (Stephen Hahn) — Wed, 15 Jun 2005 22:20:12 +0000

I’ve been happily receiving submissions of manifests for various services over the past few days. With all the traffic around other news, I believe it only fair to extend the “Manifests sans frontierès” contest to Canada Day (1 July), so that we can all monitor new mailing lists, more vigorous IRC discussion, track countless tags and feeds, and generally commune, and still write some useful code.

benr: I have an Oracle submission in hand.

One of those new forums is smf-discuss@opensolaris.org, where we’ll be having development discussions about smf(5). There’s big stuff and little stuff to do, so if you’re not certain where to start or are looking for ways to contribute, come talk with us—our learning curve is relatively gentle.

To subscribe, send a blank message to smf-discuss-subscribe@opensolaris.org.

[ T: OpenSolaris Solaris smf ]

Sharing a little bandwidth

stephen@blueslugs.com (Stephen Hahn) — Wed, 15 Jun 2005 04:53:26 +0000

I left a BitTorrent session open all day, to make a little contribution to sharing some of the code from OpenSolaris. No real load on our unexpected new server, but 413 MB shipped in 9 hours through our little pipette of a ADSL line. I’ll have to use trickle if I’m going to do that again.

[T: OpenSolaris]

libuutil and designing for debuggability

stephen@blueslugs.com (Stephen Hahn) — Tue, 14 Jun 2005 17:51:44 +0000

Going into Solaris 10, I knew we were planning to develop a troupe of new daemons; we ultimately ended up with svc.startd(1M), svc.configd(1M), and a new implementation of inetd(1M). I wanted to make sure we made some progress on daemon implementation practice, and bounced some ideas around with the afternoon coffee group and also with Mike, and probably some others—I wander around a bit.

We anticipated that most of the daemons would be multithreaded, and it became apparent that they would all present large, complicated images for postmortem debugging¹. To reduce the time to acquire familiarity with each of these daemons, we worked out three common requirements:

include Compact C Type Format (CTF) information with each daemon,
use libumem(3LIB) for memory allocation, and
use standard, debuggable, MT-safe implementations of data structures.

The problem was, of course, that there wasn’t a library with such data structures in Solaris at the time.^{2, 3}. So we began to design libuutil, which combines a number of established utility functions used in authoring Solaris commands with these new “good” implementations of useful data structures.

The library in question was named in sympathy with libumem(3LIB)—libuutil for “userland utility functions”. libuutil provides both a doubly linked list implementation and an AVL tree implementation. The list implementation is mostly located in lib/libuutil/common/uu_list.c; we’ll use that to explore the debugging assistance we designed in.

The model used is that each program is likely to have multiple lists of common structures, and that there would be multiple such structures. This led us to create an interface that is expressed in terms of pools of list. So, for each structure, you create a list pool using uu_list_pool_create(). Then, for each list of that structure, you create a list in the respective pool using uu_list_create().

That sounds complicated, but it’s for a good reason: at each call to uu_list_pool_create(), we register the newly created pool on a global list, headed by the “null pool”, uu_null_lpool:

uu_list_pool_t *
uu_list_pool_create(const char name, size_t objsize,
size_t nodeoffset, uu_compare_fn_t *compare_func, uint32_t flags)
{
uu_list_pool_t *pp, *next, *prev;
/ validate name, allocate storage, initialize members */
(void) pthread_mutex_init(&pp->ulp_lock, NULL);
pp->ulp_null_list.ul_next = &pp->ulp_null_list;
pp->ulp_null_list.ul_prev = &pp->ulp_null_list;
(void) pthread_mutex_lock(&uu_lpool_list_lock);
pp->ulp_next = next = &uu_null_lpool;
pp->ulp_prev = prev = next->ulp_prev;
next->ulp_prev = pp;
prev->ulp_next = pp;
(void) pthread_mutex_unlock(&uu_lpool_list_lock);
return (pp);
}

with similar code being used to connect each list to its pool on calls to uu_list_create().

So now we have an address space where each list pool is linked in a list, and each list in a pool is linked to a list headed at that pool. This leads us to the second part, which is to use the encoded information in a debugger. The typical debugger for kernel work in Solaris is mdb(1), the modular debugger. It’s been shipping with Solaris since 5.8, and has a rich set of extensions for kernel debugging. For userland, the modules are rarer: libumem is probably the best known.⁴

The source code for the libuutil module (or “dmod”) is located at cmd/mdb/common/modules/libuutil/libuutil.c; the function that provides the dcmd itself, uutil_listpool, is just a wrapper around the walker for uu_list_pool_t structures. The pertinent portion is the initialization function, uutil_listpool_walk_init():⁵

int
uutil_listpool_walk_init(mdb_walk_state_t *wsp)
{
uu_list_pool_t null_lpool;
uutil_listpool_walk_t *ulpw;
GElf_Sym sym;
bzero(&null_lpool, sizeof (uu_list_pool_t));
if (mdb_lookup_by_obj("libuutil.so.1", "uu_null_lpool", &sym) ==
-1) {
mdb_warn("failed to find 'uu_null_lpool'\n");
return (WALK_ERR);
}
if (mdb_vread(&null_lpool, sym.st_size, (uintptr_t)sym.st_value) ==
-1) {
mdb_warn("failed to read data from 'uu_null_lpool' address\n");
return (WALK_ERR);
}
ulpw = mdb_alloc(sizeof (uutil_listpool_walk_t), UM_SLEEP);
ulpw->ulpw_final = (uintptr_t)null_lpool.ulp_prev;
ulpw->ulpw_current = (uintptr_t)null_lpool.ulp_next;
wsp->walk_data = ulpw;
return (WALK_NEXT);
}

which safely pulls out the value of the uu_null_pool head element, and the relevant pieces we’ll need to walk the list.

This means that, for any program linked with libuutil, we can attach with mdb(1M) and display its list pools:

mdb -p `pgrep -z global startd`

Loading modules: [ svc.startd ld.so.1 libumem.so.1 libnvpair.so.1 libsysevent.so.1 libuutil.so.1 libc.so.1 ]

::uu_list_pool ADDR NAME COMPARE FLAGS 080dcf08 wait_info 00000000 D 080dce08 SUNW,libscf_datael 00000000 D 080dcd08 SUNW,libscf_iter 00000000 D 080dcc08 SUNW,libscf_transaction_entity c2b0476c D 080dc808 dict 0805749c D 080dc908 timeouts 0806ffab D 080dca08 restarter_protocol_events 00000000 D 080dcb08 restarter_instances 0806ccd7 D 080dc708 restarter_instance_queue 00000000 D 080dc608 contract_list 00000000 D 080dc508 graph_protocol_events 00000000 D 080dc408 graph_edges 00000000 D 080dc308 graph_vertices 08059844 D

and then drill down into constituent lists of interest.

Additional walkers are also provided, such that the lists and list nodes can be visited from the command line or programmatically. As an example, the ::vertex dcmd from the svc.startd module uses the walkers to display the various service graph nodes in a quasi-readable format.⁵

So, by providing extra structured information in the library and support to consume that information in the debugger, we end up with a set of data structures that, if used, leads to more debuggable programs. More work up front for less later: welcome to OpenSolaris.

Footnotes

1. By postmortem debugging, I’m referring to the operation of debugging a failed application after its failure, from a core file or other memory image captured as soon after that failure as possible. Suitability for postmortem debugging is a standard expectation for software design in Solaris, as it reduces the time to diagnose and fix software failures. In particular, multiple engineers can debug a core file in parallel; this can be contrasted with the cost of setting up a duplicate installation and trying to reproduce the failure, let alone expecting the customer to risk further downtime experimenting with “try this” scenarios.

2. Please remember that we were making these decisions three years ago, and that this choice had to fit the then-applicable constraints on the product.

3. In contrast, the kernel has had a generic, modular hash table since 5.8/2000 (uts/common/os/modhash.c), a generic AVL tree since 5.9/2002 (common/avl/avl.c), and a generic list implementation early in 5.10/2005 (uts/common/os/list.c). Of course, the kernel has used the slab allocator (uts/common/os/kmem.c) since 5.4/1994.

4. A quick listing in /usr/lib/mdb/proc/ will display the other modules valid in the process target: beyond libumem and libuutil, there’s support for the linker, libc, name-value pairs, system event, and the two main smf(5) daemons.

5. As an example, here’s the output of “::vertex on my current system, for those services related to my VNC server (and the service itself):

::vertex ! grep vnc 0x85d3380 212 I 1 svc:/application/vncserver:sch 0x85d3320 213 s - svc:/application/vncserver 0x85d3200 214 R - svc:/application/vncserver:sch>milestone 0x85d3260 215 R - svc:/application/vncserver:sch>autofs 0x85d32c0 216 R - svc:/application/vncserver:sch>nis

[ T: OpenSolaris Solaris mdb ]

Grocery changes in Redwood City

stephen@blueslugs.com (Stephen Hahn) — Wed, 08 Jun 2005 07:05:33 +0000

When we’re not looking at the fast cars, we notice other changes on our drive between Redwood City and Menlo Park. Based on coverage in the San Mateo Daily Journal, it looks like one of the independent groceries in Redwood City closed in April—confirmed by the empty parking lot just off the intersection of Fifth and El Camino Real.

The closing was at least partially due to the arrival of Whole Foods a mile and a half north, although it appears that the location itself was an issue, as that intersection is an informal spot for day workers to wait for possible job pickups. But this kind of change is the flip side of new arrivals to Redwood City—some independent businesses will need to make difficult adjustments.

Shuffling services

stephen@blueslugs.com (Stephen Hahn) — Mon, 06 Jun 2005 03:43:55 +0000

I’ve been sketching out how to gradually modernize our network infrastructure: replacing a 10Mbps combination router/hub with a 1Gb switch and a 10/100Mb VPN, or a similar combination. (I’ll leave the wireless on 802.11b until Dina upgrades her laptop.) To make this simple, I’m reducing the services on the current router, and moving them to the main system. Today, I moved DHCP to our main Solaris server, using dhcpmgr(1M), a graphical utility for managing Solaris’s bundled DHCP server.

Very easy to follow the configuration wizard’s steps, and the service is better behaved than the aging firmware on our six (seven?) year old router. (Plus, it’s managed by smf(5):

$ svcs '*dhcp*'
STATE STIME FMRI
online 12:54:07 svc:/network/dhcp-server:default

which is a pleasant bonus.)

Next up is drawing some pictures of new cable runs in the back office, and deciding on appropriate numbers of ports.

[T: Solaris smf]

Bespoke services: application/vncserver

stephen@blueslugs.com (Stephen Hahn) — Wed, 01 Jun 2005 20:00:57 +0000

In honour of the “Mugs for Manifests” contest, I thought I would spin out another custom service description I wrote some months ago.

My setup for working from home—key during the last six months of Solaris 10—is to tunnel into Sun’s network via one implementation or another of a virtual private network (VPN). In all cases, the VPN solution runs on Solaris. Although the VPN lets your system participate more or less like a regular host, I find it’s easier to use VNC to remotely present an X11 display from my main workstation, muskoka. But, of course, machine running pre-production bits can fail or be rebooted or be reinstalled regularly, so I wanted the VNC server on my system to always be up: I wanted a VNC service.

What’s distinct about running the VNC server is that it should run as me, with my environment, and not as root with init(1M)’s. svc.startd(1M), while it can run methods according to smf_method(5), doesn’t populate the environment fully in the sense of login(1). So we will need to extract some data from the name service, which is cumbersome to perform in a shell script. We’ll write our method in Perl, which implies

Tip 1: Methods need not be shell scripts.

In fact, the start method and the stop method can be totally separate commands: you could write one in Python, and one can be an executable Java .jar archive, or some even more bizarre combination.

The other trick is that, if VNC fails for some reason, I want to be aggressive about cleaning up its various leftover temporary files. For this purpose, I run the stop method with a different credential—the default of root—than the start method, which is done in our brief manifest by locating the <method_context> element on only the start method.

Tip 2: Methods need not be run with identical method contexts. Credentials, privileges, and the like may all differ from method to method.

Our manifest then looks like:

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
<service name='application/vncserver' type='service' version='0'>
<single_instance/>
<instance name='sch' enabled='true'>
<dependency name='milestone' grouping='require_all' restart_on='none' type='service'>
<service_fmri value='svc:/milestone/multi-user:default'/>
</dependency>
<dependency name='autofs' grouping='require_all' restart_on='none' type='service'>
<service_fmri value='svc:/system/filesystem/autofs:default'/>
</dependency>
<dependency name='nis' grouping='require_all' restart_on='none' type='service'>
<service_fmri value='svc:/network/nis/client:default'/>
</dependency>
<exec_method name='stop' type='method' exec='/home/sch/bin/vncserver_method stop' timeout_seconds='60'/>
<exec_method name='start' type='method' exec='/home/sch/bin/vncserver_method start' timeout_seconds='300'>
<method_context>
<method_credential user='sch' group='staff' />
</method_context>
</exec_method>
</instance>
</service>
</service_bundle>

The dependencies above are needed if you use NFS for home directories and NIS for name services; they could be reduced for less networked setups.

And, for the method, we have a short Perl program. The complete list of environment variables in login(1) would include LOGNAME, PATH, MAIL, and TZ (timezone), and exclude my silly setting of LANG, but most of these will be set up by the shell that the VNC startup script (its analgue to .xinitrc. The various print calls are just to let the service log show a little activity, and could be removed.



!/usr/perl5/bin/perl


require 5.8.3;
use strict;
use warnings;
use locale;
my ($name, $passwd, $uid, $gid, $quota, $comment, $gcos, $dir, $shell,
$expire) = getpwuid "$<";
$ENV{USER} = $name;
$ENV{HOME} = $dir;
$ENV{SHELL} = $shell;
$ENV{LANG} = "en_CA"; # Just to create havoc (i.e. expose bugs).
#


The stop method is run as root so that it can cleanup.


#
if (defined($ARGV[0]) && $ARGV[0] eq "stop") {


ksh and sh specific


print "stop method\n";
system("$ENV{SHELL}", "-c", "/opt/csw/bin/vncserver -kill :1");
if (-S "/tmp/.X11-unix/X1") {
unlink("/tmp/.X11-unix/X1");
unlink("/tmp/.X1-lock");
}
exit 0;
}
#


The start method is run with the user's identity.


#
print "start method\n";
if (-f "/tmp/.X1-lock") {
unlink("/tmp/.X1-lock");
}
if (-S "/tmp/.X11-unix/X1") {
system("logger -p 1 application/vncserver requires " .
"/tmp/.X11-unix/X1 be removed");
exit 0;
}


ksh and sh specific


{ exec "$ENV{SHELL}", "-c",
"/opt/csw/bin/vncserver -pn -geometry 1600x1200 -depth 24 :1" };
system("logger -p 1 application/vncserver can't exec /opt/csw/bin/vncserver");
exit 1;

And now we have always-on VNC service for the regular telecommuter:

$ svcs -p vncserver
STATE STIME FMRI
online 13:01:01 svc:/application/vncserver:sch
13:01:00 100577 Xvnc
13:01:17 100625 xwrits
13:01:17 100626 ctrun
13:01:17 100632 xautolock
13:11:18 102348 xlock
$ uptime
12:00pm up 23 hr(s), 4 users, load average: 0.04, 0.07, 0.07

Exercises

Remove the hard coded display numbering (“:1”, “X1”, etc.).
Make the resolution, display depth, RGB encoding, and other standard options into properties.

[ T: Solaris smf ]

smf(5) not-quite-free stuff

stephen@blueslugs.com (Stephen Hahn) — Sat, 28 May 2005 00:02:08 +0000

I’m in the middle of some longish, and one rather preachy, blog posts. These will need editing, so to pep things up…

Like one of these?

We had a bunch of custom mugs made up, to commemorate the completion of smf(5)’s integration into Solaris 10. If you’ve been at a customer or community presentation on S10 or smf(5), you might have received one: for asking a good question, for answering one from me, or for physical attendance. But these mugs—fine, solid, large capacity, high quality mugs for coffee, tea, or even pens—are heavy: too heavy for us to lug a box to all the conferences we might attend.

So instead we’re going to run a little contest.

Liane summarized our understanding of other service conversions circulating a few months ago. I’d like to get another batch done, and there’s no incentive like a ceramic container incentive, so I’m going to suggest a few categories:

Historical: Convert one (or more) of the unconverted services in /etc/rc*.d in Solaris 10.
Free/Open: Convert a F/OSS daemon to be an smf(5) service.
Commercial: Convert a commercial software package to be one or more smf(5) services.
Artistic/Offbeat: Convert something unexpected into a particularly elegant service.

The conditions are pretty simple: there are 36 mugs in the box, so the first round can have 36 winners. One mug for each converted service; the winning entry for a specific service will be judged by completeness (dependencies in particular), correctness (methods), utility (will anyone else use this?), and date received. I’ll give some no-prize honorable mentions in each category as well. This round will be quick: entries must be received by June 15th.

An entry should disclose:

Your name,
preferred email,
blog URL (optional),
mailing address,
description of the software (plus details if obscure) and
the service manifest and method(s) (if any), or
an accessible URL to same. Send it to

sch AT sun.com. I’ll assemble a few smf(5) keeners to help me evaluate the submissions.

Services on the list Liane gave are not eligible, unless you think your conversion is substantially better by the criteria above.

If your conversion wins, I’ll send you your mug via an amazing cooperative, potentially international, mechanism composed of government-granted-monopoly package delivery agencies. Winners, and their entries (or pointers) will be posted here.

[ T: Solaris smf ]

Busy few weeks

stephen@blueslugs.com (Stephen Hahn) — Fri, 27 May 2005 18:05:46 +0000

Nathaniel and Benjamin have been trading an ear infection back and forth for the past few weeks, and Nathaniel also looks like he caught the stomach bug that had been prowling through his class at daycare. (But Ben never developed chicken pox, so that’s one positive.)

I finally decided that the colds and allergies were systematic, rather than uncorrelated, and went under a part of the house to cover an open-to-ground crawlspace with 6 mil polyethelene sheeting. Air quality in the house is already better—I haven’t taken Claritin in a couple of days, and Dina mentioned her nose has been clear in the mornings. It took a little net research to figure out how to do this and, since I can write the Home Depot-based recipe, I’ll go under again and take some pictures for a longer “how to” posting.

I also did the second “clean out the hot tub when the baby turns one” chore. Icky: dead slugs (and live ones). It’s actually turning out to be a useful water experience for the boys, although Ben doesn’t like the bubbles and waves from the “air” mode. There’s a leak somewhere near the top—likely by one of the filter inlets—so I’m going to try a dynamic leak sealing system, but that will be another posting as well.

Homes are fun.

Pop-up papercasting

stephen@blueslugs.com (Stephen Hahn) — Wed, 11 May 2005 08:36:30 +0000

I think papercasting is amusing. A lot of my work at Sun is critical in nature, meaning I feel much of my added value comes during the review phase of various documents (whether that document is text or code). A week ago, I spent a morning working on paper while my car was being serviced; as a typical engineering interval, I thought my working notes might be of interest to someone.

Reading them over, I realized that the notes rely too much on my context, which papercasting wouldn’t fix. Hence, a little browser scripting and pop-up papercasting becomes a way to fill in the background on what some of the tasks mean.

[ T: papercast ]

… Or, perhaps not

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 May 2005 16:15:29 +0000

Well, the rash hasn’t progressed, and none of the constituent bumps have become vesicles, so Ben can’t have chicken pox. I suppose it’s just a viral rash—it already looks like it might be fully subsided by tomorrow.

So, another quiet day at home reading (and watching Thomas) together.

smf(5) in Hungarian

stephen@blueslugs.com (Stephen Hahn) — Mon, 09 May 2005 18:24:28 +0000

Introducing people to smf(5) can’t be limited to English-speaking audiences—everybody needs to know how the system works. On blogs.sun.com‘s lesser-known sibling site, mediacast.sun.com, Cserép János has posted his Hungarian introduction to smf(5).

In case you couldn’t guess, szolgáltatások means service.

Varicella

stephen@blueslugs.com (Stephen Hahn) — Mon, 09 May 2005 17:31:39 +0000

Benjamin has chicken pox. The pox just started to show up this morning, so I expect we’ll be at home for most of the week.

I don’t mind the red light by the Ferrari Maserati dealership

stephen@blueslugs.com (Stephen Hahn) — Thu, 05 May 2005 04:18:33 +0000

I’ve noticed that my driving habits have changed since I found out that the ugly building on El Camino in Redwood City was going to be a Ferrari Maserati dealership. Even during the construction phase, I found myself driving in the slow lane and hoping for a long red light. It turns out that the beautiful new dealership is the largest in the world with a waitlist for new cars. Once you own one, you can participate in Ferrari’s own racing series. The next event is May 13-15, 2005 at the Las Vegas Motor Speedway, Nevada. There’s also the Rally of Wine Country in August. I’d have to say that with our new Starbucks, Whole Foods, and now Ferrari Maserati dealership, Redwood City is becoming more interesting.

Banging on multiple heads

stephen@blueslugs.com (Stephen Hahn) — Wed, 27 Apr 2005 23:06:25 +0000

I spent a bit of time each of the past few weeks trying out different graphics cards to drive two displays—a multihead configuration. Presently, I’m using an older ATI Radeon 7000-based card to run two displays at 1600 × 1200 each. The radeon driver included with the X.org X server can knit these together into a single display with an effective resolution of 3200 × 1200.

Other folks are using nVidia drivers to get similar configurations.

Once I had this setup running, I started to see familiar applications fail with a pleasant message:

$ gvim
The program 'gvim' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadWindow (invalid Window parameter)'.
(Details: serial 13 error_code 3 request_code 128 minor_code 2)
(Note to programmers: normally, X errors are reported asynchronously;
that is, you will receive the error a while after causing it.
To debug your program, run it with the --sync command line
option to change this behavior. You can then get a meaningful
backtrace from your debugger if you break on the gdk_x_error() function.)

Dave Powell helped me to use xscope to watch the X11 protocol requests. This, and a little code browsing allows us a diagnosis.

It turns out that Xsun and Xorg use different implementations of the Xinerama extension (but with similar names). As far as I can tell, the standard behaviour changed after Sun developed support for a draft proposal. Now, a few years later, the applications know how to deal with both versions. With Xorg though, you now have a Sun system which doesn’t speak Sun’s Xinerama variant–hence our error message. Alan knows how to fix this for real but, after looking at libgdk startup, it’s pretty easy to work around this with a preloaded shared object.

All we do is pretend that our display can’t do Sun’s Xinerama. That means we need a function like

$ cat > xin_shim.c
int
XineramaGetState()
{
return (0);
}
^D

We then compile it into a shared object

$ cc -o xin_shim.so -G -Kpic xin_shim.c

or we could freely compile it into a shared object

$ which gcc
/usr/sfw/bin/gcc
$ gcc -o xin_shim.so -G -fpic xin_shim.c

Then, to use your shim, you use LD_PRELOAD (with an absolute path)

$ LD_PRELOAD=`pwd`/xin_shim.so gvim
[happy editing...]

Because xin_shim.so isn’t in /usr/lib/secure, you may see messages from setuid processes as the linker refuses to preload your potentially unsafe object. The message looks something like

ld.so.1: /usr/lib/utmp_update: warning:
/home/sch/src/preloads/xin_shim.so: open failed: illegal insecure pathname

smf(5): Chairman's Award

stephen@blueslugs.com (Stephen Hahn) — Wed, 27 Apr 2005 07:21:01 +0000

The smf(5) project team was honoured with the Chairman’s Award for Innovation today, along with 14 other teams or individuals at Sun. I had a camera with me, and snapped a few pictures but, since the award ceremony was televised, don’t have any from that part of the day. Afterwards, we gathered in the Menlo Park campus’s amphitheater [TerraServer] for some photos. I managed to capture

the smf(5) team;

our sister team in Predictive Self-Healing, the Fault Management Architecture team; and

the CDDL team (but after they started to move towards one end of the amphitheatre).

We had a short lunch afterwards, but longer conversations with the other award winners, the nominating committee members and other executives:

I borrowed our newer Canon PowerShot SD300 for the day, because it’s so much smaller than the S30. I had a little trouble with gphoto2, but I put the SD card directly in the slot on my Acer Ferrari running S10—which mounted directly and seamlessly.

smf(5) was the product of many people’s contributions. There are team members who could only participate for a limited period or who are no longer at Sun, and many forms of contribution from all parts of the company—the end result couldn’t have been reached without all the shared and distributed lifting. My thanks.

A small and quiet thanks to Jonathan for gesturing me into the studio camera’s field of view.

Neck kerchiefs (or neckerchiefs)

stephen@blueslugs.com (Stephen Hahn) — Tue, 26 Apr 2005 09:06:07 +0000

So, while I wore the final symmetric/named knot at the OpenSolaris CAB dinner, I haven’t stopped wearing ties to work. I’m tying a variety of knots for four-in-hand ties—Nickys and half-Windsors for the most part—plus the occasional bow-tie. As far as I can determine, it hasn’t affected my work.

This is, of course, my wrap-up post on neckties.

Tweaked back, with an inverted portent?

stephen@blueslugs.com (Stephen Hahn) — Tue, 26 Apr 2005 08:02:54 +0000

I strained my back last Thursday which showed that, while my basketball reflexes return, they do so to an older body. On the drive home, I watched a grey squirrel fall out of a tree—perhaps a twenty foot drop—and kept the car still as it limped to the grassy sward nearby. My weekend was spent chasing children of suddenly superhuman speed, and realizing that I, too, am an aging squirrel.

Loss of T-Mobile service?

stephen@blueslugs.com (Stephen Hahn) — Tue, 26 Apr 2005 04:11:52 +0000

As I drove to work in Menlo Park this morning, I was unable to operate my cell phone. I tried several numbers from my phone list and typed others. I turned the phone on and off and even tried to call T-Mobile. Each attempt was met with a message about the number being unregistered. It was a very strange feeling to be unable to talk on my drive and my car seemed particularly quiet. I have not been able to find information about the disruption on the T-Mobile website. Does anyone know anything about it?

Adrian explores libexacct

stephen@blueslugs.com (Stephen Hahn) — Fri, 22 Apr 2005 23:30:32 +0000

Adrian Cockroft has been exploring the extended accounting streams available in Solaris. By blogging about his exploration, Adrian is sharing his cognitive process—highly valuable to, say, the designer looking to improve a software subsystem for developers. I was amused by the following frustration-induced comment

The design is so abstract that it seems that reading meaningful data out of the log file is some obscure side effect of the code. You can read the data, but there is no guarantee that any specific item of data will be present. The accounting system has various options to send more or less data to the file, so it needs to be flexible, but the important thing is the meaning of the data being logged. I care about the semantic and informational content of the data source. What I get from exacct is “there are some tagged typed objects in this file”. I can’t consume the data without making assumptions about it, and the API doesn’t embody those assumptions. because some of these drawbacks were design goals: exacct is a general purpose, variable-length, grouped binary record format affording traversal in either direction through the file. I think Adrian’s criticism is valid (except that it was always meant for reading and writing data) if I read it as you haven’t written the accounting data handling layer yet.

While I wait for some time to think about how we might do that, I’ve filed

6260320 *wracct* should have some form of "for all objects" support

so we can fix at least one annoyance.

High Maintenance Mom – Trying to create a site with Scoop

stephen@blueslugs.com (Stephen Hahn) — Thu, 21 Apr 2005 22:18:48 +0000

I’ve been working on a site for mom’s (and parents or future parents) to share information. High Maintenance Mom is now in beta and I welcome all feedback. We, I needed Stephen’s help, created High Maintenance Mom using Scoop from kuro5hin. It was more difficult to create than I anticipated (mod_perl, etc.). Stephen can write more on the difficulties of getting it running. Since then, I’ve been trying to figure out how everything works. We chose Scoop for the polls and potential for people to vote on stories. Right now it’s set up so anyone who registers can post stories. The images I’ve started to create are not visible with IE and there are lots of other issues to fix. More once I’ve figured something else out. Please let me know what you think.

Thanks! dina

Cameras on Solaris

stephen@blueslugs.com (Stephen Hahn) — Wed, 06 Apr 2005 19:12:18 +0000

Dan noted a while ago the libusb/libgphoto2 stack works well on Solaris 10 (and newer Express releases, of course). I’ve been using the gphoto2 CLI to manipulate my beaten-up Canon over USB, and then gthumb as my image browser/cataloguer. There are a few bugs here and there and some obvious enhancements needed, but the trajectory in this space is impessive. Since I’ve now got a “for-work-only-toy-camera”, I’ll keep experimenting with related applications on Solaris, and make any observations here.

You can get all of the mentioned software prebuilt for Solaris at Blastwave. A working libusb is available in /usr/sfw/lib.

Tie knot: Knot 7 (Half-Windsor) [reprise].

Dinner with the board

stephen@blueslugs.com (Stephen Hahn) — Tue, 05 Apr 2005 22:03:01 +0000

I was lucky enough to be invited to the OpenSolaris community advisory board dinner last night in San Francisco. The very gracious Al Hopper and the versatile Casper Dik represented the newly formed board at our table, which enjoyed a witty and wide-ranging conversation (at least as I remember it). I also managed to meet physically Rich Teer, who I had only exchanged email with previously.

Since I’ve been handed down the old family digicam—a Canon PowerShot S30 with a broken LCD—and am now carrying it around for snaps, I took a few shots of the event:

In honour of the occasion, I wore the largest and last symmetric tie knot in Fink and Mao, and a new spread collar shirt that could contain it.

Tie knot: Bow.

For the record, 10

stephen@blueslugs.com (Stephen Hahn) — Mon, 04 Apr 2005 17:00:00 +0000

Tie knot: Knot 78 (Balthus).

For the record, 9

stephen@blueslugs.com (Stephen Hahn) — Thu, 31 Mar 2005 17:00:00 +0000

Tie knot: Bow.

Ballistics between sessions

stephen@blueslugs.com (Stephen Hahn) — Thu, 31 Mar 2005 07:20:46 +0000

Dropped in to a series of technical sessions being held in the Santa Clara Auditorium today. There was a longish lunch period and Adam, Eric, and Andrei passed the time launching water bottle lids between ever higher goal posts. Here’s a cameraphone shot from an early phase

although the posts ended up a little over a foot high, once an additional steel water flask was repurposed from a nearby table.

Tie knot: Bow (with a bow tie).

Bespoke services: application/catman

stephen@blueslugs.com (Stephen Hahn) — Tue, 29 Mar 2005 21:31:58 +0000

For various reasons—some reasonable, some suspect—Solaris doesn’t ship with a compiled set of windex databases for its manual pages. The unfortunate result is that helpful commands like apropos(1) or man -k are unhelpful:

$ apropos sort
/usr/man/windex: No such file or directory

smf(5) provides one way to address this shortcoming, via a transient service to be run during startup. Our service description would be roughly equivalent to the following:

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type='manifest' name='sch:catman'>
<service
name='application/catman'
type='service'
version='1'>
<create_default_instance enabled='false' />
<single_instance />
<!--
By default, application/catman will run in the background
during boot. If you want to run it periodically, execute</p>
/usr/sbin/svcadm restart catman
If you wish to augment the default MANPATH, use the setenv
subcommand to svccfg(1M). For instance, to add the Java
manual pages to the build:
/usr/sbin/svccfg -s application/catman
> setenv MANPATH /usr/share/man:/usr/java/man
> exit
/usr/sbin/svcadm refresh catman
If MANPATH is not defined, the default manual path is
/usr/share/man, as per catman(1M).
-->
<dependency
name='local-filesystems'
type='service'
grouping='require_all'
restart_on='none'>
<service_fmri value='svc:/system/filesystem/local' />
</dependency>
<dependency
name='remote-filesystems'
type='service'
grouping='optional_all'
restart_on='none'>
<service_fmri value='svc:/network/nfs/client' />
<service_fmri value='svc:/system/filesystem/autofs' />
</dependency>
<exec_method
type='method'
name='start'
exec='/usr/bin/catman -w'
timeout_seconds='0' />
<exec_method
type='method'
name='stop'
exec=':true'
timeout_seconds='0' />
<property_group name='startd' type='framework'>
<propval name='duration' type='astring' value='transient' />
</property_group>
<stability value='Unstable' />
<template>
<common_name>
<loctext xml:lang='C'>
manual page index generation
</loctext>
</common_name>
<documentation>
<manpage
title='catman'
section='1M'
manpath='/usr/share/man' />
</documentation>
</template>
</service>
</service_bundle>

Following my own instructions in the comment block, I defined a value for MANPATH and refreshed the service. My setting can be double-checked with svcprop(1) like so:

$ svcprop -p start application/catman
start/exec astring /usr/bin/catman\ -w
start/timeout_seconds count 0
start/type astring method
start/environment astring MANPATH=/usr/share/man:/usr/openwin/man:/usr/sfw/man:/usr/dt/man:/usr/perl5/man:/usr/java/man:/usr/apache/man:/usr/X11/man:/opt/sfw/man:/opt/csw/man

Issuing “svcadm enable catman” will cause the service to be executed immediately, and upon each subsequent boot. Our earlier query becomes fecund:

$ apropos sort
FcFontSort FcFontSort (3fontconfig) - Return list of matching fonts
aclsort aclsort (3sec) - sort an ACL
alphasort scandir (3c) - scan a directory
alphasort scandir (3ucb) - scan a directory
bsearch bsearch (3c) - binary search a sorted table
bunzip2 bzip2 (1) - a block-sorting file compressor and associated utilities
bzcat bzip2 (1) - a block-sorting file compressor and associated utilities
bzip2 bzip2 (1) - a block-sorting file compressor and associated utilities
bzip2recover bzip2 (1) - a block-sorting file compressor and associated utilities
disksort disksort (9f) - single direction elevator seek sort for buffers
ldap_sort ldap_sort (3ldap) - LDAP entry sorting functions
ldap_sort_entries ldap_sort (3ldap) - LDAP entry sorting functions
ldap_sort_strcasecmp ldap_sort (3ldap) - LDAP entry sorting functions
ldap_sort_values ldap_sort (3ldap) - LDAP entry sorting functions
libbz2 libbz2 (3) - library for block-sorting data compression
look look (1) - find words in the system dictionary or lines in a sorted list
qsort qsort (3c) - quick sort
sort sort (1) - sort, merge, or sequence check text files
sortbib sortbib (1) - sort a bibliographic database
tsort tsort (1) - topological sort
...

Exercises

Add a configuration property that makes the service also rebuild the nroffed versions of the manual pages, if set to true.
Make the service regenerate only in the case that components in the path have changed.

p class=“note”> Tie knot: Knot 54 (Hanover).

From Orthography To Pedagogy

stephen@blueslugs.com (Stephen Hahn) — Tue, 29 Mar 2005 01:07:42 +0000

The book from my father’s Festschrift, From Orthography to Pedagogy, Essays in honor of Richard L. Venezky has been published and the essays are wonderful. Prepaid copies can be ordered from the above link for $39.95.

For the record, 8

stephen@blueslugs.com (Stephen Hahn) — Mon, 28 Mar 2005 17:00:00 +0000

Tie knot: No time, no knot, alas.

For the record, 7

stephen@blueslugs.com (Stephen Hahn) — Fri, 25 Mar 2005 17:00:00 +0000

Tie knot: Knot 5 (Pratt or Shelby).

For the record, 6

stephen@blueslugs.com (Stephen Hahn) — Thu, 24 Mar 2005 17:00:00 +0000

Tie knot: Knot 25 (Christiansen).

For the record, 5

stephen@blueslugs.com (Stephen Hahn) — Wed, 23 Mar 2005 17:00:00 +0000

Tie knot: Knot 6 (Prince Albert variation).

Rained out

stephen@blueslugs.com (Stephen Hahn) — Wed, 23 Mar 2005 06:49:00 +0000

A rainy stretch here in Northern California is keeping me off the court while my body repairs itself. But I’ll need to go practise in the wet if I’m ever to get my jump shot back…

Tie knot: Knot 44 (Grantchester).

For the record, 4

stephen@blueslugs.com (Stephen Hahn) — Tue, 22 Mar 2005 06:48:22 +0000

Tie knot: Knot 31 (Windsor).

For the record, 3

stephen@blueslugs.com (Stephen Hahn) — Sat, 19 Mar 2005 06:47:02 +0000

Tie knot: Knot 18 (Plattsburgh).

On the court again

stephen@blueslugs.com (Stephen Hahn) — Fri, 18 Mar 2005 07:18:12 +0000

I’m a little sore and a bit bruised: I’ve started playing pick-up basketball on the outdoor court here at the Menlo Park campus. I pretty much stopped when we started (a) work on smf(5), and (b) parenting—three years. (Finishing my dissertation also required a break, but not nearly as long as this one.)

I feel slower, and my skills are very rusty. Regular practice will fix at least one of those.

Tie knot: Knot 23 (Cavendish).

Yellowstone supervolcano and locating coordinates for image maps

stephen@blueslugs.com (Stephen Hahn) — Thu, 17 Mar 2005 21:22:32 +0000

I’ve been working on the Yellowstone Volcano Observatory (YVO) website for the U.S. Geological Survey in anticipation of the release of the new Supervolcano movie and I needed to change the coordinates for the logos in our header. As a Mac BBEdit user, I wanted a simple program that gave me the functionality of a tool like Dreamweaver without the cost. What I found was YokMap. An easy-to-use program that ran on my Mac. Their “most stupid solitaire game ever” was an added bonus.

Supervolcano, a docudrama about a mega-eruption at Yellowstone, was shown in the UK this past weekend on BBC and will be shown here in April on the Discovery Channel.

String theory split

stephen@blueslugs.com (Stephen Hahn) — Thu, 17 Mar 2005 06:43:52 +0000

A few people mentioned an article in the San Francisco Chronicle describing the current support and skepticism around string theory in the physics community to me. It’s worth a read. I would have liked to see some comments about aspects of theory—signs of the Higgs particle or supersymmetric structure—that should become testable at CERN‘s Large Hadron Collider in 2007. It’s worth noting that, since some of its opponents are quoted in the article, we would already be exploring this energy regime at the Superconducting Super-Collider, if it had been ready by its revised completion date of 2003. (You didn’t think this was a new argument, did you?)

Tie knot: Knot 12 (St Andrew).

Knotty engineer

stephen@blueslugs.com (Stephen Hahn) — Tue, 15 Mar 2005 19:50:05 +0000

As a California-residing, code-writing Sun engineer, I don’t have a lot of sartorial restrictions beyond the standard requirements for coverage. It’s been a bit of a shock, therefore, to co-workers and long time customers to catch me wearing a necktie. The most common question is “Are you interviewing?” Which, given that whole writing-code-in-California bit, isn’t very appropriate—this isn’t standard engineering costume out here.

I think the easiest way to understand it is as an eccentricity: I’m working my way through Fink and Mao‘s The 85 ways to tie a tie. They identify thirteen suitably aesthetic knots, nine of which are in some sense novelties, not being the standard three or four. Each day, I’ve been working my way through another one of the aesthetic or historically interesting knots, by tying it on and wearing it through the work day. My head hasn’t popped off yet.

Tie knot: Knot 7 (Half-Windsor).

smf(5): manifest editing assistance

stephen@blueslugs.com (Stephen Hahn) — Tue, 15 Mar 2005 02:39:43 +0000

We had a productive wrap up meeting for Solaris 10 Platinum Beta last week, with lots of good feedback on smf(5). One point raised is that few people like to hand-edit XML—or, maybe, many people hate to—so tools for composing service manifests are needed. We’ll need to percolate on how best to improve or extend the current set of tools, but there are a few tricks out there already.

A bad manifest. Let’s take a well-formed and valid manifest file and add the nonsensical line

<french_fry>I,m a bad element.</french_fries>

to simulate a developer making a composition error during service development. How do we determine that our manifest is now broken?

svccfg(1M) validation. As I mentioned, the basic tools aren’t helpful. The logical svccfg(1M) subcommand to check a manifest for correctness is validate. Its output on our manifest is

$ svccfg validate /tmp/gdm2-login.xml
svccfg: couldn't parse document

which accurately tells us the manifest is broken but does not indicate how (at all).

xmllint(1). The XML parser implementation of svccfg(1M) is the GNOME libxml2, which includes a general validation tool in the form of xmllint(1). If we invoke this command with its --valid long option, we get

$ xmllint --valid /tmp/gdm2-login.xml
/tmp/gdm2-login.xml:26: parser error : Opening and ending tag mismatch: french_fry line 26 and french_fries
<french_fry>I,m a bad element.</french_fries>
^

which isn’t validating the document, but is telling us where and how it is not well-formed.

Graphically clear. An interesting option is to use the jEdit editor, with its XML plugin. With our document, the XML plugin will validate on save and highlight the incorrect line with red underlining:

Moreover, the error window shows both the non-well-formedness and the invalid <french_fry> element (which is absent from the non-fast food-oriented service bundle DTD).

So we see both the immediate and the deeper error, plus the plugin highlights matching tags and provides completion menus for tag selection. Civilization to most, I expect.

I happily use vim for development, but it’s important to note the value in jEdit just from using a different XML implementation. Using other tools to ease your composition of service descriptions (or profiles)? Let us know—and, rest assured, we’re working to make that svccfg(1M) output more useful.

Tie knot: Knot 6 (Victoria).

Uneven handedness sought

stephen@blueslugs.com (Stephen Hahn) — Fri, 11 Mar 2005 22:20:36 +0000

Sadly, I suspect this New York Times editorial regarding these judicial nominees’ outrageous accomplishments—numerous reversals and practising without a licence—is accurate. I can’t imagine equivalent failures in the other licensed professions not being career-ending… Not a good indicator for subscribers to the school of operational competence.

For the record, 2

stephen@blueslugs.com (Stephen Hahn) — Fri, 11 Mar 2005 17:00:00 +0000

Tie knot: Knot 3 (Kelvin).

For the record, 1

stephen@blueslugs.com (Stephen Hahn) — Thu, 10 Mar 2005 17:00:00 +0000

Tie knot: Knot 4 (Nicky).

Happy Valentine's Day

stephen@blueslugs.com (Stephen Hahn) — Tue, 15 Feb 2005 05:59:46 +0000

I was excited to find the ACME heart maker and made myself a heart. This is my first post using WordPress. I thought Moveable Type was easier for picture uploads (when I uploaded the heart, WordPress gave me the wrong location) but I don’t miss the spam.

smf(5) design admissions

stephen@blueslugs.com (Stephen Hahn) — Wed, 26 Jan 2005 22:08:01 +0000

In a Usenet thread on comp.unix.solaris and an earlier comment series in the “Comparing Linux to System VR4” on Slashdot last week, some valid smf(5) criticisms have been raised. I thought I would reply to them here, and acknowledge the impact on some practices.

The strongest recent critic has been Tim Hogard, and I hope he doesn’t mind if I collect his concerns to structure my response. Mr Hogard is a member of the class of exceptional administrators that have designed their systems’ behaviours (by removing unneeded software and writing precise configurations for what remains, among other good practices). I’ve always known that this group of experts would be most directly impacted by the changes smf(5) brings, and have tried to listen carefully to their concerns and modify the implementation to accommodate them. (Such people exist within Sun and in the Beta and Express communities.)

“On my system the binary file contains the last time when the service started. That means a file in /etc gets written with every boot.” There are actually two binary files that contain the set of information the facility acts on:

/etc/svc/repository.db, which is persistent and contains the service definitions, dependencies, and so on, and
/etc/svc/volatile/svc_nonpersist.db, which is non-persistent and contains service execution information, such as process IDs, contract IDs, and stae transition times.

(/etc/svc/volatile is a tmpfs filesystem mounted by the kernel prior to startup being turned over to the progeny of init(1M). Its contents do not persist across OS instantiations.)

I believe the concern here is that /etc/svc/repository.db change detection cannot be managed using fingerprints. This is in fact a bug:

6221934 *svc.configd* must respect repository filesystem metadata as the current implementation does an idempotent test transaction to verify writeablity (in addition to other integrity checks). In the meanwhile, one can indirectly use a hash- or checksum-based fingerprint using the output of the archive subcommand to svccfg(1M), which dumps the repository as an XML document. An example using cksum(1):

13 $ svccfg archive > /tmp/a

14 $ svcadm disable manifest-import 15 $ svccfg archive > /tmp/b 16 $ svcadm enable manifest-import 17 $ svccfg archive > /tmp/c 18 $ cksum /tmp/a /tmp/b /tmp/c 1649351107 205437 /tmp/a 2659093841 205438 /tmp/b 1649351107 205437 /tmp/c 19 $ # use diff to examine repository differences globally…

But we’ll get that bug fixed shortly.

** “I’ve found some very interesting things to do to the new system that can mess up a box in a way that the current tool set won’t even let you see what is wrong. It appears to be a script kiddies back door dream.”** I am less certain I understood this point. There are no hidden services in the repository, although you can certainly corrupt the repository deliberately. (Separate backup repositories are made automatically at stages of boot; a recovery utility is provided in /lib/svc/bin.) Because each process is a member of a process contract owned ultimately by svc.startd(1M), you can use svcs -p if you trust the repository and ptree -c if you don’t. (If you don’t trust the kernel any longer, then we’re onto intrusion containment and system reinstallation from trusted media.)

** “And yes I’m saying complexity is bad and we should stick with what works (and it isn’t DOS).”** Unfortunately, what worked once no longer works given today’s requirements. The Predictive Self-Healing initiative at Sun and equivalents elsewhere are responses to a change in availability requirements: both hardware and software components have the potential for failure, and the operating system needs to acknowledge this outcome and provide abstractions and capabilities to manage its implications. It’s my belief that smf(5) introduces a minimal amount of complexity in exchange for new and meaningful descriptive objects and, in fact, the simplification of some previously difficult operations.

This reply hasn&#8217;t explained why there&#8217;s a transactional database (which, as an implementation detail, is currently SQLite), or why hierarchical restart is a different problem than parallel startup, or why we converted the current system to use the facility rather than providing only the framework. I&#8217;m happy to expand on any of these, or the points above.

Fiddling with del.icio.us

stephen@blueslugs.com (Stephen Hahn) — Wed, 26 Jan 2005 06:45:51 +0000

I’ve started moving bookmarks onto del.icio.us, the slick social bookmarking service. Enough real work beckons that I’m resisting whipping up some Perl to glue Netscape::Bookmarks and Net::Delicious together, so for now I’m manually posting important or new bookmarks.

If you’re a del.icio.us user, and want to help build a shared link set for smf(5), please consider using “solaris smf” in your tags. The current list is incomplete.

Recovery from another hiatus; an smf(5) one-liner

stephen@blueslugs.com (Stephen Hahn) — Tue, 25 Jan 2005 07:20:34 +0000

I was out with sick children last week and have been engaged in a form of mental wrestling learning the Dvorak keyboard for the past two—both impediments to concentration, and thus to smooth blogging. Everybody is on the mend; evening wakeups are on the decline. I no longer want to hurl my keyboard across the office; my brain is remapping favourite commands and key sequences. (slrn and ls, which use the fourth and fifth fingers of the right hand (or only the fifth), feel very awkward still.)

There were some interesting smf(5)-related asides in some Slashdot discussions last week. One requires a separate, detailed reply. The other—a request for a single command that hardens your Solaris system—can be answered briefly. There are two answers: one a flip one-liner, one responsible.

The glib smf(5) one-liner answer is:

# svccfg apply /var/svc/profile/generic_limited_net.xml

which applies the “limited networking” profile to your system. This profile deactivates about three dozen services that are active in the traditional install case, including

all passwords-in-the-clear login services. There are still RPC and likely other services active on the system that serious system auditors will want to examine. (The limited networking profile was written by members of the Solaris Security Technology group and will be present in the initial release of Solaris 10. And we’re now working to make that initial install initially and increasingly secure.)

The more responsible answer is to point out that system hardening requires both minimization and reconfiguration (beyond service deactivation) and that there is a productized distillation of security practices for Solaris in the form of the Solaris Security Toolkit. Although its use might exceed a one line constraint, the toolkit makes fungible years of Solaris hardening experience.

(I’ll refrain from describing how the now-integrated IPFilter or tcpwrapper support can reduce the exposure of some of the remaining services. But I will point out that the combination of the least privileges feature and the smf(5) service description’s ability to define the appropriate privileges for a service lead to an rpcbind(1M) that cannot successfully call exec(2) (among other things):

# ppriv `pgrep -z global rpcbind`
100220: /usr/sbin/rpcbind
flags = PRIV_AWARE
E: basic,!file_link_any,net_privaddr,!proc_exec,!proc_info,!proc_session,sys_nfs
I: basic,!file_link_any,!proc_exec,!proc_fork,!proc_info,!proc_session
P: basic,!file_link_any,net_privaddr,!proc_exec,!proc_info,!proc_session,sys_nfs
L: basic,!file_link_any,!proc_exec,!proc_fork,!proc_info,!proc_session

As I noted: increasingly secure.)

Ucpoy Ekrpat lroy

stephen@blueslugs.com (Stephen Hahn) — Thu, 13 Jan 2005 20:31:40 +0000

Or First Dvorak post. After many late nights coding, my fingers ache somewhat. I decided to switch to the Dvorak keyboard layout, and also to a programmer’s keyboard. The typing is much slower going now. In a couple of weeks, I’ll review the keyboard and some of the software I used to switch.

Commercial data aggregations: New social features at Netflix

stephen@blueslugs.com (Stephen Hahn) — Sat, 08 Jan 2005 19:10:49 +0000

We’re regular, but not heavy Netflix users. For the most part, Dina and I have our own entente cordiale regarding our movie selections, meaning that we can each watch the other’s “dumb movies”. What is still difficult is the handling of whose movies will come by post next. (Netflix is a DVD rental subscription service where the DVDs are sent and returned by mail.)

Netflix has added two features; one of which they call “Profiles”, where multiple queues can be established, and each allocated some share of the account’s total number of loaned DVDs. We’ll still have a problem with splitting three discs two ways, but I hope this feature will address queue juggling, where we each reshuffle the next 10 titles based on our distinct preferences: “Why did we get recent romantic comedy? I was expecting three years’ ago’s spy movie!” (Warning: transferring movies from a shared queue to individual queues has to be done by hand. Ick.)

Since I have a few friends who are also Netflix users, I’m interested in the “Friends” feature, too, where I can see their ratings of movies, and not just the Netflix customer average rating. The extra information can help, but it’s the aspect of an asynchronous shared experience that I find interesting. I’ll issue the invitations soon.

Since these are social usage features, I suspect we’re seeing Netflix execute on its strategy for competing with Wal-Mart and Blockbuster: as a pure Internet firm, they must maintain focus on creating an data aggregation that involves their customers enriching the aggregation, as opposed to purchased content. Given financial resources, anybody can build a (500 000 DVD) library, and even an infrastructure to rent and bill that library’s contents; the innovation has to come on the experience side: pricing is one approach, but innovating to make the richest aggregation is the right network-based company strategy.

State of the domain

stephen@blueslugs.com (Stephen Hahn) — Sat, 08 Jan 2005 06:48:44 +0000

blueslugs.com turned five this week. A reasonable time to take stock. The domain’s first days were hosted over the DSL line in our apartment in Menlo Park on—wait for it—a PowerMac 7100 running MkLinux. Since then, it’s been hosted on an Ultra 10, a Netra T1, and finally on a Shuttle, running various versions of Solaris. The past year had a few events worth reviewing.

Overview. Last year, we shipped out a total of over 850 MB of data, through the thin little upstream pipe our DSL line offers. Log analysis translates to over 150 000 hits and over 10 000 visitors. Over the year, 24 blog posts were written, three by Dina. Fortnightly posting probably doesn’t mean much, but demonstrates that the domain hasn’t been abandoned. (Perhaps populated by blue sloths, blue snails, or…)

Content. Pages from Benjamin‘s alphabet book remain the most popular real content—and the most often requested by searches. We started a subsite for Nathaniel and a second blog to keep friends apprised of Dina’s father’s illness. The various syndication files are regularly requested, and the traffic from the blog aggregation sites and the search engines increased noticably for the last third of the year.

A few people are still downloading the dockapps I wrote a couple of years ago. Beyond some logfile software and a few tips, we didn’t provide very much new technical content. The distribution between technical content on my blogs.sun.com blog and on blueslugs.com is consistent; I don’t see a reason to merge the two. (For instance, reviews will remain on this site.)

I attempted some quasi-journalism/opinion about some of the local elections; as that writing is mostly related to my interest in Redwood City as a place to live, it will probably continue at some erratic rate.

System. The server hardware is unchanged, except that an APC UPS was installed to insulate the system from the local power sags we’ve been seeing. The small Shuttle system barely taxes the UPS so we should be able to survive even half hour outages easily. (By survive, I mean preserve uptime(1).) With Nathaniel’s arrival, the office (and so the server) was relocated to the cottage (where I spent many a late night the past six months).

blueslugs.com is now running a recent express release of Solaris 10 x86, complete with smf(5) service conversions for some of the software services. Much of the server software is precompiled and fetched from blastwave.org or one of its mirrors.

Future. I don’t think there’s any pressing need to modify the server’s hardware; the software we’ll upgrade occasionally with new Solaris releases. I’ve been taking a lot of pictures with the phone on my camera (or whatever)—so much so that Ben’s not satisfied with a toy cellphone that lacks a camera. Those images need to reach the web somehow. No resolutions to post more; perhaps a little one to post better.

Move to WordPress

stephen@blueslugs.com (Stephen Hahn) — Sat, 01 Jan 2005 23:29:20 +0000

With the beginning of 2005, I gave up on manually deleting blogspam: I’ve moved the content of the site to WordPress, with the AuthImage plugin installed to require some intelligence on the part of comment submission. I’ve not done all of the redirect/rewrite logic to get the archived posts links corrected; I think I’ll watch the logs and see if there are any pages that stand out. (The most popular pages, beyond the RSS feed, are both from Benjamin’s alphabet book, so the issue isn’t pressing.)

smf(5) on /.

stephen@blueslugs.com (Stephen Hahn) — Wed, 22 Dec 2004 05:01:10 +0000

smf(5) ended up in two stories on slashdot today. In “Torvalds on Opening Solaris”, elmegil observed

I’m rather amused to see Sun be the first to implement a replacement for the old init and have it done. I can’t say I know who thought it up first, but Solaris 10 SMF is the first working implementation I’m aware of that’s going to get any kind of wide deployment. I saw some linux-head saying this needed to be done a year or more ago, but I can’t even find their website in google now. And obviously if Solaris has it now, the implementation started a while back (probably more than a year)… I suspect elmegil is referring to Seth Nickell‘s System Services work, which ended up being discussed in an article on osnews.com in October 2003. But there is other work in the parallel startup area that’s been cited on slashdot, and elsewhere.

The second story is “A Diagnosis of Self-Healing Systems”, which is a discussion around Mike Shapiro’s recent overview in ACM Queue of the problems we’re working to solve in the Predictive Self-Healing effort. The comments range across a number of topics in deployment and architecture, but I was interested in the observations that self-healing in a general purpose system is a different proposition than in a limited purpose system. (I probably would also contrast open software systems and closed ones—perhaps a distinction of the past.)

Personal restarters and ctrun(1)

stephen@blueslugs.com (Stephen Hahn) — Thu, 02 Dec 2004 07:34:50 +0000

(I know I need to wrap up the cal(1) contest. I also need to finish about three smf(5) blog entries. I am also mostly keeping up with the forum/list/newsgroup traffic on smf(5). I also have a few more bugfixes to get into Solaris 10 first. But a small dispatch seems necessary.)

One of the neat things about GNOME is that it provides restarter operations for specified applications—your application references a bad address and dies, it gets restarted. I’m pretty attached to ion, however, and don’t really need all of the GNOME environment all the time. But I do want restart for a couple of applications in my session.

For instance, I wrote a little C program, osdclock, using libxosd that displays the date, the time, and those of my mail folders containing new mail. It looks like this:

Occasionally, and for reasons I lack the time to debug, new ion workspaces will obscure the on-screen display. Rather than write code into osdclock to call exec(2) (of itself) on a received signal, it’s easier to use the new commands associated with the contracts subsystem of Solaris 10, specifically the ctrun(1) command. By adding the invocation

/usr/bin/ctrun -r 0 -o noorphan -f signal,hwerr $HOME/bin/osdclock &

to my .xinitrc file, I get an osdclock that is restarted from any fatal external signal (or an uninterceptable hardware error of some kind), but is not restarted if a core file is generated (from a software error) or if the ctrun process is itself killed (like on session exit). So I avoid a home directory filled with core files. (You are using coreadm(1M) to have a meaningful core dump pattern, right?)

Since the cause of restart is ctrun‘s awareness of the process contract becoming empty, this same ctrun invocation could be used with an application that prefers to daemonize, like a personal web proxy. It could also be used to set up a restartable group of processes, like an application suite or widget collection. (ctrun has some other interesting options, and is a versatile lightweight restarter on its own—try it out.)

Having applications always available leads to predictable computing, but I think a more dramatic way to express it in this medium (but imagine appropriate stormy weather sound effects) would be

<mad_scientist>Restarters, restarters everywhere! <mad_scientist>

Strange absence

stephen@blueslugs.com (Stephen Hahn) — Tue, 09 Nov 2004 07:19:26 +0000

The Marina Shores Village site has been removed. I haven’t done enough math yet to even speculate, so I’ll merely point it out.

smf(5) GUI construction?

stephen@blueslugs.com (Stephen Hahn) — Mon, 08 Nov 2004 21:00:00 +0000

Darren has a neat example of using zenity(1) and smf(5) to make an “on-off” switch for his laptop’s network connection. The result:

where “Up” brings up network/physical and “Down” shuts it down—convenient for Solaris laptop users. zenity, by Glynn Foster, makes writing little panels like this so easy I suspect there might be a naescent cottage industry of smf–zenity panels. Let me know if you’ve written your own.

Update: Darren’s new zsvcadm example is even better!

Finding the real smf(5) conversation

stephen@blueslugs.com (Stephen Hahn) — Wed, 03 Nov 2004 22:12:41 +0000

I was a bit surprised by how little discussion of smf(5) I’d found (or how little emotional discussion, perhaps). comp.unix.solaris regulars figured things out moments after the images were posted to the Download Center, while the discussion of Predictive Self-Healing forum at BigAdmin focused on futures and further integration with management software. But it turns out that the largest discussion is at the solarisx86 Yahoo! Group.

I’ve signed up just now, and I’ll get a few more smf(5) types to join in as well—sorry for not realizing that this forum is so active, and so populated with early adopters.

(And before I neglect your favourite forum: please tell me what other forums we need to watch.)

Conversions, conversions

stephen@blueslugs.com (Stephen Hahn) — Wed, 03 Nov 2004 21:07:33 +0000

I mentioned that I’d converted a few of the F/OSS software packages that we use on our home system. Of course, the point of smf(5) isn’t to force you to rewrite all of your init.d scripts—it’s to force you to encourage your vendor to do so. (Wait, that’s not right, either.) I’ll come back to this aspect, but let’s talk a bit about service bundles.

The service bundle (or service manifest) is meant to accompany the package delivering the software. If a System V-style package uses the i.manifest class action script to identify its manifests, then these will be automatically imported into the repository upon package installation. (The class action script knows about zones, LiveUpgrade, and a couple of complicated circumstances where it’s not safe to import automatically.) In any case, upon the next reboot, unimported manifests in /var/svc/manifest will be automatically imported, and such services will be present in the graph.

Service manifests should deliver services disabled by default, so no one gets surprised by “hey, port N is now open—why’s that?” after a package installation. For these cases, all the administrator needs to do is

svcadm enable service_fmri

That is, the intentions for the service are owned by the system administrator. (There are exceptions for service complexes, where multiple services are involved, but manipulating interior services of a complex should be done by using temporary enable/disable requests. See svcadm(1M) and smf_enable_instance(3SCF).)

If you’re delivering software outside of the System V packaging, your installer can do the equivalent: use svccfg(1M) to import the service. If you support some form of upgrade, you can enable the service if you detected that the service was effectively enabled already. Remember to think about your uninstall action as well. For the default packages, we disallow deletion of a package that contains an enabled service instance. (A check on accidental removal.)

So those are the fundamentals—more detail is available in our service developer introduction. But the big question is what F/OSS software would you like to see example conversions for, first? (And tell us what commercial packages matter, too.) I’m going to start describing a few of the ones I mentioned a few days ago but, again, the goal isn’t to cause the creation of hundreds or thousands of custom descriptions of a standard software service, but to get the service described and to have that description accompany the service to deployment.

San Mateo County election results

stephen@blueslugs.com (Stephen Hahn) — Wed, 03 Nov 2004 07:12:32 +0000

The semi-official election results for San Mateo Country have been posted. For the two elections we were watching at blueslugs.com:

In the county, Ira Ruskin (D) [37,877] defeated Steve Poizner (R) [29,833]. This margin is substantially narrower than other state or national elections in San Mateo County. These results need to be added with the contributions from the portions of Santa Clara County that make up Assembly District 21; results from the California Secretary of State suggest that Ruskin will win the district.
Measure Q was defeated, with 10,214 votes for and 12,109 votes against.

smf(5) at home

stephen@blueslugs.com (Stephen Hahn) — Sun, 31 Oct 2004 22:12:22 +0000

Our electrical power at home has been sagging occassionally. It tripped yesterday for a couple of seconds: enough for everything without batteries or some capacitance to reset. (So clocks set yesterday, and again today.) Our main server went as well, and I noticed one or two services—running under very early Solaris 10—didn’t come back up properly. There’s a solution for that.

So I upgraded that that server to the 10/2004 Express release of Solaris—the first one with smf(5). I ended up writing seven service conversions very quickly (for SASL, IMAP, stunnel, and Blastwave’s Postfix, BIND9, Apache, and Squid), because otherwise Dina’s email doesn’t work. (Or my home blog.)

I need to reexamine the dependencies I elected on each of these service conversions, but if you’re interested in seeing them, let me know. With the exception of chasing a few configuration files down, it took only half an hour to get these services properly managed; that is, basically limited by typing speed.

smf(5) content flowing

stephen@blueslugs.com (Stephen Hahn) — Fri, 29 Oct 2004 19:22:26 +0000

We’ve posted an smf(5) quickstart guide and a preliminary service developer introduction (written by Liane), and are listening in the Predictive Self-Healing forum.

If you like, comment to let me know when the images hit the Download Center—they’ll most likely say “b69” somewhere in the filenames.

smf(5) seepage

stephen@blueslugs.com (Stephen Hahn) — Thu, 28 Oct 2004 22:59:48 +0000

Google (and the other search engines, I’m sure) are amusingly tenacious: if you search for “solaris” and the smf(5) command of your choice, you’ll start to find early content deep within the sun.com web tree as well as other Solaris and Unix sites. (These keywords will also deliver information about holiday resorts, although I can’t speak for their service or management or facility.)

Internet notices Q, and doesn't

stephen@blueslugs.com (Stephen Hahn) — Fri, 15 Oct 2004 21:01:00 +0000

There is also apparently questionable telemarketing behaviour associated with the pro-Q side. And jra links to another no-on-Q blog, which describes a reaction to the “real pictures” mailing similar to my own.

(I’m a little puzzled why none of these blogs are popping up on Google yet.)

More on our weighty measures

stephen@blueslugs.com (Stephen Hahn) — Fri, 15 Oct 2004 20:26:36 +0000

The other day, I noted some interesting politics in Redwood City/San Mateo County. The Economist isn’t covering this race, but is covering the U. S. Senate race in Oklahoma, and that the total spending there has been 3.8 M$. So the Poizner–Ruskin state assembly race in California and a Senate race in Oklahoma match up on absolute dollars. (Oklahoma has 1.9 M registered voters, so the per-registered voter amount for the Oklahoma race is about 2 $, or a little over 1/8 of the amount in our local race.)

On the Measure Q side, we received another two pro- mailings. One suggested looking at what was purported to be new facts on the measure, but is just a new domain registration pointing at the original site. I poked around, though, and found the following:

This is a classic graphical evasion: the coloring of the “marinas and open space” is green, suggesting grass. But note that marinas are actually on the water and, perhaps more importantly, the graphic does not separate out internal roads or parking. It would be difficult to find a common California definition of “open space” that includes roadways. If we look elsewhere on the site, we find

A very different assessment of open space, and one that raises questions about why this factor is being emphasized so strongly. I await the next mailing.

Exciting fall season

stephen@blueslugs.com (Stephen Hahn) — Thu, 14 Oct 2004 08:49:33 +0000

Although the major political news stories surround the U. S. Presidential Election, we’re having an exciting time in Redwood City, with the Poizner–Ruskin race for Assembly and the contest over Measure Q, which would approve the proposed Marina Shores Village along San Francisco Bay. Not being a citizen, I can’t vote. Moreover, I’m probably hopelessly idealistic about various behaviours, so I’ll just note some interesting aspects. (And this article covers no aspect of the fall television season—I’ll communicate my disappointment separately.)

The assembly race is of an interesting class: active new money versus established state party. As a result, our afternoon post regularly contains one or two mailings, as both sides can afford a direct mail blitz. Reading the mail, we find

Bay Area influence: Mr. Poizner is downplaying his party association and emphasizing his moderate (even left-wing) credentials, while Mr. Ruskin’s signs clearly identify Democratic affiliations.
Political celebrity appearance: San Francisco mayor, Gavin Newsom, carelessly signed some Democratic machine-authored support letter and then denied personal involvement, which threw confusion into the race, as both sides offered differing interpretations of the letter.
Money: Current total spending on the race is about 3.7 M$, split 3 M$ – 0.7 M$ in favour of Mr. Poizner. That’s roughly fifteen dollars per registered voter in the district.

Meanwhile, we have a measure to approve a new development, accompanied by zoning changes, at the bayshore: Measure Q. Evaluating the measure’s impact is complicated: there is no recent analogous development in Redwood City, so it’s difficult to acquire any insight into how the various urban systems would accomodate such a large development (especially when spread over multiple decades). The pro-Q site is low on detail, and has nothing that compares to this effective page from the anti-Q organization. The latter is a strong example of effective argument presentation on the web.

(I should note that Redwood City Water Utility Services publishes a “City of Redwood City Public Works Services Water Utility Services Periodic Customer Newsletter” (usually bi-monthly) which periodically (every other newsletter) reminds us that Redwood City is 10% overdrawing the fresh water from the San Francisco system. So local property owners are highly aware of water-related issues.)

(I should also note that my commute is along U. S. 101, which would be the highway most impacted by this development, and that that commute uses the interchanges adjacent to the development itself. Oh, and that I worked in traffic systems and am a bit skeptical that the additional trips impact is as mild as presented: it’s just as likely that 50% of the new residents will work in Santa Clara.)

The pro-Q side has been effective at press release–article conversions, but that perhaps is revealing Peninsula media outlets’ actual capacity for investigative reporting. The pro- side sent one mailing that was poorly received in our household: the card presents an array of headshots of officeholders endorsing the measure but, when so many politicians agree on something in America, even I know money is a more likely influence than any sincere, shared analysis. I’ve noted before, with some excitement, how Redwood City is changing, but Q hasn’t shown itself to be well-enough structured an argument to suggest it’s aligned with those other changes.

Puzzle #2: cal(1)

stephen@blueslugs.com (Stephen Hahn) — Wed, 29 Sep 2004 22:21:05 +0000

(I’ve been fixing little smf(5) bugs, as well as revising our documentation, presentations and–most importantly–more block diagrams for this blog. But I bumped into an annoyance and thought I should share.)

As an young old-school Unix developer, I tend to live in terminal windows. One of my favourite commands is cal(1), which has a great default mode:

$ cal
September 2004
S M Tu W Th F S
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30

But if you want to see an October calendar, you might get confused:

$ cal 10
10
Jan Feb Mar
S M Tu W Th F S S M Tu W Th F S S M Tu W Th F S
1 2 3 4 1 1
5 6 7 8 9 10 11 2 3 4 5 6 7 8 2 3 4 5 6 7 8
12 13 14 15 16 17 18 9 10 11 12 13 14 15 9 10 11 12 13 14 15
19 20 21 22 23 24 25 16 17 18 19 20 21 22 16 17 18 19 20 21 22
26 27 28 29 30 31 23 24 25 26 27 28 23 24 25 26 27 28 29
30 31
Apr May Jun
S M Tu W Th F S S M Tu W Th F S S M Tu W Th F S
1 2 3 4 5 1 2 3 1 2 3 4 5 6 7
6 7 8 9 10 11 12 4 5 6 7 8 9 10 8 9 10 11 12 13 14
13 14 15 16 17 18 19 11 12 13 14 15 16 17 15 16 17 18 19 20 21
20 21 22 23 24 25 26 18 19 20 21 22 23 24 22 23 24 25 26 27 28
27 28 29 30 25 26 27 28 29 30 31 29 30
Jul Aug Sep
S M Tu W Th F S S M Tu W Th F S S M Tu W Th F S
1 2 3 4 5 1 2 1 2 3 4 5 6
6 7 8 9 10 11 12 3 4 5 6 7 8 9 7 8 9 10 11 12 13
13 14 15 16 17 18 19 10 11 12 13 14 15 16 14 15 16 17 18 19 20
20 21 22 23 24 25 26 17 18 19 20 21 22 23 21 22 23 24 25 26 27
27 28 29 30 31 24 25 26 27 28 29 30 28 29 30
31
Oct Nov Dec
S M Tu W Th F S S M Tu W Th F S S M Tu W Th F S
1 2 3 4 1 1 2 3 4 5 6
5 6 7 8 9 10 11 2 3 4 5 6 7 8 7 8 9 10 11 12 13
12 13 14 15 16 17 18 9 10 11 12 13 14 15 14 15 16 17 18 19 20
19 20 21 22 23 24 25 16 17 18 19 20 21 22 21 22 23 24 25 26 27
26 27 28 29 30 31 23 24 25 26 27 28 29 28 29 30 31
30

It’s an interesting UI choice to assume that anyone would want the calendar for the year 10 C.E. Certainly I never do, and I’m pretty sure someone would have told me if UNIX systems were the professional historian’s first choice for computing…

So today’s puzzle is to make cal(1) more usable. If I enter cal month_num, give me the current month; If I enter “cal now” give me the 3-month window around the current month, like so:

$ cal now
August 2004 September 2004 October 2004
S M Tu W Th F S S M Tu W Th F S S M Tu W Th F S
1 2 3 4 5 6 7 1 2 3 4 1 2
8 9 10 11 12 13 14 5 6 7 8 9 10 11 3 4 5 6 7 8 9
15 16 17 18 19 20 21 12 13 14 15 16 17 18 10 11 12 13 14 15 16
22 23 24 25 26 27 28 19 20 21 22 23 24 25 17 18 19 20 21 22 23
29 30 31 26 27 28 29 30 24 25 26 27 28 29 30
31

Other than that, all other standard invocations of cal(1) should work as usual.

My example solution is a couple dozen line ksh(1) shell function, and I’ll post it along with the best submissions. (Perl folks: no non-core modules, please.)

smf(5): authorizations built-in

stephen@blueslugs.com (Stephen Hahn) — Thu, 23 Sep 2004 08:04:33 +0000

I mentioned yesterday that you can manipulate services if you have the appropriate authorizations, without needing to possess any privileges. For instance, my current shell has the following privileges and authorizations:

$ ppriv $$
117292: bash
flags = <none>
E: basic
I: basic
P: basic
L: all
$ auths
solaris.device.cdrw,solaris.profmgr.read,solaris.jobs.users,solaris.mail.mailq

And if we try to manipulate a service managed by smf(5) with this set of authorizations, we’ll get a predictable result.

$ svcadm restart network/smtp
svcadm: network/smtp: Permission denied.

However, smf(5) defines two big authorizations:

solaris.smf.manage, which allows the user to request a restart, refresh, or other state modification of any service instance, and
solaris.smf.modify, which allows the users to create or delete services and instances, as well as manipulate any of their property groups or properties.

(There are also lesser built-in authorizations—solaris.smf.modify.method, solaris.smf.modify.dependency, solaris.smf.modify.application, and solaris.smf.modify.framework—which allow the manipulation of properties within property groups of the mentioned type. And you can also customize the authorizations to allow an action at the instance level and to manipulate properties at the property group level.)

To make assigning batches of authorizations scale, the role-based access control facility (RBAC) allows the definition of rights profiles. (The definitions are contained in /etc/security/prof_attr, and the documentation is prof_attr(4).) The service management facility delives with two rights profiles we think are convenient:

$ grep ^Service /etc/security/prof_attr
Service Management:::Manage services:auths=solaris.smf.manage,solaris.smf.modify
Service Operator:::Administer services:auths=solaris.smf.manage,solaris.smf.modify.framework

We can then add the user_attr(4) database to connect a user with the appropriate profile, like:

$ grep sch /etc/user_attr
sch::::profiles=Service Management

(You can edit /etc/user_attr by hand or, if your password database is local, using the -P option to useradd(1M). The equivalent configuration via direct authorizations would read

$ grep sch /etc/user_attr
sch::::auths=solaris.smf.manage,solaris.smf.modify

and will work fine except that if the Service Management profile were to be enhanced subsequently, a user or role with the old explicity authorizations might not have the correct set for future operations.)

So now our authorization list is expanded, but our privileges are unchanged:

$ auths
solaris.smf.manage,solaris.smf.modify,solaris.device.cdrw,solaris.profmgr.read,solaris.jobs.users,solaris.mail.mailq
$ ppriv $$
117292: bash
flags = <none>
E: basic
I: basic
P: basic
L: all

and we can carry out our operation from an authorized, but unprivileged user account:

$ svcs network/smtp
STATE STIME FMRI
online Sep_21 svc:/network/smtp:sendmail
$ svcadm restart network/smtp
$ svcs network/smtp
STATE STIME FMRI
online 23:58:21 svc:/network/smtp:sendmail

And, finally, since the user_attr(4) database has network name service backends, you can actually make authorization grants that apply across an administrative domain, whether you’re giving out the big authorizations illustrated here or custom authorizations specific to a set of services running at your site.

smf(5): asking versus doing

stephen@blueslugs.com (Stephen Hahn) — Wed, 22 Sep 2004 09:34:38 +0000

Let’s consider how applications are traditionally started: we execute (or the system executes) a command, such as fooadm(1M), which in turn calls fork(2), does some detachment work, and then calls exec(2) to run food(1M) (which is what we wanted). A schematic of this sequence would look like

For long-running, always-needed applications (which we call services), this model raises some questions:

Who is responsible for food(1M)? After fooadm(1M) finishes, the intent behind the running food(1M) is murky. Is it acceptable for food(1M) to exit after some period of time? Or is exit a failure condition?
How did food(1M) get the privileges, resources, etc. that it needed to run? Either fooadm(1M) was executed holding enough privileges for food(1M) to function normally, or food(1M) is setuid-root and must verify that its requesters are authorized to execute it for some set of its offered operations. Ditto for resource assignments.
Why does each fooadm(1M)–food(1M) pair on the system (baradm(1M)–bard(1M), …) handle this relationship slightly differently? I can only speculate.

(Lest anyone assume I’m pretending to novelty: most restarters (init(1M)) or super-servers (inetd(1M)) have answered the first two of these questions by offering a single, specific application model. But many daemons we run on systems today fit neither of these application models well.)

In smf(5), the service management facility, directly forking a service is discouraged. Instead, one requests that a service be enabled, and the master restarter, svc.startd(1M) or a delegate—like inetd(1M)—will do the fork(2)–exec(2) sequence. The equivalent diagram might be drawn as

Upon receiving an enable request from smf_enable_instance(3SCF) or svcadm enable <em>fmri</em>, svc.startd(1M) determines if the service’s dependencies are satisfied and, if so, requests that the responsible restarter start an instance of the service. What the responsible restarter is doing is:

setting the privileges, resource bindings, and application-specific environment for the service,
creating a fault boundary to place the service instance within,
calling fork(2)–exec(2) to run the service*, and
watching the service for error conditions, upon which the service is terminated (and restarted if appropriate).

(The combination of the master restarter and the delegates are handling these calculations and operations for every service on the system, propagating their state changes and evaluating the impact of those state changes in turn.)

Moreover, because the smf_enable_instance(3SCF) request is evaluated based on the authorizations of the caller, fooadm(1M) can be run with no significant privileges. Since we split authorizations into action authorizations (non-persistent operations, like “restart this service”) and modify authorizations (changing configuration aspects), it becomes straightforward to create an operator role that can tend a service, but not change its configuration or affect any other independent service on the system.

More flexible administrative assignments is one aspect of inserting smf(5) into Solaris, but we’ll contrast these two approaches again—and reveal exactly what those purple rectangles represent.

Not every restarter need offer a fork(2)–exec(2) application model, but presently all smf(5) restarters do.

Had no effect, given my coloured lenses

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Sep 2004 19:37:41 +0000

The Register is displaying with a Sun Blue theme today.

smf(5): the system knowing more means…

stephen@blueslugs.com (Stephen Hahn) — Tue, 21 Sep 2004 02:05:10 +0000

you can choose to know less. For instance, if you need to know what application model your program runs under, then you have to know how to start or restart or stop your application. The common example is that, if you run a process under inetd(1M), then telling inetd(1M) to take notice of your new service (or your newly deactivated service) requires knowing that

pkill -HUP -x inetd

will cause inetd(1M) to rescan inetd.conf(4), its configuration file.

But for a service started by init(1M) out of inittab(4), you edit that configuration file and then

init Q

And for a service that is started somewhere in /etc/rc*.d, restarting looks something like



/etc/init.d/foo restart


Usage: /etc/init.d/foo {start|stop}


/etc/init.d/foo stop


/etc/init.d/foo start

and making this service start on boot consists of creating a link to the /etc/init.d file to some sequence number in the appropriate run level directory. (I’ll omit discussion of saf(1M), but that’s another distinct method for managing services.)

All of this, of course, could be simpler.

If the service describes what other service is responsible for starting it and stopping it (and restarting it or asking for its configuration to be refreshed), then a single command can relay the appropriate instructions to the responsible restarter. In smf(5), this command is svcadm(1M). Our examples above all reduce to

svcadm restart application/foo

or to

svcadm enable application/foo

depending on whether you wanted to restart or merely enable (and start) the service in question.

Of course, since we also know what other services application/foo requires, we can actually enable all required services automatically, by following our service graph. Let’s make this more concrete: to enable the SSH daemon on Solaris, all you need to do is:

svcadm enable -r network/ssh

svcs -p network/ssh

STATE STIME FMRI online Sep_14 svc:/network/ssh:default Sep_14 100152 sshd

What’s svcadm(1M) doing? We can ask it for verbose output:

svcadm -v enable -r network/ssh

svc:/network/ssh enabled. svc:/system/cryptosvc enabled. svc:/system/filesystem/minimal:default enabled. svc:/system/device/local enabled. svc:/system/filesystem/usr enabled. svc:/system/filesystem/root enabled. svc:/network/loopback enabled. svc:/system/filesystem/usr:default enabled.

svcadm(1M)’s output represents its traversal of the dependencies for sshd(1M). Taken across the many services included in Solaris, that’s a lot of knowledge we’ve formalized and moved into the system. It becomes the basis for a lot of “meta-service” administration, including automated restart.

If you didn’t know these dependencies already, you can interrogate the system (using svcs(1)) and cement your understanding; if you did, then you can answer the second-order questions like, “what is affected by a failure of system/utmp?” much more rapidly than in the past. Or you can instead know less and devote your newly freed neural capacity to understanding your application stack as a whole (or to maintaining encylopedic knowledge of Simpsons trivia…).

smf(5): a view from the moon

stephen@blueslugs.com (Stephen Hahn) — Fri, 17 Sep 2004 08:02:00 +0000

One interesting aspect of smf(5) is that we have pulled apart many of the assumed interrelationships between system services, and made them explicit. Doing this makes building availability and failure models much easier, but it also lets us see one projection of Solaris’s shambling shape. (There’s another interesting technique for dynamic discovery of relationships via DTrace, but I’ll let Bryan show the image from those experiments when he’s ready.) Everyone wanted to visualize the service graph that results, so Dan Price and David Bustos came up with a way to generate one. Here’s the result, generated on my two-way Opteron system earlier today:

Because we’ll be tweaking the graph a bit more, I’m only showing this scaled down version, but we can take a bit of a tour just from the gross features:

Generally, earliest to latest proceeds from right to left.
- The large structure just below the centre is network/rpc/bind and its dependents, the various RPC services.
  - Furthest to the right is network/pfil, which prepares the network interfaces for IP Filter.
    - Furthest to the left is system/fmd, the new Solaris Fault Manager.
    As you might guess, we’ve had to write numerous graph-aware diagnosis algorithms to make a large structure like this one navigable. We’re looking forward to further enhancing our reporting and visualization tools to make troubleshooting easier still.
    
    (Of course, knowing all the dependencies in Solaris doesn’t protect you from the occupational hazard of overdiagnosis, as John and I spent an hour poking at every possible aspect of his system, which ultimately required a new network cable. I figure once a year I still end up following an “all possible software causes” algorithm, ending up in mdb(1) poking around the kernel, rather than checking cable connections or a bad software install.)

smf(5): sun.com, new blogging, boot

stephen@blueslugs.com (Stephen Hahn) — Thu, 16 Sep 2004 00:45:12 +0000

We had our moment on sun.com last week, which was pretty gratifying: my first slides on doing this project are from June 2001, and I’m not certain how long before that—1997? 1998?—we started sketching out aspects of the problem. smf(5), which we developed under the codename “Greenline”, is now rolling out pretty well inside Sun, and with the inclusion of the Java Desktop System, makes for a nice fast workstation startup.

After being strapped into their ergonomic chairs for a year or two, people are understandably itchy to start explaining what they’ve been working on to others: Tobin‘s first entry suggests that he’s setting himself a lofty goal of touring the entire facility in his blog. I know Liane and a couple of others are ready to get the word out as well. But, being a dilettante at heart, I’ll just try to show some of the pieces I think are interesting.

The past couple of days I spent refining verbose boot, which let’s you see what services the system is starting and in what order (and what went wrong). I’m eliding some of the boring parts, but it looks roughly like this:

SunOS Release 5.10 Version smf-fixes-sparc 64-bit
Copyright 1983-2004 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
DEBUG enabled
misc/forthdebug (467758 bytes) loaded
[ system/filesystem/root:default starting (Root filesystem mount) ]
[ network/pfil:default starting (pfil) ]
[ network/loopback:default starting (Loopback network interface) ]
[ network/physical:default starting (Physical network interfaces) ]
Sep 15 16:26:33/11: system start time was Wed Sep 15 16:26:20 2004
[ system/filesystem/usr:default starting (/usr and / mounted read/write) ]
[ system/device/local:default starting (Standard Solaris device configuration.) ]
[ system/identity:node starting (System identity) ]
[ system/filesystem/minimal:default starting (Local filesystem mounts) ]
[ milestone/devices:default starting (Device configuration milestone.) ]
Hostname: gremlins-b10
[ system/sysevent:default starting (System event notification service.) ]
[ system/identity:domain starting (System identity) ]
NIS domain name is mpklab.sfbay.sun.com
[ system/cryptosvc:default starting (Cryptographic services) ]
[ system/manifest-import:default starting (Service manifest import) ]
[ system/rmtmpfiles:default starting (Remove temporary files) ]
[ system/sysidtool:net starting (sysidtool) ]
[ system/power:default starting (Power Management) ]
[ system/keymap:default starting (Keyboard defaults) ]
[ system/name-service-cache:default starting (Name service cache daemon) ]
....
[ milestone/single-user:default starting (Single-user milestone) ]
....
[ system/console-login:default starting (Console login) ]
[ milestone/multi-user:default starting (Multi-user milestone) ]
[ network/inetd:default starting (inetd) ]
gremlins-b10 console login:

(On a Java Desktop System with GDM2 enabled, it just presents the GDM graphical login shortly after system/utmp is available.) One new aspect of boot is that the service “common” names are actually localizable although, since Solaris supports various splits between the root filesystem, /usr, and other filesystems, there’s a bit of hidden acrobatics involved. As another example, the reason we print out the start time is because, on this system, prior to that message, we can’t access the timezone data to convert the time-of-day to the local timezone. Fun.

Useful software: libxosd

stephen@blueslugs.com (Stephen Hahn) — Fri, 10 Sep 2004 07:43:18 +0000

I had built an early version of libxosd for my Solaris SPARC systems a year or two ago, since it provides a very tidy way to have a ubiquitous onscreen clock with a wide variety of window managers. libxosd went, when I was distracted, through a substantial rewrite, culminating in a 2.2.x series release. And I’m running Solaris 10 (with smf(5)) on a dual Opteron now, as I’ve mentioned before.

I generally try to build OSS software with the Sun Workshop/Forte compiler suite first, but for libxosd, I recommend going directly to GCC. There are too many mistakes in the autoconf/libtool pairing to bother fixing—none of these appear to be libxosd issues. (I mean, couldn’t -Wall not be passed to cc by now?)

Recommended.

Back from scatteredness

stephen@blueslugs.com (Stephen Hahn) — Fri, 03 Sep 2004 18:16:16 +0000

August was hectic. smf(5) required care and feeding, as did our newborn Nathaniel. I took a strange kind of quasi-paternity leave, by remaining home Tuesdays and Thursdays. (Don’t do this—you’re left with very few long concentration blocks on the workdays.) I also took Ben to Muskoka to celebrate a family history centennial, and we all went to Chicago for a memorial service for Dina’s father. Plus we had a yellowjacket nest in the walls of our home. I’ll write these all up shortly.

But I’m back at work now, and cooking up some entries on smf(5), resource management, and Solaris in general. (And for all of you who’ve written me in the meanwhile, if I haven’t already replied, then I’m probably composing a reply now.)

Just because the slide opens, doesn't mean you get to use it

stephen@blueslugs.com (Stephen Hahn) — Thu, 02 Sep 2004 05:40:59 +0000

Our trip to Chicago was wonderful and on Tuesday we rushed to the airport trying not to blame each other in front of the kids for why we were late. It turned out that we were able to meet the 45 minute baggage deadline for our flight, although we both promised to do things differently next time (travel with kids IS fun). We boarded our 747 and got the kids buckled into our full row of middle seats and ready for naps. We had food, our nasal aspirator (for ‘thaniel), and more books than we really wanted to carry (our luggage was 7 lbs overweight and we were asked to remove some of the books). The plane was hot but we knew that it would be cooler once we took off. Then the announcement came…

“folks we apologize for the heat, we haven’t been hooked up to the air conditioning unit while we’re on the ground but once we take off it will be better, in the meantime we’re going to crack a few doors to cool things off”. As a Californian, all I could think was that it was a fabulous idea, with the humidity and all, but before I had finished smirking there was a strange noise followed by an announcement that a slide had been deployed and we were all going to have to exit the plane.

I waited until everyone was off to move ‘thaniel while Stephen took Benjamin. The plan was to run to another terminal at O’Hare and try to get on one of two flights. The likelihood of all four of us getting seats together on another plane after having waited for an entire 747 to deplane seemed unlikely so we let the kids sleep and waited through five announcements about departure time changes until we were told we could board again (four hours later). The flight on our very empty 747 was uneventful but we were disappointed that unlike several years ago, United did not try to compensate for the late take off by giving us food or free drinks.

The secret lives of yellow jackets

stephen@blueslugs.com (Stephen Hahn) — Wed, 01 Sep 2004 05:07:05 +0000

Every so often we have a summer so bad all we can do is try to ride it out until things start to change. This has been one of those summers. In addition to the death of my father, three days before the birth of our son, we have also had what we thought were bees in the house. Stephen’s plan was to spray insecticide and consider the situation fixed. I went along with the plan for over a month watching the bees bounce off the walls and die hitting the light fixtures while I fed Nathaniel and tried to watch (bad) television. After a month of two to four bees a day, I had had enough. Stephen caulked the “hole” from the outside and, as I expected, there were more bees… It turns out they weren’t actually coming from all the way outside. The bees appeared more angry than injured and when one flew into Nathaniel’s car seat I thought I might go insane. So in a fit of rage, over the lack of a true solution, I called an exterminator. Little did I know that I was far from a solution…

The first exterminator told me they didn’t handle bees and gave me the number for a bee person. I called and they returned my call as Stephen arrived at home. I could overhear parts of the conversation, “I have one here…let me count again…are you sure…” It turns out that the bees were actually yellow jackets, they were in the wall, and we had a few options – find the hive (2,000 – 3,000) by putting holes in the walls or do nothing and wait for them to die in the walls and the ants to eat them. Oh, we were also told that there would be 10,000 of them by October. As a side note, I am very allergic to yellow jackets. I had an allergy test done last year and was told that I needed to start shots. I didn’t because I wanted to get pregnant but I was given an epi pen, which Benjamin quickly threw away in his “the trash can is a nice place to put things” phase (in case you were wondering – Stephen and I did have the “how can you be so allergic and not be doing anything about it argument”). Here was part of our conversation after Stephen spoke with the exterminator.

D: When can they get rid of the yellow jackets? S: They are going to try to come tomorrow or Friday. D: Did you mention I’m highly allergic and we have two small kids? S: No. D: I’m calling them back.

I dropped Benjamin off at day care the next day while Stephen met with the bee guy. Here is one of my calls home (as you might imagine, I called a few times to see how things were going).

D: Did they find them? S: Bad news. He says they are behind the brick of the fireplace so we just sprayed some stuff in one of the holes. D: What! Don’t let him leave until he’s found the yellow jackets. S: You want him to take out the brick? D: I want an end to this. Get rid of the yellow jackets. I don’t care if you have to take out the whole #!@# fireplace.

So, the exterminator had to come back later to remove the bricks, he couldn’t find them, his boss came at the end of the day, he suggested somewhere else. Then the first guy came back on Friday and cut bigger holes in the wall – this time cutting through to the outside – and we spent the weekend listening to yellow jackets hitting the aluminum foil that covered the holes (they can’t chew through the foil). Meanwhile, I called mosquito abatement and talked with their lab person and another exterminator. Oh, I also scheduled an appointment with my allergist who wouldn’t give me another epi pen without seeing me.

It turns out that you have to remove the hive with bees because the honey will rot. With yellow jackets you spray a substance called drion (a neurotoxin and dessicant) at the entrance and the worker bees take it into the hive to kill the queen and larvae. So we had another exterminator spray on Tuesday and we left for Chicago on Thursday with a cupboard full of three epi pens, bendryl (for the kids and me), and steroids for all. On the way to the airport (as we rush to make our flight) I call the woman who cleans our house (every other week – I dream of more help) to tell her not to clean if she sees anything flying around the house, our neighbor who is checking on the cat requesting a call if she sees anything, and our contractor to beg him to caulk before we get home.

Comments on a brief trip to Muskoka

stephen@blueslugs.com (Stephen Hahn) — Thu, 12 Aug 2004 17:00:00 +0000

Muskoka is a vacation region two or three hours’ drive north of Toronto in Canada. I was lucky enough to be able to spend a week or two each summer there, growing up in the late Seventies, at my maternal grandfather’s cottage on Lake Rosseau. My parents now share in a small place near Baysville. It turns out that my grandfather’s family helped establish a church there at the beginning of the twentieth century. Since my uncles were flying in for the centennial celebration, I decided that Benjamin and I should attend as well. I made this decision about five days before the celebration, but between Orbitz and JetBlue put together an itinerary without too much trouble.

But I learned depressing things about Orbitz:

Fares listed in an Orbitz search may not actually have ever existed.
Listed fares related to a nonexistent fare may also not have ever existed, and Orbitz makes no attempt to remove them.
Clicking on a dozen imaginary fares as some time period expires is highly annoying. This led me to

JetBlue, which has flat pricing and real fares. (I haven’t flown Southwest, so this was my first low cost airline experience.) JetBlue is a treat: there’s a single class cabin, which I always prefer; Benjamin had his own television screen with Nickelodeon and Boomerang, which is one way to handle toddlers; and—although I didn’t try this—apparently you can bring your own wine onboard and the crew will serve it to you.

Our trip and our visit was brief, but it was great to see family, introduce Ben to his cousins, and experience Muskoka time again. (Plus Canadian food isn’t laced with high fructose corn syrup, so it actually possesses taste beyond oversweetness.)

Stephen and Benjamin’s Excellent Adventure

stephen@blueslugs.com (Stephen Hahn) — Tue, 10 Aug 2004 08:25:18 +0000

Stephen and Benjamin left on Thursday for Stephen’s family’s cottage in Northern Ontario. They drove to Oakland, flew Jet Blue to JFK and then Buffalo, spent the night in a hotel, and then drove across the border and towards the Artic Circle. Stephen sent a brief email using the Jet Blue hot zone at JFK. We were both surprised that he had cell phone coverage at the cottage causing me to reconsider its remoteness.

Benjamin greatly enjoyed meeting his family and playing in the lake. It’s unclear if he continued his recent shouting monologue in his sleep, “nooooo [name of child in daycare class]… don’t take that… that’s mine… I need it”, while at the cottage but Stephen said the following monologue in the dark prior to falling asleep greatly amused others.

B: Daddy, I can’t see your face.

B: I can’t see your face.

B: I can’t see my face.

Meanwhile, Nathaniel and I were doing our best to sleep more than four hours at a time and break our 12 diaper-a-day record.

Another quiet week

stephen@blueslugs.com (Stephen Hahn) — Thu, 29 Jul 2004 17:37:34 +0000

There’s a sizeable troupe of kernel engineers at the O’Reilly Open Source Convention this week, so once again the office is a little quieter. (Bryan‘s around, so the office isn’t silent.) Most of the blog entries are “the author has left the building” style, but I’m hoping that those of us who remain chained to our keyboards will get an opportunity to read about how the conference is progressing, particularly regarding tonight’s birds-of-a-feather session on opening Solaris source.

The tantalizing aroma of svcs(1)

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Jul 2004 17:32:17 +0000

A side point: svcs(1) is pretty fast. Our example from last week was

$ svcs *milestone*
STATE STIME FMRI
online Jul_23 svc:/milestone/devices:default
online Jul_23 svc:/milestone/single-user:default
online Jul_23 svc:/milestone/name-services:default
online Jul_23 svc:/milestone/multi-user:default
online Jul_23 svc:/milestone/multi-user-server:default

Other approaches to service management in Unix-like systems ask each service for their status. On a large system, with a complete representation of its running daemons as services, this can be thousands of fork(2)/exec(2) pairs. That’s not how smf(5) works, and so a command like svcs(1) can be quick:

$ time svcs *milestone*
STATE STIME FMRI
online Jul_23 svc:/milestone/devices:default
online Jul_23 svc:/milestone/single-user:default
online Jul_23 svc:/milestone/name-services:default
online Jul_23 svc:/milestone/multi-user:default
online Jul_23 svc:/milestone/multi-user-server:default
real 0m0.027s
user 0m0.004s
sys 0m0.009s

In fact, we’re not even calling fork(2) to get this report on service status:

$ truss -t fork,write,exit svcs *milestone*
STATE STIME FMRI
write(1, " S T A T E ".., 29) = 29
online Jul_23 svc:/milestone/devices:default
write(1, " o n l i n e ".., 55) = 55
online Jul_23 svc:/milestone/single-user:default
write(1, " o n l i n e ".., 59) = 59
online Jul_23 svc:/milestone/name-services:default
write(1, " o n l i n e ".., 61) = 61
online Jul_23 svc:/milestone/multi-user:default
write(1, " o n l i n e ".., 58) = 58
online Jul_23 svc:/milestone/multi-user-server:default
write(1, " o n l i n e ".., 65) = 65
_exit(0)

I’ll start drawing a suitable architecture diagram…

Enabling and disabling services

stephen@blueslugs.com (Stephen Hahn) — Mon, 26 Jul 2004 21:15:43 +0000

I thought I would show a different example of smf(5) today. Here’s the state of the network/telnet service on my desktop:

svcs -p network/telnet:default

STATE STIME FMRI online Jul_23 svc:/network/telnet:default

It’s easy to enable and disable service instances using svcadm(1M):

svcadm disable network/telnet

svcs -p network/telnet:default

STATE STIME FMRI disabled 13:08:15 svc:/network/telnet:default

telnet localhost

Trying 127.0.0.1... telnet: Unable to connect to remote host: Connection refused

And we can enable it just as easily, too:

svcadm enable network/telnet

svcs -p network/telnet:default

STATE STIME FMRI online 13:08:29 svc:/network/telnet:default

Note that while something is declaring telnet service is available, no processes are associated with the service instance. If we “telnet localhost” from another window, we can then see the telnet daemon and the login session:

svcs -p network/telnet:default

STATE STIME FMRI online 13:08:52 svc:/network/telnet:default 13:08:52 116400 in.telnetd 13:08:52 116403 login

Support for enabling and disabling services can be done by calling smf_enable_instance(3SCF) and smf_disable_instance(3SCF), in addition to the command line interface of svcadm(1M). Since the framework relays the enable or disable request, we don’t need privileges to signal any process (or even know which process we might have to signal to make the update…).

A peek while bug flushing

stephen@blueslugs.com (Stephen Hahn) — Thu, 22 Jul 2004 19:47:19 +0000

With only a few days of exposure on the varied system configurations around here, there have been a few bugs raised against smf(5), the new service management facility. (I suppose it’s similar to a pack of hounds flushing pheasants during a hunt (although, ultimately, whom the hounds chase and whom a gun is pointed at does differ from a hunt).) What’s more exciting is that, as the kinks get smoothed out, people are instead starting to discuss possibilities. But I thought I’d show a little tiny piece of output instead. Here’s the output of the new services listing command, svcs(1), looking only at the major milestones of the startup process:

$ svcs *milestone*
STATE STIME FMRI
online 11:03:56 svc:/milestone/devices:default
online 11:04:04 svc:/milestone/single-user:default
online 11:04:07 svc:/milestone/name-services:default
online 11:04:11 svc:/milestone/multi-user:default
online 11:04:18 svc:/milestone/multi-user-server:default

What name services am I running? Examine the name services milestone more closely:

$ svcs -d milestone/name-services:default
STATE STIME FMRI
disabled 11:03:51 svc:/network/ldap/client:default
disabled 11:03:51 svc:/network/nis/server:default
disabled 11:03:51 svc:/network/rpc/nisplus:default
online 11:04:04 svc:/network/nis/client:default
online 11:04:07 svc:/network/dns/client:default

NIS, with a bit of DNS for seasoning. What’s it take to be a NIS client these days?

$ svcs -p svc:/network/nis/client:default
STATE STIME FMRI
online 11:04:04 svc:/network/nis/client:default
11:04:04 100202 ypbind

(But you knew that already.) More later.

Checklists navigated

stephen@blueslugs.com (Stephen Hahn) — Tue, 20 Jul 2004 19:08:10 +0000

The project that I mentioned a little while back navigated our engineering processes (or, rather, we steered it through them) and integrated into the Solaris development release last week. Now it’s getting a real shakedown as more and more people in the company get access to the bits. And it will hit the various Beta programs and Software Express shortly.

They’ve found a few bumps and rough edges, but nothing we can’t address. We’re even finding that it’s straightforward to keep the hardened machines hardened. And everyone’s moving fast again, as we don’t have to keep our many, many changes in our heads anymore—people will tell us directly what’s wrong/imperfect/improvable.

The project? It’s the service management facility, which is the other major technology comprising S10’s Predictive Self-Healing feature.

How not to communicate novelty

stephen@blueslugs.com (Stephen Hahn) — Tue, 13 Jul 2004 01:49:27 +0000

Adam seems to have recovered from his initial embarrassment regarding the alleged lack of novelty around describing a Solaris 9 feature. There’s no such shame here—I look at S9 at one of our Dangerfield releases (along with S7), that didn’t get the respect it deserved. (There’s no comparison to S10.)

I mentioned in a previous entry that I wasn’t particularly proud of how I had talked about S9RM. In this vein, I dug up a paper I wrote for SUPerG 2001 in Amsterdam. SUPerG is a Sun conference for datacenter customers, and focusses a lot on best practices for large Solaris systems. I was pretty giddy after S9RM wrapped up, and wrote a paper to present there, on the various mechanisms we envisioned and were in the process of implementing. It was received very quietly.

While I was writing the paper, I was trying out various text analyzers. One that I used was the Lingua::EN::Fathom module, available at CPAN. The results?

13 $ perl fathom.pl superg-2001-paper.ltx
[ ... vocabulary list elided ... ]
Number of characters : 19918
Number of words : 2865
Percent of complex words : 26.21
Average syllables per word : 1.9763
Number of sentences : 104
Average words per sentence : 27.5481
Number of text lines : 353
Number of blank lines : 101
Number of paragraphs : 69
READABILITY INDICES
Fog : 21.5044
Flesch : 11.6817
Flesch-Kincaid : 18.4737

The Fog scale informally corresponds to the number of years of education an average reader needs to read the text once and understand it. (21.5 is somewhere in graduate school.) The Flesch scale rates text on a 100 point scale; higher is better, with 60 being a reasonable target. (It’s safe to say that 11 is not in the vicinity of 60.) The Flesch-Kincaid is meant to correlate roughly with the U.S. school grade: 18 (graduate school again) is bad. The indices agree: this text is not good, or clear, writing.

So this document is a pretty solid indicator that, indeed, I didn’t do a good job explaining the value of resource management. For posterity, I’m making the paper available. Now I work harder on not having sentences that are more than twenty seven words long, or use 25% complex words—and if there’s a resource management topic you would like to see examined, feel free to tell me and I’ll try to write something understandable.

And, yes, Tim and Andy are blameless.

Endorsing publicradiofan.com

stephen@blueslugs.com (Stephen Hahn) — Thu, 08 Jul 2004 18:45:07 +0000

I stumbled across publicradiofan.com, amongst all of the various useless internet radio sites out there. (Radio reception inside my building isn’t great.) This site is a gem. It lets you select sites by show, by category, by media streams provided, and more. Program grids, grouped listings, adjustable time windows. Very solid.

(The media stream filter is very useful if you’re running Solaris x86, as your streaming options are somewhat limited. I’ve decided that if a local public radio station isn’t willing to provide MP3 or OggVorbis streams then I won’t support them. And I was pleased to see that some of the stations that I support do indeed provide “open” streams. And some don’t—a decision to make?)

Recommended.

Why projects? Why not?

stephen@blueslugs.com (Stephen Hahn) — Thu, 08 Jul 2004 18:11:42 +0000

One of the questions I often ask myself is “why aren’t more sites using projects?”. As I wander from forum to forum, I regularly see people saying, “I want to consolidate three [application server] instances on my system”—or two [database] instances or n applications. Many of these applications need to run with identical credentials (user id, group id, authorizations, privileges, etc.) and are only distinguishable by their working directory, environment variables, or the like. Reading these requests is a bit frustrating, as this scenario is one of the key motivations we had when introducing the project(4) database—and I can only conclude that it’s my failure to really communicate its utility.

Projects let you assign a label with a specific workload. In S8 6/00 and all subsequent releases, you can explicitly launch a workload with its appropriate project using the newtask(1) command. If extended accounting has been activated using acctadm(1M) with one of the standard record groupings, then the processes within that workload will include their project ID. Writing an accounting record on every process exit can impact some workloads, so you can optionally choose to only write records when every task exits. A task is a new process collective that groups related work within a workload (so it could be a workload component, like a batch submission). acctadm(1M) will report on the current status of the extended accounting subsystem, if invoked without arguments:

$ acctadm
Task accounting: inactive
Task accounting file: none
Tracked task resources: none
Untracked task resources: extended
Process accounting: inactive
Process accounting file: none
Tracked process resources: none
Untracked process resources: extended,host,mstate
Flow accounting: inactive
Flow accounting file: none
Tracked flow resources: none
Untracked flow resources: extended

The resource line is reporting what accounting resource groups and resources we can include in each record. We can expand the resource groups for each type of accounting using the

-r option.

$ acctadm -r
process:
extended pid,uid,gid,cpu,time,command,tty,projid,taskid,ancpid,wait-status,zone,flag
basic pid,uid,gid,cpu,time,command,tty,flag
task:
extended taskid,projid,cpu,time,host,mstate,anctaskid,zone
basic taskid,projid,cpu,time
flow:
extended saddr,daddr,sport,dport,proto,dsfield,nbytes,npkts,action,ctime,lseen,projid,uid
basic saddr,daddr,sport,dport,proto,nbytes,npkts,action

So we can enable the extended task record by invoking

acctadm(1M) like

# acctadm -e extended task

acctadm -E task

acctadm -f /var/adm/exacct/task

In S10, you can optionally enable accounting without having it write to a file, such that the records are retrievable using getacct(2).

Of course, that’s all about accounting, but projects are useful even if you’re not interested in the long term resource consumption of your workloads. The project ID is useful for isolating your workload using conventional /proc-based tools like prstat(1M) and pgrep(1), as well as with DTrace. For instance to see only one’s own projects, you can use the -J option to pgrep.

$ pgrep -lf -J user.sch
728069 /usr/bin/bash
728027 /usr/bin/bash
125169 /usr/bin/bash

To see workloads on the system, you can use

prstat‘s -J option, which aggregates the activity by project ID, as well as displaying the most active processes:

$ prstat -c -J user.sch 1 1
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
653322 xx 19M 17M cpu2 0 3 166:34:10 12% setiathome/1
911046 xx 19M 17M cpu5 0 3 170:28:53 12% setiathome/1
668697 xx 19M 17M cpu4 0 3 138:53:14 12% setiathome/1
100378 daemon 2352K 1944K sleep 60 -20 30:18:23 0.2% nfsd/5
125214 sch 4472K 4152K cpu3 1 0 0:00:00 0.0% prstat/1
100066 root 7872K 6736K sleep 29 0 2:20:42 0.0% picld/13
125169 sch 2768K 2416K sleep 1 0 0:00:00 0.0% bash/1
100156 root 91M 36M sleep 59 0 0:46:59 0.0% poold/8
100249 root 6680K 4848K sleep 1 0 8:02:46 0.0% automountd/2
100254 root 5776K 3552K sleep 59 0 0:00:01 0.0% fmd/10
100262 root 4024K 3424K sleep 59 0 0:19:40 0.0% nscd/57
100265 root 1248K 776K sleep 59 0 0:00:00 0.0% sf880drd/1
100184 root 2288K 1384K sleep 1 0 0:00:00 0.0% ypbind/1
100172 daemon 2680K 1704K sleep 58 0 1:07:32 0.0% rpcbind/1
100158 root 2216K 1336K sleep 59 0 0:00:26 0.0% in.routed/1
PROJID NPROC SIZE RSS MEMORY TIME CPU PROJECT
130 3 56M 51M 0.3% 475:56:17 37% background
0 61 341M 168M 1.0% 43:21:28 0.3% system
36565 4 13M 11M 0.1% 0:00:01 0.1% user.sch
105403 14 39M 30M 0.2% 0:00:03 0.0% user.xxxxxxx
77194 17 74M 62M 0.4% 0:03:10 0.0% user.xxxxxx
Total: 133 processes, 279 lwps, load averages: 3.07, 3.07, 3.04

(This system’s pretty idle during our U.S. shutdown, so it’s doing its best to find extraterrestrial customers.)

To limit your DTrace predicates to only a project of interest, use the curpsinfo built-in variable to access the pr_projid field, like

/curpsinfo->pr_projid == $projid && ..../

where I’ve also used the

$projid scripting macro, which expands to the result of curprojid(2) for the running DTrace script. You could instead explicitly enter your project ID of interest, or use one of the argument macros if writing a script you expect to reuse.

Projects also let you place resource controls on your workload, establish its resource pool bindings, and more. We’ll make it easier to use them with the forthcoming service management facility. But I’ll summarize: projects are a precise and efficient way to label your workloads (as opposed to pattern matching on arguments or environment variables). If you are consolidating workloads, either because of machine eliminations, organizational mergers, or other reasons, they are definitely worth considering. If you think there’s a way to make them more applicable to your work, please let me know.

Answer to sort(1) puzzle #1

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Jul 2004 19:13:25 +0000

/usr/bin/sort -ur -k 1,1n -k1,2nr input.d

Well, I underspecified that problem slightly; the output I was looking for is

1305 6565
1401 8192
1408 2312

which the anonymous poster’s invocation will give. Alan’s invocation gets the correct line, but has the first field backwards (if I had specified the problem fully). You could, of course, send Alan’s output through another sort(1) stage to order the first field. The key to this puzzle is knowing that (a) sort(1) does a final comparison of the entire line using strcoll(3C), (b) that fields with specific modifiers ignore global modifiers (like the -r option here), and (c) that the Solaris implementation of sort(1) will output only the first unique line it finds in the collated sequence. The first two of these points are in the manual page; the last requires some experimenting.

True story: This puzzle grew out of a service request where a customer was moving from a platform where the last unique line was the one displayed and needed to modify their script to produce the same output on Solaris.

Please comment if you want more puzzles, or if you think I should stop before getting started!

sort(1) puzzle #1

stephen@blueslugs.com (Stephen Hahn) — Fri, 02 Jul 2004 09:26:00 +0000

(I’m waiting for a build to finish, so here’s a small Solaris sort(1) trick.)

Question: I have the following data file

Using only sort(1), how do I generate a file sorted by the first field, with only the highest valued second field for each first field value?

Note that the unique line behaviour of sort(1) isn’t well specified, so versions from other platforms may not be able to do this trick.

Resource controls and agent-influenced design

stephen@blueslugs.com (Stephen Hahn) — Wed, 30 Jun 2004 18:33:01 +0000

Eric mentioned the agent LWP technique a few days ago. I thought I would point out that the resource controls facility was expressly designed with the agent’s capabilities in mind. This choice lets us have an interface like:

int setrctl(const char *controlname, rctlblk_t *old_blk, rctlblk_t *new_blk, uint_t flags);

where the action of setting a resource control value applies to the current process or a process collective that encloses it (like a task, project, or zone). (As opposed to an interface that requires an (idtype_t, id_t) pair to specify its target.) Making this choice means we don’t end up having a lot of difficult locking scenarios, as we just insert an agent LWP into a victim process to modify the settings on its task.

Resource controls, when they were introduced in S9, brought in some new useful controls, like task.max-lwps and project.cpu-shares, as well as handling both ulimit(2) and setrlimit(2) calls in a compatible fashion. But S10 has seen the introduction of a number of very useful controls: for instance, the System V IPC tunables have been replaced by a set of resource controls, so that you can dynamically change these settings (and apply different constraints to different projects) without a reboot. Dave Powell deserves the credit for this much appreciated piece of work. New controls associated with cryptographic capabilities, zones, and general resource handling have also appeared. And Steve Lawrence spent a fair bit of time looking at making prctl(1) output actually human-readable.

So it’s all much improved from the initial work.

Assembling a collective viewpoint

stephen@blueslugs.com (Stephen Hahn) — Tue, 29 Jun 2004 20:33:19 +0000

Bryan fielded some questions at USENIX today, and in doing so <a href=“http://blogs.sun.com/roller/page/bmc/20040629"

augments some of Andy’s <a href=“http://blogs.sun.com/roller/page/tucker/20040618" first comments on why we are pursuing an open source development model for Solaris.

Quiet in the office

stephen@blueslugs.com (Stephen Hahn) — Tue, 29 Jun 2004 06:56:09 +0000

With the DTrace folks at the USENIX Technical Conference, a contingent at JavaOne, and the remainder focussed on finishing their work for Solaris 10, it’s pretty quiet at the office. Oh, except for the fact that my team’s multi-year project is wending its way through the final stages of the processes surrounding Solaris integration. Checklist, checklist, test transcript, architectural updates, performance results, status meetings. And maybe one or two more checklists. But we’re almost there.

Coffee (and other scents) in the air

stephen@blueslugs.com (Stephen Hahn) — Tue, 29 Jun 2004 03:04:23 +0000

Based on the construction and openings going on, I suspect Redwood City must have crossed some critical threshold in terms of average income or population. We’ve gone from one Starbucks and one Starbucks counter (at Safeway) to three standalone stores and two counters (one in Target). (I suspect that there’s a law of Starbucks to be formulated here.) Whole Foods has started renovating the former Albertson’s at El Camino Real and Jefferson Avenue, while Trader Joe’s is building a new store just north of the Redwood City–San Carlos border. Pasta Pomodoro and T-Mobile have opened new sites as well.

It looks like everything should be ready by mid-fall. A year past that, the downtown cinemas should open (which will be one of the few Peninsula theatre complexes within easy walking distance of the San Francisco–San Jose Caltrain line). If there is restaurant development to match, Redwood City could be a very interesting small downtown in two years. But right now it’s changing enough that I’ll be driving to Menlo Park a lot less often.

Resource pools and XML/XSLT gymnastics

stephen@blueslugs.com (Stephen Hahn) — Mon, 28 Jun 2004 07:58:23 +0000

Since I mentioned resource pools in my previous post, I thought I should also point out that Gary Pennington has been presenting some resource pool tricks. Since resource pools uses XML as a marshallable configuration format, you can apply the many “X*” technologies to the document to realize alternate views of an active configuration.

Why predictable?

stephen@blueslugs.com (Stephen Hahn) — Fri, 25 Jun 2004 22:08:10 +0000

I titled my [Sun] blog “Predictable” more out of genuine interest about making systems more so, than any particular cynicism about how things (any things) turn out. We have a pretty established language about desirable system attributes: reliability, availability, serviceability, and performance. Predictability is a courtier of all of these, but really doesn’t dominate any of them–it’s almost a separate quality.

We’ve been working on resource management for a while now, with an eye to allowing the construction of more predictable servers. Most of this work has been adding various mechanisms to the OS to prevent resource denial of service opportunities, or mechanisms that reserve or fairly schedule resources among competing services, such that the system can respond fairly smoothly until we reach some level of overcommitment. Now that we’ve got the basic mechanisms (and more are coming), we can start to examine how we can layer automation on top of them, and push out the boundary where true overcommitment occurs, without administrator intervention.

The current edition of Solaris Express contains the first of these features, dynamic resource pools, which you can use in combination with zones to have your consolidated systems smoothly react and reassign processors based on system load and relative importance. (You can also use the fair share scheduler with zones, if you don’t need to reserve or cap some absolute amount of processing capability for each zone).

We now have a system that can respond elastically across a wider range of load scenarios, without compromising some minimal expectations regarding quality of service with respect to one resource. Our solution, however, can handle multiple resources, and we’ll see how that “predictability product” allows you to run your systems closer to maximal utilization in the future.

There are lots of problems in this space, connected with specific operating system resources (and the notion of a resource itself), how resources map from one layer in a stack to the next, how end-to-end scenarios play out within a participating host, … But it’s clear that predictability engineering is a piece of operating systems development–and, if you can get some time to think about it, it’s a lot of fun.

A little quiet

stephen@blueslugs.com (Stephen Hahn) — Thu, 24 Jun 2004 22:48:34 +0000

There’s now at least two good reasons for my hiatus.

Questions for the audience: visible project leaders

stephen@blueslugs.com (Stephen Hahn) — Thu, 24 Jun 2004 22:32:48 +0000

One of the interesting attributes of open source projects is that, once they reach a certain critical size, they still may or may not have a single, visible leader. I suppose by a visible leader I mean an individual that the project’s community views as having authority over the project’s direction, architecture, and implementation. For other projects, there is a public or private group that possesses that authority. We’ve also seen projects, either at the time of a fork or in their normal course, transition from one leadership class to the other.

The primary question that we might formulate is “does the presence of a single, visible leader affect the chances for success of an open source project?”. But this doesn’t really probe the issue very deeply, as success for an open source project can be defined very differently by the project, by an external observer, or by a customer or investor in the organization(s) that use or sponsor the project. We could define success by size of the user community, increase in likelihood of the project’s longevity, introduction of desired features or elimination of defects, or grant renewals, and so forth.

If we look at the first of these, we might conclude that the size of the user community, although connected to project utility and quality, is probably also influenced by communications about the project. Since stories about individuals are usually more compelling than those about committees, perhaps it’s easiest to take advantage of the comfortable “leader-as-project” viewpoint and let those stories–be they formal news coverage or blogging or word of mount–circulate. But it’s less clear that the identification of the individual with the project (“I am the state.”) and the authority for project direction need to travel together.

What do you think? Does the leadership structure (or even the celebrity of the author) associated with a project influence your choice of that software, open source or otherwise? Are certain structures more appealing from your perspective?

Thanks.

A menu of many buffers

stephen@blueslugs.com (Stephen Hahn) — Thu, 24 Jun 2004 20:05:47 +0000

I’m often manipulating over 100 files when I’m making modifications to various software components. Generally, the buffers menu in Vim (X11/Motif) handles this okay, unless all of the files are named with the same letters. In a recent batch of changes, 90 of my files were named “Makefile”, “prototype_com”, or “pkginfo.tmpl”; this results in the buffers menu being larger than the vertical height of the screen.

I tried a few of the publically available scripts, and found BufExplorer to be all I needed and more: since it turns the list into an additional buffer, with various viewing and sorting options, it’s easy to search and select the correct buffer to modify. And your fingers need never leave the keyboard.

Recommended.

Nathaniel arrived

stephen@blueslugs.com (Stephen Hahn) — Tue, 22 Jun 2004 10:09:27 +0000

(I suppose I glossed over this in the flat tire story.) Here’s a shot of Benjamin inspecting his new younger brother the afternoon of his birth:

We’re sleeping occassionally, in case you’re wondering.

Second stream of consciousness

stephen@blueslugs.com (Stephen Hahn) — Mon, 21 Jun 2004 19:52:39 +0000

I figured I should mention that I’m going to make most of my Solaris-specific comments on my blog at blogs.sun.com. We’ll see if I can keep the topics meaningful and interesting on both, without halving my posting rate on each.

Feel the cars go by

stephen@blueslugs.com (Stephen Hahn) — Mon, 21 Jun 2004 19:45:45 +0000

Had a brief moment of suburban excitement: Dina, Nathaniel, and I were on our way to an afternoon appointment when our driver’s side rear tire blew out. On the soon-to-be-five-lanes-wide U.S. 101 South, in East Palo Alto. Busy.

Nathaniel will turn a week old soon, so he wasn’t much going to be much help; Dina had to stay inside the car with him. The AAA estimate was forty-five minutes, so I jumped out and decided to learn how to change the tires on our wagon.

The kit included a fold-up reflective safety triangle, which I put about five yards behind the car. That made me feel about one and a half times safer, since the cars were whooshing past about 10 feet behind me. I positioned the jack at the little arrow on the undercarriage that showed the jacking position. The car raised fairly easily, and I was on to worrying about whether I could generate enough leverage to loosen the lugnuts. The tire iron, at ten inches, seemed short, but I was able to break the tension on the bolts with a bounce on the iron at 90 degrees, despite being buffeted by the occasional semi or panel van.

I pulled off the tire and slid it into the included protective tire bag (an oversize plastic garbage bag) and installed the puny limited-use spare. We ended up being only ten minutes late.

(Quiet kudos to Saab for an excellent set of tire changing tools, indicators, and instructions.)

Richard Venezky 16 April 1938 - 11 June 2004

stephen@blueslugs.com (Stephen Hahn) — Sat, 12 Jun 2004 03:49:32 +0000

It is with great sadness and honor that I announce the death of my father, Richard Lawrence Venezky, on June 11th, 2004. My father was an amazing man who excelled in so many ways and I’m grateful to have had the opportunity to not only write updates on his condition but also to receive many wonderful comments in return. I have learned so much about my father over the past year and a half. Your stories and well wishes verify my feelings that he was a loving father, a caring husband, a loyal friend, a committed mentor, an inspiring teacher, an honored scholar, an avid collector, and a gifted gardener. He greatly enriched our lives and we will all miss him dearly. I would love to be able to sit with each and every person to record all the stories and I hope you will continue to email me with thoughts about Dad.

I have many stories to share of my own too. I grew up thinking that adulthood involved marriage to someone you loved filled with fun parties in junkyards and vacations to exciting places, that being well-read with multiple hobbies was important, and that no matter how brilliant or famous someone was, they were still just a person. He made these things look so easy that I thought one day it would all just happen for me too. I now understand that he worked hard to make these things happen but that he enjoyed what he did so much that it never looked like work.

In the past few weeks I have thought about the pain of losing Dad and come to realize how fortunate I am to have had him in my life for so long. He fought the cancer with such dignity and strength that we really thought there was nothing he couldn’t do. I realize that there will be many unanswered questions but that one of the gifts my father gave to me was the determination to learn on my own and the joy in doing so. I will always want to know exactly how I can best educate my children or how he made his garden grow but I will also take pleasure in knowing that he is proud of me as I go through this process of discovery. Stephen and I are going to work to make sure that our children know Dad and have some of the same benefits I have had growing up. I will think of him every time I work in the garden, eat a vegetable, read a book, or learn a new skill. Stephen, Benjamin, and I love you Dad.

A bug from a blog

stephen@blueslugs.com (Stephen Hahn) — Fri, 11 Jun 2004 07:50:19 +0000

blogs.sun.com has already improved Solaris a bit: Moazam Raja mentioned that which(1) isn’t doing what one might expect. The newly minted bug, 5061831, will give us a place to discuss how we might fix this issue, gives one suggested fix, and a workaround using ksh(1)’s whence builtin.

Introductions

stephen@blueslugs.com (Stephen Hahn) — Thu, 10 Jun 2004 23:52:31 +0000

Although I’ve been blogging privately and on the web statically for a while now, it’s pleasant to have a location to discuss Solaris and Sun specific information. I work on a couple of different projects, of varying sizes; in the past, I’ve done (I believe) interesting work in resource management, routine work fixing various bugs, and foolish things like rewriting sort(1) from scratch. (I’m particularly proud of the fact that I (with much support) integrated Perl into Solaris 8, and then found someone much more capable to make that a particular competency of Solaris.)

How Sun participates, specifically via its technical staff, in community development situations is of much interest to me; trust me when I say I’ll revisit this topic later.

Small talk: Presently, I’m an engineer in the Solaris Kernel group. I moved from Providence, RI to the Bay Area when I started work for Sun in 1997. (I was studying computational high energy physics at Brown, but decided not to pursue the series of post-doc positions traditional at the time.) My undergraduate degree is in engineering science at the University of Toronto. I was born, raised, and primarily and secondarily educated in southwestern Ontario.

Ion-2 and VNC

stephen@blueslugs.com (Stephen Hahn) — Fri, 04 Jun 2004 01:56:37 +0000

I use ion as my window manager, but have recently switched my home desktop from a G4 Cube to a PC running Solaris. I’m a regular full-screen VNC user, and getting vncviewer and ion to cooperate is pretty key. Fullscreen mode can be achieved using vncviewer and ion (from blastwave, or build them yourself) with only a few modifications to the X resources for vncviewer. These aren’t at all tricky, but require enough experimenting that writing them down will save someone somewhere five minutes some day.

Although the fullscreen transition will mostly work, ion will be confused about the keyboard focus, so we need to tell vncviewer to take additional responsibility. Add the following to your $HOME/.Xdefaults file:

vncviewer.grabKeyboard: true
vncviewer.fullScreen: true

For the latter case, you could instead use the -fullscreen option to vncviewer or the on-screen menu. That’s it: you’re immersed in a virtual reality… which for me is mostly the same as my normal reality.

Update (9 June): Five minutes saved already; see Comments.

CEOs lack midichlorians?

stephen@blueslugs.com (Stephen Hahn) — Wed, 02 Jun 2004 00:00:56 +0000

A peculiar quotation from an interview with Red Hat’s Matthew Szulik:

The issue of Intel versus AMD ultimately goes back to the distribution partner; if companies like Hewlett-Packard, for example, can be successful with the AMD parts, then AMD will benefit. Another issue for AMD will be dealing with Darth Vader in Redmond and Jabba the Hutt in Santa Clara.

(Intel is headquartered in Santa Clara, although I don’t think “Santa Clara → INTC” is standard usage like “Redmond → MSFT”.) But when did sloppy Star Wars analogies become acceptable CEO material? I can only think that there was a hapless PR liaison grateful that an even-more-confusing Axis of Evil analogy (“… between the North Korea in Redmond and…”) had been avoided.

Strange.

Ministry of Education picks StarOffice

stephen@blueslugs.com (Stephen Hahn) — Fri, 28 May 2004 23:02:35 +0000

I was happy to see that the Ontario Ministry of Education has selected StarOffice as its standard office suite. As someone who grew up a test subject under the Ministry, by learning computing on such systems as

Commodore PET, with Waterloo Structured Basic as an alternate environment,
IBM PCs with Waterloo Ports as the network applications manager, and
the Burroughs ICON running QNX,

I’m happy to see some element of real standardization. (Plus a few other exposures to Radio Shack TSR Model 1s through to CoCos, a DEC PDP-11, and submitting Fortran jobs to the mainframe at Waterloo in various classes. It seems romantic now: a new, completely different system every year (or even every semester)…)

The ICON was my favorite social hack: it came preconfigured with both administrator and teacher accounts in both English and French. A variant of RTFM gives one root privilege: read the French manual. Ultimately, I got in trouble for this, although trouble followed from viewing the administration as adversaries than causing any actual damage.

Dina passes government test

stephen@blueslugs.com (Stephen Hahn) — Thu, 27 May 2004 04:59:21 +0000

Dina passed the U. S. Department of the Interior’s Federal Information Systems Security Awareness Online Course. Huzzah.

Procyon lotor rules our backyard

stephen@blueslugs.com (Stephen Hahn) — Fri, 07 May 2004 09:15:39 +0000

We recently uprooted our home office, transplanting it from a bedroom in the front house to the room attached to our garage. With the garage building at the back of the lot, I now have a clear view of the yard. Tonight the sensor lights tripped and I had a good staring exchange with a pair of raccoons. The Redwood City raccoon is no Arctictis binturong, but these two were rudely healthy and (I argue) nonplussed to be both illuminated and observed. (I’ll skip over any rivalry with the various, large-lunged cats who’ve decided that the side of our house is the neighbourhood’s romantic nightspot—the healthy trash around here guarantees a champion’s diet to any raccoon with a bit of motivation, making him or her more than a match to a typical kitty.)

Since I now have prime suspects for last summer’s tragic shredding of Ben’s inflatable swimming pool, I’ll see if I can get a picture of these two in the coming weeks.

sun.com survey on default secure install

stephen@blueslugs.com (Stephen Hahn) — Fri, 07 May 2004 08:55:16 +0000

Here’s an interesting survey: Sun wants to know how secure the default install for Solaris 10 should be. If you listened to the NetTalk on Wednesday, then you might be able to guess that your input here probably means more work for my team and me.

(I suppose I should really split work blogging from home blogging. Oh, well: work’s particularly fun right now.)

Stephen on NetTalk panel at sun.com

stephen@blueslugs.com (Stephen Hahn) — Wed, 05 May 2004 06:16:38 +0000

Stephen’s one of the so-called “rocket scientists” in the Solaris 10 Technical Discussion being offered as a NetTalk tomorrow. (It’s at 1pm PDT.) You can register for the live session (or for details about hearing a replay later).

(You would think folks would rather hear from computer scientists or software engineers or operating systems designers than rocket scientists on this kind of topic, but perhaps not…)

Elegant assembly

stephen@blueslugs.com (Stephen Hahn) — Wed, 14 Apr 2004 09:02:58 +0000

I had an enjoyable half hour assembling a piece of furniture from IKEA, before returning for some late night text composition. It goes much faster with a power screwdriver and a good set of bits. (Also, a 16oz hammer makes tapping in dowels directly satisfying.) But if you want to grab hold of ingenuity in a tangible sense, go build some Cubitec.

Dina and I are building a wall’s worth to hold the books in the office; great fun.

Recommended.

Sounds again

stephen@blueslugs.com (Stephen Hahn) — Fri, 09 Apr 2004 19:38:42 +0000

I’ve been using a dual Opteron as my primary desktop lately, but Solaris doesn’t support the sound device included on its motherboard. After asking around about the likelihood of support arriving, I was told that one possible path was to use a USB Audio device instead.

So I drove off to Fry’s to look around and found, as is always true when I look at PC peripherals, that there is a wide variety of such devices, ranging from a cheap plastic version with only a few features to US$100+ options from Philips and Creative.

I ended up selecting from the midrange, and bought the Hercules Gamesurround MUSE Pocket, which is a little chrome number with a big volume dial (which Solaris supports) that doubles as a mute button (which it doesn’t) and 5:1 surround sound (which I can’t test). (Generally the Gamesurround gets middling reviews, but as a USB Audio pod when no sound support is available, I like it.)

Recommended.

References:

Hercules Gamesurround product page
CNET reviews the Gamesurround, and provides a larger picture. (The included picture is from the Hercules site.)
Solaris 8 2/04 USB Administration Guide on USB Audio

The joy of :bufdo

stephen@blueslugs.com (Stephen Hahn) — Tue, 27 Jan 2004 19:41:50 +0000

If you use vim, then you may have discovered that its documentation, while extensive, is nearly impenetrable. One operation I often have to do is

$ for x in [list of files]; do
> sed -e '[some substitution]' < $x > $x.bak
> mv $x.bak $x
> done

since most Unix-based editors don’t offer multi-file replace. It turns out that, sometime between 5.3 and 6.x, vim has added the :bufdo command, so this operation is much simpler as

you needn’t leave the editor, and
you needn’t maintain a list of files, provided you’ve opened them all in the editor.

I’m posting to share this small tip, since when I canvassed a few folks at the office, no one seemed to have heard of :bufdo. (See “impenetrable documentation”, above.)

Dina gives poster at AGU

stephen@blueslugs.com (Stephen Hahn) — Wed, 10 Dec 2003 02:40:03 +0000

Dina presented her poster, “Creating Access to Data of Worldwide Volcanic Unrest”, at the AGU 2003 Fall Conference (in San Francisco) today.

A call for modern phones

stephen@blueslugs.com (Stephen Hahn) — Mon, 08 Dec 2003 04:31:04 +0000

I sent the following email to Design Witin Reach. It’s opinionated, making an almost drunken attempt to spray bile at manufacturers and retailers in various markets–but they have each failed in mediocre fashion, so I’m maintaining my position.

From: Stephen Hahn Date: Tue Nov 18, 2003 21:52:05 US/Pacific To: sales[@]dwr.com Subject: Telephones?

DWRians,

A class of items I would like to see offered by DWR is modern, design-conscious telephones, philosophically similar to the Modern Fan Company ceiling fans you've offered: a high quality, appropriately featured phone for the small/home office.

Other vendors' products have limitations:

Bang and Olufsen have generally overpriced models that are occasionally elegant but can be awkward, even clunky. The materials and colors are uninspired.
The "retro" models from Pottery Barn, Crate and Barrel, and the like, are minimally featured and only rarely well made.
The consumer electronics companies are pursing a button-based roadmap (more buttons: better phone?) and are almost exclusively cordless.
I'll skip the "my entire office should be made of wood" manufacturers who entomb helpless telephones, laser printers, and computers in a cherry shell.

Something like an updated version of the Taurus telephone (Andy Davey for Browns Holdings, 1988), with a caller ID display and a headset jack. Perhaps an aluminum base or a stainless steel handset, like Vertu/Nokia is offering in the mobile space.

(If you know of a manufacturer who is already producing such items, but you don't believe the product fits in the DWR catalog, please pass along a link or other contact.)

Thanks for listening, Stephen

DWR responded promptly, promising to be on the lookout. But I hypothesize that someone must be already making such a telephone–so I open up this topic for comments, suggestions, and links to manufacturers of a modern design telephone.

software:dockables page updated

stephen@blueslugs.com (Stephen Hahn) — Mon, 08 Dec 2003 04:08:11 +0000

I’ve reformatted the software:dockables page to bring it in line with the site’s general format; it’s still one of the more active entry pages for blueslugs.com (along with the grammar entry in Ben’s ABC collection).

Fixing a common/combined Apache log

stephen@blueslugs.com (Stephen Hahn) — Thu, 27 Nov 2003 22:04:48 +0000

I was trying another web server statistics package, awstats, based on a testimonial in Alan’s blog. Unfortunately, awstats expects a log file to be in the “combined” log format, and blueslugs.com has many entries in the default “common” format (prior to a switch for superior logging). A search showed no obvious small script for this, so I hope that providing massage_log.pl will save someone a few minutes or, at least, a few thousand log entries.

Dick Venezky Updates

stephen@blueslugs.com (Stephen Hahn) — Sun, 23 Nov 2003 01:06:22 +0000

Dina’s father has been sick the past year or so, and Dina’s been relaying information to friends and family via email. I’ve archived them all.

Caching minidns

stephen@blueslugs.com (Stephen Hahn) — Sat, 22 Nov 2003 20:59:06 +0000

I’ve started using analog to build reports from the blueslugs.com server logs. One of the tools on the site is minidns, which is a small Perl script that runs through the logs, replacing any IPv4 address matches with their DNS-resolved names (if defined). I’ve improved this slightly, by adding caching inside the application (on the grounds that your site is likely visited by the same communities over time) and, via Storable, to a state file (on the grounds that you probably run the analyzer over your logs rather regularly). [Get cachedns.pl.]

Is it worth it? Read on.

Here are some simple timed runs with the original (minidns.pl) and with cachedns.pl.

$ time perl minidns.pl < ./al.1000 > /dev/null
real 1m56.431s
user 0m0.030s
sys 0m0.030s

al.1000 is the first 1000 lines of an Apache server log. Our name service cache, nscd(1), and the DNS server we’re calling (and the perl(1) text) are now reasonably warmed up for subsequent callers.

$ time perl minidns.pl < ./al.1000 > /dev/null
real 0m11.210s
user 0m0.030s
sys 0m0.020s
$ time perl minidns.pl < ./al.1000 > /dev/null
real 0m12.943s
user 0m0.050s
sys 0m0.010s

So 1000 calls take a little over 10 seconds. Let’s run the caching version:

$ time perl cachedns.pl -c dns.cache < ./al.1000 > /dev/null
real 0m8.579s
user 0m0.090s
sys 0m0.020s

So the internal caching is maybe making a little difference. But let’s rerun with the now-populated cache file.

$ time perl cachedns.pl -c dns.cache < ./al.1000 > /dev/null
real 0m0.096s
user 0m0.070s
sys 0m0.010s

Since the first part of the log file is processed every night, our cache file means that we’re likely only going to perform a DNS lookup for new visitors to the site. (There are many sophisticated DNS resolvers-for-weblogs around, that use C++ or Python or threading or whatever. I just felt that a simple, understandable Perl version, with a boost, was enough for this little site.)

RSVP, a Flash game

stephen@blueslugs.com (Stephen Hahn) — Fri, 21 Nov 2003 08:12:17 +0000

Lifetime TV’s website offers a fun solitaire variant, which they call RSVP. Apparently, regular practice will make you an excellent party hostess. (“Now turn to the young lady on your left side, and say something charming…”)

Whimsical, well-constructed Flash application: recommended.

Stash Teas, especially peppermint

stephen@blueslugs.com (Stephen Hahn) — Wed, 19 Nov 2003 06:16:16 +0000

Cutbacks at work over the past few years have forced me to take action. As the coffee and teas supplier has moved the offerings from a wide selection of Bigelow teas to a small selection of Bigelow teas to an alternate tea vendor (nameless), the quality of my tea break has dropped so that, now, the tea dust captured by each desiccated bag was only a distant echo of the resonant saga of teas (and tisanes) through the ages.

Something had to be done. Knowing of no obviously heroic possibilities, I turned to ecommerce.

Stash Tea carries a variety of loose and bagged teas. Although I brew from loose teas when deadlines allow, my current workload insists on the highest possible efficiency–a second or three could be the difference between “above average” and “more than satisfactory”. My current favourite bagged tea is the peppermint, which is made from dried Oregon-grown mint leaves. (The Darjeeling is nice, but needs multiple bags to brew anything beyond a small pot; the batch of triple ginseng is weaker than I expected.)

Recommended.

Mark Kurlansky's Cod

stephen@blueslugs.com (Stephen Hahn) — Mon, 17 Nov 2003 04:08:12 +0000

Mark Kurlanksy, Cod: a biography of the fish that changed the world, 1997. [Paperback published by Penguin.]

A short book on the role of (mostly) Atlantic cod, which, beyond being yet another species-so-abundant-we-couldn’t-help-but-nearly-extinguish-it, was apparently the seventeenth century ocean-crosser’s equivalent of interstate service areas (in that great distances could be travelled with comparatively convenient access to food). Kurlansky offers various contemporary recipes for cod-based dishes in between chapters, although I’ve never seen true salt cod for sale out here in California. (Cloves have been on a multi-century decline from their peak as the “it” herb in the 1700s.)

The discussion never convincingly reconnects to present-day fishermen who are sampling the Grand Banks cod population, although the recollection of the richness of the Banks and the fishing culture of only fifty or sixty years ago is well told. Little discussion of even small-scale aquaculture for cod, or really of the cod ecology, but an excellent survey of the role of an effective natural resource in various periods in the history of North America and Western Europe.

OK.

Gigli: petit douteux

stephen@blueslugs.com (Stephen Hahn) — Mon, 04 Aug 2003 06:07:47 +0000

We saw the Affleck-Lopez movie, Gigli, last night in Daly City. Unlike past dubious outings, the theatre was fairly full—showing the loyalty of each star’s fans. (Well, J. Lo’s fans, in any case.) Here’s proof that we, along with approximately 400 000 other domestic ticket purchasers, were there opening weekend:

Thumbs up: Chicken of the VNC

stephen@blueslugs.com (Stephen Hahn) — Fri, 13 Jun 2003 06:24:40 +0000

I’ve used pretty much every VNC client available for Mac OS X over the past year. My preferred client now is Chicken of the VNC, even though the pun makes me cringe. Recommended.

An icon for GNOME

stephen@blueslugs.com (Stephen Hahn) — Tue, 25 Mar 2003 23:24:54 +0000

I recently started using Sun’s version of GNOME 2.0 as my primary desktop environment, and decided that I needed an icon to launch my favourite mail application, mutt. I’m not much of an artist or a GIMP expert, but I was able to combine the GNOME terminal icon with the mutt in the Mutt banner:

Feel free to use it, if it’s useful to you.

Forearms, wrists stressed in bowling bachelor outing

stephen@blueslugs.com (Stephen Hahn) — Sun, 16 Mar 2003 20:26:09 +0000

My friend Mike is getting married in about a week, so his best man Bryan rounded up a group of Mike’s friends and instructed them to head directly to Sea Bowl in Pacifica. I certainly hadn’t bowled since my wedding and probably not since moving to the US. However, I managed to average over 100 for four games, so sending the ball straight is still a good fallback strategy (as in golf) for those of us who lack actual bowling technique. Here’s Mike demonstrating his more refined delivery, honed during hours of candlepin bowling in eastern Massachusetts:

Generally, Neil was the dominant player, understanding the use of bowling balls of different weights for different situations and delivering with a consistency the rest of us could only envy:

Simon, Bryan, and I were glad to hand in our shoes, what with forearms tight from overuse, fingertips rubbed raw, and egos so bruised as to make a return to the alley unlikely in the near future:

I had to return to family duties, but the boys continued on to the pool tables. Whether either activity will be a portent of marital harmony, it is much to soon to say.

Faith via multiple choice

stephen@blueslugs.com (Stephen Hahn) — Thu, 13 Mar 2003 01:14:00 +0000

I think beliefnet‘s Belief-O-Matic is a pretty interesting Internet assessment quiz. The BOM asks you for the answers to twenty-odd questions plus the importance of this particular question to construct a parameter space that is then matched against a set of declared religious systems. The name’s pretty slick, too.

A "moorish" blog arises

stephen@blueslugs.com (Stephen Hahn) — Tue, 11 Mar 2003 18:46:37 +0000

My friend Alan’s new blog at bleaklow.com talks about being a drum-bashing Ranger rambling about England’s Peak National Park. It’s worth a visit, if only to divine how wasteful the North American sprawl form of suburbia is in comparison to the island form found in the UK. (Granted, it’s also detectible that Alan is on a remote island…)

Stephen gives talk at Brown

stephen@blueslugs.com (Stephen Hahn) — Sat, 15 Feb 2003 15:36:23 +0000

Stephen was at Brown today to give a talk, “Topics in resource management”, at the Technical Center for Scientific Computation and Visualization. He had a chance to walk around the campus and its surroundings and see what’s new.

In addition, old haunts like D’Angelos, East Side Pockets, and College Hill Books, were visited.

Dina gives talk at USGS

stephen@blueslugs.com (Stephen Hahn) — Wed, 22 Jan 2003 07:00:00 +0000

Dina gave a talk on “Creating a worldwide database of volcanic unrest” for the Volcano Hazards Team seminar series at the USGS, Menlo Park.

Aristocratic pear fails under load

stephen@blueslugs.com (Stephen Hahn) — Fri, 08 Nov 2002 06:49:03 +0000

High winds (20 – 30 mph, gusting to 50+ mph) caused the aristocratic pear tree in our backyard to topple at about 7:30 P.M. tonight. Dina and I heard the walls of the house shudder in response to the deep crack of the trunk yielding. A Swiss Army knife is shown for scale in the photos below:

For the record, here’s the NWS advisory:

SONOMA AND MARIN COUNTIES-NAPA COUNTY-SAN FRANCISCO COUNTY-
SAN MATEO COUNTY-SANTA CRUZ COUNTY-ALAMEDA AND CONTRA COSTA COUNTIES-
SANTA CLARA COUNTY-COASTAL MONTEREY COUNTY-
SALINAS VALLEY/INTERIOR MONTEREY COUNTY/SAN BENITO COUNTY-
300 PM PST THU NOV 7 2002
...WIND ADVISORY THROUGH THIS EVENING...
SOUTH WINDS OF 20 TO 30 MPH WITH FREQUENT GUSTS OF 40 TO 50 MPH ARE
EXPECTED THROUGH THIS EVENING ACROSS THE SAN FRANCISCO AND MONTEREY
BAY AREAS. THE STRONGEST WINDS WILL OCCUR NEAR THE OCEAN AND IN THE
HILLS.
A POWERFUL PACIFIC STORM WAS APPROACHING THE CALIFORNIA COAST LATE
THURSDAY AFTERNOON. STRONG AND GUSTY SOUTH WINDS WILL PRECEDE THE
THE SURFACE FRONT. WINDS WILL SWITCH TO THE SOUTHWEST AND
DIMINISH AFTER FRONTAL PASSAGE AROUND MIDNIGHT. WINDS WILL INCREASE
ONCE AGAIN ON FRIDAY AHEAD OF ANOTHER FRONT...ALTHOUGH THEY ARE NOT
EXPECTED TO BE QUITE AS STRONG AS THIS EVENING.
PERSONS IN THE ADVISORY AREA ARE URGED TO TAKE ACTION TO SECURE LOOSE
OR LIGHTWEIGHT OUTDOOR OBJECTS. HIGH WINDS CAN TOPPLE TREES...BLOW
DEBRIS ONTO ROADWAYS...AND DOWN POWER LINES.

Now using Movable Type

stephen@blueslugs.com (Stephen Hahn) — Thu, 07 Nov 2002 05:37:16 +0000

blueslugs.com is now being managed using the Movable Type (http://www.movabletype.org) personal publishing system.

Mt. Shasta fly-over

stephen@blueslugs.com (Stephen Hahn) — Thu, 31 Oct 2002 07:09:01 +0000

We enjoyed a spectacular view of Mt. Shasta during a recent flight to Portland, OR. (Dina was attending a volcanology workshop; Ben and Stephen planned long walks around the grid layout of Vancouver, WA–which should use the motto of “the city that grows before your eyes” given the amount of construction we saw.) From Ben’s window seat, Shasta from crusing altitude:

Ben survives weekend with Dad

stephen@blueslugs.com (Stephen Hahn) — Mon, 28 Oct 2002 20:00:00 +0000

With Dina travelling, Ben nearly had to fend for himself under his father’s care. Here’s Ben preparing for an expedition to increase his available store of supplies at a nearby grocery store: