Arktronic.com

A redone blog, again

Fri, 28 Dec 2018 22:20:00 +0000

I'm just about due for another recreation of my blog, so here we are. Last time, I switched from a CMS to a statically generated website, using my own static site generator. This time, I'm taking it a step further and, in addition to recreating my static site generator in .NET Core, harnessing the power of GitHub Actions for should be used, with args telling the Filter action that I want the branch name to match "master".

The "do all the things" action is where it gets weird. I didn't want to publish my own GitHub Action for building a static site using my generator for a few reasons. First, as I mentioned earlier, my generator isn't ready yet. Second, I was having trouble finding the right documentation for precisely how to publish a GitHub Action, what the necessary components are, and how everything works together. And third, I fully expect the publishing process and requirements to change during the beta period, and I'm not overly interested in keeping up with all of these changes at the moment. So, I chose to do this in a bit of a backwards way. I decided to use the Docker CLI GitHub Action to build a Docker image that just happens to perform all the steps I want during the build. It's mildly horrifying, but here's what it currently looks like:

    action "Docker-Staging" {
        uses = "actions/docker/cli@76ff57a"
        needs = ["Filter to master branch"]
        args = "build --build-arg GITHUB_TOKEN --build-arg GH_SOURCE_REPO=\"arktronic/arktronic.com--source\" --build-arg GH_SOURCE_BRANCH=\"master\" --build-arg GH_DEST_REPO=\"arktronic/staging.arktronic.com\" --build-arg GH_DEST_DEPLOY_KEY --build-arg GH_CNAME=\"staging.arktronic.com\" --build-arg ACCEPT_RISK=\"1\" .github/main_support"
        secrets = ["GITHUB_TOKEN", "GH_DEST_DEPLOY_KEY"]
    }

Let's dissect that mess. First, we have the familiar uses directive for

Optimize for cognitive load

Fri, 30 Dec 2016 16:25:00 +0000

I recently read a rather interesting post by Martin Fowler regarding was developed by John Sweller, an Australian professor specializing in educational psychology. According to CLT, when humans are learning, there are three types of cognitive load that occur: intrinsic, extraneous, and germane. Intrinsic cognitive load is effectively the difficulty that the topic being learned presents by itself. Not much can be done to affect that if the topic is to be learned. Extraneous cognitive load is, as the name suggests, considered unnecessary. It is created by the manner in which information is presented. For example, the extraneous cognitive load would be much higher if I were to verbally describe a geometric shape like a square, rather than just show a picture of one. Finally, germane cognitive load is involved in processing and construction of has been growing in popularity in recent years. Numerous articles have been written about optimizing visual and interaction design to reduce extraneous cognitive load. Unfortunately, I've not seen much talk about this in regards to software design or architecture. It seems, we software engineers tend to focus more on the technical and less on the human side. But we need to take both into account when working on real (i.e., not personal/toy) projects if we want to increase maintainability and ease of development.

Application of CLT

My interpretation of cognitive load as it applies to software design is rooted in how many steps you must go through to understand the code involved in executing an API call, a workflow, a use case. This is, of course, a multifaceted problem with no clear generic solution. In most situations, in order to understand a particular flow to a sufficient degree, you don't need to know all the minutiae involved in it. For example, if you're working on a RESTful API, you rarely (hopefully never) need to debug down to the level of TCP connections or Ethernet frames. Often, you don't even need to know the exact processes your framework of choice uses to translate an HTTP request to an appropriate function call in your code. And, depending on what aspect of the codebase you're trying to understand, you can often skip other important code in order to focus on what currently matters to you.

Functions

So how does all this affect software design? Let's start with function length and go up from there. From a CLT perspective, a very long function that encompasses a significant amount of data processing will have high extraneous cognitive load when you analyze it because it will likely deal with multiple states that you have to keep in mind at all times while mentally processing the various permutations of conditionals and loops, what happens inside each of them, and how those previous decisions affect further conditionals and loops later on. This is a lot of information to keep track of, and so is inefficient for us to process.

At the same time, very small functions will also have high extraneous cognitive load. The above load of dealing with state is replaced with the load of incessant context switching when you have to look at many different functions, then back again down a stack, then forward again from the next step, and so forth. This causes the same problem of presenting too much information to keep track of, and so is still inefficient.

The ideal function length is somewhere inbetween. I hesitate to give concrete numbers, since there are multiple conflicting models of human part 1 I explained some of the basic concepts behind the hackintosh, and in part 2 I showed the easy way to create a hackintosh VM using VirtualBox. In this post I'll show the harder, but more flexible, way, which will allow you to have a custom screen resolution as well as connectivity to iCloud and iMessage.

Caveats

As I mentioned last time, there is also currently no way to have accelerated graphics in VirtualBox due to a lack of drivers, and there may be problems with audio as well. I am still unaware of any ways to work around this.

The guide

The instructions here work as of macOS Sierra 10.12.1/10.12.2 and VirtualBox 5.1.10. If you are using other versions, things may have changed, and so there is no guarantee that this guide will be accurate.

Step 0: Prerequisites

In order to follow this guide, you will need:

The 16GB or higher installer disk with the macOS installer from part 2
A real Mac with Internet access
A VM host machine with VirtualBox installed and plenty of free space

Step 1: Download and install Clover

Download the latest part 1, Clover is a bootloader capable of emulating EFI and various related firmware components. These components need to be configured so that upon bootup Clover sends the necessary information to macOS for correct operation. These configuration settings are stored in a file called config.plist. When placed in the correct location, this file (and optionally other associated files) will be read by Clover and used to configure the information sent to macOS.

In order to expedite matters, I've created a "base" configuration file that can be used as a starting point. of the Clover Configurator.

Open the downloaded config.plist file in Clover Configurator.

In the "SMBIOS" section, click the "magic wand" button to create an emulated Mac profile. Select the type of device you want to emulate, and then select the specific version from the primary dropdown. After that, click the two "shake" buttons to create a random serial number. After clicking OK, copy the "Serial Number" field to the "Board Serial Number" field and append five random letters or numbers to the end. Then copy that resulting board serial number into the "MLB" field under the "Rt Variables" section. This is all demonstrated here for reference:

After you've completed the above steps, open a new Terminal window and type in uuidgen (this will output a UUID on the next line). Copy this UUID to the clipboard. Go back to Clover Configurator and the "SMBIOS" section, and paste the UUID into the "SmUUID" field.

To set your desired screen resolution, go to the "Gui" section, and select the screen resolution you want from the dropdown. The default resolution is 1920x1080.

To set the amount of RAM you want to give to the VM, go to the "SMBIOS" section, and change the Memory information accordingly. The amount (marked "Size") is the only value that you should change if you'd like to choose anything other than the default of 4096MB.

Then you can close Clover Configurator. The config.plist file should have your modifications saved in it.

Step 3: Finalize the macOS installer changes

You'll need to copy the config.plist file to two locations: one on the main "Install macOS Sierra" partition next to the Clover installation package, and one on the EFI partition so that Clover will see it. The installer disk's EFI partition should already be mounted after Clover's installation process. If not, you can mount it like so:

sudo mkdir /Volumes/EFI
sudo mount -t msdos /dev/disk2s1 /Volumes/EFI

If necessary, replace disk2s1 above with the appropriate disk number. Note that s1 should still be used, as the EFI System Partition should always be partition number 1.

Copy the file to EFI/CLOVER on the EFI partition, replacing the existing config.plist there. After you've copied the file to both locations, eject the installer disk, unmounting all partitions.

Step 4: Create the VM

Open VirtualBox on your VM host machine and create a new VM:

For simplicity, use the above settings. In later screens, select the same amount of RAM as you chose in Clover Configurator (4096MB by default), and create a dynamically allocated 60GB VDI virtual hard drive.

Step 5: Configure the VM

Certain VirtualBox settings are easier to configure via the command line, and some are just unavailable in the GUI. The VBoxManage command line tool allows for terminal-based control over VMs.

IMPORTANT: Before using VBoxManage, close all VirtualBox windows! Otherwise, certain settings may fail to apply, or you might even end up with a corrupted VM.

Enter the following commands to further configure the macOS VM, replacing the screen resolution with what you chose in Clover Configurator and the (copyrighted Apple key) with the actual key that you can easily find online:

VBoxManage modifyvm macOS --firmware bios --vram 128 --usb on --usbehci off --usbxhci off

VBoxManage setextradata macOS "CustomVideoMode1" "1920x1080x32"

VBoxManage setextradata macOS "VBoxInternal/Devices/smc/0/Config/DeviceKey" "(copyrighted Apple key)"

Step 6: Connect your installer disk

This should be a simple step, but unfortunately VirtualBox makes it difficult. Although it is possible to connect USB devices to virtual machines, you cannot normally boot from them, and that is precisely what we need to do. Luckily, VirtualBox does provide a (and many others) on InsanelyMac, the Clover are among those I found quite useful. And it goes without saying that those who developed the tools and systems used here deserve the most credit.

Virtual Hackintosh, part 2: the easy route

Mon, 12 Dec 2016 01:15:00 +0000

In part 1 I explained some of the basic concepts behind the hackintosh. In this post I'll show the easy way to get a VirtualBox-based hackintosh system up and running.

Caveats

Although the steps in this post are relatively simple and straightforward, that simplicity comes with a price. There are at least two issues that will detract from the experience: available screen resolutions are limited, and iCloud/iMessage connectivity does not work.

Aside from the above issues, there is also currently no way to have accelerated graphics in VirtualBox due to a lack of drivers, and there may be problems with audio as well. I've not figured out any way around that, unfortunately.

The guide

The instructions here work as of macOS Sierra 10.12.1 and VirtualBox 5.1.10. If you are using other versions, things may have changed, and so there is no guarantee that this guide will be accurate.

Step 0: Prerequisites

In order to follow this guide, you will need:

An empty 16GB or higher flash drive or external hard drive
A real Mac with plenty of free space and Internet access
A VM host machine with VirtualBox installed and plenty of free space

Step 1: Create an installer disk

The first step in making a hackintosh is the creation of installation media. Start by downloading macOS Sierra from the App Store on your real Mac:

You should see it in Applications after the download has completed:

Connect your external drive and use the Disk Utility to erase it, using the following settings:

You should leave the name "Untitled" as is, since it'll make the next step easier. And besides, it'll be overwritten in the next step anyway.

Open a new Terminal window and enter the following:

sudo /Applications/Install\ macOS\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/Untitled --applicationpath /Applications/Install\ macOS\ Sierra.app

After entering your password and confirming the action, you'll need to wait while your drive is reformatted and macOS installer data is copied onto it. If you are using a slow USB flash drive, be prepared to wait.

Once the process completes, eject the drive.

Step 2: Create the VM

Open VirtualBox and create a new VM:

For simplicity, use the above settings. In later screens, select 4096MB RAM, and create a dynamically allocated 60GB VDI virtual hard drive.

Step 3: Configure the VM

Certain VirtualBox settings are easier to configure via the command line, and some are just unavailable in the GUI. The VBoxManage command line tool allows for terminal-based control over VMs.

IMPORTANT: Before using VBoxManage, close all VirtualBox windows! Otherwise, certain settings may fail to apply, or you might even end up with a corrupted VM.

Enter the following commands to further configure the macOS VM, replacing (copyrighted Apple key) with the actual key that you can easily find online:

VBoxManage modifyvm macOS --vram 128 --usb on --usbehci off --usbxhci off

VBoxManage setextradata macOS "VBoxInternal/Devices/smc/0/Config/DeviceKey" "(copyrighted Apple key)"

Next, choose a

First, connect your installer disk to your VM host machine. Next, you'll need to determine its identifier on your machine. In Windows, you can do so on the command line by entering the following:

wmic diskdrive list brief

In *nix, you can inspect the output from the mount, df -h, lsblk, or parted -l commands.

Determine which disk represents your installer disk and remember its Windows DeviceID (usually in the format of \\.\PHYSICALDRIVE#) or *nix device name (usually in the format of /dev/sdX or /dev/disk#). It is important to note that you should use the name of the device itself and not of its partitions.

Next, create the raw disk access file. In Windows, you will need to launch a new command prompt as Administrator. In *nix, you may need to prefix the command with sudo. Enter the following:

VBoxManage internalcommands createrawvmdk -filename C:\usb.vmdk -rawdisk \\.\PHYSICALDRIVE#

Replace the filename above with an appropriate *nix path if you're not on Windows, and replace the physical drive identifier as needed.

In the same administrative command prompt, enter the following:

VBoxManage storageattach macOS --storagectl SATA --port 2 --type hdd --hotpluggable on --medium C:\usb.vmdk

Replace the filename above if necessary.

Step 5: Install macOS

Launch VirtualBox as Administrator in order to allow it to access the raw disk. Then, start the macOS VM. If VirtualBox asks to pick a startup disk or ISO, just cancel out of that dialog.

Once macOS boots up, use the Disk Utility to format the 60GB disk and then install macOS onto it.

After the installer completes, there will be a first run setup wizard. When prompted to sign in with your Apple ID, select "Don't sign in" since you won't be able to anyway. Also, when prompted to send diagnostics and usage data to Apple, uncheck that box. Since this isn't real Mac hardware, it would not be helpful for Apple to try diagnosing any issues from unsupported configurations.

Finishing up

Once macOS is up and running, you're effectively done. Congratulations!

At this point, it's a good idea to shut down the VM, go to its storage settings, and remove the raw disk access VMDK file. You may want to remove it from VirtualBox's Virtual Media Manager as well.

If you would like to see the normal Apple start up screen instead of the verbose text mode, you can enter the following after closing all VirtualBox windows:

VBoxManage setextradata macOS "VBoxInternal2/EfiBootArgs" "usb=0x800 keepsyms=1 -serial=0x1"

The default boot arguments contain "-v", which enables verbose mode. The above command will overwrite those defaults to remove verbose mode.

Finally, you will want to disable power saving. VirtualBox seems to dislike it when the guest OS tries to sleep. Go to System Preferences, select Energy Saver, and disable both the computer and the display from going to sleep in the guest OS.

Next time

In part 3, I'll explain the more difficult route of creating a virtual hackintosh using Clover, which will allow for more screen resolutions and iCloud/iMessage connectivity.

Virtual Hackintosh, part 1: the concepts

Sat, 10 Dec 2016 17:35:00 +0000

Virtualization powers a lot of infrastructure today, and there have been countless advances made in this field during the last few years. CPUs support more and more advanced hypervisor scenarios, while operating systems gain better and better native virtualization capabilities. Today, even macOS is virtualizable to a significant degree using free, open source tools.

In this post, as well as parts 2 and 3, I'll be focusing on using VirtualBox as the VM host, because it is open source, cross-platform, and it generally works well for this purpose. There are ways to get VMware and QEMU/KVM to host macOS; if you're already in one of those ecosystems, you should be able to find an online guide to help. I'm unaware of any successful attempts at getting Hyper-V to host it, but it could conceivably be done.

Motivation

The only reason why I decided to attempt virtualizing macOS is because it sounded like an interesting challenge. I have no particular desire to use that operating system, but finding a way to get it working is intriguing to me.

The theory and the terminology

If you're new to the hackintosh scene, you might feel a bit overwhelmed with jargon: DSDT, Clover, kext, Chameleon, SMC, and so forth. Let's start with macOS hardware requirements and go from there. Disclaimer: I can't guarantee the accuracy of the information provided here. This is just my attempt to reconcile the various pieces of info floating around on the web. If something is incorrect, please let me know, and I'll fix it.

Hardware requirements

Over the years Apple has released quite a few devices that run modern versions of macOS. They all have a couple of things in common: they run on Intel CPUs, and they have specialized hardware to differentiate a Mac from a regular Intel-based machine.

This brings us to the first term: SMC (System Management Controller). The SMC has multiple functions, including controlling LEDs and power management, but for our purposes it only has one interesting function: identifying the hardware that it's running on as a genuine Mac. macOS checks the SMC for a specific string upon booting up, and if the string is missing or corrupted, it will crash with a kernel panic.

This string check is done using an aptly-named kext (kernel extension) called Dont_Steal_Mac_OS_X. A kernel extension is effectively a dynamically loadable kernel module - code running with very high privileges inside kernel space. There are kexts that function as hardware drivers, for example.

Macs have other common hardware found in many non-Mac machines, such as audio cards, graphics cards, wireless NICs, and so on. Since Apple only has to support a limited number of these devices, it can sometimes be a challenge to get certain functionality working on unsupported hardware. Custom kexts are often the solution.

In order to determine what hardware is present, macOS enlists the help of Intel's EFI (Extensible Firmware Interface), a modern replacement for the venerable BIOS (Basic Input/Output System) firmware, which dates back to the 1970s. In addition to allowing macOS to boot, EFI contains (or allows access to) a lot of metadata in various tables that macOS uses to look up what hardware it needs to initialize. One of these sets of tables is the SMBIOS (System Management BIOS). It contains information about the currently running firmware, as well as metadata about the motherboard and certain peripherals that are attached to it. For example, macOS uses SMBIOS information to determine how much RAM the computer has and which RAM slot(s) are occupied.

EFI also allows access to various parts of ACPI (Advanced Configuration and Power Interface), which has more information about the computer. Part of ACPI is the DSDT (Differentiated System Description Table), which contains system metadata as well as executable code to allow for correct functionality of the operating system, especially when it comes to things like power management.

Software solutions

On a real Mac, the EFI, SMBIOS, ACPI tables, and various other firmware components are all tuned for best compatibility with macOS. On regular machines, they tend to be tuned for compatibility with Windows, or sometimes not tuned at all, or even tuned incorrectly. In order to solve the various issues created by these incompatibilities, a custom bootloader can be used to tweak the firmware information that macOS sees. Chameleon is one such bootloader. Clover is another. I'll be focusing on Clover in these posts, as it is the current preferred bootloader in the hackintosh community.

Clover functions as an EFI emulator, presenting its customizable view of the computer, including all of the aforementioned tables and other firmware components, to the operating system that it boots. It's able to function within an existing EFI or UEFI environment, as well as within a legacy BIOS one.

The successful creation of a hackintosh essentially boils down to correctly configuring Clover. Although there is sometimes a need for custom kexts to enable more or better functionality of various components, that is generally not needed for basic operation. Custom DSDT modifications, which are presented by Clover to the operating system, will often solve many problems ranging from graphics card initialization to power management, and a lot more.

Configuring a virtual hackintosh is similar to configuring a physical one, with an important difference in that the emulated hardware is nearly identical across most machines. Pretty much the only hardware that is not the same is the CPU, and there are workarounds to address CPU-specific issues.

Next steps

In the next part, I'll explain the easy, though limited, route to creating a virtual hackintosh. In the third part, I'll explain the more difficult, though more functional, route. Stay tuned!

Project PiNES

Thu, 11 Feb 2016 06:00:00 +0000

There are quite a few projects out there that stuff a

The hardware

In order to use an RPi inside an NES console, we needed to figure out how to expose necessary connections - HDMI and power. So off to eBay we went and ordered some mountable connectors. Having decided not to touch the original NES connector locations due to space and cabling limitations, we opted for making new holes in the case, starting with templates:

We explored different options for hole locations, as well as the possibility of adding generic USB ports:

The board sitting on top of the RPi is the

In part 5 we discussed how integration testing is used to improve the quality of software at a higher level than unit testing and when it should - and shouldn't - be used. As I mentioned in that post, integration tests aren't suited for testing functionality that relies on slow external systems like databases and web services or functionality that touches the UI. System testing is most useful in these areas.

While integration testing provides valuable feedback on the interaction between units, system testing provides equally valuable feedback on the functionality of the application as a whole. Recall the pyramid structure from part 2. Just as integration tests see more of the application than unit tests, so do system tests see more than integration tests. Where integration tests can catch issues that unit tests cannot, system tests can catch issues that integration tests cannot: they exercise the interactions between major application components as well as interactions with external services.

Adding system testing to your process

At the most basic level, system testing involves getting your application installed in an environment that mimics production as closely as possible and using an automation tool or framework to access that application in the same way that a real user would. This type of testing is very similar to how an application would be manually tested. However, setting up an appropriate environment for system testing can be challenging.

Where to run system tests

System tests should be run on local development machines, at a minimum, in order to facilitate easy and quick development as well as maintenance of the tests themselves. Ideally, the tests should also run as part of a part 1, people may stop caring about failures if they happen so often, and then your automated testing is all but useless.

Bad timing

One of the most common and most annoying issues with system tests is a timing-related problem. The test tried clicking a button on the main screen before a modal dialog finished closing. The test didn't wait long enough before checking the value of a text box while the application was updating the screen. The test tried typing into an entry field before the screen finished loading. These are all realistic examples, and they can be rather frustrating. There is more than one way to address these timing issues. The most obvious is to add a simple delay (or "sleep") to the execution of the problematic test. Unfortunately, this is usually also the worst way, in terms of both stability and performance. A delay of one second, as an example, may solve a particular timing failure on your machine, but if someone is running the same test on a slower (or busier) machine, it may not be enough time. Conversely, if you "play it safe" and add a ten second delay, then you slow down the execution of these tests needlessly for everyone who runs them.

A better way to address timing issues is to have your tests look for specific events to happen before executing the next step. For example, you could verify that the window handle of the modal dialog no longer points to a valid window before trying to click the button on the main screen. In the case of a textual update happening too slowly, you could either wait for a signal from the application that everything is ready (if such a signal exists) or you could check for the expected value in a loop with a reasonable timeout. In this case, having a timeout of ten seconds wouldn't be nearly as bad as forcing a ten second wait on everyone because the loop would exit as soon as the correct value is seen, and if the correct value is not seen in the ten seconds, the test simply fails.

Unrealistic steps

Another issue that happens with system tests is the execution of unrealistic steps. This often involves clicking a button that is invisible or otherwise obscured such that a real user would not be able to click it. This usually happens because a test was written to push an event directly to an object via the automation framework, which generally doesn't know whether the object is truly available for the user to interact with. Executing unrealistic steps will sometimes cause failures because the application isn't yet ready to accept whatever input the test is sending it. Other times the application may get into an unpredictable state because a series of events occurs that would normally be impossible for a real user to execute.

These issues can be avoided by simulating input as realistically as possible. For example, instead of sending a click event directly to a button, you could determine the button's coordinates and then send a mouse click event to the main application window at those coordinates and let the application propagate the event down to that button - or to whatever else may be there on top of the button.

Systemic fixes

Very often the reason for many of these common issues is that the testing or automation tools don't provide a good way to do what you need to. This is a systemic problem, and the best way to avoid such problems is to create systemic fixes. If your UI automation tool requires three method calls to be able to get the coordinates of an object and send a click event to the application at those coordinates, you should create a single wrapper method for this and encourage everyone to use it. If the automation tool doesn't provide an easy way to wait for a text box to contain an expected value by constantly polling it until a timeout occurs, create this method yourself. Essentially, if you find that you're writing the same boilerplate test code in different tests, extract it to a helper class or module. This way you can improve the stability and reliability of most, if not all, of your system tests.

Wrapping up

In this series of blog posts I've explained the value of automated tests at the unit, integration, and system levels. I've also discussed the types of issues that you'll often encounter with these tests and how to solve - or at least mitigate - many of them. If you implement a robust testing strategy for your application, you will benefit immensely from it: not only will it be easier for you and others to implement changes and fixes without breaking existing functionality, but you will also have increased confidence in the overall stability of your codebase.

Go forth and boldly create!

Automated software testing, part 5: integration testing

Sun, 27 Dec 2015 03:00:00 +0000

As you will recall from part 4, we left off with a problem. All the unit tests for the

The implementation portion in the above link is unchanged from the relative time formatter v2 implementation introduced in part 4 - bug included. The tests, however, are quite different. You can see that we're still mocking out the current date, since the need for a stable and reproducible environment still exists, but instead of having separate suites testing each unit individually, there is a single integration suite whose setup function instantiates all three classes and passes the day and hour formatter instances to the primary formatter exactly as a real system would.

If you run the tests, you'll see that the first two pass, while the second two fail, thus exposing the bug. Note that, while there were a lot of unit tests for these three classes, there are only four integration tests. This is by design. Integration testing is not intended to cover every possible scenario that could occur in the individual units. Its focus is on covering the major interactions between the units. Arguably, the integration suite could have only contained two tests - one for interacting with the hour formatter, and one for interacting with the day formatter. I chose to have two tests per formatter to verify that dates in the past and in the future are working as intended. I could see a possible desire to split those up into separate classes, so I figured that a couple of extra tests might help.

Let's fix the implementation.

Avoid creating integration tests that need to communicate with slow external systems such as databases, remote file systems, and web services. Creating a temporary in-memory database and using it in an integration test is perfectly fine, but if you need to communicate with a real instance of a remote database, you're introducing a complex, relatively slow, and potentially brittle dependency. Integration tests should be reasonably fast, and they should not fail just because some database instance on another server was down for maintenance.

Most of the time you should also avoid creating integration tests that touch the UI. Those tend to get overly complicated and brittle when isolating the set of units you're trying to test. It's generally better to leave that to system testing, which will be covered in part 6.

Automated software testing, part 4: unit tests and test doubles

Thu, 03 Dec 2015 03:30:00 +0000

Navigation: part 1 | part 2 | part 3 | part 4 | part 5 | part 6

In part 3 I discussed some of the good habits surrounding unit testing, such as determining how much to cover in your tests and avoiding the temptation to test internal implementation details. In this part we'll continue to explore unit testing with a focus on verifying external interactions in a controlled and isolated environment.

First, I'd like to clarify some terminology. When I'm talking about "external interactions", I'm mostly referring to a unit interacting with another unit. Generally, this does not include a unit interacting with an underlying framework or language feature because most of the time that kind of interaction falls under the scope of "internal implementation details" and, as such, shouldn't matter to our tests.

Verify external interactions

In the last post, I showed how testing the outputs of a unit based on its inputs can work. An external interaction is just another type of input and/or output, so it's just as important to verify: units should be able to handle their respectively defined inputs and produce expected outputs, but you need to be certain of what those inputs and outputs actually are. While integration tests verify that multiple units are communicating correctly with each other, they are limited in two important ways: they cannot - and should not - see the exact data that flows between the units, and they cannot - and should not - cover all possible inter-unit communication scenarios. The second point there is especially important because it's easy to get stuck trying to create the "perfect" scenario where the data transforms and flows in just the right way to get to a very specific outcome, but that shouldn't be your focus when crafting integration tests. More on that in the next post.

Both of the above limitations can be addressed to a sufficient extent with unit testing. Instead of instantiating real dependencies, a unit test should create

The relative time formatter was first introduced in part 3. In this upgraded version, the single "class" has been split up into three different ones - the primary formatter, the hour formatter, and the day formatter. While the first version could only format days, this one can format hours as well. Getting the desired result string from the relative time formatter still consists of instantiating a RelativeTimeFormatter and calling its format function. What's different is that you now need to pass an instance of the hour formatter and an instance of the day formatter to the primary formatter's constructor. I should also mention that this isn't necessarily idiomatic JavaScript. The concepts demonstrated and discussed here are generic and applicable to many languages and frameworks, which means we can't take advantage of some of the inherent strengths unique to JavaScript.

Looking at the test section, you'll notice that there are three test suites, since we now have three classes. If you click the run button, you'll see their outputs and that they all pass. The most interesting test to look at is the one that exercises the RelativeTimeFormatter itself. In addition to the mocked out clock that was discussed in the last post, it has an hourFormatterStub and a dayFormatterStub.

Setup, execution, verification

Let's look at the "should format less than 8 hours in the future using the hour formatter" test. After defining a couple of variables, we have the following line:

hourFormatterStub.and.returnValue(expectedValue);

This line configures the hourFormatterStub to return expectedValue whenever that stub is called. Since expectedValue is defined earlier in the test as a generic string (that happens to look very different from what the real formatter might return) we can check for it later in the test to verify that this string was returned to us from our stubbed out hour formatter via the real RelativeTimeFormatter. Naturally, this line of code must be executed before we try calling RelativeTimeFormatter.format, since it sets up required functionality on a dependency. Hence, it could be called a setup step.

Once the necessary setup steps are taken, we call into our instance of the real RelativeTimeFormatter, and save the return value for later. This is the execution step.

After calling the format function on our instance of the RelativeTimeFormatter, we have some expect lines, which check that the stub was called the correct number of times, that its invocations contained the expected arguments, and finally, that the RelativeTimeFormatter returned what it got from the hourFormatterStub. Collectively, these lines could be called the test's verification step.

You may have noticed that the relative time formatter tests tend to conform to the setup-execution-verification structure. This is a pretty typical way to write unit tests that are more complex than a single-line return value verification test.

Refactoring tests

After writing your tests, you may notice that many of them have the same setup steps. That's a good indication that you should extract those steps into the beforeEach (or equivalent) of the test suite. Similarly, if common tear down steps are needed, those can go into afterEach. Such refactoring will simplify the tests, making them easier to read, modify, and debug. You can see how this common setup and tear down was done in the relative time formatter test suite. Common verification steps can also be refactored into separate functions that you call from your tests, but you should be careful with this, as it is entirely possible to over-optimize and make the tests more complicated instead of simplifying them.

An uncaught bug

Although all the tests pass, there is actually a major bug in the code of the new relative time formatter. The problem lies in the fact that the hour formatter and the day formatter take different arguments in their respective format functions: the hour formatter takes a single diffValue in milliseconds, while the day formatter takes now and target Date objects. Individually, these differences make sense, since the calculations that are performed are different between the hour formatter and the day formatter. However, the primary formatter that we've been testing with the hour and day formatter stubs isn't calling the day formatter correctly. We've assumed that the calls are identical and wrote the implementation as well as its tests with this faulty assumption in mind.

The important takeaway here is that this is not strictly a unit testing problem. While it's true that the "should format more than 8 hours [...] using the day formatter" tests are flawed, as is the implementation that is being tested, this kind of mistake is easy to make. An act as innocent as copy-pasting the previous hour formatter tests and trivially modifying them to use the day formatter would cause this problem, and copy-pasting is done a lot more often in the software development world than we like to admit.

The solution to the problem does not lie in disabling Ctrl-C and Ctrl-V. Rather, it is in testing the interactions between real units. This is integration testing, and it will catch such bugs. We'll discuss it in more detail in part 5.

Automated software testing, part 3: unit tests

Tue, 10 Nov 2015 04:00:00 +0000

The first and second parts of this blog post series are an overview of automated software testing, with part 1 focusing on the why and part 2 a pretty high-level how. Now, with the third part, let's delve deeper into the how of unit tests specifically.

As I explained in part 2, unit tests shouldn't see the "big picture". They must be purposely designed to test their units and nothing else. It's often tempting to create unit tests that span multiple units in order to get a more realistic representation of state, but you should strive to avoid this. Such tests should be added at the integration level instead.

Let's get interactive

Part of my goal is to explain many of the concepts here with real examples instead of just words, because I think that running a test for yourself and seeing it pass or fail is much more powerful than simply reading about it. To that end, I created an online JavaScript runner/tester, which lets you write JavaScript code and tests and run them immediately in your browser.

I've chosen Jasmine, a popular testing framework, to help show the concepts below. Please take a moment to look at Jasmine's

On the left, you'll see a function that returns the familiar string. This is the code that is to be tested. On the right, you'll see a test suite defined using the describe function, and two tests ("specs" in Jasmine parlance) defined using the it function inside. If you run the tests, you should see something similar to the following in the test output:

Jasmine v2.3.4 started.
Hello World function
    should exist
        [passed]
    should return the expected value
        [passed]
Hello World function: finished
Test run complete.

Both tests passed, hooray! You can play around with the helloWorld function and its tests, and run them again to see what happens when failures occur. Jasmine's syntax is intended to assist developers with using . This also serves as a decent approximation of classes with public and private methods in languages that support these concepts natively. For the next example, then, let's say we want to create a relative time formatter - something that will take a date object and return, for example, "5 days ago", based on the difference between that date object and now.

Test doubles

While the above example is relatively isolated from external dependencies, it does rely on one: the current date/time. Since we can't predict when our tests run, we have to have a way to control what the formatter thinks is the current date/time in order to have meaningful tests. Luckily, Jasmine provides a way to "stop time" at a desired point by replacing the real Date object with a fake one. Faking objects is an example of using part 4, I'll to go into more detail on test doubles (and spies) and how to use them to verify a unit's external interactions.

Automated software testing, part 2: the types

Tue, 03 Nov 2015 04:00:00 +0000

Navigation: part 1 | part 2 | part 3 | part 4 | part 5 | part 6

In part 1, I explained some of the reasoning behind automated software testing. Now let's explore a few of the more common types of automated tests that we often deal with.

An overview of the common test types

As I mentioned in the previous post, the common types that I'll focus on are unit testing, integration testing, and system testing.

A unit test, as the name implies, is intended to test a small "unit" of code. Most often, in object-oriented programming, such a unit would be equivalent to a class. It is very important to only test that unit and not increase the scope in these tests.

An integration test's purpose, fundamentally, is to test the interaction between two or more units. Realistically, integration tests are useful when multiple units (often from multiple tiers, to borrow a term from metrics. At best, they can function as merely a hint that something might be wrong. For example, 20% code coverage is probably bad. Crucially, 100% code coverage is also bad! Remember all that stuff about brittle tests? That's what 100% coverage will usually get you.

Design matters

The design and architecture of your application can make or break your ability to write good tests. In general, a more will allow stubbing/mocking to be downright simple and fun, and let you easily isolate units for testing.

Next steps

All of this theory is nice, but until you get enough practice determining what tests to write and how to write them, you'll make silly mistakes. It's okay, we've all been there. Keep practicing and you'll get better at it!

Check out part 3, a more in-depth discussion of the first type, unit tests.

Automated software testing, part 1: reasoning

Mon, 02 Nov 2015 04:00:00 +0000

Navigation: part 1 | part 2 | part 3 | part 4 | part 5 | part 6

Let's talk about automated software testing. I realize that a lot has already been said on this subject, but it seems that there's still a significant amount of misunderstanding in this important area of software development. I'm going to attempt to shine some light on a few of the core concepts of testing and how they should be applied in a typical, non-formal software project.

It should be noted that this post is not about part 3.

Why test?

Or, rather, why bother with automated tests? I know, I know, it's the obligatory introductory speech... in written form... but not everybody understands this, so it's important to mention.

There are many reasons why automated tests are valuable. Here are three:

Early detection of problems (near compile time; way before they get to production)

Increased trust in codebase

Easier maintainability and extensibility

Let's go through these one by one. Early detection of problems is good because the last thing you want is your customers complaining that you released crap. Manual testing helps, of course, but it's far from perfect, and many issues can and will be missed, so huge gaps will exist in coverage of your code - gaps in which bugs are bound to occur. Adding automated testing fills a lot of these gaps. Also, manual testing is much more expensive than automated testing, since an automated test can be run any number of times essentially for free, while you have to hire people to do manual tests.

Having automated tests increases your trust in the codebase. When you know that the critically important functionality of your product is tested every time somebody commits code, you feel more at ease. (Tests could be run by a build server when it sees new commits, and they can be run locally as part of a normal build process.) You don't dread releases and the inevitable issues that QA or your users are going to find and complain about. Maybe you even have more pride in your software.

Maintainability and extensibility are made immeasurably better, faster, and generally easier when you have automated tests. Simply put, there is very little, if any, concern that the new feature you're adding (or the bug you're fixing) is going to negatively impact other areas of the code. This is very much related to the aforementioned trust in the codebase: when you're confident that there is sufficient testing, you are not paranoid about what could possibly happen when you change some obscure value.

When does the above not apply?

The three reasons mentioned above don't magically become true when you add automated tests. They are only true when a lot of things come together. If you have terrible automated tests, they will not detect problems early. If your automated tests are so brittle that changing a single constant breaks half of them, you won't have easier maintainability. If they randomly fail for no discernible reason on every third run, you may even . There's integration testing. And don't forget about system testing. Others exist as well, but for the most part, these are the essential ones every non-toy software project needs.

Check out part 2 for an exploration of these types.

Passion in software engineering

Mon, 29 Jun 2015 03:30:00 +0000

I'm lucky enough to follow a good number of people on Twitter who, unwittingly, generate great writing prompts for me. Recently, I've seen a few discussions that focus on questioning modern criteria used by software companies in their hiring policies/processes. Passion is one such criterion. Why is it that passion appears to be so highly prized in our field? Is it, perhaps, one of those bullet point criteria that doesn't actually mean much; one that we simply put on our job descriptions because it sounds good and everyone else is doing it? What does it even mean to be passionate in a software field?

The answer is 42

Of course, I don't have the ultimate answer to the questions posed above, but I do hold a few (hopefully insightful) opinions on them. Before discussing the rationale and reasoning behind passion as a criterion for software development work, we need to start by standardizing on our definitions; otherwise, I'll be discussing one thing, and people reading this could be thinking another.

When I think about passion - in the non-romantic sense, that is - I think of it as unbridled enthusiasm for a subject: in this case, software. The connotations of this software-focused definition include, but are not limited to, the desire to make the best software that you can, the desire to improve your users' lives through the use of this software, the desire to improve your and your fellow engineers' lives by making the software as painless to develop as possible, and the frustration that is felt when dealing with software that doesn't feel like it was developed with passion.

That last connotation is quite important in my mind, as it helps you to focus on what should be done by giving you counter-examples of what should not be done in software.

A passionate engineer

Given the above definition of passion, it's easy to see why companies find it a desirable quality in employees. After all, why wouldn't a company want someone who cares deeply for what they do? And doesn't everyone want to have coworkers who are just as passionate as they are about their work?

So how can we determine if someone is passionate about software? That's where things get tricky and murky and generally uncertain. One indication that someone is passionate about software is if they actively participate in open source projects on, e.g., GitHub. Another is participation on StackOverflow or StackExchange. Having personally-developed apps or games in an app store is a great indication as well.

But what if none of these is the case? What if a candidate doesn't have a GitHub or StackOverflow account? What if they have no published apps in any app store? Should that be a negative mark against their potential for employment at a software company?

I say, no. These indications are just that - indications. They provide no clear guidance either way. All they are is hints, suggestions. There are perfectly good reasons for someone to not have a public presence as described above. NDAs and non-compete clauses are just one kind of example. Spending all your energy on your day job is another. My point is, these simple measures are inadequate in determining someone's passion for software.

Delve deeper

In order to determine whether someone is passionate about software, you have to dig. Not into their public presence, but into them. In person. By talking and getting to know them. From personal experience, I know that I wouldn't always be able to answer questions that relate to my being passionate about software during a formal interview; I tend to be somewhat terrified and mentally frozen at that stage. An informal discussion, over coffee or lunch or even beer might work better in figuring out such a complex criterion.

When someone is passionate about software, they should be able to give you examples of them going above and beyond their strict job description to improve something related to that job (in varying degrees of detail due to NDAs and such), or of helping others understand certain concepts better in the context of their job, or of them volunteering to take on a little extra (or maybe just different) work to help their team.

Not just a bullet point

Getting back to the core of this post, having passion as a criterion for hiring a software engineer can indeed make sense, given a certain definition of passion as well as a willingness to genuinely search for that passion in each candidate. However, it's very much possible for a company to simply list the passion criterion as a "me too" bullet point, or to be lazy in evaluating whether a candidate is indeed passionate about software.

To companies out there, I say, either be truthful about your desire for real passion in software development, or just remove that bullet point. And to passionate candidates, I say, remember to judge the company you're interviewing with just as critically as they should be judging you. It's easy to add such a bullet point to a job requirements document, but it's a lot harder to prove that it's genuine. Ask about it. Ask to speak to an employee whom the company considers passionate. If you have passion for software, then so should your coworkers.

Encryption and you

Fri, 08 May 2015 05:00:00 +0000

As software developers, we often have to deal with encrypting sensitive data. These days, for most of us, that simply means integrating with and enabling existing security frameworks. For web developers, it's often as easy as configuring the web server to use HTTPS. (Please note that due to the relatively narrow focus of this post I'm purposely ignoring all the other measures that need to be taken to secure websites. For a quick intro, check out the . I won't go into details on how it works (that's what the Wikipedia link is for), but suffice it to say that the best way to avoid it is to never create your own encryption implementation. There are lots and lots of frameworks and other projects that provide encryption capabilities. Use them instead of implementing the encryption code yourself.

Even if you are an amazingly talented developer, you may not know everything that you need to watch out for in order to create a secure implementation of an algorithm. For example, having constant-time byte comparisons in some situations is crucial to avoiding timing attacks. There are many other nuances (most of which I don't know either) that can affect the security of your implementation. Please, just use existing code that has been vetted and, better yet, formally audited by a trustworthy organization.

It should go without saying that creating your own encryption algorithms is a terrible idea. Even encryption schemes created by respected cryptographers have been found to be cryptographically weak before, during peer review.

Misconfiguration

Did you hear about Sony's PS3 hack a few years back? They had a pretty , you should know that encryption generally falls under the C, or confidentiality. But in addition to it, you also need the I, or integrity. (Naturally, all of this is moot without the A, or availability, but that's a tad out of scope.) Let's get a little more concrete.

Confidentiality without integrity: an example

To restate my earlier point, when you don't verify integrity, your otherwise valid encryption scheme can fall victim to malicious modification. Here's a somewhat realistic example. Feel free to follow along in your own console if you have OpenSSL installed. (Hint: it's available via Git Bash if you're on Windows.)

Let's say you're sending Alice an encrypted message about a money transfer to Bob. Start by creating a cleartext message.

$ echo Give Bob \$500 > clear.txt $ cat clear.txt Give Bob $500

Next, encrypt that message with OpenSSL, using 128-bit AES in Unfortunately, due to the way GCM is handled in OpenSSL, a proper demo of it using the openssl enc command isn't possible. Regardless, GCM and other ways to authenticate ciphertext are extremely important to know about and understand.

Go forth and encrypt

In conclusion, please be aware of the various pitfalls surrounding the correct use of encryption. Understand what encryption provides and what it doesn't provide. And finally, if you're unsure about how to implement something related to security, please seek the advice of others. Just about the worst thing you can do is blindly guess - you could easily end up with a system that looks secure to the naked eye, but is completely broken when properly inspected.

The why and how of personal VPNs

Wed, 22 Apr 2015 03:30:00 +0000

Virtual private networks, or VPNs, are used by lots of companies, big and small, to connect offices to other offices around the world and to allow people to connect from remote locations into their corporate networks for a variety of reasons - usually having something to do with productivity. But a VPN isn't just for corporate connectivity. VPNs are used by individuals around the world to securely access resources at their homes and to enhance the security of their online activities.

What's the point (-to-point)?

The way I see it, there are two major reasons to have a VPN connection to your home. The first and rather straightforward one is resource access. If you have computers or servers or printers at home that you need to access remotely, VPNs let you do that. Now, if you only have a single computer at home that you might want to connect to, it's probably easier just to set up something like GoToMyPC or LogMeIn. But if you have other resources, or if you're a little paranoid and don't want to install proprietary always-on software, then a VPN might work better.

The second, and arguably more important, reason to use a VPN is secure Internet access. Although it's not always obvious, there are many people who use the Internet insecurely without realizing it. This is usually a problem with Web browsing specifically, and especially when it's done in a public place. If you've ever accessed a public Wi-Fi hotspot, you've likely accessed some part of the Internet insecurely.

Public Wi-Fi is bad?

Public Wi-Fi hotspots that are available from your favorite coffee joint, or your hotel room, or the airport aren't bad per se. The problem is, they're not very secure. In fact, there's a hint to that effect in their very name: public. A lot of the activity you do while on a public hotspot is going to be public itself, even if nobody can see your laptop screen. If you browse to any website that isn't served over HTTPS, that entire information exchange is completely out in the open. There are free tools out there that make it extraordinarily easy to see what people nearby are sending and receiving over their wireless connections.

And then, of course, there's malicious alteration of data. Depending on a number of factors, it's possible for someone to modify your Internet requests and responses in real-time as you're browsing online. Using tools like . There is also , or better yet, a The most versatile VPN server
From my research into various VPN solutions, SoftEther appears to be the most versatile. It's open source, available for Linux (not just x86) and Windows as both server and client, provides free dynamic DNS if your home Internet connection doesn't have a static IP, and it supports multiple VPN protocols!

If you have different devices you wish to use to connect to your personal VPN, this works out great. In addition to the built-in SoftEther VPN protocol, you can enable SSTP and OpenVPN - all on the same port, even. For Android and Linux, you can download OpenVPN client software and connect to your VPN through that. This should, in theory, work for Macs as well. For Windows, you could do that too, or you can make use of the built-in support for SSTP to establish a native VPN connection.

Do it now

Whether you want to browse the Web securely in public places, or you want to access resources in your home, or you just want to play around with VPN tunneling, now is a good time to start. There are, of course, potential gotchas and other issues, as with any sufficiently complex technology. For example, depending on your VPN configuration, only some traffic might go through it, leaving your Internet traffic insecure. There are lots of resources online explaining how all of this stuff works. Check them out. Have fun. Be more secure.

API design: adequate vs. awesome

Mon, 30 Mar 2015 02:31:00 +0000

Inspired by a Twitter conversation regarding REST APIs, I wanted to better express my thoughts and feelings about API design (and I use the term API to not just mean REST API, but also frameworks and anything else that you might create that has some sort of interface for other developers to work with) in a format that allows for more than 140 characters per post. Let's get straight to the point. What makes an API good, bad, adequate, usable, unpleasant, or whatever other term you prefer that essentially still boils down to good or bad?

UX? For an API?

To say that the answer is "UX" might seem a little strange, since we don't generally think of developer-facing stuff as needing a facelift, but that really is the answer, in my opinion. The main difference is in who the "U" is in "UX" here. The developer is your user. So, UX in this instance could be called DX (not to be confused with 486DX).

But why?

Why would you want to design your API to be good, though? Isn't it enough to simply expose the functionality you're intending to expose, and then abandon the project and move on to the next thing? Some people/companies clearly think that is enough. However, I'm of the opinion that designing an API that is pleasant to use will develop goodwill between you/your company and your API users. If you're charging for this API, they'd be more willing to pay. If you're using it to show off your skills, they'd be more impressed. If you're doing it purely out of the goodness of your heart, then releasing a bad API would be a disservice to yourself, as you want to release something that people can use.

Introducing Some Random Developer

The biggest problem in designing a good API is being able to let go of all your innate knowledge of the underlying system and how all of its pieces fit together, and pretending to be Some Random Developer (or SRD for short) who has stumbled across your API in hopes of doing something useful with it. When SRD finds your API, what is the first thing they'll do? Probably try to determine whether your API is what they want in the first place. Does it seem to solve their problem? Is it available for their platform? Does it have licensing or cost restrictions? Making all of this information easily accessible makes SRD's life that much simpler.

Provide good examples

The next thing SRD will look for is examples of using your API to solve the exact problem they're trying to solve. Unless you're prescient, you won't be able to anticipate every SRD's problem, but that's all right. If you have lots of examples (that are well documented!) of various uses of your API, there's a good chance that SRD will either find the exact solution they need, or at least be able to extrapolate it from the multitude of examples you've provided.

Tedious, detailed documentation

After figuring out the API endpoints (or methods or interfaces or whatever they may be) that SRD needs to solve their problem, they'll want to see exactly how their data need to be sent in to your API and exactly what your API will output, down to the gory details such as encoding, headers, and any surprises that SRD might encounter while working with your API. This is all pretty tedious to document, and there are projects out there that strive to . Again with REST APIs, allowing access to the "latest and greatest" version through a special endpoint doesn't work well, and to the development of an API. Pretend that you are that mythical SRD. When you encounter an API that you might want to use, what would make your life easier when looking into that API? Would you appreciate excellent, accurate, in-depth documentation with plenty of examples? Would a consistent naming and data passing scheme appeal to you? Be willing to suffer a little pain and teduim in order to make all of your users suffer that much less. They'll thank you for it.

YAPoP (Yet Another Post on Pairing)

Wed, 04 Feb 2015 04:30:00 +0000

I almost named this post "Yet Another Personal Opinion On Pairing", but the resulting acronym didn't seem entirely appropriate. So, what's the big deal with pairing - or pair programming, as it were? Overall, it can be a very useful tool. It often helps to flesh out the design of a solution that's about to be implemented, since one person usually does more thinking while the other person does more typing. It helps to keep people focused on the task at hand. It can also be very effective at providing instant gut-checks of code and of concepts. And, of course, it's extremely useful in bringing new people up to speed on the codebase and domain knowledge.

The last point above is perhaps the most important reason to pair program. In my experience, the "onboarding" process for a new team member is made much more straightforward and expeditious when that person is pairing with someone more experienced or familiar with the project. In addition to providing valuable context for the code and the domain, the more experienced person will impart lots of undocumented knowledge during the task that the pair is working on, which is incredibly helpful not only for the new person, but also for the team, as interesting facts and processes that have never been formally defined come out, and this allows the team to critically examine them and perhaps to modify/improve, or at the very least to document them for future use.

Thanks to all of the benefits that pairing provides, as well as the positive press it's been getting in the last few years, it's no wonder that more and more people and companies are seriously looking into how best to use pair programming techniques. And it seems that many have decided to embrace it to such an extent as to make it a mandatory part of their development process.

But permit me to be an iconoclast for a bit and stray from the teachings of the Patron Priests of Pair Programming. Pairing is not all rainbows and sunshine. There are real reasons why it should sometimes be avoided. If we ignore those reasons and embrace pairing dogmatically, we're doing everyone - including ourselves - a great disservice. Let's start with an easy to comprehend reason: scheduling. Different people work at different times. They have different meetings, lunch hours, medical appointments, and so forth. If one person in a pair is unavailable for a couple of hours, should work on the task stop? Should another person be brought in to replace the missing one either temporarily or permanently? I would argue that the most productive answer is to be pair-less for those two hours, and once the missing person comes back, they can be quickly brought up to speed on the task's progress, since they're already familiar with it. Now let's take a closer look at the alternatives to this option.

Bringing in another person temporarily would be disruptive. Everyone has a unique way of communicating, typing (Dvorak?), and behaving in general. Adjusting to pairing with a different person for a couple of hours only to have to adjust again when the original missing person returns is a huge time sink and would cause productivity to plummet. I write from experience: when I'm working on a task, a disruption can cost me anywhere between zero and twenty minutes of downtime. If I'm , I've never read about working any better than normal pairing.

Bringing in another person as a permanent pair replacement, in addition to causing a one-time disruption, does a disservice to the missing person because they have invested a not-insignificant amount of time learning the task's requirements, challenges, and proposed solutions, only to find that knowledge unusable since they're no longer on the task. How frustrating! Everyone knows that there is a certain amount of satisfaction in implementing your vision and seeing it work. There is also a certain amount of frustration when the task ends up being implemented differently from how you envisioned it - especially after you have personally invested in it by having begun work on that task. Yes, I'm hinting at the concept of code ownership, which some feel is misguided, but I have found that it helps tremendously in keeping code quality up. The reasoning for this is rather simple: if you care about the code that you helped to design or write, you will fight to keep it clean.

Continuing on to the next reason to not pair: fatigue. Good, high-productivity pairing isn't just tiring; it's exhausting. I know that I, for one, don't want to come home every day exhausted. I've been there before, and it's not good. I'm not a robot whose job is to work tirelessly until its gears rust and disintegrate; there's only so much exhaustion that I can take. Doing too much of this is a surefire way to get burnt out. (I've been there, too, and believe me, it's so much worse than just being exhausted.) Sometimes, a nice, long break is needed from your pairing. So why not take on a new task solo and see it all the way through before going back to pairing? There is no need for unquestioning orthodox adherence to pairing.

And what about productivity? Sure, there are times when pairing increases it. But on the other hand, I know I'm not the only one who'll occasionally stay at the office late, after everyone's gone, and implement a whole lot of the task in a fraction of the time it would have taken if I were pairing with someone. I believe that's referred to as being "in the zone". It is possible for this to happen while pairing, but from my personal experience, as well as what I've seen
There is even a downside to the incredibly helpful pairing during the onboarding process. When an experienced person and one who is newer to coding (perhaps someone just out of college) are pairing on tasks in an unfamiliar codebase, this can cause issues. The reason for that is, the experienced person will want to dig into the code to determine how it currently works in order to modify it appropriately and within the established practices in that codebase. However, doing this while pairing with someone newer doesn't always work well. The newer person will often want to jump straight to coding, which is rarely what you should be doing in an unfamiliar codebase. If the more experienced person is "driving", then they'll likely be going a mile-a-minute between different functions, files, and projects, determining how everything fits together, while the newer person is lost and bored. If the newer person is at the keyboard and mouse, then the more experienced person is still doing the code exploration, but through the other's hands. That doesn't help the newer person, since they're only following directions at that point.

Finally, let's look at some of the positives from the first paragraph in a different light. The first two, to be more specific. When pairing helps to flesh out your solution's design, that could be a sign of bad practices: it's simply not a good idea to start coding before you have a plan of action. We've all done it at one point or another (I know I have) only to regret it later on, when we realize the ramifications of the choices that we implicitly made while coding up the solution with the first thoughts that came to us. I would, in fact, say that pairing is more useful when coming up with solutions to tough problems, rather than when coding them up. Indeed, I've worked on tasks in the past where the task would start with the pair figuring out the what and the how, and then splitting up and doing different parts of the work simultaneously. Toward the end of the task, we would merge our code, resolve any rarely-occurring outstanding issues, and move on. I wouldn't call this proper "pairing", but it's worked remarkably well time and again.

Along the same lines, keeping people focused through the use of pairing is another sign of bad practices (or, dare I say, bad personnel - but I won't go into that here). Personally, if I can't stay focused on a task, the reason is usually that the task itself isn't defined well, and I have to keep asking various people for clarifications. Another reason for this is tooling and environment failure, where the tests take over half an hour to run or the continuous integration server randomly fails builds. These disheartening events have a negative impact on productivity as well as morale, and they discourage people from being vigilant about keeping the codebase clean and tidy. Forcing progress to be made in these situations is nothing more than a band-aid that's used to hide the underlying issues.

So, what am I trying to say? That pairing is bad and we should stop doing it? No. Pairing has clear benefits to it; the aforementioned gut-checks and faster code familiarization are extremely useful to the pair, the team, and the project as a whole. But it also has its own set of problems such as increased fatigue, and it allows teams to sweep some severe pre-existing problems under the rug, like skipping the design phase of tasks and getting used to

SSL yourself

Thu, 06 Nov 2014 14:00:00 +0000

Hey, you! Do you have a blog? Or a website? Now, tell me this: is it accessible over HTTPS? If you're like most people, your answer is going to be something along the lines of, "No, why should I bother? I have nothing security-critical on my site, and I'm certainly not accepting credit cards for people to enter." That's a perfectly valid response. But I ask you to be just a little more forward-looking. Wouldn't it be wonderful if you didn't have to worry about whether you're really accessing your bank's website and not that of some scammer? Or if the information you're entering at the local cafe is being intercepted by someone sitting two tables away?

These are not simple issues to solve, and serving your personal blog over SSL seems almost entirely unrelated. But it is indeed related: we want to get to a point where everything on the web is secure by default. That means, among other things, that everything is transferred securely without the ability to fall back to the old, insecure protocols from the 1980s and 1990s. And you can take a small, but significant, step in that direction by serving your website over SSL. If enough people do this, then perhaps someday HTTP without the S at the end of it can be deprecated and eventually removed. I know, it's quite a lofty goal. But it can be achieved with small steps taken by everyone together.

"All right, let's say I want to be part of this Utopian vision of yours," you say, "but I don't want to pay for a dedicated IP address or for an SSL certificate. It's still not worth it for me." Well, I have good news for you. Neither of those is required anymore - thanks to
"This is all very interesting, but I don't host my own blog. Someone else does it for me." In that case, I humbly ask that you become an activist. Don't worry, I don't want to start an "occupy the blogosphere" movement. But a single email or support ticket requesting SSL support can go a long way - especially if a whole bunch of people do it.

The World Wide Web is constantly evolving, and you can help it evolve in the right direction. Taking a small step towards being secure by default right now will allow for larger steps in the future, and maybe, just maybe, someday we won't have to be concerned so much with the security of our information, and we could focus on tackling more pertinent issues.

HashProp - a better way to MD5 (and SHA) in Windows

Sat, 27 Sep 2014 15:00:00 +0000

HashProp is a Windows shell extension that I decided to write at the previous Carmel Code and Coffee meetup. Basically, it adds a new tab to the File Properties dialog for every file in Windows with the ability to calculate the file's hashes:

It's a simple interface, so people can quickly and easily calculate MD5, SHA-1, SHA-256, and SHA-512 for any file.

I created this thing after getting some inspiration from to do just that.

One thing I intentionally focused on was overall UX. Is the file being hashed very slowly? The percentage is shown while the calculation is occurring to let users know that the system is working. This is especially useful when hashing large files that are served over the network. What do I often do after hashing a file? I copy the hash to the clipboard. Clicking the textbox with the hash (or leaving the textbox selected until hashing completes) will auto-select all text in the textbox. Oh yeah, clicking the Calculate button disables it while the calculation is occurring, and the corresponding textbox is selected. And finally, if the hashing is taking a long time and the user closes the File Properties dialog, the hashing is stopped and the file handle is released. This is all a part of good UX.

HashProp is released under the ISC License. The source code and links to binaries are on to need to check out
I am willing to work around issues caused by the lack of dynamic processing, so a static blog works for me. Once I came to this realization, I needed to pick the right static site generator. There are quite a few choices out there, with varying degrees of maintenance, power, and flexibility, and many runtimes. I think it's safe to say that .

Unfortunately, I found Jekyll to be too restrictive in what it allows me to do, and most of the other generators ended up being just as restrictive, difficult to set up, or simply abandoned. All that made me come to the conclusion that I needed to write my own. So I did.

I used an existing HTML to Markdown converter (and tweaked it a bit) to convert my old posts from my Drupal blog to Markdown files, and then I created a completely new design from scratch. I think it looks a lot better than the old blog. I would appreciate any feedback on Genmaicha or on the design of the new blog!

By the way, Genmaicha is licensed under the permissive, OSI-approved
Wed, 06 Nov 2013 19:26:09 +0000

Running an elevated application at startup time on Windows 8 is rather annoying. Normally, the easiest way to run anything at startup is to simply create a shortcut to it in %AppData%\Microsoft\Windows\Start Menu\Programs\Startup. Unfortunately when UAC is on, Windows will simply refuse to launch any shortcut at startup time if its properties are configured to run the application to which it points as administrator. In Windows 7 and below, disabling UAC let everything work just fine. Windows 8 changes matters by requiring UAC to be enabled in order for the Metro app sandbox to function - in other words, disabling UAC kills all Metro apps. You can still silence UAC (I do) but it's no longer reasonable to disable it.

There are multiple ways to get around this restriction, such as using the Task Scheduler to launch the app as a startup task or writing a Windows service to launch the app. However, the most straightforward way is still to use the Startup directory. The big difference is, instead of calling a shortcut, execute a script!

In my case, I want to launch the OpenVPN GUI. It needs elevated privileges in order to control its virtual network interface. Instead of creating a shortcut, I created a new file named OpenVPN.vbs in the Startup directory with the following two lines:

Set UAC = CreateObject("Shell.Application") UAC.ShellExecute "C:\Program Files\OpenVPN\bin\openvpn-gui.exe", "", "", "runas", 1

That script launches the OpenVPN GUI with elevated privileges. Simply replace the first parameter for ShellExecute with whatever app you're trying to launch and you're good to go! And if you need to pass command-line arguments, that's what the second parameter is for.

Beware of silent failures

Mon, 09 Sep 2013 04:37:17 +0000

What's worse than experiencing a failure in a production system? I'll tell you: not knowing that it occurred.

I was inspired to write this post after seeing a neat presentation at work about a new internal tool at the office. One fact that jumped out at me during the presentation was that, if a failure occurred while running a periodic background task, an email would be fired off informing the admin(s) of this problem. On the surface, that seems like a reasonable action to take; be silent unless something's wrong.

But there's a fundamental flaw in this concept. If something does indeed go wrong, how can you be sure that email will still function? Or to say this in a more generic manner, if a failure occurs, how can you be sure that a push-based notification of this failure can still happen? The answer, of course, is that you can't. And that's why you shouldn't do it.

So, what options are there? Fundamentally, you can do two things to verify the correct functionality of your system. The first thing you can do is push success notifications, in addition to or instead of failure ones. In the particular example of the internal tool, it could send out an email every time the periodic task completed successfully. Or every 30th time it completed successfully if it runs often. The point here is to let the admin(s) know that the system is working as expected. If suddenly the emails stop or begin reporting failures (e.g., "Tasks ran; 2 of 30 failed") then appropriate actions can be taken to remedy the situation.

But what if you don't want to spam your admin(s) with useless status reports that they'll just ignore anyway? Well, it should be part of their job to monitor this stuff, so don't be afraid to do it! Alternatively, you could use a pull-based approach to this problem. If you have an external monitoring solution set up, use it to get status reports from your system. You can have your system publish a report of its background activities on a special URL or a shared network location and then have your external monitoring solution periodically check that report for problems. For websites, this could even be achieved by using uptime monitoring services like " that pointed to exists. After reading that post, I felt a visceral need to respond, so that's what I'm doing now.

The author of the aforementioned blog post essentially claims that a corporation does not deserve loyalty because it cannot be loyal to you and will necessarily let you go if doing so increases its bottom line; that the CEO's mandate is to make just those kinds of decisions; and that you have to take control of your career lest you be shackled to an oar (his words). I don't disagree with any of these points. But I do disagree with the overall message and the moral of the story, as it were.

The message should, in my opinion, be to choose your place of employment wisely. That blog post pretty much summed up the reasons why I generally dislike big, soulless corporations. But there are other options out there! I currently work at a ~100 person consulting company. Previously, I was at a <10 person consulting company. And before that I was at university, but during that time I did briefly work at a fairly large insurance company. I got a taste of the large company ethos then, and it was enough to scare me away.

Smaller companies are different. They place more value in the individual. There's little, if any, company politics and bureaucracy. And I think, for the most part, people are happier there. If I were to look at things more cynically, I could say that smaller companies have more incentive to keep employees happy because the loss of one employee at a 50-person company is more painful than the loss of one employee at a 50,000-person company. But I choose not to look at it that way.

I think the reason I felt such a strong need to respond to the blog post in question is that it hits home on some level. I believe that if I were in that author's shoes, having gone through his experiences, I could be just as jaded as he is. In a sense, I'm lucky to have had the chance to see that environment early on and seek opportunities elsewhere. People who know me have seen how cynical I can be about certain topics. But the topic of company loyalty is an exception. I truly believe that it is possible to find a company that isn't soulless, that does in fact care about its employees, and that can even deserve loyalty.

Farewell, Windows Phone

Fri, 29 Mar 2013 20:46:20 +0000

It was a gray afternoon, the kind that makes people long for rain just to break up the dreary monotony of the sky. I was nursing a scotch, which was by this point in time far too diluted for my taste, as the ice had long melted. As I lit my fifth cigarette I heard a dull thump outside. Must be the paper. Late as usual.

The winter had been strangely warm this year. Not warm enough to enjoy going outside, but not so cold as to let snow lay on the cracked pavement for any meaningful amount of time. I shivered as I opened the creaking door, and the chilly air let itself wash over me. Bringing the Gazette inside I noticed the subtle smell of damp paper and ink. The Daily Gazette was the only halfway respectable newspaper in town so everybody had a subscription, and the publisher's executives didn't feel any particular need to enforce timeliness of deliveries or proper moisture protection.

As I poured myself a fresh drink, I noticed out of the corner of my eye a familiar physiognomy. Could it be? After all this time?

Intrigued, I put down the crystal glass, half filled with that nourishing amber liquor, and walked over to the pile of newspapers that was once known as my couch. Unfolding the newspaper with ever so slight trepidation, I froze. It was her.

The year was 2004, and I was young and carefree. My interests at the time included mobile phones and software development. And she was at the center. Back then she was known as Windows Mobile 2003 SE. A long name, full of history. We hit it off immediately, and over time our love flourished. Having a troubled past, she had changed her name more than once. We pulled through those tough times of her becoming Windows Mobile 5, 6, and 6.5. During those years, she had adopted quite a few aliases, and I was intimately familiar with some more than others. Audiovox SMT5600/HTC Typhoon, HTC TyTN/HTC Hermes, and HTC Touch Pro/HTC Raphael/AT&T Fuze were my favorites.

But one day, something happened. She had been away for a while on yet another job, and we lost touch. When we finally reunited, she was... different. She called herself Windows Phone 7 and seemed to rely less on her aliases. Previously when she adopted new identities, she was still the same person underneath the makeup, the same one I fell in love with. Not this time. Her hair color and style were changed. She had adopted a new, unfamiliar accent. Even her skin tone seemed somehow different. But more than that, she was distant.

In the past we had shared everything with each other, the good and the bad. I could truly say that I knew her, and she trusted me enough to let me into her world - all of her documented and undocumented APIs were mine to explore. That trust was now gone. It was as if she had experienced something so terrible in her time away that she could no longer trust anyone at all. Except that wasn't entirely true. There were certain people who called themselves carriers and OEMs. She knew them before, but now she seemed to be perpetually closer to them than to me.

Still, we made it work. Writing software for her was still the best mobile development experience in the world. I came to truly enjoy and even be inspired by her new focus on overall user experience, something I had intuitively done before, but never really focused on. And occasionally, she would still let me use some of her APIs that she normally kept to herself, which was affectionately called "jailbreaking".

Unfortunately, even that limited, secret trust went away as she started calling herself Windows Phone 8. Her most impressive alias, Nokia Lumia 920, was a work of art indeed. Beautiful, with the world's first optical image stabilization camera in a phone. I've been with the 920 since she introduced this alias at the BUILD conference. She made me feel special, privileged to be the first to see her in this form. But as she refined her personality with new firmware updates for others, she held them back from me. So naturally, I had mixed feelings of relief, anticipation, and wonder as I saw her in the Gazette with the headline "Portico update finally available to Lumia 920s from BUILD". It was a message for me. She still wanted us to be together. It felt like our love was renewed once again.

Alas, it was not meant to last. I found out something about her update - something that, by itself, may have been all right, but combined with all her other changes, was just too much for me. She had betrayed my trust. She changed herself dynamically based on the SIM card that she thought she used. I could no longer use Wi-Fi tethering, as she thought my MVNO-provisioned SIM card was that of AT&T, and she began asking AT&T for permission to let me tether. I thought it was a fluke. I contacted the OEM that provided her Lumia 920 alias to ask about this, but they confirmed the worst. This was part of the Portico update.

This was the proverbial straw that broke the camel's back. I didn't want to admit it publicly, but I missed terribly her openness from the old Windows Mobile days. This final blow of disabling tethering, especially when she didn't even realize the true origin of the SIM card, had shattered my hopes of getting her to open up to me again. It was finally time to move on to someone who would not be so closed to me. A girl who'd been eyeing me for some time now. A girl named Android.

NOTE: The preceding was a (rather obvious) dramatization of more or less accurate events. Except, I don't smoke. Or drink scotch. Or subscribe to anything called the Daily Gazette. And I don't usually anthropomorphize my devices. However, I am indeed switching to Android. For added effect,
Let's step back a bit. Why do I think that a great UX is just as important as functionality? The single word answer is perception. I'm working under the assumption that a major goal of whatever it is you're making is that your users actually like it. Unfortunately, making users like your software is not a simple and clear-cut task. A great user experience is required - yes, required - for general likability. Take, for example, Windows Vista. For the most part, it has been disliked. The reasons for this are numerous, but functionality is not among those reasons, as far as I know. I remember Vista being somewhat slow, severely lacking in third-party drivers, and extremely annoying due to the introduction of User Account Control (UAC). Why was it slow? Probably because a lot of features were added to it since XP and turned on by default. Why weren't there many third-party drivers? Because Vista was the first widely used 64-bit Microsoft client operating system, which meant that a lot of drivers simply didn't exist, as they all had to be recompiled and signed. At the same time, Microsoft improved the Windows driver architecture, causing incompatibility for certain classes of drivers, and manufacturers were slow to respond. And, of course, UAC popping up its confirmation dialog for every minor settings change and every install was technically more secure, but obviously flawed. Windows Vista was technically a lot more functional and secure than Windows XP, but because of its bad UX, it is now remembered as a failure.

But here is the really important thing. By the time Vista SP2 came out, all of the initial issues were pretty much fixed or had widely known workarounds, and yet the negative perception of Vista remained. Windows Vista's original bad UX forever tarnished its image.

The moral, which should be pretty obvious by now, is that first impressions really do matter, and when somebody finds your incredibly functional yet barely usable public 1.0 release, that person just might give up on your software altogether and put it on a mental blacklist. Cutting features is reasonable. Cutting UX is not. (Of course, there are always exceptions to the rule. I'm not saying that this is absolutely the only way to go. I just want everyone to think hard about the choices they're making instead of simply going with the flow.)

Digital trust issues

Fri, 05 Oct 2012 02:12:24 +0000

I will freely admit that I have digital trust issues. In fact, I will go so far as to say that if you don't have these issues as well, then you are either not in the software engineering field, or you are being willfully ignorant. Allow me to explain my terminology and position.

Many incredible computing advances are being made every day. The latest piece of "that is so freaking cool" news is Ghost in the Shell. It paints a pretty realistic picture of a future world of connected machines and connected humans - and the scary things that hackers could potentially do in such a world.

Along the same lines, this is also why I've yet to enable auto-pay on any of my bills. Giving multiple companies my banking information to store for use every month to automatically withdraw funds sounds like a recipe for disaster. If even one of those companies has its data compromised, then there is suddenly a very real possibility of my bank account being emptied. You can usually escape liability for fraudulent credit card transactions, but it's not that painless with checking and savings accounts. Okay, so another reason why I don't enable auto-pay is so that I actually look at my bills to see if there are any discrepancies; otherwise, I just wouldn't bother looking at them at all. But the point stands - the code running all these systems is of unknown quality.

What can be done to improve this situation? Honestly, I don't know. There are automated systems like McAfee Secure that scan for vulnerabilities remotely and then display a . In other words, avoid overengineering. If your website or app is only going to be used internally at your company, behind a firewall, by four people, then you really don't need a very secure password system. You might even be able to get away with storing those passwords as plaintext. Of course, that won't do if your website or app is exposed to the Internet or if it may be used by a significant number of people.

Before delving into password storage security, I'd like to touch on a related topic - password complexity requirements. Many people are familiar with silly corporate password policies of "minimum 17 letters, 11 numbers, two special characters, three spaces, and a Greek god". If you're going down the path of dictating strict requirements, then you're better off requiring passphrases instead of forcing very specific minimal character type counts on your users. Passphrases may take slightly longer to type, but they are much easier to remember than complex nonsensical passwords and, consequently, much less likely to be written down on a sticky note. Alternately, you can require a minimum password complexity based on password entropy calculations, as long as you also check for dictionary words and adjust the complexity ratings accordingly.

So, what's a good, secure way to store users' passwords? It helps to think about this in "levels" of security:

Level 0 - Plaintext - Horrible.
Level 1 - Reversible encryption (AES, etc.) - There's no good reason to do this.
Level 2 - Basic hashing (MD5, SHA-256) - This is not considered secure anymore. Stop it.
Level 3 - Hashing and salting - Pretty standard today, but not great.
Level 4 - Hashing and salting using modern PBKDFs - This is best.

Let's go through each level. As I explained earlier, Level 0 should only ever be used on tiny, insignificant internal projects because it is entirely insecure.

Level 1 is marginally better because the password isn't stored as plaintext, but it is not a good idea. Usually developers do this because they want to be able to recover user passwords. However, it is generally accepted that such functionality introduces unacceptable risk. If the set of encrypted passwords is stolen by an attacker, then figuring out the encryption key will enable said attacker to decrypt all of the passwords at once.

Level 2 prevents decryption of passwords by using one-way encryption, a.k.a. hashing. When the user authenticates, you use the same hashing algorithm on their entered password and simply check if the hashes match. Unfortunately, this is still not secure. People have created what's known as "rainbow tables", or large lists of hashes and their corresponding plaintext values. It becomes trivial to run the hashes through rainbow tables and get many, if not most, passwords back as plaintext.

Level 3 makes the use of rainbow tables infeasible by appending or prepending random characters to user passwords. Those random characters are called a salt. You store the salt along with the hash resulting from the concatenation of the user's password and the salt. When the user authenticates, you perform the same concatenation of the stored salt to the entered password and then check if the hashes match. Because (good) salts are long and random, rainbow tables are highly unlikely to have any plaintext that happens to match the concatenation of the salt and the actual password. This can be slightly enhanced by using two salts - one randomly generated salt stored with the hash, and one static salt stored in the filesystem. That way, if the database is compromised, one of the salts is still hidden from the attacker. However, this is still not a great way to store passwords. Due to advances in
So there you have it. If you want to securely store your users' passwords, you know what to do. I was curious about modern methods of secure password storage, so I researched the topic a bit, and ended up with this blog post. In the process, I also implemented
Under Features/Integration, I configured the "ConEmu Here" context menu integration to the following command:

/single /cmd c:\cygwin\constart.bat -c "/conhere.sh \"`cygpath -u '%L'`\"" -new_console:an

Yes, that is one confusing command with lots of varying quotes. If there's an easier one, please tell me. Essentially, it launches Bash and makes it go to the current directory, signified by '%L'. It's in single quotes in order to handle spaces in directory names. The cygpath command is used to translate the Windows directory name to the Cygwin equivalent. The escaped double quotes surrounding that command are also for handling spaces in directory names. Finally, Bash is launched in interactive mode on the last line of conhere.sh instead of just quitting after changing directories.

Past all that, everything comes down to personal choices of Bash customization with functions, aliases, and so forth.

I disabled screen's Ctrl-A hook by adding the line "bind ^a" to /etc/screenrc. That turns screen into, for all intents and purposes, a regular virtual console.

Since nano has strange cursor scrolling behavior in Cygwin, I'm using
Finally, I added some convenient aliases:

alias ls='ls --color=auto' alias la='ls -lha' alias 2w='cygpath -w' alias 2u='cygpath -au' alias nano='echo "Using ne instead of nano!"; sleep 2s; ne'

That's about it. I might be missing a few things, but for the most part, this is a very usable console environment that's better than the default Windows one. If you have any suggestions for improvement, I gladly welcome them.

How to do a clean install of Windows 7 or 8 on Samsung Chronos laptops

Mon, 06 Aug 2012 01:04:26 +0000

First, some background. I've got a new Samsung Series 7 NP700Z3A-S06US notebook (that name just rolls off the tongue, doesn't it?) and the first thing I always do when I get a new computer that I didn't build myself is wipe everything off the hard drive(s) and install the operating system from scratch. This applies even to the "Microsoft Signature" computers, which are supposed to be bloatware-free, but still contain too much unnecessary stuff for my taste. So, when I looked over the installed software on this machine, I decided that I might as well do my usual thing and wipe it. That's where trouble struck.

This particular notebook computer, as well as similar Samsung models, contains an 8GB SSD (in addition to the 1TB HDD) that is used as a cache to speed up Windows and apps. Samsung uses Diskeeper's ExpressCache software for that purpose. Unfortunately, this SSD causes a rather large problem for the Windows installer. For whatever reason, Windows refuses to install its little "System Reserved" partition on the HDD, downright refusing to proceed with the installation if the SSD is already partitioned for ExpressCache. Once I discovered this, the only choice I had at that point was to repartition the SSD and let Windows install itself. However, things weren't that simple. After the Windows installer rebooted, the notebook went into a boot loop. I've never seen an x86-based computer do that before. I've seen many a boot error message, but never a boot loop. It appears that the BIOS really doesn't want to boot off the SSD, which is where Windows decided to install its boot partition. I had to figure out where to go from there - how to get the HDD into a state where both Windows and Samsung's BIOS were happy, and the SSD free for ExpressCache use.

Aside: I had some hardware-related trouble with this notebook, and in the process of trying to get it repaired discovered that removing or replacing the hard drive voids the warranty. I think that's idiotic. I had to wipe the HDD before sending the notebook in because it had sensitive data on it, instead of just removing the HDD.

WARNING: The following steps involve dangerous commands that delete lots and lots of data. I'm not responsible if you delete your precious memories. Only you are responsible.

Here are the approximate steps I took to get everything working again:

Boot off the Windows DVD or USB installation media

Choose the "Repair" option, and the command prompt afterwards (the way to get there is different between Windows 7 and 8)

Type in diskpart to get into the partition tool

Use the commands list disk and list part to determine which disk is what. For me, Disk 0 was the HDD and Disk 1 was the SSD. The following instructions assume this

Select the HDD: sel disk 0

Delete all partitions on it: clean

Create a 100MB partition for Windows 7 (change to 350MB for Windows 8): create part primary size=100 (or size=350)

Format it: format fs=ntfs quick

Assign it a letter: assign letter=f (if F: is in use, pick another one. Use list vol to see all volumes and their letters)

Create a partition that fills the rest of the disk: create part primary

Format it and assign it letter "C", as above

Reboot back into the Windows installation media and install Windows into the large partition that was just created

Again, reboot into the Windows installation media and go into the repair command prompt

Use diskpart's list vol and assign commands to ensure that both the boot partition (F:) and the Windows partition (C:) still have drive letters

Select the boot partition (sel part 1) and mark it as active: active

Exit diskpart and type in: bcdboot c:\windows /s f:

Reboot and you should be good to go!

After booting into Windows, you should be able to safely repartition the SSD to ExpressCache's liking, without killing Windows.

So who is to blame for this foolishness - Samsung or Microsoft? Yes. They are both to blame. Samsung should not have screwed up the system configuration to the point that using the standard Windows installer causes such problems, and Microsoft should not have made the installer so damned picky about which partitions must reside in which disks and in what order.

How to properly get a Windows Phone app's assembly version

Wed, 16 May 2012 13:40:16 +0000

I've been seeing code floating around that suggests that in order for you to get an assembly's version number in Windows Phone SDK 7.0/7.1 you have to call Assembly.ToString() or Assembly.FullName and then parse the output. Please don't do that. There is a better, more stable, and more supported way to get the information you seek:

new System.Reflection.AssemblyName(System.Reflection.Assembly.GetExecutingAssembly().FullName)

That will give you an AssemblyName object, which has not only the version number, but other information about your assembly as well.

On Visual Studio 11's redesign awkwardness

Wed, 09 May 2012 15:31:24 +0000

Most people by now are aware of the design changes from Visual Studio 2010 to VS11 Beta, and from
The phone's software is tied to AT&T, and I don't like that. I shouldn't have to be tied to AT&T's rules just because that's how the phone was initially configured. So I decided to fix these issues plaguing my phone. It turned out to be pretty easy. Here are the steps I took to enable tethering and updating.

I dev-unlocked the phone. Since I have an App Hub account, it was a piece of cake.

I interop-unlocked the phone, following the Samsung guide in .

To enable tethering without asking for AT&T's permission, I used the Root Tools' registry editor to make the following changes:

[HKLM\Comm\InternetSharing\Settings] OpenMarketEnabled=dword:1

[HKLM\Comm\InternetSharing\Settings] EntitlementURI="./Vendor/MSFT/Registry/HKLM/Comm/InternetSharing/Settings/OpenMarketEnabled" (without quotes)

To enable OS updates without AT&T's software blocks, I made these registry changes:

[HKLM\System\Platform\DeviceTargetingInfo] MobileOperator="000-88" (without quotes)

[HKLM\System\Platform\DeviceTargetingInfo] MOName="OPN" (without quotes)

Finally, to ensure the settings took hold, I rebooted the phone.

After applying the above changes, I could freely tether as well as update the phone to build 8107 through Zune. Please note that I am not advocating following the above steps because changing registry values is extremely dangerous and can potentially brick your device. However, if you're like me, and you refuse to use an artificially crippled device, you could un-cripple it.

Why the mobile "write once, run everywhere" mentality is misguided

Fri, 27 Jan 2012 02:22:32 +0000

Every so often I see articles and news blurbs about yet another product that allows people to create a mobile app once and automagically publish it on all of the major smartphone platforms. Recently, I've seen lots of buzz around PhoneGap becoming fully-featured in regards to Windows Phone. And just today I saw an . However, when laziness isn't tempered with the other two, impatience and hubris, it quickly goes from virtue to vice. I have occasionally been guilty of this myself. Sometimes, when I really want to get an app functioning, I'll forgo good techniques in favor of quick turnaround. I (usually) force myself to fix particularly bad code later on. Many others do not. When people release apps that were lazily coded, those apps should be expected to easily break. Sometimes small utilities with a very limited scope are fine to release without much error handling. They just aren't that important. But a public app that is submitted to an app store should never fall into that category. If developers are taking the time to release their apps to the public, especially if those apps aren't free, they must be resilient.

A real world counter-example

I must confess, the catalyst for my writing this blog post is that one of my apps has proven to be resilient this past weekend, and I realized that many others would not have coded it to be so. ArkWords, my free dictionary/thesaurus/translator for Windows Phone, has a popular feature called Word of the Day. It's pretty self-explanatory. What's interesting about it, though, is that I wrote a web service that sends a Live Tile image with the current word as well as the day of the week that's associated with that word. My upstream provider, the awesome
Once you've verified the requirements, you'll need to flash your router. There are plenty of tutorials on how to do this in case it's not obvious. Also, I must add the mandatory warning that I AM NOT RESPONSIBLE for what you might do to your hardware or software and the issues it might cause. Flashing can be a dangerous procedure and you may end up with a bricked router. Don't say I didn't warn you. Flash it with a version of the , will go into effect on June 3 (after the release of Mango tools).

This is, of course, important news for anyone with apps in the Marketplace as well as anyone planning on submitting new apps. Let's take a look at the major differences between the previous version and the new one.

2.14 Your application must have distinct, substantial and legitimate content and purpose other than merely launching a webpage.

This is a new requirement. It appears to be aimed at apps that are simple WebBrowser containers pointing to a publicly available site. Pretty self-explanatory.

3.7 (part) Applications that enable legal gambling in the applicable jurisdiction where legal gambling is allowed may be permitted, subject to the Application Provider's acceptance of additional contract terms.

This is an addition to the illegal gambling clause to clarify that legal gambling is generally not an issue.

4.7 Application Tile Image

This section seems to have been added for the singular purpose of telling developers to make their small and large app tiles actually have someting to do with their apps. Seems kind of obvious to me, but I suppose this rule wouldn't have been made had someone not tried to violate it.

5.1.3 Application Responsiveness: If an application performs an operation that causes the device to appear to be unresponsive for more than three seconds, such as downloading data over a network connection, the application must display a visual progress or busy indicator.

This was changed from the previous "5.1.3 Application Does not Hang". It now has a well-defined limit of three seconds maximum before an app is required to display some kind of "busy" indicator. This makes sense from a UX perspective, although the only real difference between the two versions is the explicit 3 second rule, so it's not *that *different.

5.2.3 (missing?)

Um, yeah, I have a feeling Microsoft will fix this little omission.

6.5.4 The SoundEffect class must not be used to play a continuous background music track in an application.

This was changed from a "should" in a note to a separate "must" clause. MediaPlayer is for music, and SoundEffect is for, well, sound effects.

These are all the major changes I noticed, ignoring small wording changes for clarification purposes and the like. Overall, this is not a major policy change.

Relax - Microsoft has not banned open source from Marketplace

Thu, 17 Feb 2011 15:22:40 +0000

This recent

An idea for curbing WP7 piracy

Wed, 29 Dec 2010 20:03:17 +0000

Since it has been made . I've had a few sales already, less than a dozen, but I'm not really expecting much sales from a tool like that. Since that app is now released, I'm considering making a game for WP7 as my next project. More than that, I want to open source it with the intention of having the developer community help out with additional features.

However, therein lies my dilemma. I have already done something like this in the past, and it has not worked out the way I'd hoped. My Windows Mobile app ArkSwitch, which has tens of thousands of downloads, is used by a lot of people, and is installed by default in most custom ROMs today, was open sourced soon after its release in the hopes that the community would help me fix bugs and add features. I have had exactly one person do that at one point in time. That's it. People do download the source code; I can see that on CodePlex. But nobody else is offering to help in any way.

So why should I open source my new game? On one hand, I want to do it regardless of the amount of help I get just so that there's more WP7 code out there that other developers can look at and learn from. That's important. On the other hand, though, I'd probably want to charge a small amount for the game in the Marketplace, and there is nothing preventing an unscrupulous developer from taking that exact same code and submitting it to the Marketplace as well with little or no changes and a different price (or the same price, or free, whatever). I know for a fact that I would be rather upset at such an occurrence. Not because I would be losing potential sales (well, a little because of that) but mainly because this developer did none of the work and is getting rewarded for copying that of others, be it monetarily or in terms of recognition.

So I am seeking advice from my fellow developers. What do you think about open sourcing a project like this? Am I being petty about this? Jaded? Naive? Please let me know your thoughts on this subject, either via comments below or on . That particular project is not quite portable to WP7 for obvious reasons, but I do have various ideas for new projects, as well as a new project I'm currently working on (that I'm not prepared to discuss on a public blog just yet).

Second, I have already invested time (and money) into researching and developing for WP7. I attended MIX10 for the purpose of learning first-hand about the WP7 development story. I have also almost-successfully ported the Bouncy Castle C# cryptography library to WP7 - see or via email at my first name (listed below) -at- of my first WP7 app, ArkWords. It's not the "new project" mentioned above, but it's something I decided to make, well, just because.

Thanks a lot,
Sasha

On software licenses

Fri, 14 May 2010 03:51:10 +0000

Recently, the . This is a modification of the well-known
I don't have a big problem with requiring modifications to be released under an open source license. After all, I released ArkSwitch under Ms-RL. In my opinion, if you want people to help improve your software, you can license it in a way that ensures the improvements are made freely available. However, even that impinges on other developers' freedom, though to a lesser extent.

I do have a big problem with requiring any and all software that merely links with copyleft code to be itself released under a copyleft license. That is simply not freedom. Why should I be barred from making closed software that links to open source code and selling it? That jars my thought processes. When I think of open source, I instinctively think of freedom. In a selfish sense, that includes freedom for me to do what I want with said open source code (within reason, of course). Licenses such as the GPL and, even more so, the AGPL, while claiming freedom, do quite the opposite for programmers' rights.

If you truly want to make your software free and open source, do not choose a restrictive copyleft license. At a minimum, use the LGPL, which allows for linking to proprietary software. As I stated earlier, Ms-RL does so as well. However, for even less restrictions and, consequently, more freedom, choose the Apache, BSD, or MIT/X11 license.

Thoughts on ArkSwitch, my first "popular" app

Sat, 23 Jan 2010 20:55:42 +0000

I think I can safely say that, to date, I've not really released anything that I could consider popular. Things like The Vista Syn are useful tools for a very limited audience. That's exactly what I expected when I released ArkSwitch, a finger-friendly Windows Mobile 6.5.1+ task manager. After all - it's a task manager, which there are plenty of out there; for an unreleased version of WinMo; and it doesn't even have the coolest features of other task managers, like taking over the X button.

Well, that's not entirely how it happened.

I released the app around 11 PM on 2010-01-14 on the
Next on the list is sensing open doors and windows. I would want to do this with my front door, garage door, as well as fridge and freezer doors. Maybe I would monitor downstairs windows as well, but maybe not. For all of this, I can use the
The next item on the list is home theater control. This is already mostly taken care of, as I recently purchased the Logitech and

Home automation project, part I

Mon, 02 Nov 2009 21:51:46 +0000

I have wanted to do home automation for a number of years now, but various things prevented me from experimenting with it, not the least of them being me not owning a house. Now that I actually do own a house, I've been looking at home automation technologies more and more, trying to decide what I can do, what I should do, what is feasible to do, and so forth. This is the first of a series of posts that serve as containers for both my rationale for and progress with this project.

The first issue I should address is, why am I doing this? To be perfectly honest, I don't need to automate anything in my house. However, there are many benefits to automating. The one most commonly cited is burglary prevention - that is, being able to turn on all the lights in the house at once with a single command. That's nice, but it's not a major concern for me. Here are some of the reasons I want to automate my house, not in any particular order:

Geek factor - come on, everyone knows that this would be way cool

Convenience - example: if I forgot to turn off the lights when I left the house, I can do it remotely

Energy savings - being able to turn off all the lights and turn down the heating or AC when nobody's home would save a lot of electricity

That brings us to the second issue - what features do I want in my system? Two of the above reasons bring about obvious feature requirements: remote (i.e., online) control, HVAC access, and light control. In addition to that, I want to have an integrated house monitoring system capable of detecting open windows/doors, movement, and potentially sound. Also, I would really like to control door locks, but that introduces a huge security issue that I have to take into account, so I may just have to postpone doing such a thing until later.

Now, to list all the features as a... list.

Remote (online, Bluetooth, etc) control

HVAC access with heat pump control

Light control - on/off and dimming when applicable

Shade control - controlling blinds in the house

Open door/window sensing

Movement sensing

OPTIONAL: Sound sensing

Multi-room temperature sensing (not necessarily accurate enough for presence detection, just comfort)

OPTIONAL: home theater control - at least on/off - without killing power at the wall socket

OPTIONAL: door lock and/or garage control

The above features can be broken down in a fairly clean way into hardware and software categories. In fact, pretty much the only category that isn't hardware is the first one - remote control. It's also the one posing the biggest challenge in terms of implementation. Almost everything else I can just buy and connect. However, since I have not found any reasonably-priced online home automation controllers, I will likely have to implement one myself.

Another issue that must be discussed is underlying home automation technology. I have decided to go with Z-Wave after carefully evaluating my choices. Here's how I came to this decision. In the home automation market, there are powerline-based systems and wireless ones. While powerline technology offers more security than wireless, it is very susceptible to interference from, e.g., vacuum cleaners. In addition, the most popular powerline-based standard - X10 - is unreliable in that it doesn't communicate success or failure of commands, so you don't know whether your command reached its destination. Other standards, such as UPB and Insteon, fix certain X10 issues, but ultimately, they are all affected by noisy powerlines.

On the wireless side, the most popular technologies are ZigBee and Z-Wave. ZigBee is based on IEEE 802.15.4, and is closer to an open standard than Z-Wave. However, there aren't that many products out there based on this technology, and those that are out are extremely expensive compared to their Z-Wave equivalents. I can see ZigBee eventually becoming popular and less expensive, but that won't happen for a while, so I can either wait an undetermined period of time, or just go with Z-Wave.

SQL Server sproc performance trouble?

Tue, 01 Sep 2009 17:58:06 +0000

Despite this post's title looking a bit like a spam subject line, this is a serious post about an issue we ran into today at work. We have a stored procedure that gathers some statistics for us, and a really strange thing was happening with it. When run from SSMS, it took less than one second to execute, but when run from code, it actually timed out while executing over two minutes. Why would it run so fast through SSMS and yet so slow through our code?

The answer, it turns out, is a SQL Server feature called "parameter sniffing". It is supposed to optimize the query by looking at the actual parameters that are being passed in, instead of generating a generic execution plan. However, sometimes it can cause performance issues instead of alleviating them. The reasons for that are discussed in the

Arktronic.com

A redone blog, again

Optimize for cognitive load

Application of CLT

Functions

Caveats

The guide

Step 0: Prerequisites

Step 1: Download and install Clover

Step 3: Finalize the macOS installer changes

Step 4: Create the VM

Step 5: Configure the VM

Step 6: Connect your installer disk

Virtual Hackintosh, part 2: the easy route

Caveats

The guide

Step 0: Prerequisites

Step 1: Create an installer disk

Step 2: Create the VM

Step 3: Configure the VM

Step 5: Install macOS

Finishing up

Next time

Virtual Hackintosh, part 1: the concepts

Motivation

The theory and the terminology

Hardware requirements

Software solutions

Next steps

Project PiNES

The hardware

Adding system testing to your process

Where to run system tests

Bad timing

Unrealistic steps

Systemic fixes

Wrapping up

Automated software testing, part 5: integration testing

Automated software testing, part 4: unit tests and test doubles

Verify external interactions

Setup, execution, verification

Refactoring tests

An uncaught bug

Automated software testing, part 3: unit tests

Let's get interactive

Test doubles

Automated software testing, part 2: the types

An overview of the common test types

Design matters

Next steps

Automated software testing, part 1: reasoning

Why test?

When does the above not apply?

Passion in software engineering

The answer is 42

A passionate engineer

Delve deeper

Not just a bullet point

Encryption and you

Misconfiguration

Confidentiality without integrity: an example

Go forth and encrypt

The why and how of personal VPNs

What's the point (-to-point)?

Public Wi-Fi is bad?

Do it now

API design: adequate vs. awesome

UX? For an API?

But why?

Introducing Some Random Developer

Provide good examples

Tedious, detailed documentation

YAPoP (Yet Another Post on Pairing)

SSL yourself

HashProp - a better way to MD5 (and SHA) in Windows

Beware of silent failures

Farewell, Windows Phone

Digital trust issues

How to do a clean install of Windows 7 or 8 on Samsung Chronos laptops

How to properly get a Windows Phone app's assembly version