pages tagged ick

Security isolation in CI engines

Fri, 28 Feb 2020 09:28:00 +0000

A continuous integration engine (CI) takes the source code for a software project and ensures it works. In less abstract terms, it builds it, and runs any automated tests it may have. The exact steps for that depend heavily on the CI engine and the project, but can be thought of as follows (with concrete examples of possible commands):

retrieve the desired revision of the source code (git clone, git checkout)
install build dependencies (dpkg-checkbuilddeps, apt install)
build (./configure, make)
test (make check)

This is dangerous stuff. In the specific case of an open, hosted CI service, it's especially dangerous: anyone can submit any build, and that build can do anything, including attack computers anywhere on the Internet. However, even in a CI engine that only builds projects for in-house developers, it's risky: most attacks on IT are done by insiders.

Apart from actual attacks, building software is dangerous also due to accidents: a mistake in the way software is built, or automatically tested, can result in what looks and behaves like an attack. An infinite loop can use excessive amounts of CPU resources, or block other projects from getting built.

I've been thinking about ways to deal with this, in the context of developing a CI engine, and here's a list of specific threats I've come up with:

excessive use build host resources
- e.g., CPU, GPU, RAM, disk, etc
- mitigation: use quotas or other hard limits that can't be exceeded (e.g., dedicated file system for build, virtual machine with a virtual memory limit)
- mitigation: monitor use, stop build if use goes over a limit, if a quota is infeasible (e.g., CPU time)
excessive use of network bandwidth
- mitigation: monitor use, stop build if it goes over a limit
attack on a networked target via a denial of service attack
- e.g., build joins a DDoS swarm, or sends fabricated SYN packets to prevent target from working
- mitigation: prevent direct network access for build, force all outgoing connections to go via a proxy that validates requests and stops build if anything looks suspicious
attack on build host, or other host, via network intrusion
- e.g., port scanning, probing for known vulnerabilities
- mitigation: prevent direct network access for build, force all outgoing connections to go via a proxy that validates requests and stops build if anything looks suspicious
attack build host directly without network
- e.g., by breaching security isolation using build host kernel or hardware vulnerabilities, or CI engine vulnerabilities
- this includes eavesdropping on the host, and stealing secrets
- mitigation: keep build host up to date on security updates
- mitigation: run build inside a VM controlled by CI engine (on the assumption that a VM provides better security isolation than a Linux container)

I'm sure this is not an exhaustive list. If you can think of additional risks, do tell me.

My current plan for mitigating all of the above looks as follows:

there are two, nested virtual machines
the outer VM is the manager, the inner VM is the builder
the manager creates, controls, monitors, and destroys the builder
the outer VM is probably Debian Linux, since that what I know best, using libvirt with Qemu and KVM to manage the inner VM
the inner VM can be any operating system, as long as it can run as a Qemu/KVM guest, and provides ssh access from the outer VM
the manager runs commands on the builder over ssh, or possibly via serial console (ssh would be simpler, though)
both VMs have a restricted amount of CPUs, RAM, disk space
the manager monitors the builder's use of CPU time, bandwidth use
the manager proxies and firewalls all outgoing network access to prevent any access that isn't explicitly allowed

To look at the build steps from the top of this article, they would work something like this:

retrieve the desired revision of the source code: the builder does this, but proxied via the manager, which checks that only from servers listed as allowed for this project are connected
install build dependencies: the builder downloads the build dependencies, but proxied via the manager, which checks that downloads come only from servers listed as allowed for this project
build: runs inside the builder
test: runs inside the builder

It would be awesome if the manager could disable the builder from having network access after build dependencies are installed. This would be feasible if the build recipe is structured in a way that allows the manager to know what part is doing what. (If I'm designing the CI engine, then I can probably achieve that.)

It would be even more awesome if the manager could do all the downloading, but given the guest may need to use tools specific for its operating system, which might not be available on the operating system of the manager, this might not be feasible. A filtering HTTP or HTTPS proxy may need to be enough.

What threat am I missing? Are my mitigations acceptable?

If you want to comment on this blog post, please send me email (liw@liw.fi), or respond on the fediverse on

I'd appreciate feedback, preferably by email (liw@liw.fi).

Context

I develop a number of (fairly small) programs, as a hobby. Some of them I also maintain as packages in Debian. All of them I publish as Debian packages in my own APT repository. I want to make the process for making a release of any of my programs as easy and automated as possible, and that includes building Debian packages and uploading them to my personal APT repository, and to Debian itself.

My personal APT repository contains builds of my programs against several Debian releases, because I want people to have the latest version of my stuff regardless of what version of Debian they run. (This is somewhat similar to what OEMs that provide packages of their own software as Debian packages need to do. I think. I'm not an OEM and I'm extrapolating wildly here.)

I currently don't provide packages for anything but Debian. That's mostly because Debian is the only Linux distribution I know well, or use, or know how to make packages for. I could do Ubuntu builds fairly easily, but supporting Fedora, RHEL, Suse, Arch, Gentoo, etc, is not something I have the energy for at this time. I would appreciate help in doing that, however.

I currently don't provide Debian packages for anything other than the AMD64 (x86-64, "Intel 64-bit") architecture. I've previously provided packages for i386 (x86-32), and may in the future want to provide packages for other architectures (RISC-V, various Arm variants, and possibly more). I want to keep this in mind for this discussion.

Overview

For the context of this blog post, let's assume I have a project Foo. Its source code is stored in foo.git. When I make a release, I tag it using a signed git tag. From this tag, I want to build several things:

A release tarball. I will publish and archive this. I don't trust git, and related tools (tar, compression programs, etc) to be able to reproducibly produce the same bit-by-bit compressed tarball in perpetuity. There's too many things that can go wrong. For security reasons it's important to be able to have the exact same tarball in the future as today. The simplest way to achive this is to not try to reproduce, but to archive.
A Debian source package.
A Debian binary package built for each target version of Debian, and each target hardware architecture (CPU, ABI, possibly toolchain version). The binary package should be built from the source package, because otherwise we don't know the source package can be built.

The release tarball should be put in a (public) archive. A digital signature using my personal PGP key should also be provided.

The Debian source and binary packages should be uploaded to one or more APT repositories: my personal one, and selected packages also the Debian one. For uploading to Debian, the packages will need to be signed with my personal PGP key.

(I am not going to give my CI access to my PGP key. Anything that needs to be signed with my own PGP key needs to be a manual step.)

Package versioning

In Debian, packages are uploaded to the "unstable" section of the package archive, and then automatically copied into the "testing" section, and from there to the "stable" section, unless there are problems in a specific version of a package. Thus all binary packages are built against unstable, using versions of build dependencies in unstable. The process of copying via testing to stable can take years, and is a core part of how Debian achieves quality in its releases. (This is simplified and skips consideration like security updates and other updates directly to stable, which bypass unstable. These details are not relevant to this discussion, I think.)

In my personal APT repository, no such copying takes place. A package built for unstable does not get copied into section with packages built for a released version of Debian, when Debian makes a release.

Thus, for my personal APT repository, there may be several builds of the any one version of Foo available.

foo 1.2, built for unstable
foo 1.2, built for Debian 9
foo 1.2, built for Debian 8

In the future, that list may be expanded by having builds for several architectures:

foo 1.2, built for unstable, on amd64
foo 1.2, built for Debian 9, on amd64
foo 1.2, built for Debian 8, on amd64
foo 1.2, built for unstable, on riscv
foo 1.2, built for Debian 9, on riscv
foo 1.2, built for Debian 8, on riscv

When I or my users upgrade our Debian hosts, say from Debian 8 to Debian 9, any packges from my personal APT archive should be updated accordingly. When I upgrade a host running Debian 8, with foo 1.2 built for Debian 8, gets upgraded to Debian 9, foo should be upgraded to the version of 1.2 built for Debian 9.

Because the Debian package manager works on combinations of package name and package version, that means that the version built for Debian 8 should have a different, and lesser, version than the one built for Debian 9, even if the source code is identical except for the version number. The easiest way to achieve this is probably to build a different source package for each target Debian release. That source package has no other differences than the debian/changelog entry with a new version number, so it doesn't necessarily need to be stored persistently.

(This is effectively what Debians "binary NMU" uploads do: use the same source package version, but do a build varying only the version number. Debian does this, among other reasons, to force a re-build of a package using a new version of a build depenency, for which it is unnecessary to do a whole new sourceful upload. For my CI build purposes, it may be useful to have a new source package, for cases where there are other changes than the source package. This will need further thought and research.)

Thus, I need to produce the following source and binary packages:

foo_1.2-1.dsc — source package for unstable
foo_1.2-1.orig.tar.xz — upstream tarball
foo_1.2-1.debian.tar.xz — Debian packaging and changes
foo_1.2-1_amd64.deb — binary package for unstable, amd64
foo_1.2-1_riscv.deb — binary package for unstable, riscv
foo_1.2-1~debian8.dsc — source package for Debian 8
foo_1.2-1~debian8.debian.tar.xz — Debian packaging and changes
foo_1.2-1~debian8_amd64.deb — binary package for Debian 8, amd64
foo_1.2-1~debian8_riscv.deb — binary package for Debian 8, riscv
foo_1.2-1~debian9.dsc — source package for Debian 9
foo_1.2-1~debian9.debian.tar.xz — Debian packaging and changes
foo_1.2-1~debian9_amd64.deb — binary package for Debian 9, amd64
foo_1.2-1~debian9_riscv.deb — binary package for Debian 9, riscv

The orig.tar.xz file is a bit-by-bit copy of the upstream release tarball. The debian.tar.xz files have the Debian packaging files, plus any Debian specific changes. (For simplicity, I'm assuming a specific Debian source package format. The actual list of files may vary, but the .dsc file is crucial, and references the other files in the source package. Again, these details don't really matter for this discussion.)

To upload to Debian, I would upload the foo_1.2-1.dsc source package from the list above, after downloading the files and signing them with my PGP key. To upload to my personal APT repository, I would upload all of them.

Where should Debian packaging be stored in version control?

There seems to be no strong consensus in Debian about where the packaging files (the debian/ subdirectory and its contents) should be stored in version control. Several approaches are common. The examples below use git as the version control system, as it's clearly the most common one now.

The "upstream does the packaging" approach: upstream's foo.git also contains the Debian packaging. Packages are built using that. This seems to be especially common for programs, where upstream and the Debian package maintainer are the same entity. That's also the OEM model.
The "clone upstream and add packaging" approach: the Debian package maintainer clonse the upstream repository, and adds the packaging files in a separate branch. When upstream makes a release, the master branch in the packaging repository is updated to match the upstream's master branch, and the packaging branch is rebased on top of that.
The "keep it separate" approach: the Debian packager puts the packaging files in their own repository, and the source tree is constructed from botht the upstream repository and the packaging repository.

For my own use, I prefer the "upstream does packaging" approach, as it's the least amount of friction for me. For ick, I want to support any approach.

There are various tools for maintaining package source in git (e.g., dgit and git-buildpackage), but those seem to not be relevant to this blog post, so I'm not discussing them in any detail.

The build process

Everything starts from a signed git tag in the foo.git plus additional tags in any packaging repository. The tags are made by the upstream developers and Debian package maintainers. CI will notice the new tag, and build a release from that.

Create the upstream tarball (foo-1.2.tar.gz).
Manully download and sign the upstream tarball with PGP.
Manully publish the upstream tarball and its signature in a suitable place.
Create the Debian source package for unstable (foo_1.2-1.dsc), using a copy of the upstream tarball, renamed.
Using the Debian source package, build a Debian binary package for unstable for each target architecture (foo_1.2-1_amd64.deb etc).
For each target Debian release other than unstable, create a new source package by unpacking the source package for unstable, and adding a debian/changelog entry with ~debianN appended to the version number. If there is a need, make any additional Debian release specific changes to the source package.
Build each of those source packages for each target architecture, in a build environment with the target Debian release. (foo_1.2-1~debianN_amd64.deb etc).
Upload all the Debian source and binary packages to an APT repository that allows upload by CI. Have that APT repository sign the resulting Packages file with its own PGP key.
Manully download the Debian packages and sign the unstable build to Debian, and upload it to Debian. (Source package only, except in cases where the binary package also needs to be uploaded, such as for new packages.)

Ick version 0.53 released: CI engine

Wed, 18 Jul 2018 18:14:00 +0000

I have just made a new release of ick, my CI system. The new version number is 0.53, and a summary of the changes is below. The source code is pushed to my git server (git.liw.fi), and Debian packages to my APT repository (code.liw.fi/debian). See A notable change from previous releases should be invisible to users: the release is built by ick2 itself, instead of my old mostly-manual CI script. This means I can abandon the old script and live in a brave, new world with tea, frozen-bubble, and deep meaningful relationships with good people.

Version 0.53, released 2018-07-18

Notification mails now include controller URL, so it's easy to see which ick instance they come from. They also include the exit code (assuming the notification itself doesn't fail), and a clear SUCCESS or FAILURE in the subject.

Icktool shows a more humane error message if getting a token fails, instead of a Python stack trace.

Icktool will now give a more humane error message if user triggers the build of a project that doesn't exist, instead of a Python stack trace.

Icktool now looks for credentials using both the controller URL, and the authentication URL.

Icktool can now download artifacts from the artifact store, with the new get-artifact subcomand.

The archive: workspace action now takes an optional globs field, which is a list of Unix filename globs, for what to include in the artifact. Also, optionally the field name_from can be used to specify the name of a project parameter, which contains the name of the artifact. The default is the artifact_name parameter.

A Code of Conduct has been added to the ick project.
Ick can, right now:
- Build system trees for containers.
- Use system trees to run builds in containers.
- Build Debian packages.
- Publish Debian packages via its own APT repository.
- Deploy to a production server.
There's still many missing features. Ick is by no means ready to replace your existing CI/CD system, but if you'd like to have a look at ick, and help us make it the CI/CD system of your dreams, now is a good time to give it a whirl.

(Big missing features: web UI, building for multiple CPU architectures, dependencies between projects, good documentation, a development community. I intend to make all of these happen in due time. Help would be welcome.)

Ick: a continuous integration system

Mon, 22 Jan 2018 20:11:00 +0000

TL;DR: Ick is a continuous integration or CI system. See

Architecture

Ick has an architecture consisting of several components that communicate over HTTPS using RESTful APIs and JSON for structured data. See the

Please give feedback

If you try ick, or even if you've just read this far, please share your thoughts on it. See the

I am currently using the first generation of Ick and it is a rigid, cumbersome, and fragile thing. It works well enough that I don't miss Jenkins, but I would like something better. That's the second generation of Ick, or Ick2, and that's what we discussed with Daniel.

Where pretty much everything in Ick1 is hardcoded, everything in Ick2 will be user-configurable. It's my last, best chance to go completely overboard in the

Ick2 builds static websites from source in a git repository, using ikiwiki, and published to a web server using rsync. A change to the git repository triggers a new build. It can handle many separate websites, and if given enough worker machines, can build many of them concurrently.

This is a real task, and something we already do with Ick1 at work. It's a reasonable first step for the new program.

Some decisions we made:

The Ick2 controller, which decides which projects to build, and what's the next build step at any one time, will be reactive only. It will do nothing except in response to an HTTP API request. This includes things like timed events. An external service will need to poke the controller at the right time.
The controller will be accompanied by worker manager processes, which fetch instructions of what to do next, and control actual worker over ssh.
Provisioning of the workers is out of scope for the MVP. For the MVP we are OK with a static list of workers. In the future we might make worker registration be a dynamic things, but not for the MVP. (Parts or all of this decision may be changed in the future, but we need to start somewhere.)
The MVP publishing will happen by running rsync to a web server. Providing credentials for the workers to do that is the sysadmin's problem, not something the MVP will handle itself.
The MVP needs to handle more than one worker, and more than one pipelines, and needs to build things concurrently when there's call for it.
The MVP will need to read the pipelines (and their steps and any other info) from YAML config files, and can't have that stuff hardcoded.
The MVP API will have no authentication or authorization stuff yet.

The initial pipelines will be basically like this, but expressed in some way by the user:

Clone the source repoistory.
Run ikiwiki --build to build the website.
Run rsync to publish the website on a server.

Assumptions:

Every worker can clone from the git server.
Every worker has all the build tools.
Every worker has rsync and access to every web server.
Every pipeline run is clean.

Actions the Ick2 controller API needs to support:

List all existing projects.
Trigger a project to build.
Query what project builds are running.
Get build logs for a project: current log (from the running build), and the most recent finished build.

A sketch API:

POST /projects/foo/+trigger

Trigger build of project foo. If the git hasn't changed, the build runs anyway.
GET /projects

List names of all projects.
GET /projects/foo

On second thought, I can't think of anything useful for this to return for the MVP. Scratch.
GET /projects/foo/logs/current

Return entire known build log captured so far for the currently running build.
GET /projects/foo/logs/previous

Return entire build log for latest finished build.
GET /work/bar

Used by worker bar: return next not-yet-finished step to run as a JSON object containing fields "project" (name of project for which to run the step) and "shell" (a shell command to run). The call will return the same JSON object until the worker reports it as having finished.
POST /work/bar/snippet

Used by worker bar to report progress on the currently running step: a JSON object containing fields "stdout" (string with output from the shell command's stdout), "stderr" (ditto but stderr), and "exit_code" (the shell command's exit code, if it's finished, or null).

Sequence:

Git server has a hook that calls "GET /projects/foo/+trigger" (or else this is simulated by user).
Controller add a build of project foo to queue.
Worker manager calls "GET /work/bar", gets a shell command to run, and starts running it on its worker.
While worker runs shell command, every second or so, worker manager calls "POST /work/bar/snippet" to report progress including collected output, if any.
Controller responds with OK or KILL, and if the latter, worker kills the command it is running. Worker manager continues reporting progress via snippet until shell command is finished (on its own or by having been killed).
Controller appends any output reported via .../snippet. When it learns a shell command has finished, it updates its idea of the next step to run.
When controller learns a project has finished building, it rotates the current build log to be the previous one.

The next step will probably be to sketch a yarn test suite of the API and implement a rudimentary one.

Thinking about CI, maybe writing ick2

Fri, 09 Sep 2016 18:03:59 +0000

A year ago I got tired of Jenkins and wrote a CI system for myself, At this point I'm still thinking about very high level concepts. I've not written any code, and probably won't in the next couple of months. But I had to get this out of my brain.