Googlier.com | Andrea Corbellini

Advertise with Googlier.com Andrea Corbellinihttps://andrea.corbellini.blog/Tue, 20 Oct 2026 08:10:00 +0000This blog has moved to a new domainhttps://andrea.corbellini.blog/2026/10/20/new-domain/TL;DR: This blog has moved from <a href="https://andrea.corbellini.name">https://andrea.corbellini.name</a> to <a href="https://andrea.corbellini.blog">https://andrea.corbellini.blog</a>. Please update all URLs pointing to this blog, and sorry for the inconvenience. <h1 id="me-and-name">Me and .name</h1> Back in 2014 (12 years ago), I wanted to claim my small slice of the internet, and so I purchased my first personal domain name, where I would have hosted my blog: <code>andrea.corbellini.name</code>. To be honest with you, I would have liked a more creative domain name, but I’m notoriously not good at giving names to things, and so I couldn’t come up with anything better than <code>first.last.name</code>. For those who are not familiar with the matter, <code>.name</code> domains have always been special and different from other domains: when they launched (back in 2002, according to Wikipedia), they only allowed third-level domain registrations, meaning that you could register the third-level domain <code>first.last.name</code>, but could not register the second-level domain <code>last.name</code>. Second-level domains would be somewhat shared between the folks sharing the same last name. A few years later (in 2009, still according to Wikipedia), this restriction was lifted, and the ability to register second-level domains was offered. So when I went to register my domain name I had two options: register <code>corbellini.name</code> or <code>andrea.corbellini.name</code>. I wanted to be kind to other people and so I decided to register the third-level domain, leaving <code>corbellini.name</code> subdomains available to other people as well. I don’t think my last name is very common, and in fact I would bet that my domain is the only one under <code>corbellini.name</code> even today, so perhaps we could say that my kindness was totally unnecessary. What’s sure is that my kindness did not get rewarded. Recently, Verisign, the operator of <code>.name</code> domains, decided to discontinue third-level domains like mine. This means that sometime in the future, <code>andrea.corbellini.name</code> will disappear from the internet. It will cease to exist. Any attempt to resolve that domain name and to reach my blog will fail. Visitors will get a generic error from their browser, without any hints about what happened, why, and where my blog can be found. No redirect, no “click here to visit the new site”, nothing. The void. <figure> <img src="https://andrea.corbellini.blog/images/librewolf-server-not-found.png" alt="Server Not Found error from LibreWolf"> </figure> <h1 id="a-new-home-blog">A new home: .blog</h1> I recently registered a new domain at <code>andrea.corbellini.blog</code> and moved my blog there. Needless to say, this time I own the whole <code>corbellini.blog</code> domain (also because there’s no other choice with <code>.blog</code>). Right now, my <code>.name</code> domain redirects to my new <code>.blog</code>, but this won’t last forever. I haven’t published a ton of stuff on this blog (not nearly as much as I wished), but some of the articles I published have become quite popular in their niche ecosystem, in particular my articles about Elliptic Curve Cryptography and Authenticated Encryption. I see a ton of visits from people who found links to my articles from other sites. All those links will be broken once my <code>.name</code> domain will be terminated, which is a shame. It’s still not very clear to me when this will happen, but it will happen at some point, so I need to ask everyone who has links pointing to my blog: please change all the URLs to my articles from <code>.name</code> to <code>.blog</code>. <h1 id="the-perks-of-blog">The perks of .blog</h1> I must say, while it’s certainly disappointing to have a domain name (that has served me and others for so many years) disappear in a blip without any control from me, or any concrete possibility to appeal, owning a second-level domain comes with its own perks. One of them is that now I can enable DNSSEC, which is something I could never do with my third-level <code>.name</code> domain, as I had no control over the second-level domain (who knows, maybe there was a way, but I never figured it out). And in fact I’m happy to say that I did enable DNSSEC on <code>andrea.corbellini.blog</code>. Another perk is that I can now use services that were not previously available to me, like Cloudflare. Services like Cloudflare in fact do not understand third-level domains and did not allow me to register with them because they wanted proof of ownership of the second-level domain. Now this is no longer an obstacle. Reaching my blog also becomes a bit easier: <a href="https://corbellini.blog">https://corbellini.blog</a> in fact redirects to <a href="https://andrea.corbellini.blog">https://andrea.corbellini.blog</a>. Perhaps an amusing fact about <code>.blog</code> specifically is that it’s owned and operated by a company called “Knock Knock WhoIS There, LLC”, and I find that name quite funny. Having said that, I don’t know anything about this company or their reputation. <h1 id="final-remarks">Final remarks</h1> I know there are a few active attempts at keeping third-level <code>.name</code> domains alive. A very prominent reconsideration request was recently sent by <a href="https://neil.fraser.name/news/2026/10/04/">Neil Fraser</a>. I will very likely join these attempts, although honestly I’ve witnessed so much enshittification in the past few years that I’ve lost hope in most (if not all) corporations, and I seriously doubt that things will change in this time frame. What is happening to <code>.name</code> domains in my opinion is a small symptom of something much bigger: the fast decay of modern corporate culture, focused on profits above anything else, to the point that screwing customers over and over has become standard practice. I doubt Verisign is going to sacrifice revenue, keep on the costs of maintaining third-level <code>.name</code> domains, and reverse their course of action. And to those who are going to point out that it’s ICANN who has the power of reversing this decision, and ICANN is a non-profit organization, not one of the corporations I’m complaining about: sure, you may have a point. I don’t want to either defend nor attack ICANN’s role in this matter. The thing is that I don’t see a world where ICANN forces Verisign to do something they don’t want to do anymore, regardless of what ICANN’s intentions are. I’m happy to be proven wrong. I truly hope this decision gets reversed and I get to keep my <code>.name</code> domain. But at this point I lost all hope.andreacorbelliniTue, 20 Oct 2026 08:10:00 +0000tag:andrea.corbellini.blog,2026-10-20:/2026/10/20/new-domain/blogTesting crash recovery features in a CI environmenthttps://andrea.corbellini.blog/2025/09/09/testing-crash-recovery-features-in-ci/About two years ago I wrote a Rust crate to fulfill this promise: <blockquote> If a crash occurs while updating a file, the file either contains the old contents, or the new contents, nothing in between. </blockquote> This crate essentially solves the following problem: when you update a file, generally you open it, truncate it, write to it block-by-block, and eventually close it. The problem with this process is that if a crash occurs at any point during this process (the program segfaults, the kernel panics, the machine loses power, …), your file will be left in an intermediate state where the old contents are completely lost, and the new contents are only partially written (if at all). My crate solves this problem by making file updates atomic. This crate is publicly available as <a href="https://crates.io/crates/atomic-write-file">atomic-write-file</a> and you can read its description for more details (sorry, I’m very bad at naming things). The way the crate works is simple: <ol> <li>Instead of opening the target file directly, the crate opens a temporary file.</li> <li>You write to the temporary file.</li> <li>Once all changes are in, the temporary file contents are synced to the storage, and it is atomically renamed so that it replaces the target file.</li> </ol> This is not by any chance a new technique that I invented, and it’s a very common strategy to solve the problem. This technique guarantees that if a crash occurs at or before step 3, then the file will have the old contents, unchanged. If a crash occurs after step 3, then the file will have the new contents. There’s a small caveat with this technique: if a crash occurs between steps 1 and 3, then the crate will leave behind a temporary file that occupies some space on the storage device with no purpose. This is where Linux anonymous temporary files come into play: this Linux-specific feature allows you to create temporary files that are written on the filesystem, but are not given a path in the filesystem. If a crash occurs between step 1 and 3, the temporary file will simply be forgotten into oblivion. <h1 id="meet-the-opponent-btrfs">Meet the opponent: btrfs</h1> Using Linux anonymous temporary files seemed very appealing for atomic-write-file to avoid leaving leftovers behind, although I knew that they’re not the perfect solution: there’s limited support (they’re supported only on Linux, and only on some filesystems), they require the use of Linux features that may not be available all the time (the <code>/proc</code> filesystem), and they also have the problem that, sooner or later, they will need to be given a name if we want to be able to replace the target file, and this leaves a small time window during which we could leave some cruft behind. But none of these are huge problems, and in fact atomic-write-file is able to do a best effort attempt at using anonymous temporary files, and reverting to regular files if that doesn’t work. One day I was notified about <a href="https://github.com/andreacorbellini/rust-atomic-write-file/issues/6">an issue on using my crate on the btrfs file system</a>: after a crash, my crate could break its promise and leave the file contents empty. I did some investigation on the issue that <a href="https://github.com/andreacorbellini/rust-atomic-write-file/issues/6#issuecomment-2048836275">you can read about if you’re interested</a>, and then fixed it. This problem was specific to btrfs, and I argue that it’s actually caused by a btrfs bug, rather than a bug in the crate itself, but nonetheless this issue had revealed a critical flaw of my crate: I did not have any tests to check if my crate was fulfilling its promise or not! Therefore I decided to create a small test suite for Linux to simulate crashes and inspect the contents of files after those crashes occurred on a variety of filesystems, which is what this blog post is about. This test suite now runs on my laptop and <a href="https://github.com/andreacorbellini/rust-atomic-write-file/actions/workflows/crash-tests.yml">on GitHub Actions</a>, and potentially any other CI environment. <h1 id="testing-strategy">Testing strategy</h1> Here’s the idea I came up with: I can create a virtual machine, with a virtual storage device attached to it, and a filesystem initialized with a file. The virtual machine updates the file using atomic-write-file, and then triggers a kernel panic. After that, the file is inspected for consistency. So here is how it works in practice: there are 3 main pieces: 1. A test binary that is responsible for updating the test file using atomic-write-file. 1. An <code>init.sh</code> script that runs inside the virtual machine. 1. The <code>run-tests.sh</code> script that puts everything together and starts up the virtual machine. <h2 id="the-test-binary">The test binary</h2> The test binary that is a <a href="https://github.com/andreacorbellini/rust-atomic-write-file/blob/master/crash-tests/linux/src/main.rs">short Rust program</a> that I can paste here: <div class="highlight"><pre><code>use atomic_write_file::AtomicWriteFile; use std::io::Write; fn main() { let mut file = AtomicWriteFile::open("/test/file").expect("open failed"); file.write_all(b"hello").expect("write failed"); file.commit().expect("commit failed"); } </code></pre></div> Nothing special to see here: this is the minimal code required to use atomic-write-file. <h2 id="the-initsh-script">The init.sh script</h2> <a href="https://github.com/andreacorbellini/rust-atomic-write-file/blob/master/crash-tests/linux/init.sh">The <code>init.sh</code> script</a> is more interesting: this script in fact is meant to run twice. The first time it runs, <code>init.sh</code> will run the test binary above and trigger the kernel panic. The second time, it will read the test file contents and report them back <code>run-tests.sh</code> script for inspection. Let’s take a closer look at what it does: first, it mounts the <code>/proc</code> and <code>/sys</code> virtual filesystems. These are needed for some basic system functions, and also to enable the Linux anonymous temporary file feature. <div class="highlight"><pre><code>mount -t proc none /proc mount -t sysfs none /sys </code></pre></div> Then it mounts the filesystem containing the test file. Before it can do that, however, it needs to load the correct kernel module for the filesystem. Nothing too fancy here: the filesystem type is simply specified on the kernel command line (read through <code>/proc/cmdline</code>): <div class="highlight"><pre><code>fstype=$(grep -Eo 'test.fs=\w+' /proc/cmdline | cut -d= -f2) modprobe "$fstype" || true mount -t "$fstype" /dev/sdb /test </code></pre></div> Then, if this script is run for the first time, it will run the test executable and quit: <div class="highlight"><pre><code>echo 'Running test binary' atomic-write-file-test </code></pre></div> <code>init.sh</code> (as the name suggests) is run as the init script of the virtual machine. In Linux, if the init script quits without invoking the proper shutdown sequence, the kernel will panic. So we don’t actually need to do anything special to trigger a panic. I could have inserted a <code>echo c > /proc/sysrq-trigger</code> line to make it more explicit, but then I figured that having that line wouldn’t be nice for people who want to test the script on their system. You might ask: how does <code>init.sh</code> know if it’s the first time it gets called or the second? Again, nothing fancy here: <code>run-tests.sh</code> gives that hint using a kernel command line argument: <div class="highlight"><pre><code>if grep -q test.verify /proc/cmdline; then # ... fi </code></pre></div> If <code>test.verify</code> is specified on <code>/proc/cmdline</code>, then it means this is the second time this script is run. In that case, the file contents are just printed on the console: <div class="highlight"><pre><code>if grep -q test.verify /proc/cmdline; then echo 'Verifying test file contents' echo '-----' xxd /test/file echo '-----' poweroff -f fi </code></pre></div> This uses <code>xxd</code> so in case there are garbled characters on screen, we can comfortably analyze the output and figure out what happened. At the end, the system is more gracefully shut down using <code>poweroff</code> to avoid another kernel panic (this is not really necessary). <h1 id="the-run-testssh-script">The run-tests.sh script</h1> And then there’s <a href="https://github.com/andreacorbellini/rust-atomic-write-file/blob/master/crash-tests/linux/run-test.sh">the <code>run-tests.sh</code> script that crates the virtual storage, launches the virtual machine, and inspects the file contents</a>. This is where most the complexity is, so let’s go step-by-step (the code below has been simplified from the original for readability): The first thing it does is compiling the test binary above. The virtual machine runs in a very barebone environment, so I used static linking to avoid having to copy additional libraries like <code>libc</code>: <div class="highlight"><pre><code>target=$(uname -m)-unknown-linux-gnu RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --features "$cargo_features" --target "$target" </code></pre></div> Then <code>run-tests.sh</code> creates a minimal <a href="https://en.wikipedia.org/wiki/Initial_ramdisk">initramfs</a> for the virtual machine. The initramfs is a filesystem that gets mounted by Linux early at boot. It’s generally used by Linux distributions to load the main operating system, but in my case the initramfs contains everything needed for the test. In particular, it contains: <ul> <li>The test binary.</li> <li>The <code>init.sh</code> script.</li> <li><a href="https://en.wikipedia.org/wiki/BusyBox">BusyBox</a>, to get the needed utilities like <code>xxd</code> and <code>poweroff</code>.</li> <li>The kernel modules to mount filesystems and their dependencies.</li> </ul> The first 3 items are easy to set up: <div class="highlight"><pre><code>cp "init.sh" -T "$initramfs_build_dir/sbin/init" cp "atomic-write-file-test" -t "$initramfs_build_dir/bin" cp "$(which busybox)" -t "$initramfs_build_dir/bin" "$busybox" --install -s "$initramfs_build_dir/bin" </code></pre></div> The kernel modules are a bit more complicated. First of all: where can we get them from? Ideally, I would have liked to download them from some minimal Linux distribution, but this turned out to be more complicated than expected. In the end, I decided to just steal them from the host operating system: <div class="highlight"><pre><code>modules=/usr/lib/modules/$(uname -r) cp -a "$modules" -t "$initramfs_build_dir/lib/modules" while read -r mod_path; do if [[ "$mod_path" = *.ko.gz ]]; then gunzip "$mod_path" -o "$mod_path.uncompressed" elif [[ "$mod_path" = *.ko.xz ]]; then unxz "$mod_path" -o "$mod_path.uncompressed" elif [[ "$mod_path" = *.ko.zst ]]; then unzstd "$mod_path" -o "$mod_path.uncompressed" else continue fi mv "$mod_path.uncompressed" "$mod_path" done < <(modprobe --dirname "$initramfs_build_dir" --show-depends "$filesystem_type" | grep ^insmod | cut -d' ' -f2) </code></pre></div> What this does is copying all the module files from the host, getting all the dependencies for the filesystem module using <code>modprobe</code>, and then uncompressing those modules if they’re compressed (I found this much easier than adding compression support in the virtual machine). An alternative approach would be to simply decompress all the modules, but that’s much slower. Once everything has been copied over and uncompressed, the initramfs is created: <div class="highlight"><pre><code>initramfs=rootfs.img mksquashfs "$initramfs_build_dir" "$initramfs" </code></pre></div> Now it’s time to create the virtual storage device with the filesystem, which is a trivial task: <div class="highlight"><pre><code>testfs=testfs.img dd if=/dev/zero of="$testfs" bs=1M count=300 "mkfs.$filesystem_type" "$testfs" </code></pre></div> Finally, we can start our VM using <a href="https://en.wikipedia.org/wiki/QEMU">QEMU</a>: <div class="highlight"><pre><code>qemu=qemu-system-$(uname -m) kernel=/boot/vmlinuz-$(uname -r) "$qemu" \ -kernel "$kernel" \ -append "root=/dev/sda ro panic=-1 console=ttyS0 quiet test.fs=$filesystem_type" \ -drive "index=0,media=disk,format=raw,file=$initramfs" \ -drive "index=1,media=disk,format=raw,file=$testfs" \ -no-reboot \ -nographic </code></pre></div> Again we’re stealing the kernel from the host operating system, which is also where the kernel modules come from. The <code>drive</code> with <code>index=0</code> is the initramfs and will be visible as <code>/dev/sda</code> in the guest; the other drive with <code>index=1</code> is our test storage and will be visible as <code>/dev/sdb</code>. <code>root=/dev/sda</code> tells the kernel to boot from our initramfs image. <code>panic=-1</code> tells the kernel not to reboot in case of a panic (<code>panic=N</code> specifies the number of seconds to wait before rebooting in case of kernel panic; specifying a negative value blocks that behavior). <code>console=ttyS0</code> allows us to capture the output printed by the init script. <code>test.fs=...</code> is the option that <code>init.sh</code> is going to look for to understand what filesystem type to use. Once that runs, the virtual machine is expected to do its job and crash with a kernel panic. We then need to restart it to verify the file contents. The QEMU command is exactly the same except for the additional <code>test.verify</code> option: <div class="highlight"><pre><code>output=output.txt "$qemu" \ -kernel "$kernel" \ -append "root=/dev/sda ro panic=-1 console=ttyS0 quiet test.fs=$filesystem_type test.verify" \ -drive "index=0,media=disk,format=raw,file=$initramfs" \ -drive "index=1,media=disk,format=raw,file=$testfs" \ -no-reboot \ -nographic \ | tee "$output" </code></pre></div> The output from this command is captured in a file that can be analyzed. If you remember how <code>init.sh</code> is written, you might have noticed that it writes the test file contents between two markers: <div class="highlight"><pre><code>echo '-----' xxd /test/file echo '-----' </code></pre></div> So what we have to do to verify the contents of the file is simply look for those two <code>-----</code> markers, get the content in between, and parse it through <code>xxd -r</code>: <div class="highlight"><pre><code>if [[ $(sed -n '/-----/, /-----/p' "$output" | xxd -r) = hello ]]; then echo "Success" else echo "Failure" exit 1 fi </code></pre></div> And that’s it! <h1 id="running-in-github-actions">Running in GitHub Actions</h1> The next step for me was to set up some automation to make sure that my test script was run whenever I made any change to the crate. Because my project was already hosted on GitHub, I decided to go with GitHub Actions. I was a bit worried that I would have had to struggle to make it work because my script works by stealing the kernel from the host, and I thought that the GitHub runners could have some protections in place to prevent me from reading the kernel. To my surprise, there were no such restrictions. In fact, I did not have to modify a single line of code to make my tests work in GitHub Actions: check out the <a href="https://github.com/andreacorbellini/rust-atomic-write-file/blob/master/.github/workflows/crash-tests.yml">workflow file</a> if you’re curious. <h1 id="result">Result</h1> In the end, the test suite was able to reproduce the issue with btrfs and anonymous temporary files, as well as show that the fix was working as intended. This is what it looks like (output in the screenshot was trimmed down a bit): <figure> <img src="https://andrea.corbellini.blog/images/crash-tests-failure.png" alt="Output showing a failure from the test suite"> <figcaption>Output from the <code>run-test.sh</code> script before applying the fix, showing a failure.</figcaption> </figure> <figure> <img src="https://andrea.corbellini.blog/images/crash-tests-success.png" alt="Output showing a success from the test suite"> <figcaption>Output from the <code>run-test.sh</code> script after applying the fix, showing a success.</figcaption> </figure> <h1 id="future">Future</h1> I’m pretty satisfied with the general approach of running in a virtual machine and simulating a crash, but the implementation has a huge limitation: because it steals the kernel from the host, it cannot simulate crashes on other kernels, operating systems, or platforms. I think in the future I’m going to take a look at <a href="https://github.com/cross-rs/cross">cross-rs</a>, a cross-platform tool for Rust crates. Cross-rs works kinda in a similar way as my test suite, in that it uses QEMU emulation to run tests. Maybe I can reuse their QEMU images and tweak them to run my crash tests. If that is feasible, then I will be able to extend my test suite to all major platforms and operating systems.andreacorbelliniTue, 09 Sep 2025 00:05:00 +0000tag:andrea.corbellini.blog,2025-09-09:/2025/09/09/testing-crash-recovery-features-in-ci/information-technologyIt will take decades to undo the damage done by "AI"https://andrea.corbellini.blog/2025/08/20/it-will-take-decades-to-undo-the-damage-done-by-ai/Many business owners in the software development industry are investing a lot into LLM bots, marketed as “generative AI”. Some of them go as far as forcing software developers to use such tools under the threat of being fired if they don’t comply. After all, why shouldn’t they? This technology has a catchy name, it produces very convincing output that sometimes is correct (or close to being correct), and companies producing these tools are very good at downplaying their limitations and at promising that the “next version” will be astonishingly better than the one before. My day-to-day experience, and scientific research, however show a quite big problem: while senior developers get little or no gain from “generative AI” (<a href="https://www.reuters.com/business/ai-slows-down-some-experienced-software-developers-study-finds-2025-07-10/">1</a>, <a href="https://fortune.com/2025/07/20/ai-hampers-productivity-software-developers-productivity-study/">2</a>, <a href="https://blueheadline.com/software-dev/ai-helps-new-developers-but-seniors/">3</a>, <a href="https://blog.stackademic.com/ai-coding-tools-slow-senior-devs-by-19-why-experts-are-still-using-them-98d855b81c4e">4</a>), junior developers get massive boosts in productivity. Why am I describing a boost in productivity as a problem? Well, it’s simple: the new generations of software developers (the ones that are just entering the job market, and the ones that are still in school) are relying heavily on these LLM tools for nearly all work-related tasks. And, again, why shouldn’t they? The entire world is telling them to do so! And as a result of that, they are advancing without gaining any actual skill. Me and other (ex-)coworkers have seen it first hand: a junior developer completes a task using an LLM bot. More senior developers find major problems with the result. Junior developers go back to the LLM tool asking for a fix, wasting hours or days without a positive outcome. Junior developers are becoming unable to perform tasks independently. In addition to that, there’s a problem that I feel like is not talked about extensively: these LLM tools can only regurgitate what they’ve been trained with. They’re good at finding patterns and re-applying strategies that have been used in prior work, but by their nature they cannot create or innovate. So, if this trend continues, the new generations of software developers are not just going to be skill-less, they’re also going to be incapable of solving new problems that haven’t been seen before. My prediction? Within 10 or 20 years, the older generation of software developers will be asked to come out of retirement to fix the unmaintainable mess created by the newer generations, and to make advancements in the information technology sector. A bit like old COBOL developers are asked to come out of retirement to maintain banking systems, but on a much larger scale. This, unless there will be advancements towards true Artificial Intelligence, or unless the current trend in education is broken. Now don’t get me wrong: I’m not opposed to LLM bots (although I’m against calling them “AI”, because they’re very far from fulfilling the AI promise), and I think they can be very powerful tools. What I’m worried about is that new generations are being told to rely on them almost exclusively, and this can only lead to an evident skill gap. In fact, I think that if there’s a country that in the future will be able to harness the power of the LLM bots and at the same time maintain a good enough level of education, thus addressing the skill gap problem, they will be the dominant economic power of the future, because they will be able to automate tasks that are time/energy-consuming for humans, while at the same time use the human brain to innovate and advance.andreacorbelliniWed, 20 Aug 2025 17:17:00 +0000tag:andrea.corbellini.blog,2025-08-20:/2025/08/20/it-will-take-decades-to-undo-the-damage-done-by-ai/information-technologyRunning the operating system that you're currently using in a virtual machine (with Secure Boot and TPM emulation)https://andrea.corbellini.blog/2023/11/19/running-current-os-inside-vm/In this article I will show you how to start your current operating system inside a virtual machine. That is: launching the operating system (with all your settings, files, and everything), inside a virtual machine, while you’re using it. This article was written for Ubuntu, but it can be easily adapted to other distributions, and with appropriate care it can be adapted to non-Linux kernels and operating systems as well. <h1 id="motivation">Motivation</h1> Before we start, why would a sane person want to do this in the first place? Well, here’s why I did it: <ul> <li> To test changes that affect Secure Boot without a reboot. Recently I was doing some experiments with Secure Boot and the Trusted Platform Module (TPM) on a new laptop, and I got frustrated by how time consuming it was to test changes to the boot chain. Every time I modified a file involved during boot, I would need to reboot, then log in, then re-open my terminal windows and files to make more modifications… Plus, whenever I screwed up, I would need to manually recover my system, which would be even more time consuming. I thought that I could speed up my experiments by using a virtual machine instead. </li> <li> To predict the future TPM state (in particular, the values of PCRs 4, 5, 8, and 9) after a change, without a reboot. I wanted to predict the values of my TPM PCR banks after making changes to the bootloader, kernel, and initrd. Writing a script to calculate the PCR values automatically is in principle not that hard (and I actually did it before, in a different context), but I wanted a robust, generic solution that would work on most systems and in most situations, and emulation was the natural choice. </li> <li> And, of course, just for the fun of it! </li> </ul> To be honest, I’m not a big fan of Secure Boot. The reason why I’ve been working on it is simply that it’s the standard nowadays and so I have to stick with it. Also, there are no real alternatives out there to achieve the same goals. I’ll write an article about Secure Boot in the future to explain the reasons why I don’t like it, and how to make it work better, but that’s another story… <h1 id="procedure">Procedure</h1> The procedure that I’m going to describe has 3 main steps: <ol> <li>create a copy of your drive</li> <li>emulate a TPM device using swtpm</li> <li>emulate the system with QEMU</li> </ol> I’ve tested this procedure on Ubuntu 23.04 (Lunar) and 23.10 (Mantic), but it should work on any Linux distribution with minimal adjustments. The general approach can be used for any operating system, as long as appropriate replacements for QEMU and swtpm exist. <h2 id="prerequisites">Prerequisites</h2> Before we can start, we need to install: <ul> <li><a href="https://www.qemu.org/">QEMU</a>: a virtual machine emulator</li> <li><a href="https://github.com/stefanberger/swtpm/wiki">swtpm</a>: a TPM emulator</li> <li><a href="https://wiki.ubuntu.com/UEFI/OVMF">OVMF</a>: a UEFI firmware implementation</li> </ul> On a recent version of Ubuntu, these can be installed with: <div class="highlight"><pre><code>sudo apt install qemu-system-x86 ovmf swtpm </code></pre></div> Note that OVMF only supports the x86_64 architecture, so we can only emulate that. If you run a different architecture, you’ll need to find another UEFI implementation that is not OVMF (but I’m not aware of any freely available ones). <h2 id="create-a-copy-of-your-drive">Create a copy of your drive</h2> We can decide to either: <ul> <li> Choice #1: <a href="#early-boot-components-only">run only the components involved early at boot</a> (shim, bootloader, kernel, initrd). This is useful if you, like me, only need to test those components and how they affect Secure Boot and the TPM, and don’t really care about the rest (the init process, login manager, …). </li> <li> Choice #2: <a href="#entire-system">run the entire operating system</a>. This can give you a fully usable operating system running inside the virtual machine, but may also result in some instability inside the guest (because we’re giving it a filesystem that is in use), and may also lead to some data loss if we’re not careful and make typos. Use with care! </li> </ul> <h3 id="early-boot-components-only">Choice #1: Early boot components only</h3> If we’re interested in the early boot components only, then we need to make a copy the following from our drive: the GPT partition table, the EFI partition, and the <code>/boot</code> partition (if we have one). Usually all these 3 pieces are at the “start” of the drive, but this is not always the case. To figure out where the partitions are located, run: <div class="highlight"><pre><code>sudo parted -l </code></pre></div> On my system, this is the output: <div class="highlight"><pre><code>Model: WD_BLACK SN750 2TB (nvme) Disk /dev/nvme0n1: 2000GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 525MB 524MB fat32 boot, esp 2 525MB 1599MB 1074MB ext4 3 1599MB 2000GB 1999GB lvm </code></pre></div> In my case, the partition number 1 is the EFI partition, and the partition number 2 is the <code>/boot</code> partition. If you’re not sure what partitions to look for, run <code>mount | grep -e /boot -e /efi</code>. Note that, on some distributions (most notably the ones that use <code>systemd-boot</code>), a <code>/boot</code> partition may not exist, so you can leave that out in that case. Anyway, in my case, I need to copy the first 1599 MB of my drive, because that’s where the data I’m interested in ends: those first 1599 MB contain the GPT partition table (which is always at the start of the drive), the EFI partition, and the <code>/boot</code> partition. Now that we have identified how many bytes to copy, we can copy them to a file named <code>drive.img</code> with <code>dd</code> (maybe after running <code>sync</code> to make sure that all changes have been committed): <div class="highlight"><pre><code># replace '/dev/nvme0n1' with your main drive (which may be '/dev/sda' instead), # and 'count' with the number of MBs to copy sync && sudo -g disk dd if=/dev/nvme0n1 of=drive.img bs=1M count=1599 conv=sparse </code></pre></div> <h3 id="entire-system">Choice #2: Entire system</h3> If we want to run our entire system in a virtual machine, then I would recommend creating a QEMU copy-on-write (COW) file: <div class="highlight"><pre><code># replace '/dev/nvme0n1' with your main drive (which may be '/dev/sda' instead) sudo -g disk qemu-img create -f qcow2 -b /dev/nvme0n1 -F raw drive.qcow2 </code></pre></div> This will create a new copy-on-write image using <code>/dev/nvme0n1</code> as its “backing storage”. Be very careful when running this command: you don’t want to mess up the order of the arguments, or you might end up writing to your storage device (leading to data loss)! The advantage of using a copy-on-write file, as opposed to copying the whole drive, is that this is much faster. Also, if we had to copy the entire drive, we might not even have enough space for it (even when using sparse files). The big drawback of using a copy-on-write file is that, because our main drive likely contains filesystems that are mounted read-write, any modification to the filesystems on the host may be perceived as data corruption on the guest, and that in turn may cause all sort of bad consequences inside the guest, including kernel panics. Another drawback is that, with this solution, later we will need to give QEMU permission to read our drive, and if we’re not careful enough with the commands we type (e.g. we swap the order of some arguments, or make some typos), we may potentially end up writing to the drive instead. <h2 id="emulate-a-tpm-device-using-swtpm">Emulate a TPM device using swtpm</h2> There are various ways to run the swtpm emulator. Here I will use the “vTPM proxy” way, which is not the easiest, but has the advantage that the emulated device will look like a real TPM device not only to the guest, but also to the host, so that we can inspect its PCR banks (among other things) from the host using familiar tools like <code>tpm2_pcrread</code>. First, enable the <code>tpm_vtpm_proxy</code> module (which is not enabled by default on Ubuntu): <div class="highlight"><pre><code>sudo modprobe tpm_vtpm_proxy </code></pre></div> If that worked, we should have a <code>/dev/vtpmx</code> device. We can verify its presence with: <div class="highlight"><pre><code>ls /dev/vtpmx </code></pre></div> swtpm in “vTPM proxy” mode will interact with <code>/dev/vtpmx</code>, but in order to do so it needs the <code>sys_admin</code> capability. On Ubuntu, swtpm ships with this capability explicitly disabled by AppArmor, but we can enable it with: <div class="highlight"><pre><code>sudo sh -c "echo ' capability sys_admin,' > /etc/apparmor.d/local/usr.bin.swtpm" systemctl reload apparmor </code></pre></div> Now that <code>/dev/vtpmx</code> is present, and swtpm can talk to it, we can run swtpm in “vTPM proxy” mode: <div class="highlight"><pre><code>sudo mkdir /tpm/swtpm-state sudo swtpm chardev --tpmstate dir=/tmp/swtpm-state --vtpm-proxy --tpm2 </code></pre></div> Upon start, swtpm should create a new <code>/dev/tpmN</code> device and print its name on the terminal. On my system, I already have a real TPM on <code>/dev/tpm0</code>, and therefore swtpm allocates <code>/dev/tpm1</code>. The emulated TPM device will need to be readable and writeable by QEMU, but the emulated TPM device is by default accessible only by root, so either we run QEMU as root (not recommended), or we relax the permissions on the device: <div class="highlight"><pre><code># replace '/dev/tpm1' with the device created by swtpm sudo chmod a+rw /dev/tpm1 </code></pre></div> Make sure not to accidentally change the permissions of your real TPM device! <h2 id="emulate-the-system-with-qemu">Emulate the system with QEMU</h2> Inside the QEMU emulator, we will run the OVMF UEFI firmware. On Ubuntu, the firmware comes in 2 flavors: <ul> <li>with Secure Boot enabled (<code>/usr/share/OVMF/OVMF_CODE_4M.ms.fd</code>), and</li> <li>with Secure Boot disabled (in <code>/usr/share/OVMF/OVMF_CODE_4M.fd</code>)</li> </ul> (There are actually even more flavors, see <a href="https://askubuntu.com/q/1409590">this AskUbuntu question</a> for the details.) In the commands that follow I’m going to use the Secure Boot flavor, but if you need to disable Secure Boot in your guest, just replace <code>.ms.fd</code> with <code>.fd</code> in all the commands below. To use OVMF, first we need to copy the EFI variables to a file that can be read & written by QEMU: <div class="highlight"><pre><code>cp /usr/share/OVMF/OVMF_VARS_4M.ms.fd /tmp/ </code></pre></div> This file (<code>/tmp/OVMF_VARS_4M.ms.fd</code>) will be the equivalent of the EFI flash storage, and it’s where OVMF will read and store its configuration, which is why we need to make a copy of it (to avoid modifications to the original file). Now we’re ready to run QEMU: <ul> <li> If you <a href="#early-boot-components-only">copied only the early boot files (choice #1)</a>: <div class="highlight"><pre><code># replace '/dev/tpm1' with the device created by swtpm qemu-system-x86_64 \ -accel kvm \ -machine q35,smm=on \ -cpu host \ -smp cores=4,threads=1 \ -m 4096 \ -vga virtio \ -bios /usr/share/ovmf/OVMF.fd \ -drive if=pflash,unit=0,format=raw,file=/usr/share/OVMF/OVMF_CODE_4M.ms.fd,readonly=on \ -drive if=pflash,unit=1,format=raw,file=/tmp/OVMF_VARS_4M.ms.fd \ -drive if=virtio,format=raw,file=drive.img \ -tpmdev passthrough,id=tpm0,path=/dev/tpm1,cancel-path=/dev/null \ -device tpm-tis,tpmdev=tpm0 </code></pre></div> </li> <li> If you have <a href="#entire-system">a copy-on-write file for the entire system (choice #2)</a>: <div class="highlight"><pre><code># replace '/dev/tpm1' with the device created by swtpm sudo -g disk qemu-system-x86_64 \ -accel kvm \ -machine q35,smm=on \ -cpu host \ -smp cores=4,threads=1 \ -m 4096 \ -vga virtio \ -bios /usr/share/ovmf/OVMF.fd \ -drive if=pflash,unit=0,format=raw,file=/usr/share/OVMF/OVMF_CODE_4M.ms.fd,readonly=on \ -drive if=pflash,unit=1,format=raw,file=/tmp/OVMF_VARS_4M.ms.fd \ -drive if=virtio,format=qcow2,file=drive.qcow2 \ -tpmdev passthrough,id=tpm0,path=/dev/tpm1,cancel-path=/dev/null \ -device tpm-tis,tpmdev=tpm0 </code></pre></div> Note that this last command makes QEMU run as the <code>disk</code> group: on Ubuntu, this group has the permission to read and write all storage devices, so be careful when running this command, or you risk losing your files forever! If you want to add more safety, you may consider using an <a href="https://manpages.ubuntu.com/manpages/mantic/en/man5/acl.5.html">ACL</a> to give the user running QEMU read-only permission to your backing storage. </li> </ul> In either case, after launching QEMU, our operating system should boot… while running inside itself! In some circumstances though it may happen that the wrong operating system is booted, or that you end up at the EFI setup screen. This can happen if your system is not configured to boot from the “first” EFI entry listed in the EFI partition. Because the boot order is not recorded anywhere on the storage device (it’s recorded in the EFI flash memory), of course OVMF won’t know which operating system you intended to boot, and will just attempt to launch the first one it finds. You can use the EFI setup screen provided by OVMF to change the boot order in the way you like. After that, changes will be saved into the <code>/tmp/OVMF_VARS_4M.ms.fd</code> file on the host: you should keep a copy of that file so that, next time you launch QEMU, you’ll boot directly into your operating system. <h2 id="reading-pcr-banks-after-boot">Reading PCR banks after boot</h2> Once our operating system has launched inside QEMU, and after the boot process is complete, the PCR banks will be filled and recorded by swtpm. If we choose to <a href="#early-boot-components-only">copy only the early boot files (choice #1)</a>, then of course our operating system won’t be fully booted: it’ll likely hang waiting for the root filesystem to appear, and may eventually drop to the initrd shell. None of that really matters if all we want is to see the PCR values stored by the bootloader. Before we can extract those PCR values, we first need to stop QEMU (Ctrl-C is fine), and then we can read it with <code>tpm2_pcrread</code>: <div class="highlight"><pre><code># replace '/dev/tpm1' with the device created by swtpm tpm2_pcrread -T device:/dev/tpm1 </code></pre></div> Using the method described here in this article, PCRs 4, 5, 8, and 9 inside the emulated TPM should match the PCRs in our real TPM. And here comes an interesting application of this method: if we upgrade our bootloader or kernel, and we want to know the future PCR values that our system will have after reboot, we can simply follow this procedure and obtain those PCR values without shutting down our system! This can be especially useful if we use TPM sealing: we can reseal our secrets and make them unsealable at the next reboot without trouble. <h2 id="restarting-the-virtual-machine">Restarting the virtual machine</h2> If we want to restart the guest inside the virtual machine, and obtain a consistent TPM state every time, we should start from a “clean” state every time, which means: <ol> <li>restart swtpm</li> <li>recreate the <code>drive.img</code> or <code>drive.qcow2</code> file</li> <li>launch QEMU again</li> </ol> If we don’t restart swtpm, the virtual TPM state (and in particular the PCR banks) won’t be cleared, and new PCR measurements will simply be added on top of the existing state. If we don’t recreate the drive file, it’s possible that some modifications to the filesystems will have an impact on the future PCR measurements. We don’t necessarily need to recreate the <code>/tmp/OVMF_VARS_4M.ms.fd</code> file every time. In fact, if you need to modify any EFI setting to make your system bootable, you might want to preserve it so that you don’t need to change EFI settings at every boot. <h1 id="automating-the-entire-process">Automating the entire process</h1> I’m (very slowly) working on turning this entire procedure into a script, so that everything can be automated. Once I find some time I’ll finish the script and publish it, so if you liked this article, stay tuned, and let me know if you have any comment/suggestion/improvement/critique!andreacorbelliniSun, 19 Nov 2023 16:33:00 +0000tag:andrea.corbellini.blog,2023-11-19:/2023/11/19/running-current-os-inside-vm/information-technologyHow to run Remark42 on Fly.iohttps://andrea.corbellini.blog/2023/09/19/running-remark42-on-flyio/As I wrote on my <a href="https://andrea.corbellini.blog/2023/09/05/disqus-to-remark42/">previous post</a>, I recently switched from <a href="https://disqus.com/">Disqus</a> to <a href="https://remark42.com/">Remark42</a> for the comments on my blog. Here I will explain how I set it up on <a href="https://fly.io/">Fly.io</a>. <h1 id="overview">Overview</h1> The setup that I ended up with looks like the following: <figure> <img src="https://andrea.corbellini.blog/images/remark42-setup.svg" alt="Diagram of the components for the Remark42 setup"> </figure> Something to note about this setup is that the “machine” (more on that later) and the storage volume are both a single instance. This is not a distributed setup. This is because Remark42 stores comments in a single file and does not make use of a distributed database. This is listed as a “feature” on the <a href="https://remark42.com/">Remark42 website</a>. How one is supposed to implement replication? I have no idea. Thankfully Fly.io seems to be fast to provision machines, and the Remark42 daemon also seems fast to start, so hopefully if a problem occurs (or when updates are required), the downtime will be minimal. It is imperative however to understand that, because of the non-distributed/non-replicated nature of this setup, backups should be made periodically to avoid the risk of losing your comments forever. <h1 id="preliminaries">Preliminaries</h1> Before setting up Remark42, I had never used <a href="https://fly.io/">Fly.io</a> before. As Fly.io newbie, I would describe it as a cloud provider focused on Docker containers. Fly.io uses some concepts (like “apps” and “machines”) that make sense after you practice a bit with them, but as a beginner they are not the easiest to learn. Most of the complexity I think comes from the fact that the Fly.io documentation is poorly written. On top of that, it appears that Fly.io is migrating their offering from “V1 apps” to “V2 apps”, and today some documentation applies only to “V1 apps”, other pieces apply only to “V2 apps”, resulting in a big mess. The error messages you get are also far from clear. But don’t get too scared: once you get to know Fly.io, it can actually be fun to use. Creating resources on Fly.io requires installing their command line client: <code>flyctl</code>. Because I do not like to run unknown software unconfined, I <a href="https://snapcraft.io/andrea-flyctl">packaged it as a snap</a> that you can install using: <div class="highlight"><pre><code>snap install andrea-flyctl </code></pre></div> Another source of confusion that I had the beginning was that, by reading the documentation, it looked like a second command line tool named <code>fly</code> was needed in addition to <code>flyctl</code>. It turns out that <code>fly</code> and <code>flyctl</code> are the same thing, it’s just that they’re transitioning from a name to another. If you installed the tool through the snap, you can set up these aliases so that you can copy and paste commands without trouble: <div class="highlight"><pre><code>alias fly=/snap/bin/andrea-flyctl.fly alias flyctl=/snap/bin/andrea-flyctl.fly </code></pre></div> According to the documentation (and assuming it’s up-to-date), <code>flyctl</code> does not support everything that Fly.io supports, so sometimes <code>curl</code> is used to interact directly with the Fly.io API. In order to use that, you’ll need to download an authentication token from the Fly.io interface and store it in a file (that I’ll call <code>~/fly-token</code> from now on). I’m going to skip over the steps to create and configure a Fly.io account, obtaining an authentication token, as those were easy steps in my opinion. <h1 id="creating-a-machine">Creating a machine</h1> A Fly.io “machine” is a virtual machine running a single Docker container with a persistent volume attached to it. In order to create my Fly.io machine to run Remark42 in it, I loosely followed this page from the Fly.io documentation: <a href="https://fly.io/docs/machines/guides-examples/functions-with-machines/">Run User Code on Fly Machines </a>. “Loosely” because it turned out that some pieces on that page are not fully correct, but anyway… Before creating a machine, you first need to create an “app”. A Fly.io app is basically an endpoint, which consists of a DNS name (in the form <code>${app_name}.fly.dev</code>), and a set of IP addresses. Behind these IP addresses there are Fly.io load balancers that will forward requests to the machines inside the app. You can do that through the API like this: <div class="highlight"><pre><code>curl -X POST \ -H "Authorization: Bearer $(<~/fly-token)" \ -H 'Content-Type: application/json' \ 'https://api.machines.dev/v1/apps' \ -d '{ "app_name": "${app_name}", "org_slug": "personal" }' </code></pre></div> (Replace <code>${app_name}</code> with some identifier of your choice; I chose <code>remark42</code> without knowing that this would have removed the possibility for other people to register an app with the same name.) IP addresses need to be manually allocated: <div class="highlight"><pre><code>fly ips allocate-v4 --app=${app_name} --shared fly ips allocate-v6 --app=${app_name} </code></pre></div> The <code>--shared</code> option to <code>allocate-v4</code> tells Fly.io to allocate an IP address that may be shared with other Fly.io apps, even outside of your account/organization. Remove <code>--shared</code> if you want to use a dedicated IP, but note that dedicated IPv4 addresses is a paid feature. Allocating IPs is an important step: it can be done later, after creating the machine, but it must be done, otherwise your machine will be unreachable and it won’t be obvious why. You should now create a persistent volume for your machine: <div class="highlight"><pre><code>fly volume create remark42_db_0 --app=${app_name} --size=1 </code></pre></div> This will display a warning about replication, but you can ignore it because, sadly, Remark42 does not support replication. Remark42 needs to be given a secret key (I guess for the purpose of signing <a href="https://en.wikipedia.org/wiki/JSON_Web_Token">JWT tokens</a>). Fly.io has a handy feature to manage secrets, and make them available to machines, albeit poorly documented. You can set the Remark42 secret like this: <div class="highlight"><pre><code>fly secrets set --app=${app_name} SECRET='a very secret string' </code></pre></div> (You can generate a random secret string with a command like <code>cat /dev/urandom | tr -Cd 'a-zA-Z0-9' | head -c64</code>, which means: get some random bytes, keep only alphanumeric characters, get the first 64 characters.) You may be wondering: how is the container running inside the machine supposed to access this secret? The Fly.io documentation doesn’t say a word about it, but after experimenting I was able to find that all the app secrets are passed as environment variables, which is great, because this is exactly what Remark42 expects. Note: it’s important to set <code>SECRET</code> before creating the machine, or Remark42 will refuse to start. Now you’re ready to spin up the machine: create a configuration file for it… <div class="highlight"><pre><code>{ "name": "remark42-0", "config": { "image": "umputun/remark42:latest", "env": { "SITE": "andrea.corbellini.name", "REMARK_URL": "https://${app_name}.fly.dev", "ALLOWED_HOSTS": "'self',https://andrea.corbellini.name", "AUTH_SAME_SITE": "none", "AUTH_ANON": "true", "AUTH_EMAIL_ENABLE": "true", "AUTH_EMAIL_FROM": "Andrea's Blog <hi@andrea.corbellini.name>", "AUTH_EMAIL_SUBJ": "Andrea's Blog - Email Confirmation", "NOTIFY_USERS": "email", "NOTIFY_ADMINS": "email", "NOTIFY_EMAIL_FROM": "Andrea's Blog <hi@andrea.corbellini.name>", "ADMIN_SHARED_EMAIL": "corbellini.andrea@gmail.com", }, "mounts": [ { "volume": "${volume_id}", "path": "/srv/var" } ], "services": [ { "ports": [ { "port": 443, "handlers": [ "tls", "http" ] }, { "port": 80, "handlers": [ "http" ] } ], "protocol": "tcp", "internal_port": 8080 } ], "checks": { "httpget": { "type": "http", "port": 8080, "method": "GET", "path": "/ping" "interval": "15s", "timeout": "10s", } }, "metadata": { "fly_platform_version": "v2", } } } </code></pre></div> …and give it to Fly.io: <div class="highlight"><pre><code>curl -X POST \ -H "Authorization: Bearer $(<~/fly-token)" \ -H 'Content-Type: application/json' \ "https://api.machines.dev/v1/apps/${app_name}/machines" -d @config.json </code></pre></div> There’s a lot here, so let me break it down for you: <ul> <li> <code>"image": "umputun/remark42:latest"</code>: this is the Docker image for Remark42. </li> <li> <code>"env": { ... }</code>: these are all the environment variables to pass to our container. They are briefly documented on the <a href="https://remark42.com/docs/configuration/parameters/">Remark42 website</a>, and here’s a bit more detailed explanation of some of them: <ul> <li> <code>"SITE": "andrea.corbellini.name"</code>: this is the internal identifier for the site, it can be an arbitrary string, it won’t be visible, and you can omit it. </li> <li> <code>"REMARK_URL": "https://${app_name}.fly.dev"</code>: this is the URL where Remark42 will be serving requests from. I set it to the Fly.io app endpoint. It’s important that you do not put a trailing slash, or Remark42 will error out later on. It’s also important that the protocol (http or https) matches your blog’s protocol, or Remark42 will refuse to display comments (this makes local testing a bit annoying). </li> <li> <code>"ALLOWED_HOSTS": "'self',https://andrea.corbellini.name"</code>: this is the list of sources that will be put into the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/frame-ancestors"><code>Content-Security-Policy: frame-ancestors</code> header</a>) of HTTP responses. Essentially, this defines where the Remark42 comments can be displayed. </li> <li> <code>"AUTH_SAME_SITE": "none"</code>: this disable the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie#samesitesamesite-value">“same site” policy for cookies</a>. Disabling it is necessary because, in my setup, comments are served from one domain (<code>remark42.fly.dev</code>) to another domain (<code>andrea.corbellini.name</code>). </li> <li> <code>"AUTH_ANON": "true"</code>: allows anonymous commenters. You may or may not want it. </li> <li> <code>"AUTH_EMAIL_ENABLE": "true"</code> and friends: allows email-based authentication of commenters. </li> <li> <code>"NOTIFY_USERS" "email"</code>: allows readers and commenters to be notified of new comments via email. </li> <li> <code>"NOTIFY_ADMINS" "email"</code> and <code>"ADMIN_SHARED_EMAIL": "corbellini.andrea@gmail.com"</code>: makes Remark42 send me an email every time there’s a new comment. </li> </ul> </li> <li> <code>"mounts": [ ... ]</code>: this tells Fly.io to attach the volume that you created earlier to the container at the path <code>/srv/var</code>, which is what Remark42 uses to store its database as well as daily backups. </li> <li> <code>"services": [ ... ]</code>: this tells Fly.io what to expose through the load balancer. With the configuration that I provided, the Fly.io endpoint (<code>${app_name}.fly.dev</code>) will provide both HTTP and HTTPS to the internet. However, the load balancer will talk to the machine over plain HTTP on port 8080 (meaning that TLS is terminated at the load balancer). I think in the future I will setup <a href="https://certbot.eff.org/">certbot</a> inside the container so that I can do TLS termination on the machine, but not today. </li> <li> <code>"checks": { ... }</code>: this tells Fly.io to check if the Remark42 daemon is healthy by using its <code>/ping</code>endpoint. </li> <li> <code>"metadata": { "fly_platform_version": "v2" }</code>: this tells Fly.io to use a “V2 machine”, or something like that. Setting this metadata is very important, or certain things won’t work later on. The Fly.io documentation doesn’t tell you to do it, but this is needed if you need to update the environment variables or the secrets inside the machine. </li> </ul> Note that all of this configuration can be changed at any time, so if you make any mistakes or you just want to experiment, you don’t have to overly worry. You can even destroy your machine and recreate it from scratch if you want. To view the configuration of an existing machine use the following: <div class="highlight"><pre><code>curl \ -H "Authorization: Bearer $(<~/fly-token)" \ "https://api.machines.dev/v1/apps/${app_name}/machines/${machine_id}" </code></pre></div> And to update it: <div class="highlight"><pre><code>curl -X POST \ -H "Authorization: Bearer $(<~/fly-token)" \ -H 'Content-Type: application/json' \ "https://api.machines.dev/v1/apps/${app_name}/machines/${machine_id}" \ -d @new-config.json </code></pre></div> I was also successful at changing configuration using <code>fly machines update</code>, although it can’t be used for everything (for example: it can be used to add or change environment variables, but not to remove them). <h1 id="testing-the-setup">Testing the setup</h1> If everything went well, you should be able to interact with Remark42 at <code>https://${app_name}.fly.dev/web</code>. This should let you read and post new comments. <h1 id="configuring-remark42-to-send-emails">Configuring Remark42 to send emails</h1> For sending emails, I chose to use <strike>Elastic Email</strike> <a href="https://mailtrap.io/">Mailtrap</a>, which is an email-delivery service that supports SMTP with STARTTLS. Creating a Mailtrap account, setting up <a href="https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail">DKIM</a> and <a href="https://en.wikipedia.org/wiki/Sender_Policy_Framework">SPF</a>, and obtaining SMTP credentials was extremely easy, so I won’t cover it here. <blockquote> UPDATE: I initially chose to go with Elastic Email, but I found it to be garbage. They force the insertion of tracking URLs every one of your emails, and they refuse to disable tracking if you ask them to. </blockquote> Setting up email delivery with Remark42 is pretty easy once you have the SMTP credentials. Set the necessary (non-secret) configuration like this: <div class="highlight"><pre><code>fly machines update ${machine_id} --app=${app_name} \ -e SMTP_HOST=live.smtp.mailtrap.io \ -e SMTP_PORT=587 \ -e SMTP_STARTTLS=true \ -e SMTP_USERNAME=... </code></pre></div> And then set the SMTP password as a Fly.io secret: <div class="highlight"><pre><code>fly secrets set --app=${app_name} SMTP_PASSWD='a very secret password' </code></pre></div> Doing both <code>machines update</code> and <code>secrets set</code> will automatically restart the machine so that Remark42 can pick up the new configuration. Pretty neat, heh? <h1 id="configuring-authentication-providers-for-remark42">Configuring authentication providers for Remark42</h1> Remark42 can let your users log in from a variety of providers, including: GitHub, Google, Facebook, Telegram, and more. There are specific instructions for each provider in the <a href="https://remark42.com/docs/configuration/authorization/">Remark42 documentation</a>. There’s really not much to add on top of what’s already written there. Just remember: set non-secret environment variables with <code>fly machines update</code>, and set secrets with <code>fly secrets set</code>. <h1 id="creating-an-administrator-account">Creating an administrator account</h1> If you want to be able to moderate comments, you’ll need an administrator account. With Remark42, this is a 3 step process: first you create an account (like any other user would do), then you copy the ID of the user you just created, and lastly you add that user ID to the <code>ADMIN_SHARED_ID</code> environment variable: <div class="highlight"><pre><code>fly machines update ${machine_id} --app=${app_name} -e ADMIN_SHARED_ID=... </code></pre></div> As step-by-step guide is on the <a href="https://remark42.com/docs/manuals/admin-interface/">Remark42 documentation</a>. <h1 id="importing-comments-from-disqus-or-any-other-platform">Importing comments from Disqus (or any other platform)</h1> In order to import comments into Remark42, first you need to temporarily set an “admin password” for Remark42 (here the word “admin” has nothing to do with the administrator account you just created; it’s a totally separate concept): <div class="highlight"><pre><code>fly secrets set --app=${app_name} ADMIN_PASSWD='this is super secret' </code></pre></div> You can now copy your Disqus (or equivalent) backup on the machine and import it. I could not find an easy way to do it through <code>flyctl</code> (but I also did not spend too much time looking for an option), I did however find a way to open a console on the machine, so what I did was simply copying and pasting the base64-encoded backup: <div class="highlight"><pre><code># on my laptop base64 < disqus-export.xml.gz # copy the output # attach to the machine fly console --app=${app_name} --machine=${machine_id} # on the machine cd /srv/var base64 -d > disqus-export.xml.gz # paste the output from earlier gunzip disqus-export.xml.gz import --provider=disqus --file=/srv/var/disqus-export.xml --url=http://localhost:8080 rm disqus-export.xml </code></pre></div> Note: importing comments will clear the Remark42 database. Any pre-existing comment will be deleted. See also the <a href="https://remark42.com/docs/backup/migration/">Remark42 documentation</a> for more information. Another note: for some reason, my Disqus export referenced my blog posts using <code>http://</code> URLs instead of <code>https://</code>. Because of that, Remark42 did correctly import all the Disqus comments in its database, but would not display them under my blog posts. Remember: Remark42 is very picky when it comes to URL schemes. To fix this, I simply <a href="https://remark42.com/docs/backup/backup/">created a backup from Remark42</a>, modified the backup to change all <code>http</code> entries to <code>https</code>, and then <a href="https://remark42.com/docs/backup/restore/">restored the backup</a>. This was quite trivial given that the format used by the backups is extremely intuitive. <h1 id="final-remarks">Final remarks</h1> That was it! Setting up Remark42 on Fly.io wasn’t particularly difficult, but it took me way more time than expected due to the poor documentation of both Remark42 and Fly.io. I had to resort to trial-and-error multiple times to make things work. One big drawback of Remark42 is that it does not allow replication. This means that: <ul> <li>if the machine running my instance of Remark42 goes down, or becomes unreachable for any reason, there will be downtime;</li> <li>some people who are “far away” from the Remark42 instance may experience higher latency than others;</li> <li>I need to periodically take backups of my Remark42 database and copy it somewhere, otherwise if my single storage volume is lost, I will lose all the comments.</li> </ul> Nonetheless I think both Remark42 and Fly.io are very interesting products. I love Remark42’s features, and Fly.io is easy enough to use once you get familiar with it. I think I’m gonna stick with them for a long time.andreacorbelliniTue, 19 Sep 2023 02:12:00 +0000tag:andrea.corbellini.blog,2023-09-19:/2023/09/19/running-remark42-on-flyio/blogMy journey from Disqus to Remark42https://andrea.corbellini.blog/2023/09/05/disqus-to-remark42/Readers of this blog might have noticed a few changes recently. For example, I’ve been working on improving the look of the blog (maybe with questionable results), as well as improving the experience on mobile. But one of the biggest changes that perhaps some have noticed is that all of the comments on all of my articles have suddenly disappeared since February 2023. Now, almost 7 months later, all comments have finally been restored. The reason for this 7 months blackout of comments is that I decided to change the platform that hosts comments: I got rid of <a href="https://disqus.com/">Disqus</a>, and eventually replaced it with <a href="https://remark42.com/">Remark42</a>. Here I will describe why I did it. There will be another (more technical) blog post about my new setup. <h1 id="premise">Premise</h1> My blog is a static website that has been using Disqus as a commenting platform for a long time: since at least 2015 (8 years ago), or maybe even more (back when my blog was on WordPress). Disqus at that time was gaining a lot of popularity, it was free, and it was very attractive to me because easy to set up. I might be wrong, but at that time, Disqus did not look to me like the data-savvy, privacy-invading, revenue-oriented company that it is today. Maybe I just naive, but so I kept using Disqus all these years without paying too much attention to it: after all, it worked, so why would I spend any time thinking about it? <h1 id="advertisements-on-my-blog">Advertisements on my blog!?</h1> Fast-forward to February 2023: one day, a person very close to me, with the utmost kindness that characterizes her, came to me and said: “the ads on your blog suck! They’re the worst kind of ads!” At the beginning I had no idea what she was talking about. I have never intentionally run any sort of advertisements on my blog. I hate advertisements! Then I realized what was going on: precisely because I hate advertisements, I run ad-blockers on all my devices. Maybe there were ads on my blog, but I never noticed because I block those ads. The only third-party service that I used to run on my blog was Disqus, so I immediately turned my attention to it. I disabled my ad-blockers, refreshed my blog, scrolled down to the comments section, and… the sad truth was revealed: Disqus was showing ads to my readers. And yes, those ads were some of the worst kind of ads. And I knew that, together with those ads, there was massive tracking, collection of data, and maybe even data sharing with third-parties. People who know me, know that I deeply care about privacy, and having Disqus on my blog tracking my readers was the complete opposite of what I wanted. I was extremely disappointed. <h1 id="leaving-disqus">Leaving Disqus</h1> I did some quick research and I discovered that (1) I could not disable Disqus ads without paying, and (2) Disqus was no longer that nice commenting platform that I met in 2015. It had mutated into something obsessed about revenue, and it was clear that their business model was completely based on ads. My fears about tracking were <a href="https://techcrunch.com/2021/05/05/disqus-facing-3m-fine-in-norway-for-tracking-users-without-consent/">quickly confirmed</a>. Let’s just say that Disqus turned out to something that does not really align with my values. I made the difficult decision to completely <a href="https://github.com/andreacorbellini/andreacorbellini.github.io/commit/4f0e450d31441cab387a0e70f884fb65f19693fd">remove Disqus</a> from my blog on the same day. But I firmly believe that <a href="https://blog.codinghorror.com/a-blog-without-comments-is-not-a-blog/">a blog without comments is not a blog</a>, and so I had to find an alternative. <h1 id="looking-for-a-new-platform">Looking for a new platform</h1> I quickly started to look for new commenting platforms that could replace Disqus. The basic criteria that this new platform had to meet were (in no particular order): <ul> <li>be free of charge</li> <li>display only comments, no ads</li> <li>respect the privacy of users</li> <li>allow users to comment anonymously (at least to some extent)</li> </ul> The last time that I searched for a commenting platform was in 2015. Back in those days, there were not many solutions, and that’s one reason why I ended up with Disqus. I thought: 8 years have passed since then, surely the space must have improved, and alternatives must be proliferating, right? Well, no, not really. I struggled to find a managed platform that met those criteria. I did find some solutions that were using Mastodon or GitHub as a backend to store comments, but I did not like at all the idea of forcing my readers to have a Mastodon or GitHub account to comment on my blog. <h1 id="trying-cactus-comments">Trying Cactus Comments</h1> One platform that came up multiple times during my search was <a href="https://cactus.chat/">Cactus Comments</a>. Quoting the homepage of the project: <blockquote> Cactus Comments is a federated comment system built on Matrix. It respects your privacy, and puts you in control. The entire thing is completely free and open source. </blockquote> That sounded interesting, although I did not really know what <a href="https://matrix.org/">Matrix</a> was to begin with (if you, like me earlier this year, do not know what Matrix is: it is a team communication platform, somewhat similar to <a href="https://slack.com/">Slack</a>). I thought that I could give Cactus a try. So, a few days after removing Disqus, I onboarded on Cactus Comments. Onboarding was not hard, but it was not trivial either, mostly because I was not familiar with Matrix. The frontend shown to readers was a bit disappointing: even though Matrix supports threads, Cactus Comments does not. Overall, the number of features that commenters could use was scarce: people could only post a comment, and not much else; they had no ability to edit their comments, or delete them. But it did allow people to post even without creating a Matrix account, and that was great for me. The “administrative interface” (if we can call it this way) was also disappointing. All the administration and moderation had to be done through Matrix, sometimes by communicating with a bot, and could not be done by clicking buttons on my blog. Every blog post had to have its own Matrix channel and I (the author) had to manually join each channel in order to get some sort of notification for new comments. I needed a Matrix client to spot new comments, and to perform moderation actions, and I chose <a href="https://element.io/">Element</a> for that purpose. Sadly, Element was totally unreadable on small displays like my phone. And apparently there’s no web-based Matrix client that works well on mobile. I could have installed an app for my phone, but I hate installing apps, especially for activities that can in theory be done through a web browser. Cactus Comments also did not support importing comments from Disqus, so moving to this platform meant that all the conversations that happened over the years on my blog were lost. But because Cactus Comments is free & open source software, I thought that I could add support for importing comments from Disqus if I decided to settle with Cactus Comments, so this was not a deal breaker. Overall my experience with Cactus Comments was not great, but I was willing to accept that in exchange for a platform that was free, managed by someone else, and respecting the privacy of my readers. There was however one big problem that eventually led me to remove Cactus Comments from my blog: Cactus did not support sending email notifications. This meant that if you left a comment on this blog, I would not get notified. And if I responded to your comment, you would not get notified. In order to spot new comments, I had to check the Matrix channels periodically, and readers and to check my blog periodically. Maybe if I installed a Matrix app I could have received push notifications on my phone, but that’s not what I wanted, and this wouldn’t have solved the problem for my commenters anyway. I was pretty bad at checking for new comments on Cactus. What happened multiple times is that people would leave comments or questions on my blog, but I wouldn’t notice until 2 weeks later. At that point, it was pointless for me to respond because so much time had passed that those commenters surely wouldn’t be checking my blog for a response… I would say that with Cactus I had a blog that allowed comments, but did not allow conversations. Not allowing conversations made the comments pointless in my opinion. I might as well have had no comments at all: at least people would stop leaving questions there that were destined to be unanswered, and instead they would have emailed me directly. <h1 id="meet-remark42">Meet Remark42</h1> Between August and September 2023, I decided that I had to restart my quest for a commenting platform. This time I knew that I had to look for a solution that I had to install and manage myself. I was not super-excited about it, but from my first search for a Disqus alternative, I couldn’t find any managed solution that I really liked. Initially I thought about writing my own commenting platform in Rust with a key-value store, but then I figured that if I looked for a software to install instead of a managed platform, maybe I could find something I liked. After some research, I decided to go with <a href="https://remark42.com/">Remark42</a>. There were a few contenders, but Remark42 won because it looked like it had all of the features I needed, and more: <ul> <li>it supports sending of email notifications, both to me, and to my readers;</li> <li>it supports various authentication mechanisms, including: email, GitHub, Google, Facebook, etc (it’s nice to give commenters a choice);</li> <li>it supports leaving comments anonymously, without logging in or leaving an email address;</li> <li>commenters can edit and delete comments;</li> <li>it supports importing comments from Disqus;</li> <li>in fact, it supports importing comments from any platform: the format it uses for restoring backups is JSON-based and very easy to replicate (in theory I could import the comments from Cactus, even though I have not done that yet);</li> <li>it’s privacy-focused, and it looks like it’s implemented with security in mind.</li> </ul> I decided to host it on <a href="https://fly.io/">Fly.io</a>, which offers some compute and storage capacity for free. I was introduced to Fly.io on Mastodon, but I had never used it before. For sending emails, I chose <a href="https://elasticemail.com/">Elastic Email</a>, which also offers the features I needed for free. I also had never used this service before, and did not know much about it: it showed up while searching for a free SMTP provider. Elastic Email describes itself as a marketing service, which does not sound great from the point of view of privacy, but I figured that all the emails being sent here contain only public information (all comments are public after all), so there’s not much to protect besides email addresses. And people are free to use temporary email providers like <a href="https://www.mailinator.com/">Mailinator</a> if they don’t want to leave their real email, or even leave no email address at all. (Should I be concerned about Elastic Email, like I should have been concerned about Disqus? Let me know… in the <a href="#comments">comments</a> below.) Setting up Remark42 on Fly.io was relatively easy, but it took me way longer than I had expected, mostly because the Fly.io documentation was quite inconsistent and confusing, and also the Remark42 documentation was not fully clear. In the end I managed to make everything work and I’m pretty happy with the setup I ended up with. I’m going to publish details about my setup in a future blog post, in case you’re interested (update: said blog post is now <a href="https://andrea.corbellini.blog/2023/09/19/running-remark42-on-flyio/">published</a>). <h1 id="conclusion">Conclusion</h1> That’s all I have to say for now! Remark42 has been running on my blog for a few days, so it’s too early for me to say whether I’ll stick with it or I will look for a new solution, but so far it looks very promising, and I’m very happy with it. I hope this is the beginning of a long journey!andreacorbelliniTue, 05 Sep 2023 08:30:00 +0000tag:andrea.corbellini.blog,2023-09-05:/2023/09/05/disqus-to-remark42/blogOn ignoring mistakes, resilience, and the hidden dangers thereinhttps://andrea.corbellini.blog/2023/03/18/mistakes/As a scuba diver who often explores new places, I can say that I have found myself in some dangerous situations, but I always made it back to the surface without facing any negative consequences. Does this mean that I never made any mistakes? Absolutely not: mistakes were made, and lessons were learned. We can all agree that learning from mistakes is good, but sometimes, when mistakes happen and consequences don’t manifest themselves immediately, we run the risk of not noticing them, not learning from them, repeating them, and over time developing a false sense of confidence, which can drive us to believe that our repeated mistakes are actually good practices. Why do we ignore mistakes? Because sometimes outcomes are positive even if we make mistakes. “I made it out of water even this time, this means that my dive was executed perfectly.” This is a common way of reasoning, but in reality, things are much more complex than that. There is a difference between correct execution and successful outcome, and the two should not be confused. In fact, everyone should know from experience that goals can be achieved even if the execution was sloppy and full of mistakes. Catastrophic consequences may happen if we fail to see that. An example of the consequences of ignoring mistakes is given by the two space shuttle disasters: the <a href="https://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disaster">Challenger disaster of 1986</a>, and the <a href="https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster">Columbia disaster of 2003</a>. Both these instances were caused by NASA leadership ignoring the concerns from the engineering teams. Problems that occurred in previous shuttle launches should have been a wake-up call for NASA leadership. Instead, all the previous successful launches and re-entries despite the problems were seen as accomplishments, and nourished the leadership’s overconfidence. “We have made it this time too, this means that all those concerns that engineers raised were excessive.” The tendency of diverting from proper procedures, dismissing valid concerns, and ignoring problems, has a name: it’s called <a href="https://en.wikipedia.org/wiki/Normalization_of_deviance">normalization of deviance</a>. The driving force of normalization of deviance is overconfidence and the false belief that positive outcomes are inherently caused by correct executions. Overconfidence and normalization of deviance can spread like a virus in an organization. It is important to be vigilant for signs of overconfidence in individuals, before it infects other people. I once had to deal with a manager who was a self-declared micromanager (and proud to be) but lacked technical foundations and knowledge of the product. He would consistently and quickly dismiss anything that he did not understand, and focus on short-term goals of questionable usefulness. Whenever his team would accomplish a goal, he would send a pumped-up announcement, often containing inaccuracies, and carefully skipping over the shortcomings of the solutions implemented. Given the apparent success of this management style, other managers started to follow his example. Soon after (in less than a year), the entire organization became a toxic environment where raising even the minimal concern was seen as an attack on the “great new vision”. I see many parallels between this manager story and what is happening with ‘Twitter 2.0’ right now (although, I must say, in my case engineers did not get fired on the spot for speaking the truth). And with that manager, just like with ‘Twitter 2.0’, whenever problems occurred, those problems would either be ignored or blamed on the preexisting components built before the manager joined, never on the new, careless developments. The truth however was that problems that occurred had been preannounced weeks, or months before, but concerns around them had been promptly dismissed due to being too challenging to address, and because “everything works right now, so that’s not a concern”. The idea that everything must be correct because everything works, goals are achieved, and outcomes are successful, is a dangerous idea that can potentially have catastrophic consequences. It’s important to be critical and analytical, regardless of the outcome. This does not mean that success shouldn’t be celebrated, but that mistakes should be captured so that lessons can be learned from them, even if the final outcome was successful. Not learning from mistakes does not allow us to advance, and on the contrary can only lead us to repeat them. And if we keep repeating the same mistakes, sooner or later, those will have some negative consequences. A common practice in the aviation industry is to write reports on incidents, close calls, and near misses, whenever they occur, even if the flight was concluded successfully and no injuries or damages occurred. These reports are collected in databases like the <a href="https://asrs.arc.nasa.gov/">Aviation Safety Reporting System</a> (which can be freely consulted online), so that flight safety experts and regulators can identify common failure scenarios and eventually introduce mechanisms to improve safety in the aviation industry. A key element of these reports is that they are not meant to put the blame on certain people, but rather focus on what chain of events led to a certain mistake. “Human mistake” is generally not a valid root cause: if a human was able to make a mistake, it means that a mechanism is missing that can either prevent the mistake or detect it before it causes any negative consequences. Some companies in other industries have similar processes for writing reports or retrospectives when a mistake happens (regardless of the outcome), with the goal of finding proper root causes and preventing future mistakes. Amazon with its <a href="https://aws.amazon.com/blogs/mt/why-you-should-develop-a-correction-of-error-coe/">Correction of Error</a> practice is a famous example. I think introducing these practices in an organization can help to establish a healthy culture where finding mistakes and raising concerns is encouraged, rather than being oppressed. However these practices, by themselves, may not be enough to ensure that such a culture can be maintained over time, because people can always disagree on what is considered a ‘mistake’. Empathy is probably the key to a truly healthy culture that allows people to learn and advance. There are also cases where we are aware of problems, and we see them as such, but we deliberately choose not to do anything about them. This is where resilience comes into play. Resilience is generally a good quality to have. Resilience can give us the strength to go through long-term hardships, and can have positive effects on our tenancy and determination. But even resilience, when taken to the extreme, can be dangerous. Resilience can lead us to ignore problems, and not react to them. Resilience can make us tolerate a negative situation, without finding a proper strategy to cope with it. Poor planning forces you to consistently work extra hours? Resist and keep going, until you burn out. The relationship with your partner doesn’t satisfy you? Resist and think that things will get better, while the relationship slowly deteriorates. Feel pain in your knee every time you run for more than 30 minutes? Resist and don’t go to see a doctor, the pain will go away, sooner or later… until you cannot run anymore. When we let resilience become an excuse to avoid solving problems, we can end up in situations from which it’s difficult to recover. It’s important to make a distinction between what is under our control and what is not. We can fix problems that are under our control, but in situations where we cannot directly change the course of things, finding an alternative strategy is the only way. Resisting and hoping that things will get better often does not give the expected outcome–on the contrary, it can be detrimental. In the end, I think that the ‘practice’ of ignoring mistakes (because of the overconfidence built from successful outcomes) or ignoring problems (because of resilience taken to the extreme) are hidden time bombs, silently ticking, waiting for the right conditions before exploding. We need to be aware that just because things seem to work today, it doesn’t mean that we’re making the right decisions, and this can have consequences in the future. Being critical, analytical, empathetic, and honest is important to avoid these behaviors and the dangers that come with them.andreacorbelliniSat, 18 Mar 2023 08:20:00 +0000tag:andrea.corbellini.blog,2023-03-18:/2023/03/18/mistakes/behaviorAuthenticated encryption: why you need it and how it workshttps://andrea.corbellini.blog/2023/03/09/authenticated-encryption/In this article I want to explore a common problem of modern cryptographic ciphers: malleability. I will explain that problem with some hands-on examples, and then look in detail at how that problem is solved through the use of authenticated encryption. I will describe in particular two algorithms that provide authenticated encryption: ChaCha20-Poly1305 and AES-GCM, and briefly mention some of their variants. <h1 id="the-problem">The problem</h1> If we want to encrypt some data, a very common approach is to use a symmetric cipher. When we use a symmetric cipher, we hold a secret key, which is generally a sequence of bits chosen at random of some fixed length (nowadays ranging from 128 to 256 bits). The symmetric cipher takes two inputs: the secret key, and the message that we want to encrypt, and produces a single output: a ciphertext. Decryption is the inverse process: it takes the secret key and the ciphertext as the input and yields back the original message as an output. With symmetric ciphers, we use the same secret key both to encrypt and decrypt messages, and this is why they are called symmetric (this is in contrast with public key cryptography, or asymmetric cryptography, where encryption and decryption are performed using two different keys: a public key and a private key). Generally speaking, symmetric ciphers can be divided into two big families: <ul> <li>stream ciphers, which can encrypt data bit-by-bit;</li> <li>block ciphers, which can encrypt data block-by-block, where a block has a fixed size (usually 128 bits).</li> </ul> As we will discover soon, both these two families exhibit the same fundamental problem, although they slightly differ in the way this problem manifests itself. To understand this problem, let’s take a close look at how these two families of algorithms work and how we can manipulate the ciphertexts they produce. <h2 id="stream-ciphers">Stream ciphers</h2> A good way to think of a stream cipher is as a deterministic random number generator that yields a sequence of random bits. The secret key can be thought of as the seed for the random number generator. Every time we initialize the random number generator with the same secret key, we will get exactly the same sequence of random bits out of it. The bits coming out of the random number generator can then be XOR-ed together with the data that we want to encrypt: <code>ciphertext = random sequence XOR message</code>, like in the following example: <pre> random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F ⊕ message: I would really like an ice cream right now = ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1 </pre> The XOR operator acts as a toggle that can either flip bits or keep them unchanged. Let me explain with an example: <ul> <li><code>a XOR 0 = a</code></li> <li><code>a XOR 1 = NOT a</code></li> </ul> If we XOR “something” with a 0 bit, we get “something” out; if we XOR “something” with a 1 bit, we get the opposite of “something”. And if we use the same toggle twice, we return to the initial state: <ul> <li><code>a XOR b XOR b = a</code></li> </ul> This works for any <code>a</code> and any <code>b</code> and it’s due to the fact that <code>b XOR b</code> is always equal to 0. In more technical terms, each input is its own self-inverse under the XOR operator. The self-inverse property gives us a way to decrypt the message that we encrypted above: all we have to do is to replay the random sequence and XOR it together with the ciphertext: <pre> random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F ⊕ ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1 = message: I would really like an ice cream right now </pre> This works because <code>ciphertext = random sequence XOR message</code>, therefore <code>random sequence XOR ciphertext = random sequence XOR random sequence XOR message</code>. The two <code>random sequence</code> are the same, so they cancel each other (self-inverse), leaving only <code>message</code>: <pre> random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F ⊕ random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F ⊕ message: I would really like an ice cream right now = message: I would really like an ice cream right now </pre> Only the owner of the secret key will be able to generate the random sequence, therefore only the owner of the secret key should, in theory, be able to recover the message using this method. <h3 id="playing-with-stream-ciphers">Playing with stream ciphers</h3> The self-inverse property not only allows us to recover the message from the random sequence and the ciphertext, but it also allows us to recover the random sequence if can correctly guess the message: <pre> message: I would really like an ice cream right now ⊕ ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1 = random sequence: 3bAWC5ThFSPXX1W8P94q3XV35TG6CRVTNAPW27Q69F </pre> This “feature” opens the door to at least two serious problems. If we are able to correctly guess the message or a portion of it, then we can: <ol> <li>decrypt other ciphertexts produced by the same secret key (or at least portions of them, depending on what portions of the random sequence we were able to recover);</li> <li>modify ciphertexts.</li> </ol> And we can do all of this without any knowledge of the secret key. The first problem implies that key reuse is forbidden with stream ciphers. Every time we want to encrypt something with a stream cipher, we need a new key. This problem is easily solved by the use of a nonce (also known as initialization vector, IV, or starting variable, SV): a random value that is generated before every encryption, and that is combined in some way with the secret key to produce a new value to initialize the random number generator. If the nonce is unique per encryption, then we can be sufficiently confident that the random sequence generated will also be unique. The nonce value does not necessarily need to be kept secret, and needs to be known at decryption time. Nonces are usually generated at random at encryption time and stored alongside the ciphertext. The second problem is a bit more subtle: if we have a ciphertext and we can correctly guess the original message that produced it, we can modify it using the XOR operator to “cancel” the original message and “insert” a new message, like in this example: <pre> ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1 ⊕ message: I would really like an ice cream right now ⊕ altered message: I would really like to go to bed right now = tampered ciphertext: zB686Y0H46144HwT9RQQG7vTZt3Yc030n390ZCqXV1 </pre> This message, when correctly decrypted with the secret key, will return the tampered ciphertext without detection! Note that I do not need to know the full message to carry out this technique, in fact, the following example (where unknown parts have been replaced by hyphens) produces the same result as the above one: <pre> ciphertext: zB686Y0H46144HwT9RQQR6vZV1gU1779n390ZCqXV1 ⊕ message: --------------------an ice cream---------- ⊕ altered message: --------------------to go to bed---------- = tampered ciphertext: zB686Y0H46144HwT9RQQG7vTZt3Yc030n390ZCqXV1 </pre> This problem is known as <a href="https://en.wikipedia.org/wiki/Malleability_(cryptography)">malleability</a>, and it’s a serious issue in the real world because most of the messages that we exchange are in practice relatively easy to guess. Suppose for example that I have control over a WiFi network, and I can inspect and alter the internet traffic that passes through it. Suppose that I know that a person connected to my WiFi network is visiting an e-commerce website and that they’re interested in a particular item. The traffic that your browser exchanges with the e-commerce website may be encrypted, and therefore I won’t be able to decrypt its contents, but I might be able to guess certain parts of it, like the HTTP headers sent by the website, or some parts of the HTML that are common to all pages on that website, or even the name and the price of the item you want to buy. If I can guess that information (which is public information, not a secret, and it’s generally easy to guess), then I might be able to alter some parts of the web page, showing you false information, and altering the price that you see in an attempt to trick you into buying that item. <details> <summary>An example of malleability using ChaCha20 with OpenSSL</summary> Here’s a practical example of how we can take the output of a stream cipher, and alter it as we wish without knowledge of the secret key. I’m going to use the OpenSSL command line interface to encrypt a message with a stream cipher: <a href="https://en.wikipedia.org/wiki/ChaCha20">ChaCha20</a>. This is a modern, fast, stream cipher with a good reputation: <div class="highlight"><pre><code>openssl enc -chacha20 \ -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \ -iv 0123456789abcdef0123456789abcdef \ -in <(echo 'I would really like an ice cream right now') \ -out ciphertext.bin </code></pre></div> The <code>-K</code> option specifies the key in hexadecimal format (256 bits, or 32 bytes, or 64 hex characters), the <code>-iv</code> is the nonce, also known as initialization vector (128 bits, or 16 bytes, or 32 hex characters). This trivial Python script can tamper with the ciphertext: <div class="highlight"><pre><code>with open('ciphertext.bin', 'rb') as file: ciphertext = file.read() guessed_message = b'--------------------an ice cream----------\n' replacement_message = b'--------------------to go to bed----------\n' tampered_ciphertext = bytes(x ^ y ^ z for (x, y, z) in zip(ciphertext, guessed_message, replacement_message)) with open('tampered-ciphertext.bin', 'wb') as file: file.write(tampered_ciphertext) </code></pre></div> This script is using partial knowledge of the message. It knows (thanks to an educated guess) that the original message contained the words “an ice cream” at a specific offset, and uses that knowledge to replace those words with new ones (“to go to bed”) which add up to the same length. Note that this technique cannot be used to remove or add parts from the message, only to modify them without changing their length. Now if we run this script and we decrypt the <code>tampered-ciphertext.bin</code> file with the same key and nonce as before, we get “to go to bed” instead of “an ice cream”, without any error indicating that tampering occurred: <div class="highlight"><pre><code>openssl enc -d -chacha20 \ -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \ -iv 0123456789abcdef0123456789abcdef \ -in tampered-ciphertext.bin </code></pre></div> </details> <h2 id="block-ciphers">Block ciphers</h2> We have seen that stream ciphers alone have a serious problem (malleability) that allows anyone to modify arbitrary portions of ciphertexts without detection. Let’s take a look at the alternative: block ciphers. Will they have the same problem? While a stream cipher can encrypt variable amounts of data, a block cipher can only take as input a block of data of a fixed size, and produce as output another block of data. A good block cipher produces an output that is indistinguishable from random. The block size is generally small, usually 128 bits (16 bytes), so if we want to encrypt larger amounts of data, we have to split the data into multiple blocks, and encrypt each block individually. If the data is too short to fit in a block, the data will also need to be padded. <pre> message: The cat is on th e table......... |______________| |______________| block #1 block #2 (padded) ciphertext: c2TNPW3r09hZ6f1P Vc32VX41XSy579Y9 </pre> This approach however has a problem: if we encrypt multiple blocks with the same secret key, then portions of messages that are the repeated will produce the same output. This gives the ability to analyze a ciphertext and find patterns in it without knowledge of the secret key. This problem is famously evident when encrypting pictures: <figure> <img src="https://andrea.corbellini.blog/images/lack-of-diffusion.webp" alt="Ubuntu logo before and after encryption with a block cipher"> <figcaption>Example of applying a block cipher to an uncompressed image. The original colors are lost, but the overall layout of the image is still understandable. That's because multiple blocks of the image (containing the RGB values of each pixel), for example from the white background, are repeated multiple times, yielding the same exact encrypted blocks. The inspiration for making this image came from <a href="https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation">Wikipedia</a>.</figcaption> </figure> <details> <summary>How this image was generated</summary> Before jumping into how I encrypted the image, let me spend a few words on how I did NOT encrypt the image: I did not use a modern image format. Modern image formats are very sophisticated, they’re not a simple sequence of RGB values. Instead, they have some control structures mixed in the image, they implement compression to reduce the image size, etc. This complexity means that if I simply take an image in any format and encrypt it, the result won’t be visualizable by an image viewer: the image viewer would just throw an error because it would find invalid data structures. Note that this does not imply that encrypting modern image formats is more secure: people can still analyze patterns in them, but it simply means that a modern image format, once encrypted, would not produce the sensational visualization that I showed above. In order to produce this visualization I had to find an uncompressed image format without too much metadata in it. Thankfully the Wikipedia article on <a href="https://en.wikipedia.org/wiki/Image_file_format">image file formats</a> provided a list, which included the <a href="https://en.wikipedia.org/wiki/Netpbm">Netpbm</a> family of formats (something I never heard of before). Among the formats in this family, I chose PPM, because it’s the one that supports colors. The PPM file format is very simple: it has 3 lines of metadata, followed by the RGB values for each pixel. No compression. Definitely the right format for this kind of experiment! So here’s what I did: first of all I downloaded an image (the Ubuntu “Circle of Friends” logo, obtained from Wikipedia) and converted it to PPM with <a href="https://imagemagick.org/script/convert.php">ImageMagick</a>: <div class="highlight"><pre><code>convert UbuntuCoF.png img.ppm </code></pre></div> I separated the header from the RGB values: <div class="highlight"><pre><code>head -n3 img.ppm > ppm-header tail -n+4 img.ppm > ppm-image </code></pre></div> The reason why I separated the header from RGB values is that I won’t encrypt the header. If I did, then the image won’t be visualizable by an image viewer, just like if I used a modern image format. In a real-world scenario, a person would be able to easily guess the header if it was encrypted. I encrypted the RGB values with <a href="https://en.wikipedia.org/wiki/Advanced_Encryption_Standard">AES-256</a>, a modern, strong block cipher with a good reputation: <div class="highlight"><pre><code>openssl enc -aes-256-ecb \ -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \ -in ppm-image \ -out ppm-image-encrypted </code></pre></div> Then I joined the header and the encrypted RGB values in a PPM file: <div class="highlight"><pre><code>cat ppm-header ppm-image-encrypted > img-encrypted.ppm </code></pre></div> This results in a randomized image that can be viewed without problems on an image viewer. It’s interesting to see that, if you change the encryption key, you will get a different randomized image! </details> The problem we have just seen is known as lack of diffusion. This is kinda analogous to the first problem we identified with stream ciphers: at the root of both problems there is key reuse. We solved this problem for stream ciphers by combining a key with a random nonce. We could use the same strategy here, but it would be an expensive approach, as initializing a block cipher with a new key is a relatively expensive operation. It’s much cheaper to initialize the block cipher once, and reuse it for every block encryption. We need a way to “link” blocks to each other, so that if two linked blocks contain the same plaintext, their encryption will give different results. There are various strategies do that. These strategies are known as <a href="https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation">mode of operation</a> of block ciphers. Let’s take a look at two of them: Cipher Block Chaining (CBC) and Counter Mode (CTR). <h3 id="cipher-block-chaining-cbc">Cipher Block Chaining (CBC)</h3> This mode of operation, as the name suggests, ‘chains’ each block to the next one. The way it works is by using the XOR operator in the following way: <ul> <li> First of all, a random nonce is generated. The purpose of the nonce is the same as before (with stream ciphers): ensuring that using the same secret key to perform multiple encryptions yields different results each time, so that secret information or patterns are not revealed. The nonce does not need to be kept secret and is normally stored alongside the ciphertext so that it can be easily used during decryption. It is however important that the nonce is unique. </li> <li> The first block of message <code>m[0]</code> is XOR-ed with the nonce, and then encrypted the block cipher, producing the first block of ciphertext <code>c[0] = block_encrypt(m[0] XOR nonce)</code> </li> <li> The second block of message <code>m[1]</code> is XOR-ed with <code>c[0]</code>, and then encrypted with the block cipher: <code>c[1] = block_encrypt(m2 XOR c[0])</code> </li> <li> … </li> <li> The last block of message <code>m[n]</code> is XOR-ed together with <code>c[n-1]</code>, and then encrypted with the block cipher: <code>c[n] = block_encrypt(m[n] XOR c[n-1])</code> </li> </ul> <figure> <img src="https://andrea.corbellini.blog/images/cipher-block-chaining.svg" alt="Visualization of the Cipher Block Chaining (CBC) mode of operation"> </figure> The XOR operator is back! With stream ciphers, the XOR operator was allowing us to tamper with ciphertexts. Can we do the same thing here? Yes of course! The approach is slightly different though: instead of acting directly on the block that we want to change, we will act on the block that precedes it. For example, if we want to change the sentence “I came home in the afternoon and the cat was on the table” so that it reads ‘dog’ instead of ‘cat’, we would need to change the block right before the one that contains the word ‘cat’. If we want to change the very first block, for example to change the word ‘came’ to ‘left’, we would need to change the nonce instead. <pre> nonce+ciphertext: yzURZRbP6X1w3ZRL XRDnPbEkx3JUP2Fv C2ZWt19EdAXDi76H pkbk8qTgaSdzerbF 8CWYqscBqE6cSLmx ⊕ message (shifted 1 block to the left): I came home in t he afternoon and the cat was on the table....... ⊕ altered message (shifted 1 block to the left): I left home in t he afternoon and the dog was on the table....... = tampered nonce+ciphertext: yzZVQCbP6X1w3ZRL XRDnPbEkx3JUP2Fv C2ZWt67VdAXDi76H pkbk8qTgaSdzerbF 8CWYqscBqE6cSLmx </pre> If we do the above, and then decrypt the tampered ciphertext, we will get something like this: <div class="highlight"><pre><code>I left home in t��the dog was on the table </code></pre></div> <details> <summary>How to get this result using AES-CBC with OpenSSL</summary> Here’s a step-by-step guide on how to tamper with a ciphertext encrypted with AES-256 in CBC mode. First, generate a valid ciphertext: <div class="highlight"><pre><code>openssl enc -aes-256-cbc \ -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \ -iv 0123456789abcdef0123456789abcdef \ -in <(echo 'I came home in the afternoon and the cat was on the table') \ -out ciphertext.bin </code></pre></div> Like in the stream cipher example, <code>-K</code> is the key in hexadecimal format (256 bits), while <code>-iv</code> is the nonce (128 bits). We can perform the tampering with this Python script: <div class="highlight"><pre><code>with open('ciphertext.bin', 'rb') as file: ciphertext = file.read() guessed_message = b'---------------------cat----------------------------------------' replacement_message = b'---------------------dog----------------------------------------' tampered_ciphertext = bytes(x ^ y ^ z for (x, y, z) in zip(ciphertext, guessed_message, replacement_message)) with open('tampered-ciphertext.bin', 'wb') as file: file.write(tampered_ciphertext) </code></pre></div> Note that OpenSSL does not store the nonce along with the ciphertext, but instead expects it to be passed as a command line argument. We need to modify it separately, so here’s another Python script just for the nonce: <div class="highlight"><pre><code>nonce = bytes.fromhex('0123456789abcdef0123456789abcdef') guessed_message = b'--came----------' replacement_message = b'--left----------' tampered_nonce = bytes(x ^ y ^ z for (x, y, z) in zip(nonce, guessed_message, replacement_message)) print(tampered_nonce.hex()) </code></pre></div> If we run that script, we get: <code>01234a6382bacdef0123456789abcdef</code>. Now to decrypt the tampered ciphertext with the tampered nonce: <div class="highlight"><pre><code>openssl enc -d -aes-256-cbc \ -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \ -iv 01234a6382bacdef0123456789abcdef \ -in tampered-ciphertext.bin </code></pre></div> </details> It’s interesting to see that we were successful in changing the word ‘cat’ to ‘dog’ but in doing so we had to sacrifice a block, which, when decrypted, resulted in random bytes. In a real world scenario, seeing some random bytes could raise some suspicion, and maybe generate some errors in applications, however that’s not always the case (how many times have we seen garbled text on our monitors, and we never worried that somebody was tampering with our communications). Also, when dealing with formats like HTML, one could conceal tampering attempts using comment blocks, or using JavaScript. One example of what I’m describing is the <a href="https://efail.de/">EFAIL vulnerability</a>: discovered in 2017, it affected some popular email clients including Gmail, it targeted the use of AES in CBC mode (as well as another mode very similar to it: Cipher Feedback, CFB), and allowed the injection of malicious content in HTML emails. We can conclude that block ciphers in CBC mode, just like stream ciphers, are also malleable. <h3 id="counter-mode-ctr">Counter Mode (CTR)</h3> Are other modes of operation all malleable like CBC, or will they be different? Let’s take a look at another, very common, mode of operation: Counter Mode (CTR), so that we can get a better sense of how the problem of malleability can affect the world of block ciphers. The mechanism behind Counter Mode is very simple: <ol> <li> A random nonce is generated. The purpose of the nonce is the usual one: make sure that repeated encryptions using the same key produce different results. </li> <li> A counter (usually an integer) is initialized to 1 (or any other starting value of your choice). </li> <li> The nonce is concatenated with the counter, and encrypted using the block cipher: <code>r[0] = block_encrypt(nonce || counter)</code>. Because the block cipher can only accept as input a block of a fixed size, it follows that the length of the nonce plus the length of the counter must be equal to the block size. For example, for a 128-bit block cipher, a common choice is to have a 96-bit nonce and a 32-bit counter. </li> <li> The counter is incremented: <code>counter = counter + 1</code> (the increment does not necessarily need to be by 1, but that’s a common choice). The nonce and the new counter are concatenated again, and encrypted using the block cipher: <code>r[1] = block_encrypt(nonce || counter)</code>. </li> <li> The counter is incremented again (<code>counter = counter + 1</code>), and a new block is encrypted, just like before: <code>r[2] = block_encrypt(nonce || counter)</code>. </li> <li> … </li> </ol> This mechanism produces a sequence of blocks <code>r[0]</code>, <code>r[1]</code>, <code>r[2]</code>, … which are indistinguishable from random. This sequence of random blocks can be XOR-ed with the message to produce the ciphertext. It’s important that the values for the counter never repeat. If, for example, we’re using a 32-bit counter, the counter will “reset” (go back to the starting value) after 232 iterations, and will start repeating the same sequence of random blocks as it did at the beginning. This introduces the problem of lack of diffusion that we have seen before, just at a larger scale. If we’re using a 32-bit counter with a 128-bit block cipher, we cannot encrypt more than 128·232 bits = 64 GiB of data at once. This is a very important detail: exceeding these limits may allow the decryption of portions of ciphertext without knowledge of the secret key. <figure> <img src="https://andrea.corbellini.blog/images/counter-mode.svg" alt="Visualization of the Counter Mode (CTR) mode of operation"> </figure> What Counter Mode is doing is effectively turning a block cipher into a stream cipher. As such, a block cipher in Counter Mode has the exact same malleability problems of stream ciphers that we have seen before. <details> <summary>An example of malleability using AES-CTR with OpenSSL</summary> This example is going to be very similar (almost identical) to the example with ChaCha20 that I showed in the stream cipher section, just that this time I’m going to use AES-256 in CTR mode. Let’s produce a valid ciphertext: <div class="highlight"><pre><code>openssl enc -aes-256-ctr \ -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \ -iv 0123456789abcdef0123456700000001 \ -in <(echo 'Can you give me a ride to the party?') \ -in <(echo 'Do not give me a ride to the party!') \ -out ciphertext.bin </code></pre></div> Tamper it with Python: <div class="highlight"><pre><code>with open('ciphertext.bin', 'rb') as file: ciphertext = file.read() guessed_message = b'--------------------an ice cream----------\n' replacement_message = b'--------------------to go to bed----------\n' tampered_ciphertext = bytes(x ^ y ^ z for (x, y, z) in zip(ciphertext, guessed_message, replacement_message)) with open('tampered-ciphertext.bin', 'wb') as file: file.write(tampered_ciphertext) </code></pre></div> And now we can decrypt it: <div class="highlight"><pre><code>openssl enc -d -aes-256-ctr \ -K 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \ -iv 0123456789abcdef0123456700000001 \ -in tampered-ciphertext.bin </code></pre></div> </details> <h1 id="the-solution-authenticated-encryption-ae">The solution: Authenticated Encryption (AE)</h1> We have seen that stream ciphers and block ciphers (in their mode of operation) both exhibit the same problem (in different flavors): malleability. I’ve shown some examples of how this problem can be exploited with modern ciphers like ChaCha20 and AES. These ciphers, alone, cannot guarantee the integrity or authenticity of encrypted data. In this context, <a href="https://en.wikipedia.org/wiki/Data_integrity">integrity</a> is the assurance that data is not corrupted or modified in any way. <a href="https://en.wikipedia.org/wiki/Message_authentication">Authenticity</a> can be thought of as a stronger version of integrity, and it’s the assurance that a given ciphertext was produced only with knowledge of the secret key. Does this mean that modern ciphers, like ChaCha20 and AES, should be considered insecure and avoided? Absolutely not! The correct answer is that those ciphers cannot be used alone. You should think of them as basic building blocks, and you need some additional building blocks in order to construct a complete and secure cryptosystem. One of these additional building blocks, that we are going to explore in this article, is an algorithm that provides integrity and authentication: welcome Authenticated Encryption (AE). When using authenticated encryption, an adversary may be able to modify a ciphertext using the techniques described above, but such modification would be detected by the authentication algorithm, and decryption will fail with an error. The decrypted message at that point should be discarded, preventing the use of tampered data. There are many different methods to implement authenticated encryption. The most common approach is to use an authentication algorithm to authenticate the ciphertext produced by a cipher. Here I will describe two very popular authentication algorithms: <ul> <li>Poly1305, which is often used in conjunction with the stream cipher ChaCha20 to form ChaCha20-Poly1305;</li> <li>Galois/Counter Mode (GCM), which is often used with the block cipher AES to form AES-GCM.</li> </ul> These authentication algorithms work by computing a hash of the ciphertext, which is then stored alongside the ciphertext. This hash is not a regular hash, but it’s a keyed hash. A regular hash is a function that takes as input some data and returns a fixed-size bit string: $$\operatorname{hash}: data \rightarrow bits$$ A keyed hash instead takes two inputs: a secret key and some data, and produces a fixed-size bit string: $$\operatorname{keyed-hash}: (key, data) \rightarrow bits$$ The output of the keyed hash is more often called Message Authentication Code (MAC), or authentication tag, or even just tag. During decryption, the same authentication algorithm is run again on the ciphertext, and a new tag is produced. If the new tag matches the original tag (that was stored alongside the ciphertext), then decryption succeeds. Else, if the tags don’t match, it means that the ciphertext was modified (or the stored tag was modified), and decryption fails. This gives us a way to detect tampering and gives us the opportunity to reject ciphertexts that were not produced by the secret key. The secret key passed to the keyed hash function is not necessarily the same secret key used for the encryption. In fact, both ChaCha20-Poly1305 and AES-GCM operate on a subkey derived from the key used for encryption. <h2 id="poly1305">Poly1305</h2> Poly1305 is a keyed hash function proposed by <a href="https://en.wikipedia.org/wiki/Daniel_J._Bernstein">Daniel J. Bernstein</a> in 2004, who is also the author of ChaCha20. It works by using polynomials evaluated modulo the prime 2130 - 5, hence the name. The key to Poly1305 is a 256-bit string, and it’s split into two halves: <ul> <li>the first half (128 bits) is called $r$;</li> <li>the second half (128 bits) is called $s$.</li> </ul> We’ll see later how this key is generated when Poly1305 is used to implement authenticated encryption. For now, let’s assume that the key is a random (unpredictable) bit string provided as an input. The first half $r$ is also clamped by setting some of its bits to 0. This is a performance-related optimization that some Poly1305 implementations can take advantage of when doing multiplication using 64-bit registers. Clamping is performed by applying the following hexadecimal bitmask: <div class="highlight"><pre><code>0ffffffc0ffffffc0ffffffc0fffffff </code></pre></div> The message to authenticate is split into chunks of 128 bits each: $m_1$, $m_2$, $m_3$, … $m_n$. If the length of the message is not a multiple of 128 bits, then the last block may be shorter. The authentication tag is then calculated as follows: <ul> <li> Interpret $r$ and $s$ as two 128-bit little-endian integers. </li> <li> Initialize the Poly1305 state $a_0$ to the integer 0. As we shall see later, this state will need to hold at most 131 bits. </li> <li> For each block $m_i$: <ul> <li> Interpret the block $m_i$ as a little-endian integer. </li> <li> Compute $\overline{m}_i$ by appending a 1-bit to the end of the block $m_i$. If $m_i$ is 128 bits long, then this is equivalent to computing $\overline{m}_i = 2^{128} + m_i$. In general, if the length of the block $m_i$ in bits is $\operatorname{len}(m_i)$, then this is equivalent to $\overline{m}_i = 2^{\operatorname{len}(m_i)} + m_i$. This step ensures that the resulting block $\overline{m}_i$ is always non-zero, even if the original block $\overline{m}_i$ is zero. This is important for the security of the algorithm, as explained later. </li> <li> Compute the new state $a_i = (a_{i-1} + \overline{m}_i) \cdot r \pmod{2^{130} - 5}$. Note that, because the operation is modulo $2^{130} - 5$, the result will always fit in 130 bits. </li> </ul> </li> <li> Once each block has been processed, compute the final state $a_{n+1} = a_n + s$. Note that the state $a_n$ is at most 130 bits long, and $s$ is at most 128 bits long, hence the result will be at most 131 bits long. </li> <li> Truncate the final state $a_{n+1}$ to 128 bits by removing the most significant bits. </li> <li> Return the truncated final state $a_{n+1}$ as a little-endian byte string. </li> </ul> What this method is doing is computing the following polynomial in $r$ and $s$: $$\begin{align*} tag & = ((((((\overline{m}_1 \cdot r) + \overline{m}_2) \cdot r) + \cdots + \overline{m}_n) \cdot r) \bmod{(2^{130} - 5)}) + s \\ & = (\overline{m}_1 r^n + \overline{m}_2 r^{n-1} + \cdots + \overline{m}_n r) \bmod{(2^{130} - 5)} + s \end{align*}$$ $r$ and $s$ are secrets, and they come from the Poly1305 key. Note that if we didn’t add $s$ at the end, then the resulting polynomial would be a polynomial in $r$, and one could use polynomial root-finding methods to figure out $r$ from the authentication tag, without knowledge of the key. Therefore it’s important that $s$ is non-zero. In Python, this is what a Poly1305 implementation could look like (disclaimer: this is for learning purposes, and not necessarily secure or optimized for performance): <div class="highlight"><pre><code>def iter_blocks(message: bytes): """ Splits a message in blocks of 16 bytes (128 bits) each, except for the last block, which may be shorter. """ start = 0 while start < len(message): yield message[start:start+16] start += 16 def poly1305(key: bytes, message: bytes): assert len(key) == 32 # 256 bits # Prime for the evaluation of the polynomial p = (1 << 130) - 5 # Split the key into two parts r and s r = int.from_bytes(key[:16], 'little') # 128 bits s = int.from_bytes(key[16:], 'little') # 128 bits # Clamp r r = r & 0x0ffffffc0ffffffc0ffffffc0fffffff # Initialize the state a = 0 # Update the state with every block for block in iter_blocks(message): # Append a 1-bit to the end of each block block = block + b'\1' # Convert the block to an integer c = int.from_bytes(block, 'little') # Update the state a = ((a + c) * r) % p # Add s to the state and truncate it to 128 bits, removing the most # significant bits and keeping only the least significant 128 bits a = (a + s) & ((1 << 128) - 1) # Convert the state from an integer to a 16-byte string (128 bits) return a.to_bytes(16, 'little') </code></pre></div> And here is an example of how that code could be used: <div class="highlight"><pre><code>key = bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef') msg = b'I had a very nice day today at the beach' print(poly1305(key, msg).hex()) </code></pre></div> This returns <code>b0c4cb74b3089e9a982e3baa90c1bb5f</code>, which is the same result that we would get using OpenSSL: <div class="highlight"><pre><code>openssl mac \ -macopt hexkey:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \ -in <(echo -n 'I had a very nice day today at the beach') \ poly1305 </code></pre></div> A few things to note: <ul> <li> The same key cannot be reused to construct two distinct tags. In fact, suppose that we use the same hash key to compute <code>tag1 = Poly1305(key, msg1)</code> and <code>tag2 = Poly1305(key, msg2)</code>. Then, because $s$ is the same for both, we could subtract the two tags (<code>tag1 - tag2</code>) to remove the $s$ part and obtain a polynomial in $r$. From there, we could use algebraic methods to figure out $r$. Once we have $r$, we can use either one of the tags and compute $s$, therefore recovering the full secret key. Similarly, if the keys were generated using a predictable algorithm (for example, incrementally: <code>key[i+1] = key[i] + 1</code>), it would still be possible to use a similar approach to figure out the secret key. For this reason, Poly1305 keys must be unique and unpredictable. Generating Poly1305 keys randomly or pseudo-randomly is an acceptable approach. Authentication functions like Poly1305 are called one-time authenticators because they can be used only one time with the same key. </li> <li> If we didn’t add the 1-bits at the end of each block (in other words, if we used the $m_i$ blocks instead of $\overline{m}_i$), then encrypting a message full of zero bits would be the equivalent of encrypting an empty message. Adding the 1-bits is a way to ensure that the length of the message always has an effect on the output. </li> </ul> <h3 id="use-of-poly1305-with-chacha20-chacha20-poly1305">Use of Poly1305 with ChaCha20 (ChaCha20-Poly1305)</h3> Let’s see how we can combine ChaCha20 and Poly1305 to construct an authenticated cipher. To recap: <ul> <li>ChaCha20 is a stream cipher;</li> <li>Poly1305 is a one-time authenticator;</li> <li>ChaCha20, like most ciphers, requires the use of a unique nonce to allow key reuse.</li> </ul> Putting the two together gives birth to ChaCha20-Poly1305. Here I’m going to describe how to implement it as standardized in <a href="https://www.rfc-editor.org/rfc/rfc8439">RFC 8439</a>. The inputs to the ChaCha20-Poly1305 encryption function are: <ul> <li>a 256-bit secret key;</li> <li>a 96-bit nonce;</li> <li>a variable-length plaintext message.</li> </ul> The outputs from the ChaCha20-Poly1305 encryption function are: <ul> <li>a variable-length ciphertext (same length as the input plaintext);</li> <li>a 128-bit authentication tag.</li> </ul> The ChaCha20-Poly1305 decryption function will accept the same secret key, nonce, ciphertext, and authentication tag as the input, and produce either the plaintext or an error as the output. The error is returned in case the authentication fails. <figure> <img src="https://andrea.corbellini.blog/images/chacha20-poly1305-encryption.svg" alt="Diagram of data flow during encryption with ChaCha20-Poly1305"> <figcaption>Data flow during a ChaCha20-Poly1305 encryption. This shows the inputs in blue, the outputs in green, and the intermediate objects in red.</figcaption> </figure> ChaCha20-Poly1305 works in the following way: <ol> <li> The ChaCha20 stream cipher is initialized with the 256-bit secret key and the 96-bit nonce. </li> <li> The stream cipher is used to encrypt a 256-bit string of all zeros. The result is the Poly1305 subkey. If you recall how a stream cipher works, you should know that encrypting using a stream cipher is equivalent to performing the XOR of a random bit stream with the plaintext. Here the plaintext is all zeros, so the process of generating the Poly1305 subkey is equivalent to grabbing the first 256 bits from the ChaCha20 bit stream. We previously saw that the Poly1305 subkey must be unpredictable and unique in order for Poly1305 to be secure. The use of ChaCha20 with a unique nonce ensures that: because ChaCha20 is a stream cipher, its output will be random and unpredictable. Therefore, with this construction, the subkey will be unpredictable even if the nonce is predictable. </li> <li> The stream cipher is used to encrypt another 256-bit string. The result is discarded. This is equivalent to advancing the stream cipher state by 256 bits. This step may seem weird, and in fact is not needed for security purposes, but it’s a mere implementation detail. This step is here because ChaCha20 has an internal state of 512 bits. In the previous step we obtained the first 256 bits of the state, and this next step is to discard the rest of the state to start with a fresh state. There is no particular reason for requiring a fresh state. The reason why RFC 8439 does that is because… spoiler alert: ChaCha20 is a block cipher under the hood. Its block size is 512 bits. If you read the RFC, you’ll see that it asks to call the ChaCha20 block encryption function once, grab the first 256 bits, and discard the rest. Here I’m treating ChaCha20 as a stream cipher, so I have to include this extra step to discard the bits. </li> <li> The plaintext is encrypted using the stream cipher. Note that this is done without resetting the state of the cipher. We are continuing to use the same stream cipher instance that was used to generate the Poly1305 subkey. </li> <li> The ciphertext is padded with zeros to make its length a multiple of 16 bytes (128 bits) and is authenticated using Poly1305, via the subkey generated in step 2. This step may be done in parallel to the previous one, that is: every time we generate a chunk of ciphertext, we feed it to the Poly1305 authentication function. Why pad the ciphertext before passing it to Poly1305? After all, ChaCha20 is a stream cipher, and Poly1305 can accept arbitrary-sized messages. Again, this is an detail of RFC 8439 and padding does not serve any specific purpose. </li> <li> The length of the ciphertext (in bytes) is fed into the Poly1305 authenticator. This length is represented as a 64-bit little-endian integer padded with 64 zero bits. The reason why the length is represented as 64 bits and padded (instead of representing it as 128 bits) will be clearer later: what I have given you so far is a simplified view of ChaCha20-Poly1305 and authenticated encryption in general. I will give you the full picture when talking about <a href="#authenticated-encryption-with-associated-data-aead">associated data</a> later on, and at that point this step will be slightly modified. </li> <li> The ciphertext from ChaCha20 and the authentication tag from Poly1305 are returned. </li> </ol> The decryption algorithm works in a very similar way: ChaCha20 is initialized in the same way, the subkey is generated in the same way, the Poly1305 authentication tag is calculated from the ciphertext in the same way. The only difference is that ChaCha20 is used to decrypt the ciphertext (instead of encrypting the plaintext) and that the input authentication tag is compared to the calculated authentication tag before returning. Here is a Python implementation of ChaCha20-Poly1305, based on the implementations of ChaCha20 and Poly1305 from <a href="https://pypi.org/project/pycryptodome/">pycryptodome</a> (usual disclaimer: this code is for educational purposes, and is not necessarily secure or optimized for performance): <div class="highlight"><pre><code>from Crypto.Cipher import ChaCha20 from Crypto.Hash import Poly1305 def chacha20poly1305_encrypt(key, nonce, message): assert len(key) == 32 # 256 bits assert len(nonce) == 12 # 96 bits # Initialize the ChaCha20 cipher with the key and nonce cipher = ChaCha20.new(key=key, nonce=nonce) # Derive the Poly1305 subkey using the ChaCha20 cipher subkey = cipher.encrypt(b'\0' * 32) # 256 bits subkey_r = subkey[:16] subkey_s = subkey[16:] # Initialize the Poly1305 authenticator with the subkey authenticator = Poly1305.Poly1305_MAC(r=subkey_r, s=subkey_s, data=None) # Discard the rest of the internal ChaCha20 state cipher.encrypt(b'\0' * 32) # 256 bits # Encrypt the message ciphertext = cipher.encrypt(message) # Authenticate the ciphertext authenticator.update(ciphertext) # Pad the ciphertext with zeros (to make it a multiple of 16 bytes) if len(ciphertext) % 16 != 0: authenticator.update(b'\0' * (16 - len(ciphertext) % 16)) # Authenticate the length of the associated data (0 for simplicity) authenticator.update((0).to_bytes(8, 'little')) # 64 bits # Authenticate the length of the ciphertext authenticator.update(len(ciphertext).to_bytes(8, 'little')) # 64 bits # Generate the authentication tag tag = authenticator.digest() return (ciphertext, tag) def chacha20poly1305_decrypt(key, nonce, ciphertext, tag): assert len(key) == 32 # 256 bits assert len(nonce) == 12 # 96 bits assert len(tag) == 16 # 128 bits # Initialize the ChaCha20 cipher and the Poly1305 authenticator, in the # same exact way as it was done during encryption cipher = ChaCha20.new(key=key, nonce=nonce) subkey = cipher.encrypt(b'\0' * 32) subkey_r = subkey[:16] subkey_s = subkey[16:] authenticator = Poly1305.Poly1305_MAC(r=subkey_r, s=subkey_s, data=None) cipher.encrypt(b'\0' * 32) # Generate the authentication tag, like during encryption authenticator.update(ciphertext) if len(ciphertext) % 16: authenticator.update(b'\0' * (16 - len(ciphertext) % 16)) authenticator.update((0).to_bytes(8, 'little')) authenticator.update(len(ciphertext).to_bytes(8, 'little')) expected_tag = authenticator.digest() # Compare the input tag with the generated tag. If they're different, the # plaintext must not be returned to the caller if tag != expected_tag: raise ValueError('authentication failed') # The two tags match; decrypt the plaintext and return it to the caller # Note that, because ChaCha20 is a symmetric cipher, there is no difference # between the encrypt and decrypt method: here we are reusing the same # exact code used during decryption message = cipher.encrypt(ciphertext) return message </code></pre></div> And here is how it can be used: <div class="highlight"><pre><code>key = bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef') nonce = bytes.fromhex('0123456789abcdef01234567') message = b'I wanted to go to the beach, but now I changed my mind' ciphertext, tag = chacha20poly1305_encrypt(key, nonce, message) decrypted_message = chacha20poly1305_decrypt(key, nonce, ciphertext, tag) assert message == decrypted_message print(f'ciphertext: {ciphertext.hex()}') print(f' tag: {tag.hex()}') print(f' plaintext: {decrypted_message}') </code></pre></div> Running it produces the following output: <div class="highlight"><pre><code>ciphertext: 5d9b09cc5d90ca9ddff2d3470cfd6b563c5158e952bfae6acf1ebf9a3b968a488a41969567ef5ccfe05dcf9e548567028ff374a754af tag: dac3c05d261920e278ceb22e2800aa95 plaintext: b'I wanted to go to the beach, but now I changed my mind' </code></pre></div> This is the same output we would obtain by using the ChaCha20-Poly1305 implementation from pycryptodome directly: <div class="highlight"><pre><code>from Crypto.Cipher import ChaCha20_Poly1305 key = bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef') nonce = bytes.fromhex('0123456789abcdef01234567') message = b'I wanted to go to the beach, but now I changed my mind' cipher = ChaCha20_Poly1305.new(key=key, nonce=nonce) ciphertext, tag = cipher.encrypt_and_digest(message) print(f'ciphertext: {ciphertext.hex()}') print(f' tag: {tag.hex()}') </code></pre></div> As already stated, it is extremely important that the nonce passed to ChaCha20-Poly1305 is unique. It may be predictable, but it must be unique. If the same nonce is reused twice or more, we can: <ul> <li> Decrypt arbitrary messages without using the secret key, if we can guess at least one message from its ciphertext. This can be done using the techniques described at the beginning of this article: by recovering the random bit string from the XOR of the ciphertext with the guessed message. </li> <li> Recover the Poly1305 subkey, and, at that point, tamper with ciphertexts and forge new, valid authentication tags. This can be done by using algebraic methods on the polynomial of the authentication tag. </li> </ul> There is also a variant of ChaCha20-Poly1305, called XChaCha20-Poly1305, that features an extended 192-bit nonce (the X stands for ‘extended’). This is described in an <a href="https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-xchacha">RFC draft</a> but so far it hasn’t been accepted as a standard yet. I won’t cover XChaCha20 in detail here, because it’s slightly more complex and does not add much to the topic of this article, but XChaCha20-Poly1305 has better security properties than ChaCha20-Poly1305, so you should prefer it in your applications if you can use it. The reason why XChaCha20-Poly1305 has better properties than ChaCha20-Poly1305 is that, having a longer nonce, the probability of generating two random nonces with the same value are much lower. <h2 id="galoiscounter-mode-gcm">Galois/Counter Mode (GCM)</h2> Let’s now take a look at Galois/Counter Mode (GCM). This is commonly used with the Advanced Encryption Standard (AES), to construct the authenticated cipher AES-GCM. One main difference between Poly1305 and GCM is that Poly1305 can work with any stream or block cipher, while GCM is designed to work with block ciphers with a block size of 128 bits. GCM was proposed by David McGrew and John Viega in 2004 and is standardized in <a href="https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf">NIST Special Publication 800-38D</a> as well as <a href="https://www.rfc-editor.org/rfc/rfc5288">RFC 5288</a>. It takes its name from Galois fields, also known as <a href="https://en.wikipedia.org/wiki/Finite_field">finite fields</a>, which in turn get their name from the French mathematician <a href="https://en.wikipedia.org/wiki/%C3%89variste_Galois">Évariste Galois</a>, who introduced the concept of finite fields as we know them today. As we did before with Poly1305, we are going to first see how the keyed hash function used by GCM works, and then we will see how to use it to construct an authenticated cipher like AES-GCM on top of it. Before we can do that though, we need to understand what finite fields are, and what specific types of finite fields are used in GCM. <h3 id="finite-fields-galois-fields">Finite Fields (Galois Fields)</h3> What is a field? A field is a mathematical structure that contains a bunch of elements, and those elements can interact with each other using addition and multiplication. For both these operations there’s an identity element and an inverse element. Addition and multiplication in a field must obey the usual properties that we’re used to: commutativity, associativity, and distributivity. A well-known example of a field is the field of fractions. Here is why fractions form a field: <ul> <li>the elements of the field are the fractions;</li> <li>addition is well-defined: if we add two fractions, we get a fraction out (example: $5/3 + 3/2 = 19/6$);</li> <li>multiplication is also well-defined: if we multiply two fractions, we get a fraction out (example: $1/2 \cdot 8/3 = 4/3$);</li> <li>the additive identity element is 0: if we add 0 to any fraction, we get the same fraction back;</li> <li>the additive inverse element is the negated fraction (example: $5/1$ is the additive inverse of $-5/1$ because $5/1 + (-5/1) = 0$);</li> <li>the multiplicative identity element is 1: multiplying any fraction by 1 yields the same fraction back;</li> <li>the multiplicative inverse element is what we get if we swap the numerator with the denominator (example: $3/2$ is the multiplicative inverse of $2/3$ because $3/2 \cdot 2/3 = 1$)—except for 0, which does not have a multiplicative inverse.</li> <li>and so on…</li> </ul> On top of addition, multiplication, and inverse elements, we can define derived operations like subtraction and division. Subtracting $a$ from $b$ is equivalent to adding $a$ to the additive inverse of $b$: $a - b = a + (-b)$. Similarly, division can be defined in terms of multiplication with multiplicative inverses ($a / b = a b^{-1}$). Fields are a generalization of structures where addition, multiplication, subtraction, and division behave according to the rules that we’re used to. Field elements do not necessarily need to be numbers. An example of something that is not a field is the integers. That’s because integers don’t have multiplicative inverses (for example, there’s no integer that multiplied by 5 makes the result equal to 1). However, there is a way to turn the integers into a field: if we take the integers and a prime number p, then we can construct the field of integers modulo p. When we work with the integers modulo a prime p, whenever we see p appear in any of our expressions, we can replace it with 0. In other words, in such a field, p and 0 are two different ways to write the same element–they are two different representations of the same element. Here is an example: in the field of integers modulo 7, the expression 5 + 3 equals 1, because: <ul> <li>5 + 3 evaluates to 8;</li> <li>8, by definition, is 7 + 1;</li> <li>if 7 and 0 are the same element, then 7 + 1 is equal to 0 + 1</li> <li>0 + 1 evaluates to 1</li> </ul> What we have just seen is that 8 is just a different representation of 1, just like 7 is a different representation of 0. Different symbols, same object. Just like, in programming languages, we can have multiple variables pointing to the same memory location: here the numbers are like variables, and what they point to is what really matters. In the field of integers modulo 7, the additive inverse for 5 is 2, because 5 + 2 = 7 = 0. If we manipulate the equation, we get that 5 = −2. In other words, 5 and −2 are two different representations for the same element, and similarly 2 and −5 are also two different representations of the same element. A similar story holds for multiplication: the multiplicative inverse for 5 is 3 because: 5 · 3 = 15 = 7 + 7 + 1 = 1, so we can write 5 = 3−1 as well as 3 = 5−1. What we have just seen is an example of a finite field. It’s different from a general field because it contains a finite number of elements (unlike fractions, which do not have a limit). In the case of the integers modulo 7, the number of elements is 7, and the list of elements is: {0, 1, 2, 3, 4, 5, 6}, or {−3, −2, −1, 0, 1, 2, 3}, or {0, 1, 2, 3, 2−1, 3−1, 6}, depending on what representation we like the most. <details> <summary>A few words about terminology, notation, and equivalences of finite fields</summary> There can be many ways to construct a finite field (or even a general field). I have given an example using numbers, but a field does not necessarily need to be formed from numbers. We can also use vectors, matrices, polynomials, and anything you would like. As long as addition, multiplication, identity elements, and inverse elements are well-defined, you can get a field. Using programming terms, you can think of a field as an interface or a trait that can have arbitrary implementations. An important result in algebra is that finite fields with the same number of elements are “unique up to isomorphism”. This means that if two finite fields have the same number of elements, then there is an equivalence relation between the two. The number of elements of a field is therefore enough to define a field. It’s not enough to tell us what the elements of the field look like, or how they can be represented, but it’s enough to know how it behaves. To denote a field with $n$ elements, there are two major notations: $GF(n)$ and $\mathbb{F}_{n}$. Another important result in algebra is that $n$ may be either a prime number, or a power of a prime. For example, we can have finite fields with 2 elements, or with 9 (= 32) elements, but we cannot have a field with 6 (= 2·3) elements. For this reason, you will often find finite fields denoted as $GF(p^k)$ or $\mathbb{F}_{p^k}$, where $p$ is a prime and $k$ is an integer greater than 0. The prime $p$ is called characteristic of the field, while $n = p^k$ is called order of the field. Some common fields also have their own notation: in particular, the field of integers modulo a prime $p$ is denoted as $Z/pZ$. This notation encodes the “building instructions” to construct the field, in fact: <ul> <li>$Z$ denotes the integers: $Z = \{\dots, -2, -1, 0, 1, 2, \dots\}$;</li> <li>$pZ$ denotes the integers multiplied by $p$: $pZ = \{\dots, -2p, -p, 0, p, 2p\}$ (example: $2Z = \{\dots, -4, -2, 0, 2, 4, \dots\}$);</li> <li>$A/B$ is a quotient. This is a way to define an equivalence relation between elements, and its meaning is: within $A/B$, all the elements of $B$ are equivalent to 0. In the case of $Z/pZ$, all the multiples of $p$ are equivalent to 0, which is indeed what happens with the integers modulo $p$. The way I described this equivalence relation earlier is by saying that multiples of $p$ are different representations for 0.</li> </ul> Note that the integers modulo a power of a prime ($Z/p^kZ$, with $k$ greater than 1) do not form a field. The problem is that elements in $Z/p^kZ$ sometimes do not have a multiplicative inverse. For example, in $Z/4Z$, the number 2 does not have a multiplicative inverse (there is no element that multiplied by 2 gives 1). A field $GF(p^k)$ with $k$ greater than 1 needs to be constructed in a different way. One such way is to use polynomials, as described in the next section. </details> <h4 id="polynomial-fields">Polynomial fields</h4> Let’s now move our attention from integers to polynomials, like this one: $$x^7 + 5x^3 - 9x^2 + 2x + 1$$ Polynomials are a sum of coefficients multiplied by a variable (usually denoted by the letter x) raised to an integral power. Let’s restrict our view to polynomials that have integer coefficients, like the one shown above. Something that is not a polynomial with integer coefficients is $1/2 x^2 + x$, because it has a fraction in it. Integers and polynomials with integer coefficients are somewhat similar to each other. They kinda behave the same in many aspects. One important property of integers is the <a href="https://en.wikipedia.org/wiki/Fundamental_theorem_of_arithmetic">unique factorization theorem</a>: if we have an integer, there’s a way to write it as a multiplication of some primary factors. For example, the integer 350 can be factored as the multiplication of 2, 5, 5, and 7. $$350 = 7 \cdot 5 \cdot 5 \cdot 2$$ This factorization is unique: we can change the order of the factors, but it’s not possible to obtain a different set of factors (there’s no way to make the number 3 appear in the factorization of 350, or to make the number 7 disappear). Polynomials with integer coefficients also have a unique factorization. In the case of integers, We call the unique factors “prime numbers”; in the case of polynomials we have “irreducible polynomials”. And just like we can have a field of integers modulo a prime, we can have a field of polynomials modulo an irreducible polynomial. <table> <thead> <tr> <th>Integers</th> <th>Polynomials (with integer coefficients)</th> </tr> </thead> <tbody> <tr> <td>Unique factorization: $42 = 7 \cdot 3 \cdot 2$</td> <td>Unique factorization: $x^3 - 1 = (x^2 + x + 1)(x - 1)$</td> </tr> <tr> <td>Prime numbers: 2, 3, 5, 7, 11, …</td> <td>Irreducible polynomials: $x + 1$, $x^2 - 2$, $x^2 + x + 1$, …</td> </tr> <tr> <td>Integers modulo a prime number</td> <td>Polynomials modulo an irreducible polynomial</td> </tr> </tbody> </table> Let’s take a look at how arithmetic in polynomial fields works. Let’s take, for example, the field of polynomials with integer coefficients modulo $x^3 + x + 1$, and try to compute the result of $(x^2 + 1)(x^2 + 2)$. If we expand the expression, we get: $$(x^2 + 1)(x^2 + 2) = x^4 + 3x^2 + 2$$ This expression can be reduced. Reducing a polynomial expression is the equivalent of what we were doing with the integers modulo a prime, when we were saying that 8 = 7 + 1 = 1 (mod 7). That “conversion” from 8 to 1 is the equivalent of the reduction that we’re talking about here. To reduce $x^4 + 3x^2 + 2$, first note that $x^4 = x \cdot x^3$. Also note that $x^3 = x^3 + x + 1 - x - 1$. Here we have just added and removed the term $x + 1$: the result hasn’t changed, but now the irreducible polynomial $x^3 + x + 1$ appears in the expression, and so we can substitute it with 0. Putting everything together, we get: $$\begin{align*} (x^2 + 1)(x^2 + 2) & = x^4 + 3x^2 + 2 \\ & = x \cdot x^3 + 3x^2 + 2 \\ & = x \cdot (x^3 + x + 1 - x - 1) + 3x^2 + 2 \\ & = x \cdot (0 - x - 1) + 3x^2 + 2 \\ & = -x^2 - x + 3x^2 + 2 \\ & = 2x^2 - x + 2 \end{align*}$$ It’s interesting to note that, if the polynomial field is formed by an irreducible polynomial with degree $n$, then all the polynomials in that field will all have degree less than $n$. That’s because if any $x^n$ (or higher) appears in a polynomial expression, then we can use the substitution trick I just showed to reduce its degree. <h4 id="binary-fields">Binary fields</h4> Let’s now look at polynomials where coefficients are from the field of integers modulo 2, meaning that they can be either 0 or 1. This is an example of such a polynomial: $$x^7 + x^4 + x^2 + 1$$ or, in a more explicit form, where we can clearly see all the coefficients: $$1 x^7 + 0 x^6 + 0 x^5 + 1 x^4 + 0 x^3 + 1 x^2 + 0 x^1 + 1 x^0$$ These are called binary polynomials. It’s interesting to note that if we ignore the variables and the powers, and keep only the coefficients, then what we get is a bit string: $$(1 0 0 1 0 1 0 1)$$ This suggests that there’s an interesting duality between binary polynomials and bit strings. This means, in particular, that binary polynomials can be represented in a very compact and natural way on computers. The duality between binary polynomials and bit strings also suggests that perhaps we can use bitwise operations to perform arithmetic on binary polynomials. And this turns out to be true, in fact: <ul> <li>binary polynomial addition can be computed using the XOR operator on the two corresponding bit strings;</li> <li>binary polynomial multiplication can be computed using XOR, AND and bit-shifting.</li> </ul> Computers are pretty fast at performing these bitwise operations, and this makes binary polynomials quite attractive for use in computer algorithms and cryptography. <details> <summary>Arithmetic with binary polynomials</summary> The arithmetic of such polynomials is quite interesting: in fact, because $1 + 1 = 0$ (modulo 2), then also $x^k + x^k = 0$, in fact: $$1 \cdot x^k + 1 \cdot x^k = (1 + 1) x^k = 0 \cdot x^k = 0$$ It’s easy to see that addition modulo 2 is equivalent to the XOR binary operator. And addition of two binary polynomials is equivalent to the bitwise XOR of their corresponding bit strings: $$\begin{array}{ccccc} (x^3 + x^2 + 1) & + & (x^2 + x) & = & x^3 + x + 1 \\ \updownarrow & & \updownarrow & & \updownarrow \\ (1101) & \oplus & (0110) & = & (1011) \end{array}$$ Multiplication of binary polynomials can also be implemented as a bitwise operation on bit strings. First, note that multiplying a polynomial by a monomial is equivalent to bit-shifting: $$\begin{array}{ccccc} (x^3 + x + 1) & \cdot & x^2 & = & x^5 + x^3 + x^2 \\ \updownarrow & & \updownarrow & & \updownarrow \\ (1011) & \ll & 2 & = & (101100) \end{array}$$ Then note that multiplication of two polynomials can be expressed as the sum of multiplications by monomials: $$(x^3 + 1)(x^2 + x + 1) = (x^3 + 1) \cdot x^2 + (x^3 + 1) \cdot x^1 + (x^3 + 1) \cdot x^0$$ Putting everything together, we have multiplications by monomials (equivalent to bit-shifts) and sums (equivalent to bitwise XOR). This suggests that multiplication can be implemented on top of bitwise XOR and bit-shifting. Here is some Python code to implement binary polynomial multiplication, where each polynomial is represented compactly as an <code>int</code>: <div class="highlight"><pre><code>def multiply(a, b): """ Compute a*b, where a and b are two integers representing binary polynomials. a and b are expected to have their most significant bit set to the monomial with the highest power. For example, the polynomial x^8 is represented as the integer 0b10000. """ assert a >= 0 assert b >= 0 result = 0 while b: result ^= a * (b & 1) a <<= 1 b >>= 1 return result </code></pre></div> Other than XOR and bit-shifting, this code also uses AND to “query” whether a certain monomial is present or not. Here is an example of how to use the code: <div class="highlight"><pre><code>a = 0b0101_0111 # x^6 + x^4 + x^2 + x + 1 b = 0b0001_1010 # x^4 + x^3 + x c = multiply(a, b) assert c == 0b0111_0110_0110 # x^10 + x^9 + x^8 + x^6 + x^5 + x^2 + x </code></pre></div> </details> Now that we have introduced binary polynomials, we can of course form binary polynomials modulo a binary irreducible polynomial. These form a finite field, which is more concisely called: binary field. Note that in a binary field where the modulo is an irreducible polynomial of degree $n$, all polynomials in the field can be represented as $n$-bit strings, and all $n$-bit strings have a corresponding binary polynomial in the field. <details> <summary>Arithmetic in binary fields</summary> If we have three integers $a$, $b$, and $p$, we can compute $(a + b) \bmod{p}$ or $a \cdot b \bmod{p}$ by performing the binary operation (addition or multiplication) and then taking the remainder of the division by $p$. This is a method that returns the results of addition or multiplication using a representation with the lowest number of digits possible. What if instead of having 3 integers we have three binary polynomials $A$, $B$, and $P$ and we want to compute $(A + B) \bmod{P}$ or $A \cdot B \bmod{P}$? It turns out that these operations can be implemented with code that is even easier than the integer counterpart: no division needs to be involved! Let’s start with addition: we have already seen that addition with binary polynomials can be implemented with a simple XOR operation. This means that if the degree of $A$ and $B$ is lower than the degree of $P$, then the result of $A + B$ is also going to have degree less than $P$, hence no reduction is needed. We can use the result as-is, without any transformation: adding two binary field elements can be implemented with a single XOR operation. With multiplication the story is different: the product $A \cdot B$ may have degree equal to or higher than $P$. For example, if $A = B = x$ and $P = x^2 + 1$, the product $A \cdot B$ is equal to $x^2$, which has the same degree as $P$. We need to find a way to efficiently reduce the higher-degree terms of this product. To see one way to do that, note that we can write $P$ like this: $$P = x^n + Q$$ where $n$ is the degree of $P$ (the maximum power of $P$) and $Q$ is another binary polynomial, with degree strictly lower than $n$. Rearranging the equation, we get: $$x^n = P + Q$$ Note that subtraction and addition are the same operations in a binary field. Because $P$ equals 0, we can write: $$x^n = Q$$ This equivalence gives us a way to eliminate higher-level terms that appear during multiplication: whenever we see an $x^n$ appearing in the result, we can remove that term and add $Q$ instead. One way to do that, using binary strings, is to discard the highest bit (the one corresponding to $x^n$) and XOR with the binary string corresponding to $Q$. Another way to do it is to just add $P$ (XOR by the binary string corresponding to $P$). This is equivalent to adding 0, results in the more compact representation that we’re interested in. We could use similar tricks to eliminate terms like $x^{n+1}$, but these tricks are not necessary if we eliminate $x^n$ terms as soon as they appear in an iterative way. Here is some Python code for multiplication in binary fields that uses the “add $P$” trick just described: <div class="highlight"><pre><code>def multiply(a, b, p): """ Compute a*b modulo p, where a, b and c are three integers representing binary polynomials. a, b and p are expected to have their most significant bit set to the highest power monomial. For example, the polynomial x^8 is represented as 0b10000. """ bit_length = p.bit_length() - 1 assert a >= 0 and a < (1 << bit_length) assert b >= 0 and b < (1 << bit_length) result = 0 for i in range(bit_length): result ^= a * (b & 1) a <<= 1 a ^= p * ((a >> bit_length) & 1) b >>= 1 return result </code></pre></div> This code is essentially the same as the binary polynomial multiplication code we had before, except for this line in the <code>for</code> loop: <div class="highlight"><pre><code>a ^= p * ((a >> bit_length) & 1) </code></pre></div> This line is what “adds $P$” whenever adding the shifted $A$ would result in a $x^n$ term to appear. Again, we achieved implementing multiplication using only XOR, AND and bit-shifting. Note that the binary polynomial $P$ here does not necessarily need to be an irreducible polynomial for this algorithm to work. However, the resulting algebraic structure won’t be a field unless $P$ is irreducible. A similar story holds for integers: we can have integers modulo a non-prime number, but that’s not a field. </details> <h3 id="the-ghash-keyed-hash-function">The GHASH keyed hash function</h3> GCM uses a binary field. The irreducible binary polynomial that defines the binary field used by GCM is: $$P = x^{128} + x^7 + x^2 + x + 1$$ We will call this field the GCM field. Note that this polynomial has degree 128, hence the GCM field elements can be represented as 128-bit strings, and each 128-bit string has a corresponding element in the GCM field. The keyed hash function used by GCM is called GHASH and takes as input a 128-bit key. We will call this key $H$. This key is interpreted as an element of the GCM field. The message to authenticate is split into blocks of 128 bits each: $M_1$, $M_2$, $M_3$, … $M_n$. If the length of the message is not a multiple of 128 bits, then the last block is padded with zeros. Each block of message is also interpreted as an element of the GCM field. Here is how the authentication tag is computed from $H$ and the padded message blocks $M_1$, …, $M_n$: <ul> <li>The initial state (a GCM field element) is initialized to 0: $A_0 = 0$.</li> <li>For every block of message $M_i$, the next state $A_i$ is computed as $A_i = (A_{i-1} + M_i) \cdot H \bmod{P}$.</li> <li>The final state $A_n$ is returned as a 128-bit string.</li> </ul> What this function is doing is computing the following polynomial in $H$: $$\begin{align*} Tag & = (((M_1 \cdot H + M_2) \cdot H + \cdots M_n) \cdot H) \bmod{P} \\ & = (M_1 H^n + M_2 H^{n-1} + \cdots M_n H) \bmod{P} \end{align*}$$ This construction is somewhat similar to the one from Poly1305, although there are important differences: <ul> <li>In Poly1305, the elements of the tag polynomial are integers modulo a prime, in GHASH they are elements of a binary field.</li> <li>GHASH does not perform any step to encode the length of the message, hence the tag for an empty message will be the same as the tag for a sequence of zero blocks. We will see later that GCM fixes this problem by appending the length of the message to the end of the input passed to GHASH.</li> <li>Most importantly, the final $Tag$ polynomial is a polynomial in one unknown, and as such $H$ may be easily recoverable using algebraic methods. For this reason, GHASH is not suitable as a secure one-time authenticator. We will see that GCM fixes this problem by encrypting the output of GHASH.</li> </ul> <h3 id="use-of-gcm-with-aes-aes-gcm">Use of GCM with AES (AES-GCM)</h3> GCM is the combination of a block cipher, Counter Mode (CTR), and the GHASH function that we have just seen. The block cipher is often AES. When we combine AES with GCM, the what we get is AES-GCM, which is described below. However the block cipher does not necessarily need to be AES: what is important is that the block size of the cipher is 128 bits, and that’s because GHASH only works on 128-bit blocks. The inputs to the AES-GCM encryption function are: <ul> <li>a secret key (the length of the key depends on the variant of AES used: if AES-128, this will be 128 bits);</li> <li>a 96-bit nonce;</li> <li>a variable-length plaintext message.</li> </ul> The outputs of the AES-GCM encryption function are: <ul> <li>a variable-length ciphertext (same length as the input plaintext);</li> <li>a 128-bit authentication tag.</li> </ul> The AES-GCM decryption function will accept the same secret key, nonce, ciphertext, and authentication tag as the input, and produce either the plaintext or an error as the output. The error is returned in case the authentication fails. <figure> <img src="https://andrea.corbellini.blog/images/aes-gcm-encryption.svg" alt="Diagram of data flow during encryption with AES-GCM"> <figcaption>Data flow during an AES-GCM encryption. This shows the inputs in blue, the outputs in green, and the intermediate objects in red.</figcaption> </figure> AES-GCM works in the following way: <ol> <li> The GHASH subkey $H$ is generated by encrypting a zero-block: $H = \operatorname{Encrypt}(key, \underbrace{000\dots0}_\text{128 bits})$. </li> <li> The block cipher AES is initialized in Counter Mode (AES-CTR) with the key, the nonce, and a 32-bit, big-endian counter starting at 2. </li> <li> The plaintext is encrypted using the instance of AES-CTR just created. </li> <li> The GHASH function is run with the following inputs: <ul> <li>the subkey $H$, computed in step 1;</li> <li>the ciphertext padded with zeros to make its length a multiple of 16 bytes (128 bits), concatenated to the length (in bits) of the ciphertext represented as a 128-bit big-endian integer.</li> </ul> The result is a 128-bit block $S = \operatorname{GHASH}(H, ciphertext || padding || length)$. </li> <li> The AES-CTR counter is set to 1. </li> <li> The block $S$ is then encrypted using AES-CTR. The result of the encryption is the authentication tag. Note that, because $S$ matches the block size of the cipher, this encryption won’t cause the counter value 2 to be reused. </li> <li> The ciphertext and authentication tag are returned. </li> </ol> Here is how AES-GCM and GHASH can be implemented in Python, using the AES implementation from <a href="https://pypi.org/project/pycryptodome/">pycryptodome</a> (usual disclaimer: this code is for educational purposes, and it’s not necessarily secure or optimized for performance): <div class="highlight"><pre><code>from Crypto.Cipher import AES def multiply(a: int, b: int) -> int: """ Compute a*b in the GCM field, where a and b are two integers representing elements of the GCM field. a and b are expected to have their least significant bit set to the highest power monomial. For example, the polynomial x^125 is represented as 0b100. """ bit_length = 128 q = 0xe1000000000000000000000000000000 assert a >= 0 and a < (1 << bit_length) assert b >= 0 and b < (1 << bit_length) result = 0 for i in range(bit_length): result ^= a * ((b >> 127) & 1) a = (a >> 1) ^ (q * (a & 1)) b <<= 1 return result def pad_block(data: bytes) -> bytes: """ Pad data with zero bytes so that the resulting length is a multiple of 16 bytes (128 bits). """ if len(data) % 16 != 0: data += b'\0' * (16 - len(data) % 16) return data def iter_blocks_padded(data: bytes): """ Split the given data into blocks of 16 bytes (128 bits) each, padding the last block with zeros if necessary. """ start = 0 while start < len(data): yield pad_block(data[start:start+16]) start += 16 def ghash(subkey: bytes, message: bytes) -> bytes: subkey = int.from_bytes(subkey, 'big') assert subkey < (1 << 128) state = 0 for block in iter_blocks_padded(message): block = int.from_bytes(block, 'big') state = multiply(state ^ block, subkey) return state.to_bytes(16, 'big') def aes_gcm_encrypt(key: bytes, nonce: bytes, message: bytes): assert len(key) in (16, 24, 32) assert len(nonce) == 12 # Initialize a raw AES instance and encrypt a 16-byte block of all zeros to # derive the GHASH subkey H cipher = AES.new(mode=AES.MODE_ECB, key=key) h = cipher.encrypt(b'\0' * 16) # Encrypt the message with AES in CTR mode, with the counter composed by # the concatenation of the 12 byte (96 bits) nonce and a 4 byte (32 bits) # integer, starting from 2 cipher = AES.new(mode=AES.MODE_CTR, key=key, nonce=nonce, initial_value=2) ciphertext = cipher.encrypt(message) # Compute the GHASH of the ciphertext plus the ciphertext length in bits s = ghash(h, pad_block(ciphertext) + (len(ciphertext) * 8).to_bytes(16, 'big')) # Encrypt the GHASH value using AES in CTR mode, with the counter composed # by the concatenation of the 12 byte (96 bits) nonce and a 4 byte (32 # bits) integer set at 1. The GHASH value fits in one block, so the counter # won't be increased during this round of encryption cipher = AES.new(mode=AES.MODE_CTR, key=key, nonce=nonce, initial_value=1) tag = cipher.encrypt(s) return (ciphertext, tag) def aes_gcm_decrypt(key: bytes, nonce: bytes, ciphertext: bytes, tag: bytes): assert len(key) in (16, 24, 32) assert len(nonce) == 12 assert len(tag) == 16 # Compute the GHASH subkey, the GHASH value, and the authentication tag, in # the same exact way as it was done during encryption cipher = AES.new(mode=AES.MODE_ECB, key=key) h = cipher.encrypt(b'\0' * 16) s = ghash(h, pad_block(ciphertext) + (len(ciphertext) * 8).to_bytes(16, 'big')) cipher = AES.new(mode=AES.MODE_CTR, key=key, nonce=nonce, initial_value=1) expected_tag = cipher.encrypt(s) # Compare the input tag with the generated tag. If they're different, the # plaintext must not be returned to the caller if tag != expected_tag: raise ValueError('authentication failed') # The two tags match; decrypt the plaintext and return it to the caller. # Note that, because AES-CTR is a symmetric cipher, there is no difference # between the encrypt and decrypt method: here we are reusing the same # exact code used during decryption cipher = AES.new(mode=AES.MODE_CTR, key=key, nonce=nonce, initial_value=2) message = cipher.encrypt(ciphertext) return message </code></pre></div> And here is how the code can be used: <div class="highlight"><pre><code>key = bytes.fromhex('0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef') nonce = bytes.fromhex('0123456789abcdef01234567') message = b'I went to the zoo yesterday but not today' ciphertext, tag = aes_gcm_encrypt(key, nonce, message) print(f'ciphertext: {ciphertext.hex()}') print(f' tag: {tag.hex()}') decrypted_message = aes_gcm_decrypt(key, nonce, ciphertext, tag) print(f' plaintext: {decrypted_message}') assert message == decrypted_message </code></pre></div> This snippet produces the following output: <div class="highlight"><pre><code>ciphertext: e0c32db2962f9b729c69028d9a1fdfb2c93839fc1188f314c58ee97fd6a242404953bb208df609a33c tag: 9fa6fe2f77a0c98282868924ace0e4ec plaintext: b'I went to the zoo yesterday but not today' </code></pre></div> This is the same output we would obtain by using the AES-GCM implementation from pycryptodome directly: <div class="highlight"><pre><code>from Crypto.Cipher import AES cipher = AES.new(mode=AES.MODE_GCM, key=key, nonce=nonce) ciphertext, tag = cipher.encrypt_and_digest(message) print(f'ciphertext: {ciphertext.hex()}') print(f' tag: {tag.hex()}') </code></pre></div> Nonce reuse is catastrophic for AES-GCM in two ways: <ul> <li> Because the ciphertext produced by AES-GCM is just a variant of AES-CTR, nonce reuse with GCM can have the same consequences as nonce reuse with AES-CTR, or any other stream cipher: if someone is able to guess the plaintext, they can recover the random stream, and use that to decrypt other messages (or portions of them). </li> <li> If the same nonce is used twice or more, the GHASH subkey $H$ will always be the same. Even if the output of GHASH is encrypted in step 7, we can use the XOR of two authentication tags to “cancel” the encryption and obtain a polynomial in $H$. From there, we can use algebraic methods to recover $H$. This gives us the ability to forge new, valid authentication tags. </li> </ul> It’s worth mentioning that there’s a variant of AES-GCM, called AES-GCM-SIV, (Synthetic Initialization Vector) specified in <a href="https://www.rfc-editor.org/rfc/rfc8452">RFC 8452</a>. This differs from AES-GCM in that it uses a little-endian version of GHASH called POLYVAL (which is faster on modern CPUs), and in that it allows nonce reuse without the two catastrophic consequences that I mentioned above. (Nonce reuse with AES-GCM-SIV however still presents a problem, just not as serious as the two ones above: specifically, it breaks <a href="https://en.wikipedia.org/wiki/Ciphertext_indistinguishability">ciphertext indistinguishability</a>.) <h1 id="authenticated-encryption-with-associated-data-aead">Authenticated Encryption with Associated Data (AEAD)</h1> The way I have described authenticated encryption, and in particular the constructions ChaCha20-Poly1305 and AES-GCM, is accurate, but incomplete. What I have told you is that when you use an authenticated encryption cipher, the ciphertext is checked for integrity and authenticity. But we can use the same technique to authenticate anything, not just ciphertexts: we can, for example, authenticate some plaintext data, or authenticate a piece of plaintext data and a piece of ciphertext altogether. When we use a method to authenticate a plaintext message only, what we get is a Message Authentication Code (MAC). We don’t use the word “encryption” in this context, because the confidentiality of the message is not ensured (only its authenticity). When we use a method to authenticate both a ciphertext and a plaintext message, what we get is Authenticated Encryption with Associated Data (AEAD). In this construction, there are two messages involved: one to be encrypted (resulting in a ciphertext), and one to be kept in plaintext. The plaintext message is called “associated data” (AD) or “additional authenticated data” (AAD). Both the ciphertext and the associated data are authenticated at encryption time, so their integrity and authenticity will be enforced. The inputs to the encryption function of an AEAD cipher are, generally speaking: <ul> <li>a key;</li> <li>a nonce;</li> <li>the additional data;</li> <li>the message to encrypt.</li> </ul> The outputs of the encryption are: <ul> <li>the ciphertext;</li> <li>the authentication tag.</li> </ul> Note that there’s only one authentication tag that covers both the additional data and the ciphertext. The inputs to the decryption function are: <ul> <li>the key used for encryption;</li> <li>the nonce used for encryption;</li> <li>the additional data used for encryption;</li> <li>the ciphertext.</li> </ul> And the output of the decryption is either an error or the decrypted message. It’s important to note that the associated data must be both at encryption time and decryption time. Changing a single bit of it will make the entire decryption operation fail. Both ChaCha20-Poly1305 and AES-GCM (and their variants, XChaCha20-Poly1305 and AES-GCM-SIV) are AEAD ciphers. Here’s how they implement AEAD: <ul> <li>When the Poly1305 or GHASH authenticator is first initialized, they are fed the additional data, padded with zeros to make its size a multiple of 16 bytes (128 bits).</li> <li>Then the padded ciphertext is fed into the authenticator.</li> <li>The length of the additional data and the length of the ciphertext are represented as two 64-bit integers, concatenated, and fed into the authenticator.</li> </ul> <figure> <img src="https://andrea.corbellini.blog/images/chacha20-poly1305-aead-encryption.svg" alt="Diagram of data flow during encryption with ChaCha20-Poly1305, including the Associated Data (AE)"> <figcaption>Updated data flow during a ChaCha20-Poly1305 encryption which shows where the Associated Data (AE) is placed.</figcaption> </figure> <figure> <img src="https://andrea.corbellini.blog/images/aes-gcm-aead-encryption.svg" alt="Diagram of data flow during encryption with AES-GCM, including the Associated Data (AE)"> <figcaption>Updated data flow during an AES-GCM encryption which shows where the Associated Data (AE) is placed.</figcaption> </figure> If the additional data is empty, then what you get are exactly the constructions that I described earlier in this article. Authenticated Encryption with Associated Data is useful in situations where you want to encode some metadata along with your encrypted data. For example: an identifier for the resource that is encrypted, or the type of data encrypted (text, image, video, …), or some information that indicates what key and algorithm was used to encrypt the resource, or maybe the expiration of the data. The associated data is in plaintext so systems that do not have access to the secret key can gather some properties about the encrypted resource. It must however be understood that the associated data cannot be trusted until it’s verified using the secret key. Systems that analyze the associated data must be designed in such a way that, if the associated data is tampered, nothing bad will happen, and such tampering attempt will be detected sooner or later. <h1 id="a-word-of-caution">A word of caution</h1> Something very important to understand is that when using authenticated encryption ciphers like ChaCha20-Poly1305 or AES-GCM, decryption can in theory succeed even if the verification of the authentication tag fails. For example, we can decrypt a ciphertext encrypted with ChaCha20-Poly1305 by using ChaCha20 and ignoring the authentication tag. Similarly, we can decrypt a ciphertext encrypted with AES-GCM by using AES-CTR and, again, ignoring the authentication tag. This possibility opens the doors to all the nasty scenarios that we have seen at the beginning of this article, removing all the benefits of authenticated encryption. Perhaps the most important thing to remember when using authenticated encryption is: never use decrypted data until you have verified its authenticity. Why am I emphasizing this? Because some AE or AEAD implementations do return plaintext bytes before verifying their authenticity. The code samples that I have provided do the following: they first calculate the authentication tag, compare it to the input tag, and only if the comparison succeeds they perform the decryption. This is a simple approach, but it may be expensive when encrypting large amounts of data (for example: several gigabytes). The reason why this approach is expensive is that, if the ciphertext is too large, it may not fit all in memory, and the ciphertext would have to be read from the storage device twice: once for calculating the tag, and once for decrypting the ciphertext. Also, chances are that by the time the application has computed the tag, the underlying ciphertext may have changed without detection. What some authenticated encryption implementations do when dealing with large amounts of data is that they calculate the tag and perform the decryption in parallel. They read the ciphertext chunk-by-chunk, and pass each chunk to both the authenticator and the decryption function, returning a chunk of decrypted bytes to the caller at each iteration. Only at the end, when the full ciphertext has been read, the authenticity is checked, and the application may return an error only at that point. With such implementations, it is imperative that the exit status of the application is checked before using any of the decrypted bytes. An implementation that works like that (returning decrypted bytes before authentication is complete) is GPG. Here is an example of the output that GPG produces when decrypting a tampered message: <div class="highlight"><pre><code>gpg: AES256.CFB encrypted data gpg: encrypted with 1 passphrase This is a very long message. gpg: WARNING: encrypted message has been manipulated! </code></pre></div> The decrypted message (“This is a very long message”) got printed, together with a warning, and the exit status is 2, indicating that an error occurred. It is important in this case that the decrypted message is not used in any way. Other implementations avoid this problem by simply not encrypting large amounts of data. If given a large file to encrypt, the file is first split into multiple chunks of a few KiB, then each chunk is encrypted independently, with its own nonce and authentication tag. Because each chunk is small, authentication and decryption can happen in memory, one before the other. If a chunk was tampered, decryption would stop, returning truncated output, but never tampered output. It’s still important to check the exit status of such an implementation, but the consequences are less catastrophic than before. The drawback of this approach is that the total size of the ciphertext increases, because each chunk requires a nonce, an authentication tag, and some information about the position of the chunk (to prevent the chunks from being reordered). Storing the nonces or the positions can be avoided by using an algorithm to generate them on the fly, but storing the tag cannot be avoided. The method of splitting that I have just described (of splitting long messages into chunks that are individually encrypted and authenticated) is used for example in <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a>, as well as the command line tool <a href="https://github.com/FiloSottile/age">AGE</a>. <h1 id="summary-and-final-considerations">Summary and final considerations</h1> At the beginning of this article we have seen some risks of using bare encryption ciphers: one of them in particular was malleability, that is: the property that ciphertexts may be modified without detection. This problem was addressed by using Authenticated Encryption (AE) or Authenticated Encryption with Associated Data (AEAD), which are methods to provide integrity and authenticity in addition to confidentiality when encrypting data. We have seen the details of the two most popular authenticated encryption ciphers and briefly mentioned some of their variants. Their features are summarized here: <table> <thead> <tr> <th>Cipher</th> <th>Cipher Type</th> <th>Key Size</th> <th>Nonce Size</th> <th>Nonce Reuse</th> <th>Tag Size</th> </tr> </thead> <tbody> <tr> <td>ChaCha20-Poly1305</td> <td>Stream, AEAD</td> <td>256 bits</td> <td>96 bits</td> <td>Catastrophic</td> <td>128 bits</td> </tr> <tr> <td>XChaCha20-Poly1305</td> <td>Stream, AEAD</td> <td>256 bits</td> <td>192 bits</td> <td>Catastrophic</td> <td>128 bits</td> </tr> <tr> <td>AES-GCM</td> <td>Stream, AEAD</td> <td>128, 192, 256 bits</td> <td>96 bits</td> <td>Catastrophic</td> <td>128 bits</td> </tr> <tr> <td>AES-GCM-SIV</td> <td>Stream, AEAD</td> <td>128 or 256 bits</td> <td>96 bits</td> <td>Reduced risk</td> <td>128 bits</td> </tr> </tbody> </table> Authenticated encryption is used in most of our modern protocols, including TLS, S/MIME, PGP/GPG, and many more. Failure to implement authenticated encryption correctly has lead to some serious issues in the past. Whenever you’re using encryption, ask yourself: how is integrity and authentication verified? And remember: it’s essential to verify the authenticity of data before using it. I hope you enjoyed this article! As usual, if you have any suggestions or spotted some mistakes, let me know in the comments or by contacting me!andreacorbelliniThu, 09 Mar 2023 18:35:00 +0000tag:andrea.corbellini.blog,2023-03-09:/2023/03/09/authenticated-encryption/cryptographyWhat time is it? A simple question with a complex answer. How computers synchronize timehttps://andrea.corbellini.blog/2023/01/23/what-time-is-it/Ever wondered how your computer or your phone displays the current date and time accurately? What keeps all the devices in the world (and in space) in agreement on what time it is? What makes applications that require precise timing possible? In this article, I will explain some of the challenges with time synchronization and explore two of the most popular protocols that devices use to keep their time in sync: the Network Time Protocol (NTP) and the Precision Time Protocol (PTP). <h1 id="what-is-time">What is time?</h1> It wouldn’t be a good article about time synchronization without spending a few words about time. We all have an intuitive concept of time since childhood, but stating precisely what ‘time’ is can be quite a challenge. I’m going to give you my idea of it. Here is a simple definition to start with: time is how we measure changes. If the objects in the universe didn’t change and appeared to be fixed, without ever moving or mutating, I think we could all agree that time wouldn’t be flowing. Here by ‘change’ I mean any kind of change: from objects falling or changing shape, to light diffusing through space, or our memories building up in our mind. This definition may be a starting point but does not capture all we know about time. Something that it does not capture is our concept of past, present, and future. From our day-to-day experience, we know in fact that an apple would fall off the tree due to gravity, under the normal flow of time. If we observed an apple rising from the ground, attaching itself to the tree (without the action of external forces), we could perhaps agree that what we’re observing is time flowing backward. And yet, both the apple falling off the tree and the apple rising from the ground are two valid changes from an initial state. This is where causality comes into place: time flows in such a way that the cause must precede the effect. We can now refine our definition of time as an ordered sequence of changes, where each change is linked to the previous one by causality. <h1 id="how-do-we-measure-time">How do we measure time?</h1> Now we have a more precise definition of time, but we still don’t have enough tools to define what is a second, an hour, or a day. This is where things get more complicated. If we look at the definition of ‘second’ from the international standard, we can see that it is currently defined from the emission frequency of caesium-133 (133Cs) atoms. If you irradiate caesium-133 atoms with some light having sufficient energy, the atoms will absorb the light, get excited, and release the energy back in the form of light at a specific frequency. That frequency of emission is defined as 9192631770 Hz, and the second is defined as the inverse of that frequency. This definition is known as the <a href="https://en.wikipedia.org/wiki/Caesium_standard">caesium standard</a>. Here’s a problem to think about: how do we know that a caesium-133 atom, after getting excited, really emits light at a fixed frequency? The definition of second is implying that the frequency is constant and the same all over the world, but how do we know it’s really the case? This assumption is supported by quantum physics, according to which atoms can only transition between discrete (quantified) energy states. When an atom gets excited, it transitions from an energy state $E_1$ to an energy state $E_2$. Atoms like to be in the lowest energy state, so the atom will not stay in the state $E_2$ for long, and will want to go back to $E_1$. When doing that, it will release an amount of energy of exactly $E_2 - E_1$ in the form of a photon. According to the <a href="https://en.wikipedia.org/wiki/Planck_relation">Planck formula</a>, the photon will have frequency $f = (E_2 - E_1) / h$ where $h$ is the Planck constant. Because the energy levels are fixed, the resulting emission frequency is fixed as well. By the way, this process of absorption and emission of photons is the same process that causes fluorescence. <figure> <img src="https://andrea.corbellini.blog/images/atomic-clock-emission.svg" alt="Visualization of the absorption and emission process for an atom transitioning between two energy states"> <figcaption>Visualization of the absorption and emission process for an atom transitioning between a ground state $E_1$ to an excited state $E_2$.</figcaption> </figure> Assuming that caesium-133 atoms emit light at a single, fixed frequency, we can now build extremely accurate caesium atomic clocks and measure spans of time with them. Existing caesium atomic clocks are estimated to be so precise that they may lose one second every 100 million years. The same approach can be applied to other substances as well: atomic clocks have been constructed using rubidium (Rb), strontium (Sr), hydrogen (H), krypton (Kr), ammonia (NH3), ytterbium (Yb), each having its own emission frequency, and their own accuracy. The <a href="https://www.theverge.com/2015/4/22/8466681/most-accurate-atomic-clock-optical-lattice-strontium">most accurate clock ever built</a> is a strontium clock which may lose one second every 15 billion years. <h1 id="time-dilation">Time dilation</h1> If we have two atomic clocks and we let them run for a while, will they show the same time? This might sound like a rhetorical question: we just established that the frequencies of emission of atoms are fixed, so why would two identical atomic clocks ever get out of sync? Well, as a matter of fact, two identical atomic clocks may get out of sync, and this problem is not due to the clocks, but with time itself: it appears that time does not always flow in the same way everywhere. Many experiments have shown this effect on our planet, the most famous one probably being the <a href="https://en.wikipedia.org/wiki/Hafele%E2%80%93Keating_experiment">Hafele-Keating experiment</a>. In this experiment, a set of caesium clocks was placed on an airplane flying around the world west-to-east, another set was placed on an airplane flying east-to-west, and another set remained on ground. The 3 sets of clocks, which were initially in sync before the planes took off, were showing different times once reunited after the trip. This experiment and similar ones have been repeated and refined multiple times, and they all showed consistent results. These effects were due to <a href="https://en.wikipedia.org/wiki/Time_dilation">time dilation</a>, and the results were consistent with the predictions of <a href="https://en.wikipedia.org/wiki/Special_relativity">special relativity</a> and <a href="https://en.wikipedia.org/wiki/General_relativity">general relativity</a>. <h2 id="time-dilation-due-to-special-relativity">Time dilation due to special relativity</h2> Special relativity predicts that if two clocks are moving with two different velocities, they are going to measure different spans of time. Special relativity is based on two principles: <ul> <li>the speed of light is constant;</li> <li>there are no privileged reference frames.</li> </ul> To understand how these principles affect the flow of time, it’s best to look at an example: imagine that a passenger is sitting on a train with a laser and a mirror in front of them. Another person is standing on the ground next to the railroad and observing the train passing. The passenger points the laser perpendicular to the mirror and turns it on. What the passenger will observe is the beam of light from the laser to hit the mirror and come back in a straight line: <figure> <img src="https://andrea.corbellini.blog/images/special-relativity-train-reference-frame.webp" alt="Beam of light in the train reference frame"> <figcaption>Portion of the beam of light in the train reference frame, emitted from the laser (bottom) and bouncing from the mirror (top). Note how it follows a vertical path.</figcaption> </figure> From the observer perspective, however, things are quite different. Because the train is moving relative to the observer, the beam looks like it’s taking a different, slightly longer path: <figure> <img src="https://andrea.corbellini.blog/images/special-relativity-observer-reference-frame.webp" alt="Beam of light in the observer reference frame"> <figcaption>The same portion of light beam as before, but this time in the observer reference frame. Note how it follows a diagonal path, longer than the vertical path in the train reference frame.</figcaption> </figure> If both the passenger and the observer measure how long it took for the light beam to hit back at the source, and if the principles of special relativity hold, then the two persons will record different measurements. If the speed of light is constant, and there is no privileged reference frame, then the speed of light $c$ must be the same in both reference frames. From the passenger’s perspective, the beam has traveled a distance of $2 L$, taking a time $2 L / c$. From the observer’s perspective, the beam has traveled a longer distance $2 M$, with $M > L$, taking a longer time $2 M / c$. <figure> <img src="https://andrea.corbellini.blog/images/special-relativity-reference-frame-comparison.webp" alt="Beam of light in the observer reference frame"> <figcaption>Comparison of the light beams as seen from the two reference frames. In the train reference frame, the light beam is a vertical line of length $L$ (therefore traveling a path of length $2 L$ after bouncing from the mirror). In the observer reference frame, the light beam is distorted due to the velocity of the train. If the train moves at speed $v$, then the light beam travels a total length of $2 M = 2 L c / \sqrt{c^2 - v^2}$.</figcaption> </figure> How can we reconcile these counterintuitive measurements? Special relativity does it is by stating that time flows differently in the two reference frames. Time runs “slower” inside the train and runs “faster” for the observer. One consequence of that is that the passenger ages less than the observer. Time dilation due to special relativity is not easily detectable in our day-to-day life, but it can still cause problems with high-precision clocks. This time dilation may in fact cause clock drifts in the order of hundreds of nanoseconds per day. <h2 id="time-dilation-due-to-general-relativity">Time dilation due to general relativity</h2> Experimental data shows that clocks in a gravitational field do not follow (solely) the rules of special relativity. This does not mean that special relativity is wrong, but it’s a sign that it is incomplete. This is where general relativity comes into play. In general relativity, gravity is not seen as a force, like in classical physics, but rather as a deformation of <a href="https://en.wikipedia.org/wiki/Spacetime">spacetime</a>. All objects that have mass bend spacetime, and the path of objects traveling through spacetime is affected by its curvature. An apple falling from a tree is not going towards the ground because there’s a force “pushing” it down, but rather because that’s the shortest <a href="https://en.wikipedia.org/wiki/World_line">path in spacetime</a> (a straight line in bent spacetime). <figure> <img src="https://andrea.corbellini.blog/images/apple-falling-classical-physics.webp" alt="Apple falling according to classical physics, following a parabolic motion"> <figcaption>Apple falling according to classical physics, following a parabolic motion.</figcaption> </figure> <figure> <img src="https://andrea.corbellini.blog/images/apple-falling-general-relativity.webp" alt="Apple falling according to general relativity, following a straight path in distorted spacetime"> <figcaption>Apple falling according to general relativity, following a straight path in distorted spacetime.</figcaption> </figure> The larger the mass of objects, the larger the curvature of spacetime they produce. Time flows “slower” near large masses, and “faster” away from it. Interesting facts: people on a mountain age faster than people on the sea level, and it has been <a href="https://phys.org/news/2016-05-earth-core-younger-thought.html">calculated</a> that the core of the Earth is 2.5 years younger than the crust. The time dilation caused by gravity on the surface of the Earth may amount to clock drifts in the order of hundreds of nanoseconds per day, just like special relativity. <h1 id="can-we-actually-synchronize-clocks">Can we actually synchronize clocks?</h1> Given what we have seen about time dilation, and that we may experience time differently, does it even make sense to talk about time synchronization? Can we agree on time if time flows differently for us? The short answer is yes: the trick is to restrict our view to a closed system, like the surface of our planet. If we place some clocks scattered across the system, they will almost certainly experience different flows of time, due to different velocities, different altitudes, and other time dilation phenomena. We cannot make those clocks agree on how much time has passed since a specific event; what we can do is aggregate all the time measurements from the clocks and average them out. This way we end up with a value that is representative of how much time has passed on the entire system—in other words, we get an “overall time” for the system. Very often, the system that we consider is not restricted to just the surface of our planet, but involves the Sun, and sometimes the moon as well. In fact, what we call one year is roughly the time it takes for the Earth to complete an orbit around the Sun; one day is roughly the time it takes for the Earth to spin around itself once and face the Sun in the same position again. Including the Sun (or the moon) in our time measurements is complicated: in part this complexity comes from the fact that precise measurements of the Earth’s position are difficult, and in part from the fact that the Earth’s rotation is not regular, not fully predictable, and it’s slowing down. It’s worth noting that climate and geological events affect the Earth’s rotation in a measurable way, and such events are very hard to model accurately. What is important to understand here is that the word ‘time’ is often used to mean different things. Depending on how we measure it, we can end up with different definitions of time. To avoid ambiguity, I will classify ‘time’ into two big categories: <ul> <li> Elapsed time: this is the time measured directly by a clock, without using any extra information about the system where the clock lies into or about other clocks. We can use elapsed time to measure durations, latencies, frequencies, as well as lengths. </li> <li> Coordinated time: this is the time measured by using a clock, paired with information about the system where it’s located (like position, velocity, and gravity), and/or information from other clocks. This notion of time is mostly useful for coordinating events across the system. Some practical examples: scheduling the execution of tasks in the future, checking the expiration of certificates, real-time communication. </li> </ul> <h1 id="time-standards">Time standards</h1> Over the centuries several <a href="https://en.wikipedia.org/wiki/Time_standard">time standards</a> have been introduced to measure coordinated time. Nowadays there are three major standards in use: TAI, UTC, and GNSS. Let’s take a brief look at them. <h2 id="tai">TAI</h2> <a href="https://en.wikipedia.org/wiki/International_Atomic_Time">International Atomic Time (TAI)</a> is based on the weighted average of the elapsed time measured by several atomic clocks spread across the world. The more a clock in TAI is precise, the more it contributes to the weighted average. The fact that the clocks are spread in multiple locations, and the use of an average, mitigates relativistic effects and yields a value that we can think of as the overall time flow experienced by the surface of the Earth. Note that the calculations for TAI does not include the Earth’s position with respect to the Sun. <figure> <img src="https://andrea.corbellini.blog/images/tai-equipment-distribution.webp" alt="Distribution of the laboratories that contribute to TAI all over the world"> <figcaption>Distribution of the laboratories that contribute to International Atomic Time (TAI) all over the world as of 2020. Map taken from the <a href="https://webtai.bipm.org/ftp/pub/tai/annual-reports/bipm-annual-report/annual_report_2020.pdf">BIPM Annual Report on Time Activities</a>.</figcaption> </figure> <h2 id="utc">UTC</h2> <a href="https://en.wikipedia.org/wiki/Coordinated_Universal_Time">Coordinated Universal Time (UTC)</a> is built upon TAI. UTC, unlike TAI, is periodically adjusted to synchronize it with the Earth’s rotation around itself and the Sun. The goal is to make sure that 24 UTC hours are equivalent to a solar day (within a certain degree of precision). Because, as explained earlier, the Earth’s rotation is irregular, not fully predictable, and slowing down, periodic adjustments have to be made to UTC at irregular intervals. The adjustments are performed by inserting <a href="https://en.wikipedia.org/wiki/Leap_second">leap seconds</a>: these are extra seconds that are added to the UTC time to “slow down” the UTC time flow and keep it in sync with Earth’s rotation. On days when a leap second is inserted, UTC clocks go from 23:59:59 to 23:59:60. <figure> <img src="https://andrea.corbellini.blog/images/leap-seconds-timeline.svg" alt="Visualization of leap seconds inserted into UTC, and a comparison with TAI"> <figcaption>A visualization of leap seconds inserted into UTC until the end of 2022. Each orange dot represents a leap second (not in scale). When UTC was started in 1972, it started with 10 seconds of offset from TAI. As you can see, the insertion of leap seconds is very irregular: some decades have seen many leap seconds, others have seen much more.</figcaption> </figure> It’s worth noting that the practice of inserting leap seconds is most likely <a href="https://en.wikipedia.org/wiki/Leap_second#Future_of_leap_seconds">going to be discontinued</a> in the future. The main reason is that leap seconds have been the source of complexity and bugs in computer systems, and the benefit-to-pain ratio of leap seconds is not considered high enough to keep adding them. If leap seconds are discontinued, UTC will become effectively equivalent to TAI, with an offset: UTC will always differ from TAI by a few seconds, but this difference will always be constant, if no more leap seconds are inserted. <h2 id="gnss">GNSS</h2> <a href="https://en.wikipedia.org/wiki/GNSS">Global Navigation Satellite System (GNSS)</a> is based on a mix of accurate atomic clocks on ground and less accurate atomic clocks on artificial satellites orbiting around the Earth. The clocks on the satellites, being less accurate and subject to a variety of relativistic effects, are updated about twice a day from ground stations to correct clock drifts. Nowadays there are several implementations of GNSS around the world, including: <ul> <li>the United States’ <a href="https://en.wikipedia.org/wiki/Global_Positioning_System">Global Positioning System (GPS)</a>;</li> <li>the European <a href="https://en.wikipedia.org/wiki/Galileo_(satellite_navigation)">Galileo</a> system;</li> <li>China’s <a href="https://en.wikipedia.org/wiki/BeiDou">BeiDou (BDS)</a>;</li> <li>the Russian <a href="https://en.wikipedia.org/wiki/GLONASS">GLONASS</a>.</li> </ul> When GPS was launched, it was synchronized with UTC, however GPS, unlike UTC, is not adjusted to follow the Earth’s rotation, and due to that, GPS today differs from UTC by 18 seconds (because 18 leap seconds have been inserted since GPS was launched in 1980). BeiDou also does not implement leap seconds. GPS and BeiDou are therefore compatible with TAI. Other GNSS systems like Galileo and GLONASS do implement leap seconds and are therefore compatible with UTC. <h1 id="time-synchronization-protocols">Time synchronization protocols</h1> Dealing with coordinated time is not trivial. Different ways to deal with relativistic effects and Earth’s irregular rotation result in different time standards that are not always immediately compatible with each other. Nonetheless, once we agree on a well-defined time standard, we have a way to ask the question “what time is it?” and receive an accurate answer all around the world (within a certain degree of precision). Let’s now take a look at how computers on a network can obtain an accurate value for the coordinated time given by a time standard. I will describe two popular protocols: NTP and PTP. The two are using similar algorithms, but offer different precision: milliseconds (NTP) and nanoseconds (PTP). Both use UDP/IP as the transport protocol. <h2 id="network-time-protocol-ntp">Network Time Protocol (NTP)</h2> The way time synchronization works with NTP is the following: a computer that wants to synchronize its time periodically queries an NTP server (or multiple servers) to get the current coordinated time. The server that provides the current coordinated time may have obtained the time from an accurate source clock connected to the server (like an atomic clock synchronized with TAI or UTC, or a GNSS receiver), or from a previous synchronization from another NTP server. To record how “fresh” the coordinated time from an NTP server is (how distant the NTP server is from the source clock), NTP has a concept of stratum: this is a number that indicates the number of ‘hops’ from the accurate clock source: <ul> <li>stratum 0 is used to indicate an accurate clock;</li> <li>stratum 1 is a server that is directly connected to a stratum 0 clock;</li> <li>stratum 2 is a server that is synchronized from a stratum 1 server;</li> <li>stratum 3 is a server that is synchronized from a stratum 2 server;</li> <li>and so on…</li> </ul> The maximum stratum allowed is 15. There’s also a special stratum 16: this is not a real stratum, but a special value used by clients to indicate that time synchronization is not happening (most likely because the NTP servers are unreachable). <figure> <img src="https://andrea.corbellini.blog/images/ntp-strata.svg" alt="Visualization of NTP strata in a distributed network"> <figcaption>Examples of different NTP strata in a distributed network. A stratum n server obtains its time from stratum n - 1 servers.</figcaption> </figure> The major problem with synchronizing time over a network is latency. Networks can be composed of multiple links, some of which may be slow or overloaded. Simply requesting the current time from an NTP server without taking latency into account would lead to an imprecise response. Here is how NTP deals with this problem: <ol> <li>The NTP client sends a request via a UDP packet to an NTP server. The packet includes an originate timestamp $t_0$ that indicates the local time of the client when the packet was sent.</li> <li>The NTP server receives the request and records the receive timestamp $t_1$, which indicates the local time of the server when the request was received.</li> <li>The NTP server processes the request, prepares a response, and records the transmit timestamp $t_2$, which indicates the local time of the server when the response was sent. The timestamps $t_0$, $t_1$ and $t_2$ are all included in the response.</li> <li>The NTP client receives the response and records the timestamp $t_3$, which indicates the local time of the client when the response was received.</li> </ol> <figure> <img src="https://andrea.corbellini.blog/images/ntp-sync-algorithm.svg" alt="Visualization of the NTP time synchronization algorithm"> <figcaption>The NTP synchronization algorithm.</figcaption> </figure> Our goal is now to calculate an estimate for the network latency and processing delay and use that information to calculate, in the most accurate way possible, the offset between the NTP client clock and the NTP server clock. The difference $t_3 - t_0$ is the duration of the overall exchange. The difference $t_2 - t_1$ is the duration of the NTP server processing delay. If we subtract these two durations, we get the total network latency experienced, also known as round-trip delay: $$\delta = (t_3 - t_0) - (t_2 - t_1)$$ If we assume that the transmit delay and the receive delay are the same, then $\delta / 2$ is the average network latency (this assumption may not be true in a general network, but that’s the assumption that NTP makes). Under this assumption, the time $t_0 + \delta/2$ is the time on the client’s clock that corresponds to $t_1$ on the server’s clock. Similarly, $t_3 - \delta/2$ on the client’s clock corresponds to $t_2$ on the server’s clock. These correspondences let us calculate two estimates for the offset between the client’s clock and the server’s clock: $$\begin{align*} \theta_1 & = t_1 - (t_0 + \delta/2) \\ \theta_2 & = t_2 - (t_3 - \delta/2) \end{align*}$$ We can now calculate the client-server offset $\theta$ as an average of those two estimates: $$\begin{align*} \theta & = \frac{\theta_1 + \theta_2}2 \\ & = \frac{t_1 - (t_0 + \delta/2) + t_2 - (t_3 - \delta/2)}2 \\ & = \frac{t_1 - t_0 - \delta/2 + t_2 - t_3 + \delta/2}2 \\ & = \frac{(t_1 - t_0) + (t_2 - t_3)}2 \\ \end{align*}$$ Note that the offset $\theta$ may be a positive duration (meaning that the client clock is in the past), a negative duration (meaning that the client clock is in the future) or zero (meaning that the client clock agrees with the server clock, which is unlikely). After calculating the offset $\theta$, the client can update its local clock by shifting it by $\theta$ and from that point the client will be in sync with the server (within a certain degree of precision). Once the synchronization is done, it is expected that the client’s clock will start drifting away from the server’s clock. This may happen due to relativistic effects and more importantly because often clients do not use high-precision clocks. For this reason, it is important that NTP clients synchronize their time periodically. Usually NTP clients start by synchronizing time every minute or so when they are started, and then progressively slow down until they synchronize time once every half an hour or every hour. There are some drawbacks with this synchronization method: <ul> <li>The request and response delays may not be perfectly symmetric, resulting in inaccuracies in the calculations of the offset $\theta$. Network instabilities, packet retransmissions, change of routes, queuing may all cause unpredictable and inconsistent delays.</li> <li>The timestamps $t_1$ and $t_3$ must be set as soon as possible (as soon as the packets are received), and similarly $t_0$ and $t_2$ must be set as late as possible. Because NTP is implemented at the software level, there may be non-negligible delays in acquiring and recording these timestamps. These delays may be exacerbated if the NTP implementation is not very performant, or if the client or server are under high load.</li> <li>Errors propagate and add up when increasing the number of strata.</li> </ul> For all these reasons, NTP clients do not synchronize time just from a single NTP server, but from multiple ones. NTP clients take into account the round-trip delays, stratum, and jitter (the variance in round-trip delays) to decide the best NTP server to get their time from. Under ideal network conditions, an NTP client will always prefer a server with a low stratum. However, an NTP client may prefer an NTP server with high stratum and more reliable connectivity over an NTP server with low stratum but a very unstable network connection. The precision offered by NTP is in the order of a few milliseconds. <h2 id="precision-time-protocol-ptp">Precision Time Protocol (PTP)</h2> PTP is a time synchronization protocol for applications that require more accuracy than the one provided by NTP. The main differences between PTP and NTP are: <ul> <li>Precision: NTP offers millisecond precision, while PTP offers nanosecond precision.</li> <li>Time standard: NTP transmits UTC time, while PTP transmits TAI time and the difference between TAI and UTC.</li> <li>Scope: NTP is designed to be used over large networks, including the internet, while PTP is designed to be used in local area networks.</li> <li>Implementation: NTP is mainly software based, while PTP can be implemented both via software and on specialized hardware. The use of specialized hardware considerably reduces delays and jitter introduced by software.</li> </ul> <figure> <img src="https://andrea.corbellini.blog/images/timecard.jpg" alt="Picture of a Time Card device"> <figcaption><a href="http://www.timingcard.com/">Time Card</a>: an open-source hardware card with a PCIe interface that can be plugged into a computer that can serve as a PTP master. It can be optionally connected to a GNSS receiver and contains a rubidium (Rb) clock.</figcaption> </figure> <ul> <li>Hierarchy: NTP can support a complex hierarchy of NTP servers, organized via strata. While PTP does not put a limitation on the number of nodes involved, the hierarchy is usually only composed of master clocks (the source of time information) and slave clocks (the receivers of time information). Sometimes boundary clocks are used to relay time information to network segments that are unreachable by the master clocks.</li> <li>Clock selection: in NTP, clients select the best NTP server to use based on the NTP server clock quality and the network connection quality. In PTP, slaves do not select the best master clock to use. Instead, master clocks perform a selection between themselves using a method called best master clock algorithm. This algorithm takes into account the clock’s quality and input from system administrators, and does not factor network quality at all. The master clock selected by the algorithm is called grandmaster clock.</li> <li> Algorithm: in NTP, clients poll the time information from servers periodically and calculate the clock offset using the algorithm described above (based on the timestamps $t_0$, $t_1$, $t_2$ and $t_3$). With PTP, the algorithm used by slaves to calculate the offset from the grandmaster clock is somewhat similar to the one used in NTP, but the order of operations is different: <ol> <li>the grandmaster periodically broadcasts its time information $T_0$ over the network;</li> <li>each slave records the time $T_1$ when the broadcasted time was received;</li> <li>each slave sends a packet to the grandmaster at time $T_2$;</li> <li>the grandmaster receives the packet at time $T_3$ and sends that value back to the slave.</li> </ol> The average network delay can be calculated as $\delta = ((T_3 - T_0) - (T_2 - T_1)) / 2$. The clock offset can be calculated as $\theta = ((T_1 - T_0) + (T_2 - T_3)) / 2$. </li> </ul> <figure> <img src="https://andrea.corbellini.blog/images/ptp-sync-algorithm.svg" alt="Visualization of the PTP time synchronization algorithm"> <figcaption>The PTP time synchronization algorithm.</figcaption> </figure> <h1 id="summary">Summary</h1> <ul> <li>Synchronizing time across a computer network is not an easy task, and first of all requires agreeing on a definition of ‘time’ and on a time standard.</li> <li>Relativistic effects make it so that time may not flow at the same speed all over the globe, and this means that time has to be measured and aggregated across the planet in order to get a suitable value that can be agreed on.</li> <li>Atomic clocks and GNSS are the clock sources used for most applications nowadays.</li> <li>NTP is a time synchronization protocol that can be used on large and distributed networks like the internet and provides millisecond precision.</li> <li>PTP is a time synchronization protocol for local area networks and provides nanosecond precision.</li> </ul>andreacorbelliniMon, 23 Jan 2023 19:15:00 +0000tag:andrea.corbellini.blog,2023-01-23:/2023/01/23/what-time-is-it/information-technologytimerelativityperformanceclocksntpptpCan we encrypt data using Elliptic Curves?https://andrea.corbellini.blog/2023/01/02/ec-encryption/From time to time, I hear people saying that Elliptic Curve Cryptography (ECC) cannot be used to directly encrypt data, and you can only do key agreement and digital signatures with it. This is a common misconception, but it’s not actually true: you can indeed use elliptic curve keys to encrypt arbitrary data. And I’m not talking about hybrid-encryption schemes (like <a href="https://en.wikipedia.org/wiki/Integrated_Encryption_Scheme">ECIES</a> or <a href="https://datatracker.ietf.org/doc/rfc9180/">HPKE</a>): I’m talking about pure elliptic curve encryption, and I’m going to show an example of it in this article. It’s true however that pure elliptic curve encryption is not widely used or standardized because, as I will explain at the end of the article, key agreement is more convenient for most applications. <h1 id="quick-recap-on-elliptic-curve-cryptography">Quick recap on Elliptic Curve Cryptography</h1> I wrote an <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/">in-depth article about elliptic curve cryptography</a> in the past on this blog, and here is a quick recap: points on an elliptic curve from an interesting algebraic structure: a cyclic group. This group lets us do some algebra with the points of the elliptic curve: if we have two points $A$ and $B$, we can add them ($A + B$) or subtract them ($A - B$). We can also multiply a point by an integer, which is the same as doing repeated addition ($n A$ = $A + A + \cdots + A$, $n$ times). We know some efficient algorithms for doing multiplication, but the reverse of multiplication is believed to be a “hard” problem for certain elliptic curves, in the sense that we know efficient methods for computing $B = n A$ given $n$ and $A$, but we do not know very efficient methods to figure out $n$ given $A$ and $B$. This problem of reversing a multiplication is known as Elliptic Curve Discrete Logarithm Problem (ECDLP). Elliptic Curve Cryptography is based on multiplication of elliptic curve points by integers and its security is given mainly by the difficulty of solving the ECDLP. In order to use Elliptic Curve Cryptography, we first have to generate a private-public key pair: <ul> <li>the private key is a random integer $s$;</li> <li>the public key is the result of multiplying the integer $s$ with the generator $G$ of the elliptic curve group: $P = s G$.</li> </ul> Let’s now see a method to use Elliptic Curve Cryptography to encrypt arbitrary data, so that we can demystify the common belief that elliptic curves cannot be used to encrypt. <h1 id="elliptic-curve-elgamal">Elliptic Curve ElGamal</h1> One method to encrypt data with elliptic curve keys is <a href="https://en.wikipedia.org/wiki/ElGamal_encryption">ElGamal</a>. This is not the only method, of course, but it’s the one that I chose because it’s well known and simple enough. ElGamal is a cryptosystem that takes the name from <a href="https://en.wikipedia.org/wiki/Taher_Elgamal">its author</a> and works on any cyclic group, not just elliptic curve groups. If we want to encrypt a message using the public key $P$ via ElGamal, we can do the following: <ol> <li>map the message to a point $M$ on the elliptic curve</li> <li>generate a random integer $t$</li> <li>compute $C_1 = t G$</li> <li>compute $C_2 = t P + M$</li> <li>return the tuple $(C_1, C_2)$</li> </ol> To decrypt an encrypted tuple $(C_1, C_2)$ using the private key $s$, we can do the following: <ol> <li>compute $M = C_2 - s C_1$</li> <li>map the point $M$ back to a message</li> </ol> The scheme works because: $$\begin{align*} s C_1 & = s (t G) \\ & = t (s G) \\ & = t P \end{align*}$$ therefore: $$\begin{align*} C_2 - s C_1 & = (t P + M) - (t P) \\ & = M \end{align*}$$ There’s however a big problem with this scheme: how do we map a message to a point, and vice versa? How can we perform step 1 of the encryption algorithm, or step 2 of the decryption algorithm? <h1 id="mapping-a-message-to-a-point">Mapping a message to a point</h1> A message can be an arbitrary byte string. An elliptic curve point is, generally speaking, a pair of integers $(x, y)$ belonging to the elliptic curve field. How can we transform a byte string into a pair of field integers? Well, as far as computers are concerned, both byte strings and integers have the same nature: they are just sequences of bits, so there’s a natural map between the two. We could take the message, split it into two parts, and interpret the first part as an integer $x$ and the second part as an integer $y$. This would work for obtaining two arbitrary integers, but there’s a problem: the coordinates $x$ and $y$ of an elliptic curve point are related by a mathematical equation (the curve equation), so we cannot choose two arbitrary $x$ and $y$ and expect them to identify a valid point on the curve. In fact, for curves in Weierstrass form, given $x$ there are at most two possible choices for $y$, so it’s very unlikely that this splitting method will yield a valid point. Let’s change our strategy a little bit: instead of transforming the message to a pair $(x, y)$, we transform it to $x$ and then we compute a valid $y$ from the curve equation. This is a much better method, but there’s still a problem: generally speaking, not every $x$ will have a corresponding $y$. Not every $x$ can satisfy the curve equation. Luckily, most of the popular elliptic curves used in cryptography have an interesting property: about half of the possible field integers are valid $x$-coordinates. To see this, let’s take a look at an example: the curve <code>secp384r1</code>. This is a Weierstrass curve that has the following order: <div class="highlight"><pre><code>0xffffffffffffffffffffffffffffffffffffffffffffffffc7634d81f4372ddf581a0db248b0a77aecec196accc52973 </code></pre></div> I remind you that the order is the number of valid points that belong to the elliptic curve group. Because this is a Weierstrass curve, for each $x$ there are 2 possible points, so the number of valid $x$-coordinates is <code>order / 2</code>. Given an arbitrary 384-bit integer, what are the chances that this is a valid $x$-coordinate? The answer is <code>(order / 2) / (2 ** 384)</code> which is approximately 0.5 or 50%. OK, but how does this help with our goal: mapping an arbitrary message to a valid $x$-coordinate? It’s simple: we can append a random byte (or multiple bytes) to the message. We call this extra byte (or bytes): padding. If the resulting padded message does not translate to a valid $x$-coordinate, we choose another random padding and try again, until we find one that works. Given that there’s 50% chance of finding a valid $x$ coordinate, this method will find a valid $x$-coordinate very quickly: on average, this will happen on the first or the second try. <figure> <img src="https://andrea.corbellini.blog/images/ec-elgamal-padding.svg" alt="Padding a message to obtain a valid elliptic curve point" width="500" height="120"> <figcaption>Example of how to use padding to obtain a valid elliptic curve point from an arbitrary message.</figcaption> </figure> This operation can be easily reversed: if you have a point $(x, y)$, in order to recover the message that generated it, just take the $x$ coordinate and remove the padding. That’s it! It’s worth noting that there are some standard curves where all the possible byte strings (of the proper size) can be translated to elliptic curve points, without any random padding needed. For example, with <a href="https://en.wikipedia.org/wiki/Curve25519">Curve25519</a>, every 32-byte string is a valid elliptic curve point. Another curve like that is <a href="https://en.wikipedia.org/wiki/Curve448">Curve448</a>. It’s also important to note that the padding does not need to be truly random. In the image above I show a padding that is simply a constantly increasing sequence of numbers: 1, 2, 3, … That’s enough to find a valid point. <h1 id="putting-everything-together">Putting everything together</h1> We have seen how to map a message to a point and how ElGamal works, so now we have all the elements to write some working code. I’m choosing <a href="https://www.python.org/">Python</a> and the <a href="https://github.com/cslashm/ECPy">ECPy</a> package to work with elliptic curves, which you can install with <code>pip install ecpy</code>. <div class="highlight"><pre><code>import random from ecpy.curves import Curve, Point def message_to_point(curve: Curve, message: bytes) -> Point: # Number of bytes to represent a coordinate of a point coordinate_size = curve.size // 8 # Minimum number of bytes for the padding. We need at least 1 byte so that # we can try different values and find a valid point. We also add an extra # byte as a delimiter between the message and the padding (see below) min_padding_size = 2 # Maximum number of bytes that we can encode max_message_size = coordinate_size - min_padding_size if len(message) > max_message_size: raise ValueError('Message too long') # Add a padding long enough to ensure that the resulting padded message has # the same size as a point coordinate. Initially the padding is all 0 padding_size = coordinate_size - len(message) padded_message = bytearray(message) + b'\0' * padding_size # Put a delimiter between the message and the padding, so that we can # properly remove the padding at decrypt time padded_message[len(message)] = 0xff while True: # Convert the padded message to an integer, which may or may not be a # valid x-coordinate x = int.from_bytes(padded_message, 'little') # Calculate the corresponding y-coordinate (if it exists) y = curve.y_recover(x) if y is None: # x was not a valid coordinate; increment the padding and try again padded_message[-1] += 1 else: # x was a valid coordinate; return the point (x, y) return Point(x, y, curve) def encrypt(public_key: Point, message: bytes) -> bytes: curve = public_key.curve # Map the message to an elliptic curve point message_point = message_to_point(curve, message) # Generate a random number seed = random.randrange(0, curve.field) # Calculate c1 and c2 according to the ElGamal algorithm c1 = seed * curve.generator c2 = seed * public_key + message_point # Encode c1 and c2 and return them return bytes(curve.encode_point(c1) + curve.encode_point(c2)) def point_to_message(point: Point) -> bytes: # Number of bytes to represent a coordinate of a point coordinate_size = curve.size // 8 # Convert the x-coordinate of the point to a byte string padded_message = point.x.to_bytes(coordinate_size, 'little') # Find the padding delimiter message_size = padded_message.rfind(0xff) # Remove the padding and return the resulting message message = padded_message[:message_size] return message def decrypt(curve: Curve, secret_key: int, ciphertext: bytes) -> bytes: # Decode c1 and c2 and convert them to elliptic curve points c1_bytes = ciphertext[:len(ciphertext) // 2] c2_bytes = ciphertext[len(ciphertext) // 2:] c1 = curve.decode_point(c1_bytes) c2 = curve.decode_point(c2_bytes) # Calculate the message point according to the ElGamal algorithm message_point = c2 - secret_key * c1 # Convert the message point to a message and return it return point_to_message(message_point) </code></pre></div> And here is an usage example: <div class="highlight"><pre><code>curve = Curve.get_curve('secp384r1') secret_key = 0x123456789abcdef public_key = secret_key * curve.generator message = 'hello' print(' Message:', message) encrypted = encrypt(public_key, message.encode('utf-8')) print('Encrypted:', encrypted.hex()) decrypted = decrypt(curve, secret_key, encrypted).decode('utf-8') print('Decrypted:', decrypted) </code></pre></div> Which produces the following output: <div class="highlight"><pre><code> Message: hello Encrypted: 04fa333c6a03994c5bce4627de4447c5cdd358415f8db2745b67836932a0d5e81f19... Decrypted: hello </code></pre></div> <h1 id="some-considerations-on-padding-and-security">Some considerations on padding and security</h1> It’s important to note that padding is a very delicate problem in cryptography. There exist many <a href="https://en.wikipedia.org/wiki/Padding_(cryptography)">padding schemes</a>, and not all of them are secure. The padding scheme that I wrote in this article was just for demonstration purposes and may not be the most secure, so don’t use it in production systems. Take a look at <a href="https://en.wikipedia.org/wiki/Optimal_asymmetric_encryption_padding">OAEP</a> if you’re looking for a modern and secure padding scheme. Another thing to note is that the decryption method that I wrote does not check if the decryption was successful. If you try to decrypt an invalid ciphertext, or use the wrong key, you won’t get an error but instead a random result, which is not desiderable. A good padding scheme like OAEP will instead throw an error if decryption was unsuccessful. (Receiving an error when decryption is not successful is very important due to the fact that schemes like ElGamal are <a href="https://en.wikipedia.org/wiki/Malleability_(cryptography)">malleable</a>. Check out my post about <a href="https://andrea.corbellini.blog/2023/03/09/authenticated-encryption/">authenticated encryption</a> for examples and details about why this is important.) <h1 id="cost-of-elliptic-curve-encryption">Cost of elliptic curve encryption</h1> With Elliptic Curve ElGamal, if we are using an n-bit elliptic curve, we can encrypt messages that are at most n-bit long (actually less than that, if we’re using padding), and the output is at least 2n-bit long (if the resulting points $C_1$ and $C_2$ are encoded using point compression). This means that encryption using Elliptic Curve ElGamal doubles the size of the data that we want to encrypt. It also requires a fair amount of compute resources, because it involves a random number generation and 2 point multiplications. In short, Elliptic Curve ElGamal is expensive both in terms of space and in terms of time and compute power, and this makes it unattractive in applications like <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a> or general purpose encryption. So what can we use Elliptic Curve ElGamal for? We can use it to encrypt symmetric keys, such as <a href="https://en.wikipedia.org/wiki/Advanced_Encryption_Standard">AES</a> keys or <a href="https://en.wikipedia.org/wiki/ChaCha20">ChaCha20</a> keys, and then use these symmetric keys to encrypt our arbitrary data. Symmetric keys are relatively short (ranging from 128 to 256 bits nowadays), so they can be encrypted with one round of Elliptic Curve ElGamal with most curves. It’s worth noting that this is the same approach that we use with <a href="https://en.wikipedia.org/wiki/RSA_(cryptosystem)">RSA</a> encryption: for most applications, we don’t use RSA to encrypt data directly, but rather we use RSA to encrypt symmetric keys which are later used for encrypting data. These are the reason why schemes like Elliptic Curve ElGamal, or other methods of encryption with elliptic curves, are not used in practice: <ul> <li>elliptic curve encryption is more expensive than hybrid encryption;</li> <li>hybrid encryption scales better and is more performant;</li> <li>elliptic curve key exchange is simpler and has fewer pitfalls than encryption.</li> </ul> In conclusion, there are no practical benefits from elliptic curve encryption compared to hybrid encryption with key agreement, and that’s why we don’t use it. However, the idea that elliptic curves cannot be used for encryption is a myth, and I hope this article will help clarify that confusion.andreacorbelliniMon, 02 Jan 2023 06:30:00 +0000tag:andrea.corbellini.blog,2023-01-02:/2023/01/02/ec-encryption/cryptographyeccencryptionelgamalThe curious case of bad blocks on an SSD, and how I got rid of themhttps://andrea.corbellini.blog/2022/12/29/curious-ssd-badblocks/I recently inherited a laptop that was broken by pouring some hot coffee on it. When I dissected it, it was pretty clear that most of it was unrecoverable: the CPU was completely fried, and its thermal paste splashed everywhere on the motherboard. (I wish I took a picture of it that I could share.) There were however a few pieces that looked in a good state. One of those components was a NVMe Solid State Drive (SSD). I decided to take this SSD and recycle it in my own laptop, maybe to join my LVM pool. When I plugged it in my laptop however the SSD I tried to navigate the filesystem, and it appeared to be working quite slowly. Opening certain files sometimes would hang indefinitely. Upon inspection of the SMART data and the kernel logs, it was clear that the drive was returning plenty of read errors. Here is a sample of the kernel logs: <div class="highlight"><pre><code>$ dmesg ... [ 860.465707] ata2.00: exception Emask 0x0 SAct 0x8 SErr 0x0 action 0x0 [ 860.465726] ata2.00: irq_stat 0x40000008 [ 860.465733] ata2.00: failed command: READ FPDMA QUEUED [ 860.465737] ata2.00: cmd 60/08:18:58:c5:28/00:00:00:00:00/40 tag 3 ncq dma 4096 in [ 860.465737] res 41/40:08:58:c5:28/00:00:00:00:00/00 Emask 0x409 (media error) <F> [ 860.465750] ata2.00: status: { DRDY ERR } [ 860.465754] ata2.00: error: { UNC } [ 860.467010] ata2.00: configured for UDMA/133 [ 860.467046] sd 1:0:0:0: [sda] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s [ 860.467054] sd 1:0:0:0: [sda] tag#3 Sense Key : Medium Error [current] [ 860.467060] sd 1:0:0:0: [sda] tag#3 Add. Sense: Unrecovered read error - auto reallocate failed [ 860.467066] sd 1:0:0:0: [sda] tag#3 CDB: Read(10) 28 00 00 28 c5 58 00 00 08 00 [ 860.467069] I/O error, dev sda, sector 2671960 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 ... [ 1057.914608] ata2: softreset failed (device not ready) [ 1057.914623] ata2: hard resetting link [ 1063.230631] ata2: found unknown device (class 0) [ 1067.934891] ata2: softreset failed (device not ready) [ 1067.934911] ata2: hard resetting link [ 1073.270826] ata2: found unknown device (class 0) [ 1078.486604] ata2: link is slow to respond, please be patient (ready=0) [ 1102.970841] ata2: softreset failed (device not ready) [ 1102.970860] ata2: limiting SATA link speed to 1.5 Gbps [ 1102.970865] ata2: hard resetting link [ 1108.034602] ata2: found unknown device (class 0) [ 1108.194622] ata2: softreset failed (device not ready) [ 1108.194638] ata2: reset failed, giving up [ 1108.194642] ata2.00: disable device [ 1108.194677] ata2: EH complete [ 1108.194726] sd 1:0:0:0: [sda] tag#6 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=232s [ 1108.194740] sd 1:0:0:0: [sda] tag#6 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00 [ 1108.194748] I/O error, dev sda, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 ... </code></pre></div> These logs show that the SSD was returning errors (exceptions) to the operating system, and also that the SSD would sometimes become so slow to respond that the kernel would attempt to reset it (which didn’t really work, I can tell you). Here is an excerpt of the SMART data: <div class="highlight"><pre><code>$ smartctl -a /dev/sda ... SMART Attributes Data Structure revision number: 0 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 166 001 006 Pre-fail Always In_the_past 0 5 Retired_Block_Count 0x0032 100 100 036 Old_age Always - 76 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1740 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 2247 100 Total_Erase_Count 0x0032 100 100 000 Old_age Always - 7654272 168 Min_Erase_Count 0x0032 253 096 000 Old_age Always - 0 169 Max_Erase_Count 0x0032 083 083 000 Old_age Always - 181 171 Program_Fail_Count 0x0032 253 253 000 Old_age Always - 0 172 Erase_Fail_Count 0x0032 253 253 000 Old_age Always - 0 174 Unexpect_Power_Loss_Ct 0x0030 100 100 000 Old_age Offline - 14 175 Program_Fail_Count_Chip 0x0032 253 253 000 Old_age Always - 0 176 Unused_Rsvd_Blk_Cnt_Tot 0x0032 253 253 000 Old_age Always - 0 177 Wear_Leveling_Count 0x0032 090 090 000 Old_age Always - 116 178 Used_Rsvd_Blk_Cnt_Chip 0x0032 100 100 000 Old_age Always - 399 179 Used_Rsvd_Blk_Cnt_Tot 0x0032 100 100 000 Old_age Always - 2460 180 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 2980 184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 9919 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 10051 188 Command_Timeout 0x0032 253 253 000 Old_age Always - 0 194 Temperature_Celsius 0x0002 038 000 000 Old_age Always - 38 (Min/Max 16/48) 195 Hardware_ECC_Recovered 0x0032 100 085 000 Old_age Always - 715203 196 Reallocated_Event_Count 0x0032 100 100 036 Old_age Always - 76 198 Offline_Uncorrectable 0x0032 253 253 000 Old_age Always - 0 199 UDMA_CRC_Error_Count 0x0032 253 253 000 Old_age Always - 0 204 Soft_ECC_Correction 0x000e 100 001 000 Old_age Always - 13 212 Phy_Error_Count 0x0032 253 253 000 Old_age Always - 0 234 Unknown_SK_hynix_Attrib 0x0032 100 100 000 Old_age Always - 32297 241 Total_Writes_GB 0x0032 100 100 000 Old_age Always - 3715 242 Total_Reads_GB 0x0032 100 100 000 Old_age Always - 3680 250 Read_Retry_Count 0x0032 096 096 000 Old_age Always - 176835377 ... </code></pre></div> This table show various attributes for the operational status of the SSD. The meaning of the numeric values is pretty much vendor-specific, so trying to understand those number exactly is quite a challenge, but what matters is that the numbers under the <code>VALUE</code> column are higher than the <code>THRESH</code> (threshold) column. The <code>WORST</code> column indicates the lowest <code>VALUE</code> that has ever been observed. To my surprise, despite all the errors and hangs that the SSD was experiencing, the SMART values looked pretty good. Sure, there’s a very low <code>WORST</code> value for <code>Raw_Read_Error_Rate</code> (001, much lower than the threshold 001), and there is also and indication that this attribute failed in the past, but besides that everything looked acceptable enough. Of course the SMART log was recording the read errors as well. Here’s another excerpt from the output: <div class="highlight"><pre><code>$ smartctl -a /dev/sda ... SMART Error Log Version: 1 ATA Error Count: 1875 (device log contains only the most recent five errors) ... Error 1875 occurred at disk power-on lifetime: 1737 hours (72 days + 9 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 41 00 00 00 00 00 Error: UNC at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 08 70 98 31 af 40 40 00:02:32.920 READ FPDMA QUEUED 47 00 01 30 08 00 a0 a0 00:02:32.920 READ LOG DMA EXT 47 00 01 30 00 00 a0 a0 00:02:32.920 READ LOG DMA EXT 47 00 01 00 00 00 a0 a0 00:02:32.920 READ LOG DMA EXT ef 10 02 00 00 00 a0 a0 00:02:32.920 SET FEATURES [Enable SATA feature] ... </code></pre></div> Give the lack of concrete signs of old age or extended damage to the SSD, I wondered if it could be a link problem: maybe I did not insert the drive correctly, or maybe a pin was dirty. But no: upon inspection I did not find any issue, and after carefully reseating the drive, the problem was persisting. I proceeded to run a SMART self test, here are the results (from most recent to oldest): <div class="highlight"><pre><code>SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short captive Completed: read failure 90% 1736 5712 # 2 Short offline Completed: read failure 90% 1736 5712 # 3 Extended offline Completed: read failure 90% 1733 50117792 # 4 Extended captive Interrupted (host reset) 90% 1730 - # 5 Short captive Interrupted (host reset) 90% 1730 - </code></pre></div> The first two tests were interrupted by Linux, which tried to reset the device while the tests were running. A self-test (as the name suggests) is completely self contained and does not involve sharing of data between the SSD and the operating system in the process. The fact that the self-test was failing due to bad blocks was therefore a sign that this was not a link error, but that the blocks were really damaged. I decided therefore to give up on trying to fix the SSD, but I still wanted to use it. After all, it was working for the most part: as long as you didn’t access the bad blocks, the SSD would behave fine. So here is my plan: I would format the SSD and create an ext4 filesystem on it, using <code>mkfs.ext4 -c</code>, which would scan for and exclude bad blocks so that they wouldn’t be used. The resulting filesystem would have less storage available than the advertised capacity of the SSD, but that was an acceptable trade-off for me. And here is the most interesting part: <code>mkfs.ext4 -c</code> discarded all blocks before creating the filesystem. After that, it scanned for bad blocks and, shockingly, it found none! SMART self-tests also did not report any error: <div class="highlight"><pre><code>SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 1740 - # 2 Short offline Completed without error 00% 1738 - </code></pre></div> All the read errors, exceptions and the hanging problem that kept appearing before disappeared! I’m not fully sure how to explain how this happened, but I did some research and the general consensus is that discarding bad blocks won’t recover them. My theory is that, when the coffee was poured on the laptop, a spike of voltage led to incorrect values to be written to a few blocks that were in use at that time, causing uncorrectable discrepancies between the data and the error-correcting-codes of the SSD. Discarding the blocks reset both the data cells and the ECC cells, removing all the inconsistencies. Do you have a better explanation? Let me know in the comments!andreacorbelliniThu, 29 Dec 2022 04:00:00 +0000tag:andrea.corbellini.blog,2022-12-29:/2022/12/29/curious-ssd-badblocks/information-technologyubuntustoragessdbadblocksHow to use the same DNS for all connections in Ubuntu (and other network privacy tricks)https://andrea.corbellini.blog/2020/04/28/ubuntu-global-dns/<h1 id="problem">Problem</h1> Currently Ubuntu does not offer an easy way to set up a “global” DNS for all network connections: whenever you connect to a new WiFi network, if you don’t want to use the DNS server provided by the WiFi, you are forced to go to the network settings and manually set your preferred DNS server. With this brief guide I want to show how you can setup a global DNS to be used for all the WiFi and network connections, both old and new ones. I will also show you how to use DNSSEC, DNS-over-TLS and randomized MAC addresses for all connections. This guide is written for Ubuntu 20.04, but in general it will work on every distribution using systemd-resolved and NetworkManager. <h1 id="step-1-setup-the-global-dns-in-resolved">Step 1: setup the Global DNS in resolved</h1> In Ubuntu (as well as many other distributions), DNS is managed by systemd-resolved. Its configuration is in <code>/etc/systemd/resolved.conf</code>. Open that file and add a <code>DNS=</code> line inside the <code>[Resolve]</code> section listing your preferred DNS servers. For example, if you want to use <a href="https://1.1.1.1/"><code>1.1.1.1</code></a>, your <code>resolved.conf</code> should look like this: <div class="highlight"><pre><code>[Resolve] DNS=1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001 #FallbackDNS= #Domains= #LLMNR=no #MulticastDNS=no #Cache=yes #DNSStubListener=yes #ReadEtcHosts=yes </code></pre></div> Once you are done with the changes, reload systemd-resolved: <div class="highlight"><pre><code>sudo systemctl restart systemd-resolved.service </code></pre></div> You can check your changes with <code>resolvectl status</code>: you should see your DNS servers on top of the output, under the Global section: <div class="highlight"><pre><code>$ resolvectl status Global LLMNR setting: no MulticastDNS setting: no DNSOverTLS setting: opportunistic DNSSEC setting: allow-downgrade DNSSEC supported: no Current DNS Server: 1.1.1.1 DNS Servers: 1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001 ... </code></pre></div> This however won’t be enough to use that DNS! In fact, the Global DNS of systemd-resolved is just a default option that is used whenever no DNS servers are configured for an interface. When you connect to a WiFi network, NetworkManager will ask the access point for a list of DNS servers and will communicate that list to systemd-resolved, effectively overriding the settings that we just edited. If you scroll down the output of <code>resolvectl status</code>, you will see the DNS servers added by NetworkManager. We have to tell NetworkManager to stop doing that. <h1 id="step-2-disable-dns-processing-in-networkmanager">Step 2: Disable DNS processing in NetworkManager</h1> In order for systemd-resolved to consider our global DNS, we need to tell NetworkManager not to provide any DNS information for new connections. Doing that is easy: just create a new file <code>/etc/NetworkManager/conf.d/dns.conf</code> (or any name you like) with this content: <div class="highlight"><pre><code>[main] # do not use the dhcp-provided dns servers, but rather use the global # ones specified in /etc/systemd/resolved.conf dns=none systemd-resolved=false </code></pre></div> To apply the settings either restart your computer or run: <div class="highlight"><pre><code>sudo systemctl reload NetworkManager.service </code></pre></div> Now, when you connect to a new network connection, NetworkManager won’t push the list of DNS servers to systemd-resolved and only the global ones will be used. If you check <code>resolvectl status</code>, you should see that, for every interface, there is no DNS server specified. If you specified <code>1.1.1.1</code> as your DNS servers, then you can also head over to <a href="https://1.1.1.1/help">https://1.1.1.1/help</a> to verify that they’ve been correctly set up. <h1 id="dnssec-and-dns-over-tls">DNSSEC and DNS-over-TLS</h1> If you would like to enable DNSSEC and/or DNS-over-TLS, the file to edit is <code>/etc/systemd/resolved.conf</code>. You can add the following options: <ul> <li><code>DNSSEC=true</code> if you want all queries to be DNSSEC-validated. The default is <code>DNSSEC=allow-downgrade</code>, which attempts to use DNSSEC if it works properly, and falls back to disabling validation otherwise.</li> <li><code>DNSOverTLS=true</code> if you want all queries to go through TLS. You can also specify <code>DNSOverTLS=opportunistic</code> to attempt to use TLS if it supported, and fall back to the plaintext DNS protocol if it’s not.</li> </ul> With those options, my <code>/etc/systemd/resolved.conf</code> looks like this: <div class="highlight"><pre><code>[Resolve] DNS=1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001 #FallbackDNS= #Domains= #LLMNR=no #MulticastDNS=no DNSSEC=true DNSOverTLS=opportunistic #Cache=yes #DNSStubListener=yes #ReadEtcHosts=yes </code></pre></div> Note that I’m using <code>DNSOverTLS=opportunistic</code> because I found that some access points with captive portals don’t work properly when using <code>DNSOverTLS=true</code>. Also note that <code>DNSSEC=true</code> may cause some pain because there are still many misconfigured domain records out there that will make make DNSSEC validation fail. Like before, to apply the changes, run: <div class="highlight"><pre><code>sudo systemctl restart systemd-resolved.service </code></pre></div> And to verify the changes: <div class="highlight"><pre><code>resolvectl status </code></pre></div> If you’re using <code>1.1.1.1</code>, you can also go to <a href="https://1.1.1.1/help">https://1.1.1.1/help</a> to verify DNS-over-TLS. <h1 id="random-mac-address">Random MAC address</h1> NetworkManager supports 3 options to have a random MAC address (also known as “cloned” or “spoofed” MAC address): <ol> <li><code>wifi.scan-rand-mac-address</code> controls the MAC address used when scanning for WiFi devices. This goes into the <code>[device]</code> section</li> <li><code>wifi.cloned-mac-address</code> controls the MAC address for WiFi connections. This goes into the <code>[connection]</code> section</li> <li><code>ethernet.cloned-mac-address</code> controls the MAC address for Ethernet connections. This goes into the <code>[connection]</code> section</li> </ol> The first option can take either <code>yes</code> or <code>no</code>. The last two can take various values, but if you want a randomized MAC address you are interested in these two: <ul> <li><code>random</code>: generate a new random MAC address each time you establish a connection</li> <li><code>stable</code>: this generates a MAC address that is kinda random (it’s a hash), but will be reused when you connect to the same network again.</li> </ul> <code>random</code> is better if you don’t want to be tracked, but it has the disadvantage that captive portals won’t remember you. Instead <code>stable</code> allows captive portals to remember you and therefore won’t show up whenever you reconnect. Whatever options you want to go with, put them into a file <code>/etc/NetworkManager/conf.d/mac.conf</code> (or any other name you like). Mine looks like this: <div class="highlight"><pre><code>[device] # use a random mac address when scanning for wifi networks wifi.scan-rand-mac-address=yes [connection] # use a random mac address when connecting to a network ethernet.cloned-mac-address=random wifi.cloned-mac-address=random </code></pre></div> To apply the settings either run restart your computer or run: <div class="highlight"><pre><code>sudo systemctl reload NetworkManager.service </code></pre></div> You can test your changes with: <div class="highlight"><pre><code>ip link </code></pre></div>andreacorbelliniTue, 28 Apr 2020 06:30:00 +0000tag:andrea.corbellini.blog,2020-04-28:/2020/04/28/ubuntu-global-dns/ubuntuubuntusystemdresolvednetwork-managerdnsmac-addressprivacy11 years of Ubuntu membershiphttps://andrea.corbellini.blog/2018/05/12/11-years-of-ubuntu-membership/It’s been 11 years and 1 month since I was awarded with <a href="https://wiki.ubuntu.com/Membership">official Ubuntu membership</a>. I will never forget that day: as a kid I had to write about myself on <a href="https://en.wikipedia.org/wiki/Internet_Relay_Chat">IRC</a>, in front of the Community Council members and answer their questions in a language that was not my primary one. I must confess that I was a bit scared that evening, but once I made it, it felt so good. It felt good not just because of the award itself, but rather because that was the recognition that I did something that mattered. I did something useful that other people could benefit from. And for me, that meant a lot. So much time has passed since then. So many things have changed both in my life and around me, for better or worse. So many that I cannot even <a href="https://en.wikipedia.org/wiki/Cantor%27s_diagonal_argument">enumerate</a> all of them. Nonetheless, deep inside of me, I still feel like that young kid: curious, always ready to experiment, full of hopes and uncertain (but never scared) about the future. Through the years I received the support of a bunch of people who believed in me, and I thank them all. But if today I feel so hopeful it’s undoubtedly thanks to one person in particular, a person who holds a special place in my life. A big thank you goes to you.andreacorbelliniSat, 12 May 2018 21:30:00 +0000tag:andrea.corbellini.blog,2018-05-12:/2018/05/12/11-years-of-ubuntu-membership/ubuntuubuntuRunning Docker Swarm inside LXC (outdated)https://andrea.corbellini.blog/2016/04/13/docker-swarm-inside-lxc/UPDATE: This article was written in 2016 and refers to a version of Docker Swarm that is now known as “legacy Swarm”. The newer Docker Swarm won’t work in LXC as described in this article. I’ve been using <a href="https://docs.docker.com/swarm/">Docker Swarm</a> inside <a href="https://linuxcontainers.org/lxc/introduction/">LXC</a> containers for a while now, and I thought that I could share my experience with you. Due to their nature, LXC containers are pretty lightweight and require very few resources if compared to virtual machines. This makes LXC ideal for development and simulation purposes. Running Docker Swarm inside LXC requires a few steps that I’m going to show you in this tutorial. Before we begin, a quick premise: LXC, Docker and Swarm can be configured in many different ways. Here I’m showing just my preferred setup: LXC with AppArmor disabled, Docker with the OverlayFS storage driver, Swarm with etcd discovery. There exist many other kind of configurations that can work under LXC — leave a comment if you want to know more. Overview: <ol> <li><a href="#step-1">Create the Swarm Manager container</a></li> <li><a href="#step-2">Modify configuration for the Swarm Manager container</a></li> <li><a href="#step-3">Load the OverlayFS module</a></li> <li><a href="#step-4">Start the container and install Docker</a></li> <li><a href="#step-5">Check if Docker is working</a></li> <li><a href="#step-6">Set up the Swarm Manager</a></li> <li><a href="#step-7">Create the Swarm Agents</a></li> <li><a href="#step-8">Play with the Swarm</a></li> </ol> Terminology: <ul> <li>the host is the system that will create and start the LXC containers (e.g. your laptop);</li> <li>the manager is the LXC container that will run the Swarm manager (it’ll run the <code>swarm manage</code> command);</li> <li>an agent is one of the many LXC containers that will run a Swarm agent node (it’ll run the <code>swarm join</code> command);</li> </ul> To avoid ambiguity, all commands will be prefixed with a prompt such as <code>root@host:~#</code>, <code>root@swarm-manager:~#</code> and <code>root@swarm-agent-1:~#</code>. Prerequisites: This tutorial assumes that you have at least a vague idea of what Docker and Docker Swarm are. You should also be familiar with the shell. This tutorial has been successfully tested on Ubuntu 15.10 (that ships with Docker 1.6) and Ubuntu 16.04 LTS (Docker 1.10), but it may work on other distributions and Docker versions as well. <h1 id="step-1">Step 1: Create the Swarm Manager container</h1> Create a new LXC container with: <div class="highlight"><pre><code>root@host:~# lxc-create -t download -n swarm-manager </code></pre></div> When prompted, choose your favorite distribution and architecture. I chose <code>ubuntu</code> / <code>xenial</code> / <code>amd64</code>. <code>lxc-create</code> needs to run as root, <a href="https://www.stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/">unprivileged containers</a> won’t work. We could actually make Docker start inside an unprivileged container, the problem is that we wouldn’t be allowed to create block and character devices, and many Docker containers need this ability. <h1 id="step-2">Step 2: Modify the configuration for the Swarm Manager container</h1> Before starting the LXC container, open the file <code>/var/lib/lxc/swarm-manager/config</code> on the host and add the following configuration to the bottom of the file: <div class="highlight"><pre><code># Distribution configuration # ... # Container specific configuration # ... # Network configuration # ... # Allow running Docker inside LXC lxc.aa_profile = unconfined lxc.cap.drop = </code></pre></div> The first rule (<code>lxc.aa_profile = unconfined</code>) disables AppArmor confinement. The second one (<code>lxc.cap.drop =</code>) gives all capabilities to the processes in LXC container. These two rules may seem harmful from a security standpoint, and in fact they are. However we must remember that we will be running Docker inside the LXC container. Docker already ships with its own AppArmor profile and the two rules above are needed exactly for the purposes of letting Docker talk to AppArmor. So, while Docker itself won’t be confined, Docker containers will be confined, and this is an encouraging fact. <h1 id="step-3">Step 3: Load the OverlayFS module</h1> OverlayFS is shipped with Ubuntu, but not enabled by default. To enable it: <div class="highlight"><pre><code>root@host:~# modprobe overlay </code></pre></div> It is important to do this step before installing Docker. Docker supports various storage drivers and when Docker is installed for the first time it tries to detect the most appropriate one for the system. If Docker detects that OverlayFS is not loaded, it’ll fall back to the device mapper. There’s nothing wrong with the device mapper, we can make it work, however, as I said at the beginning, in this tutorial I’m focusing only on OverlayFS. If you want to load OverlayFS at boot, instead of doing it manually after every reboot, add it to <code>/etc/modules-load.d/modules.conf</code>: <div class="highlight"><pre><code>root@host:~# echo overlay >> /etc/modules-load.d/modules.conf </code></pre></div> <h1 id="step-4">Step 4: Start the container and install Docker</h1> It’s time to see if we did everything right! <div class="highlight"><pre><code>root@host:~# lxc-start -n swarm-manager root@host:~# lxc-attach -n swarm-manager root@swarm-manager:~# apt update root@swarm-manager:~# apt install docker.io </code></pre></div> Installation should complete without any problem. If you get an error like this: <div class="highlight"><pre><code>Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details. invoke-rc.d: initscript docker, action "start" failed. dpkg: error processing package docker.io (--configure): subprocess installed post-installation script returned error exit status 1 </code></pre></div> It means that Docker failed to start. Try checking <code>systemctl status docker</code> as suggested, or run <code>docker daemon</code> manually. You might get an error like this: <div class="highlight"><pre><code>root@swarm-manager:~# docker daemon WARN[0000] devmapper: Udev sync is not supported. This will lead to unexpected behavior, data loss and errors. For more information, see https://docs.docker.com/reference/commandline/daemon/#daemon-storage-driver-option ERRO[0000] There are no more loopback devices available. ERRO[0000] [graphdriver] prior storage driver "devicemapper" failed: loopback attach failed FATA[0000] Error starting daemon: error initializing graphdriver: loopback attach failed </code></pre></div> In this case, Docker is using the devicemapper storage driver and is complaining about the lack of loopback devices. If that’s the case, check whether OverlayFS is loaded and reinstall Docker. Or you might get an error like this: <div class="highlight"><pre><code>root@swarm-manager:~# docker daemon ... FATA[0000] Error starting daemon: AppArmor enabled on system but the docker-default profile could not be loaded. </code></pre></div> It this other case, Docker is complaining about the fact that it can’t talk to AppArmor. Check the configuration for the LXC container. <h1 id="step-5">Step 5: Check if Docker is working</h1> Once you are all set, you should be able to use Docker: try running <code>docker info</code>, <code>docker ps</code> or launch a container: <div class="highlight"><pre><code>root@swarm-manager:~# docker run --rm docker/whalesay cowsay burp! Unable to find image 'docker/whalesay:latest' locally latest: Pulling from docker/whalesay ... Status: Downloaded newer image for docker/whalesay:latest _______ < burp! > ------- \ \ \ ## . ## ## ## == ## ## ## ## === /""""""""""""""""___/ === ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~ \______ o __/ \ \ __/ \____\______/ </code></pre></div> It appears to be working. By the way, we can check whether Docker is correctly confining containers. Try running a Docker container and check on the host the output of <code>aa-status</code>: you should see a process running with the <code>docker-default</code> profile. For example: <div class="highlight"><pre><code>root@swarm-manager:~# docker run --rm ubuntu bash -c 'while true; do sleep 1; echo -n zZ; done' zZzZzZzZzZzZzZzZ... # On another shell root@host:~# aa-status apparmor module is loaded. 5 profiles are loaded. 5 profiles are in enforce mode. /sbin/dhclient /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/NetworkManager/nm-dhcp-helper /usr/lib/connman/scripts/dhclient-script docker-default 0 profiles are in complain mode. 4 processes have profiles defined. 4 processes are in enforce mode. /sbin/dhclient (797) /sbin/dhclient (2832) docker-default (6956) docker-default (6973) 0 processes are in complain mode. 0 processes are unconfined but have a profile defined. root@host:~# ps -ef | grep 6956 root 6956 4982 0 17:17 ? 00:00:00 bash -c while true; do sleep 1; echo -n zZ; done root 6973 6956 0 17:17 ? 00:00:00 sleep 1 root 6982 6808 0 17:17 pts/3 00:00:00 grep --color=auto 6956 </code></pre></div> Yay! Everything is running as expected: we launched a process inside a Docker container, and that process is running with the <code>docker-default</code> AppArmor profile. Once again: even if LXC is running unconfined, our Docker containers are not. <h1 id="step-6">Step 6: Set up the Swarm Manager</h1> That was the hardest part. Now we can proceed setting up Swarm as we would usually do. As I said at the beginning, Swarm can be configured in many ways. In this tutorial I’ll show how to set it up with etcd discovery. First of all, we need the IP address of the LXC container: <div class="highlight"><pre><code>root@swarm-manager:~# ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:16:3e:8e:cb:43 inet addr:10.0.3.154 Bcast:10.0.3.255 Mask:255.255.255.0 inet6 addr: fe80::216:3eff:fe8e:cb43/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:23177 errors:0 dropped:0 overruns:0 frame:0 TX packets:20859 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:147652946 (147.6 MB) TX bytes:1455613 (1.4 MB) </code></pre></div> <code>10.0.3.154</code> is my IP address. Let’s start etcd: <div class="highlight"><pre><code>root@swarm-manager:~# SWARM_MANAGER_IP=10.0.3.154 root@swarm-manager:~# docker run -d --restart=always --name=etcd -p 4001:4001 -p 2380:2380 -p 2379:2379 \ quay.io/coreos/etcd -name etcd0 \ -advertise-client-urls http://$SWARM_MANAGER_IP:2379,http://$SWARM_MANAGER_IP:4001 \ -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \ -initial-advertise-peer-urls http://$SWARM_MANAGER_IP:2380 \ -listen-peer-urls http://0.0.0.0:2380 \ -initial-cluster-token etcd-cluster-1 \ -initial-cluster etcd0=http://$SWARM_MANAGER_IP:2380 \ -initial-cluster-state new Unable to find image 'quay.io/coreos/etcd:latest' locally latest: Pulling from coreos/etcd ... Status: Downloaded newer image for quay.io/coreos/etcd:latest e742278a97d2ad3f88658aa871903d20b4094e551969a03aa8332d3876fe5d0d root@swarm-manager:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e742278a97d2 quay.io/coreos/etcd "/etcd -name etcd0 -a" 32 seconds ago Up 31 seconds 0.0.0.0:2379-2380->2379-2380/tcp, 0.0.0.0:4001->4001/tcp, 7001/tcp etcd </code></pre></div> Replace <code>10.0.3.154</code> with the IP address of your LXC container. Note that I’ve started etcd with <code>--restart=always</code>, so that every time etcd is automatically started when the LXC container starts. With this option, etcd will restart even if you explicitly stop it. Drop <code>--restart=always</code> if that’s not what you want. Now we can start the Swarm manager: <div class="highlight"><pre><code>root@swarm-manager:~# docker run -d --restart=always --name=swarm -p 3375:3375 \ swarm manage -H 0.0.0.0:3375 etcd://$SWARM_MANAGER_IP:2379 Unable to find image 'swarm:latest' locally latest: Pulling from library/swarm ... Status: Downloaded newer image for swarm:latest 8080c93c544ff92cc2cf682ff0bbc82e0d2dfb01e1f98f202c3a0801d3427330 root@swarm-manager:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 46b556e73e87 swarm "/swarm manage -H 0.0" 3 seconds ago Up 2 seconds 2375/tcp, 0.0.0.0:3375->3375/tcp swarm e742278a97d2 quay.io/coreos/etcd "/etcd -name etcd0 -a" 7 minutes ago Up 7 minutes 0.0.0.0:2379-2380->2379-2380/tcp, 0.0.0.0:4001->4001/tcp, 7001/tcp etcd </code></pre></div> Our Swarm manager is up and running. We can connect to it and issue a few commands: <div class="highlight"><pre><code>root@swarm-manager:~# docker -H localhost:3375 info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: swarm/1.1.3 Role: primary Strategy: spread Filters: health, port, dependency, affinity, constraint Nodes: 0 Plugins: Volume: Network: Kernel Version: 4.4.0-15-generic Operating System: linux Architecture: amd64 CPUs: 0 Total Memory: 0 B Name: d39c33295ef3 </code></pre></div> As you can see there are no nodes connected, as we would expect. Everything looks good. <h1 id="step-7">Step 7: Create the Swarm Agents</h1> Our Swarm manager can’t do anything interesting without agent nodes. Creating new LXC containers for the agents is not much different from what we already did with the manager. To set up new agents in an automatic fashion I’ve created a script, so that you don’t need to repeat the steps manually: <div class="highlight"><pre><code>#!/bin/bash set -eu SWARM_MANAGER_IP=10.0.3.154 DOWNLOAD_DIST=ubuntu DOWNLOAD_RELEASE=xenial DOWNLOAD_ARCH=amd64 for LXC_NAME in "$@" do LXC_PATH="/var/lib/lxc/$LXC_NAME" LXC_ROOTFS="$LXC_PATH/rootfs" # Create the container. lxc-create -t download -n "$LXC_NAME" -- \ -d "$DOWNLOAD_DIST" -r "$DOWNLOAD_RELEASE" -a "$DOWNLOAD_ARCH" cat <<EOF >> "$LXC_PATH/config" # Allow running Docker inside LXC lxc.aa_profile = unconfined lxc.cap.drop = EOF # Start the container and wait for networking to start. lxc-start -n "$LXC_NAME" sleep 10s # Install Docker. lxc-attach -n "$LXC_NAME" -- apt-get update lxc-attach -n "$LXC_NAME" -- apt-get install -y docker.io # Tell Docker to listen on all interfaces. sed -i -e 's/^#DOCKER_OPTS=.*$/DOCKER_OPTS="-H 0.0.0.0:2375"/' "$LXC_ROOTFS/etc/default/docker" lxc-attach -n "$LXC_NAME" -- systemctl restart docker # Join the Swarm. SWARM_AGENT_IP="$(lxc-attach -n "$LXC_NAME" -- ifconfig eth0 | grep -Po '(?<=inet addr:)\S+')" lxc-attach -n "$LXC_NAME" -- docker run -d --restart=always --name=swarm \ swarm join --addr="$SWARM_AGENT_IP:2375" "etcd://$SWARM_MANAGER_IP:2379" done </code></pre></div> Be sure to change the values for <code>SWARM_MANAGER_IP</code>, <code>DOWNLOAD_DIST</code>, <code>DOWNLOAD_RELEASE</code> and <code>DOWNLOAD_ARCH</code> to fit your needs. Thanks to this script, creating 10 new agents is as simple as running one command: <div class="highlight"><pre><code>root@host:~# ./swarm-agent-create swarm-agent-{0..9} </code></pre></div> Here’s an explanation of what the script does: <ul> <li> It first sets up a new LXC container following steps 1-5 above, that is: create a new LXC container (with <code>lxc-create</code>), apply the LXC configuration (<code>lxc.aa_profile</code> and <code>lxc.cap.drop</code> rules), start the container and install Docker. <div class="highlight"><pre><code>LXC_PATH="/var/lib/lxc/$LXC_NAME" LXC_ROOTFS="$LXC_PATH/rootfs" # Create the container. lxc-create -t download -n "$LXC_NAME" -- \ -d "$DOWNLOAD_DIST" -r "$DOWNLOAD_RELEASE" -a "$DOWNLOAD_ARCH" cat <<EOF >> "$LXC_PATH/config" # Allow running Docker inside LXC lxc.aa_profile = unconfined lxc.cap.drop = EOF # Start the container and wait for networking to start. lxc-start -n "$LXC_NAME" sleep 10s # Install Docker. lxc-attach -n "$LXC_NAME" -- apt-get update lxc-attach -n "$LXC_NAME" -- apt-get install -y docker.io </code></pre></div> </li> <li> Our Swarm agents need to be reachable by the manager. For this reason we need to configure them so that they bind to a public interface. To do so, the script adds <code>DOCKER_OPTS="-H 0.0.0.0:2375"</code> and restarts Docker. <div class="highlight"><pre><code># Tell Docker to listen on all interfaces. sed -i -e 's/^#DOCKER_OPTS=.*$/DOCKER_OPTS="-H 0.0.0.0:2375"/' "$LXC_ROOTFS/etc/default/docker" lxc-attach -n "$LXC_NAME" -- systemctl restart docker </code></pre></div> </li> <li> Lastly, the script checks the IP address for the LXC container and it launches Swarm. <div class="highlight"><pre><code># Join the Swarm. SWARM_AGENT_IP="$(lxc-attach -n "$LXC_NAME" -- ifconfig eth0 | grep -Po '(?<=inet addr:)\S+')" lxc-attach -n "$LXC_NAME" -- docker run -d --restart=always --name=swarm \ swarm join --addr="$SWARM_AGENT_IP:2375" "etcd://$SWARM_MANAGER_IP:2379" </code></pre></div> </li> </ul> <h1 id="step-8">Step 8: Play with the Swarm</h1> Now, if we check <code>docker info</code> on the Swarm manager, we should see 10 healthy nodes: <div class="highlight"><pre><code>root@swarm-manager:~# docker -H localhost:3375 info Containers: 10 Running: 10 Paused: 0 Stopped: 0 Images: 10 Server Version: swarm/1.1.3 Role: primary Strategy: spread Filters: health, port, dependency, affinity, constraint Nodes: 10 swarm-agent-0: 10.0.3.73:2375 └ Status: Healthy └ Containers: 1 └ Reserved CPUs: 0 / 4 └ Reserved Memory: 0 B / 4.052 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.4.0-15-generic, operatingsystem=Ubuntu 16.04, storagedriver=overlay └ Error: (none) └ UpdatedAt: 2016-04-13T15:32:35Z swarm-agent-1: 10.0.3.97:2375 └ Status: Healthy └ Containers: 1 └ Reserved CPUs: 0 / 4 └ Reserved Memory: 0 B / 4.052 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.4.0-15-generic, operatingsystem=Ubuntu 16.04, storagedriver=overlay └ Error: (none) └ UpdatedAt: 2016-04-13T15:31:49Z swarm-agent-2: 10.0.3.58:2375 └ Status: Healthy └ Containers: 1 └ Reserved CPUs: 0 / 4 └ Reserved Memory: 0 B / 4.052 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.4.0-15-generic, operatingsystem=Ubuntu 16.04, storagedriver=overlay └ Error: (none) └ UpdatedAt: 2016-04-13T15:31:54Z swarm-agent-3: 10.0.3.195:2375 └ Status: Healthy └ Containers: 1 └ Reserved CPUs: 0 / 4 └ Reserved Memory: 0 B / 4.052 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.4.0-15-generic, operatingsystem=Ubuntu 16.04, storagedriver=overlay └ Error: (none) └ UpdatedAt: 2016-04-13T15:32:03Z swarm-agent-4: 10.0.3.235:2375 └ Status: Healthy └ Containers: 1 └ Reserved CPUs: 0 / 4 └ Reserved Memory: 0 B / 4.052 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.4.0-15-generic, operatingsystem=Ubuntu 16.04, storagedriver=overlay └ Error: (none) └ UpdatedAt: 2016-04-13T15:32:22Z swarm-agent-5: 10.0.3.174:2375 └ Status: Healthy └ Containers: 1 └ Reserved CPUs: 0 / 4 └ Reserved Memory: 0 B / 4.052 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.4.0-15-generic, operatingsystem=Ubuntu 16.04, storagedriver=overlay └ Error: (none) └ UpdatedAt: 2016-04-13T15:32:16Z swarm-agent-6: 10.0.3.222:2375 └ Status: Healthy └ Containers: 1 └ Reserved CPUs: 0 / 4 └ Reserved Memory: 0 B / 4.052 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.4.0-15-generic, operatingsystem=Ubuntu 16.04, storagedriver=overlay └ Error: (none) └ UpdatedAt: 2016-04-13T15:32:21Z swarm-agent-7: 10.0.3.140:2375 └ Status: Healthy └ Containers: 1 └ Reserved CPUs: 0 / 4 └ Reserved Memory: 0 B / 4.052 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.4.0-15-generic, operatingsystem=Ubuntu 16.04, storagedriver=overlay └ Error: (none) └ UpdatedAt: 2016-04-13T15:31:43Z swarm-agent-8: 10.0.3.95:2375 └ Status: Healthy └ Containers: 1 └ Reserved CPUs: 0 / 4 └ Reserved Memory: 0 B / 4.052 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.4.0-15-generic, operatingsystem=Ubuntu 16.04, storagedriver=overlay └ Error: (none) └ UpdatedAt: 2016-04-13T15:32:17Z swarm-agent-9: 10.0.3.125:2375 └ Status: Healthy └ Containers: 1 └ Reserved CPUs: 0 / 4 └ Reserved Memory: 0 B / 4.052 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.4.0-15-generic, operatingsystem=Ubuntu 16.04, storagedriver=overlay └ Error: (none) └ UpdatedAt: 2016-04-13T15:32:30Z Plugins: Volume: Network: Kernel Version: 4.4.0-15-generic Operating System: linux Architecture: amd64 CPUs: 40 Total Memory: 40.52 GiB Name: d39c33295ef3 </code></pre></div> Let’s try running a command on the Swarm: <div class="highlight"><pre><code>root@swarm-manager:~# docker -H localhost:3375 run -i --rm docker/whalesay cowsay 'It works!' ___________ < It works! > ----------- \ \ \ ## . ## ## ## == ## ## ## ## === /""""""""""""""""___/ === ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~ \______ o __/ \ \ __/ \____\______/ </code></pre></div> <h1 id="conclusion">Conclusion</h1> We created a Swarm cluster consisting of one manager and 10 agents, and we kept memory and disk usage low thanks to LXC containers. We also succeeded in confining our Docker containers with AppArmor. Overall, this setup is probably not ideal for use in a production environment, but very useful for simulating clusters on your laptop. I hope you enjoyed the tutorial. Feel free to leave a comment if you have questions!andreacorbelliniWed, 13 Apr 2016 18:00:00 +0000tag:andrea.corbellini.blog,2016-04-13:/2016/04/13/docker-swarm-inside-lxc/information-technologydockerswarmlxccontainersdistributed-computingWhen bureaucracy hits the web: the cookie lawhttps://andrea.corbellini.blog/2015/09/22/cookie-law/For a few years now, every first of April I hoped to read between the news something on the lines of “the cookie law was a joke, sorry for that”. You know, bureaucracy is slow, and it’s reasonable to think that it takes time for them to reveal jokes. Yet, many firsts of April have passed, and no such announcement has been made. Many missed opportunities for Europe to show their love for progress and their competence with the web. Being compliant with the EU cookie law is hard to do. It’s not just a matter of showing a boring banner, it’s a matter of defacing your web pages, writing long privacy policies that nobody will read, implementing ways to prevent certain cookies from being set. The truth is: if you, as a webmaster, want to avoid wasting time and avoid headaches, you just have to avoid cookies. This is what I have done with most websites I maintain: I have removed all analytics, all social sharing buttons, all YouTube videos, all comments. This was a sad thing to do, but it was the only thing I could do: I maintain websites for free mainly as a favor for friends and no-profits I’m involved with — it’s not my day job. Also, I do not want other people being sued because of mistakes from my side: cookies may be set in the most unexpected situations and disabling every feature that could potentially set them seems the safest choice. The only exception is this blog. Here, I use cookies for Google Analytics, for social sharing buttons and for Disqus. I may live without Google Analytics (even though it gives useful insights, such as performance statistics and tips), but I can’t really remove social buttons and Disqus: this is a blog and it wouldn’t make any sense to remove social features and comments. Being compliant with the EU cookie law has been on my todo list for a while, and I never found the time (nor the desire) to look into it. Today I did. I spent a few hours of my time to discover that Google Analytics is “OK” (in the sense that I do not have to display an ugly banner, nor have to ask for explicit permission from the user before setting the cookies) and to discover that social buttons and Disqus are “bad” (in the sense that I have to display a banner and ask for explicit consent from the user before setting the cookies). In the end, the only service that I could remove is the less problematic service. As I said, I really do not want to remove social buttons, Disqus or whatever third-party content I’ll want to display in the future. Therefore, in order to comply with the cookie law, I’m forced to write code, write a privacy policy, waste another bunch of hours of my time. But not today, as I’ve already had enough sense of sadness and impotence. At least for now, I guess that the EU cookie law compliance will stay on my todo list for some more time. Probably if I worked on compliance instead of writing this rant, I could have already finished (but then what’s the point of having a blog if you don’t blog?) The cookie law wants to be “on the side of the users,” and it is based on noble principles: it wants users to be well-informed about how their data is used and by whom. However, as it is today, it’s against both users and webmasters. Webmasters have to lose their time working on compliance, and users receive a degraded experience due to silly regulations. I’d like to do <a href="http://nocookielaw.com/">what Silktide did</a>: actively protesting against the law, but I wouldn’t be so happy if I were sued. I’d like to read “the cookie law was a joke” in the news, but I’m starting to believe that it’s not going to happen any time soon. It seems that accepting the sadness of the reality is the only option I’m left with. End of rant, let’s move on.andreacorbelliniTue, 22 Sep 2015 18:35:00 +0000tag:andrea.corbellini.blog,2015-09-22:/2015/09/22/cookie-law/blogblogcookie-lawHello Pelican!https://andrea.corbellini.blog/2015/08/02/hello-pelican/Today I switched from WordPress.com to <a href="http://getpelican.com/">Pelican</a> and <a href="https://pages.github.com/">GitHub Pages</a>. First off, let me say: almost all URLs that were previously working should still work. Only the feed URLs are broken, and this is not something I can fix. If you were following my blog via a feed reader, you should update to the new feed. Sorry for the inconvenience. Having said that, I’d like to share with you the motivation that made me move and the details of the migration. <h1 id="the-bad-things-of-wordpress">The bad things of WordPress</h1> Now, this doesn’t want to be a rant, so I’ll be pretty concise. WordPress, the content management system, is an excellent platform for blogging. Easy to start with, easy to maintain, easy to use. WordPress.com makes things even easier. It also comes with many useful features, like comments and social networks integration. The problem is: you can’t customize things or add features without paying. Of course, this is business, and I do not want to discuss business decisions made at WordPress.com. Not only that, but I could live fine with most of the major limitations. Also, I was perfectly conscious of this kind of problems with WordPress.com when I started (after all, this is not <a href="https://andrea.corbellini.blog/2015/02/15/new-blog-again/">the first blog I started</a>). I actually become upset of WordPress.com when writing the series of blog posts about <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/">Elliptic Curve Cryptography</a>. When writing these articles, I spent a lot of time employing workarounds to overcome WordPress.com limitations. Being used to Vim and its advanced features, I also found the editors (both the old and the new one) as a great obstacle for getting things done quickly. I do not want to enter the details of the problems I’m referring to, what matters is that, eventually, I gave up and I realized it was time to move on and seek for an alternative. <h1 id="why-pelican">Why Pelican</h1> Pelican is a static site generator. I’ve always thought that a static site had too many limitations for me. But while seeking an alternative to WordPress.com, I realized that many of those limitations were not affecting me in any way. Actually, with a static site I can do everything I want: edit my articles with Vim, render my equations with MathJax, customize my theme, version control my content, write scripts to post process my content. The only bad thing about Pelican is that it does not come with any theme I truly like. I decided to make my own. I’m not entirely satisfied with it, as I feel it is too “anonymous”, but I believe it is fully responsive, fast, readable and offers all the features I want. Perhaps I’ll tweak it a little more to make it more “personal”. Setting up Pelican and migrating everything required some time, but at least this time I worked on true solutions, not on ugly hacks and workarounds like I did with WordPress. This implies that when writing articles I will be able to focus more on content than other details. <h1 id="why-not-other-static-site-generators">Why not other static site generators</h1> In short: Pelican is written in Python and to my eyes it looked better than the other Python static site generators. I’ll be honest and say that I did not truly evaluate all of the alternatives: I knew <a href="...">list.org</a> switched to Pelican and that made me try Pelican before all other solutions. <h1 id="conclusion">Conclusion</h1> In the end I decided to leave WordPress for Pelican hosted on GitHub Pages. I’m pretty satisfied with the result I got. The nature of GitHub Pages prevents me from using HTTP redirects (and therefore the old feed links are broken), however in exchange I’ve got much more freedom, and this is what matters to me.andreacorbelliniSun, 02 Aug 2015 18:55:00 +0000tag:andrea.corbellini.blog,2015-08-02:/2015/08/02/hello-pelican/blogblogpelicanwordpressLet's Encrypt is going to start soonhttps://andrea.corbellini.blog/2015/06/16/lets-encrypt-is-going-to-start-soon/<a href="https://letsencrypt.org/">Let’s Encrypt</a> (the free, automated and open certificate authority) has just <a href="https://letsencrypt.org/2015/06/16/lets-encrypt-launch-schedule.html">announced its launch schedule</a>. According to it, certificates will be released to the public starting from the week of September 14, 2015. Their intermediate certificates, which <a href="https://letsencrypt.org/2015/06/04/isrg-ca-certs.html">were generated a few days ago</a>, will be signed by <a href="https://www.identrustssl.com/">IdenTrust</a>. What this means is that if you browse a web page secured by Let’s Encrypt, you won’t get any scary message, but the usual green lock. <figure> <img src="https://andrea.corbellini.blog/images/green-lock.png" alt="Green lock" width="612" height="188"> <figcaption>You will see this...</figcaption> </figure> <figure> <img src="https://andrea.corbellini.blog/images/red-lock.png" alt="Red lock" width="612" height="300"> <figcaption>... not this.</figcaption> </figure> In case you are curious: the root certificate is a 4096-bit RSA key, the two intermediate certificates are both 2048-bit RSA keys. But they are also <a href="https://letsencrypt.org/certificates/">planning to generate ECDSA keys later this year</a> as well. Technical aspects aside, this will be a great opportunity for the entire web. As I have <a href="https://andrea.corbellini.blog/2015/04/12/lets-encrypt-the-road-towards-a-better-web/">already written</a>, I always dreamed of an encrypted web, and I truly believe that Let’s Encrypt — or at least its approach to the problem — is the way to go. So, will you get a Let’s Encrypt certificate when the time comes? I will do. Not for this blog (I can’t put a certificate without paying), but for other websites I manage. Perhaps I’ll also show a “Proudly secured by Let’s Encrypt” badge.andreacorbelliniTue, 16 Jun 2015 18:20:00 +0000tag:andrea.corbellini.blog,2015-06-16:/2015/06/16/lets-encrypt-is-going-to-start-soon/cryptographyecdsalet's encryptrsasecuritytlsElliptic Curve Cryptography: breaking security and a comparison with RSAhttps://andrea.corbellini.blog/2015/06/08/elliptic-curve-cryptography-breaking-security-and-a-comparison-with-rsa/This post is the fourth and last in the series <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/">ECC: a gentle introduction</a>. In the <a href="https://andrea.corbellini.blog/2015/05/30/elliptic-curve-cryptography-ecdh-and-ecdsa/">last post</a> we have seen two algorithms, ECDH and ECDSA, and we have seen how the discrete logarithm problem for elliptic curves plays an important role for their security. But, if you remember, we said that <a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/#discrete-logarithm">we have no mathematical proofs</a> for the complexity of the discrete logarithm problem: we believe it to be “hard”, but we can’t be sure. In the first part of this post, we’ll try to get an idea of how “hard” it is in practice with today’s techniques. Then, in the second part, we will try to answer the question: why do we need elliptic curve cryptography if RSA (and the other cryptosystems based on modular arithmetic) work well? <h1 id="breaking-the-discrete-logarithm-problem">Breaking the discrete logarithm problem</h1> We will now see the two most efficient algorithms for computing discrete logarithms on elliptic curve: the baby-step, giant-step algorithm, and Pollard’s rho method. Before starting, as a reminder, here is what the discrete logarithm problem is about: given two points $P$ and $Q$ find out the integer $x$ that satisfies the equation $Q = xP$. The points belong to a subgroup of an elliptic curve, which has a base point $G$ and which order is $n$. <h2 id="baby-step-giant-step">Baby-step, giant-step</h2> Before entering the details of the algorithm, a quick consideration: we can always write any integer $x$ as $x = am + b$, where $a$, $m$ and $b$ are three arbitrary integers. For example, we can write $10 = 2 \cdot 3 + 4$. With this in mind, we can rewrite the equation for the discrete logarithm problem as follows: $$\begin{align*} Q & = xP \\ Q & = (am + b) P \\ Q & = am P + b P \\ Q - am P & = b P \end{align*}$$ The baby-step giant-step is a “meet in the middle” algorithm. Contrary to the brute-force attack (which forces us to calculate all the points $xP$ for every $x$ until we find $Q$), we will calculate “few” values for $bP$ and “few” values for $Q - amP$ until we find a correspondence. The algorithm works as follows: <ol> <li>Calculate $m = \left\lceil{\sqrt{n}}\right\rceil$</li> <li>For every $b$ in ${0, \dots, m}$, calculate $bP$ and store the results in a hash table.</li> <li>For every $a$ in ${0, \dots, m}$:<ol> <li>calculate $amP$;</li> <li>calculate $Q - amP$;</li> <li>check the hash table and look if there exist a point $bP$ such that $Q - amP = bP$;</li> <li>if such point exists, then we have found $x = am + b$.</li> </ol> </li> </ol> As you can see, initially we calculate the points $bP$ with little (i.e. “baby”) increments for the coefficient $b$ ($1P$, $2P$, $3P$, …). Then, in the second part of the algorithm, we calculate the points $amP$ with huge (i.e. “giant”) increments for $am$ ($1mP$, $2mP$, $3mP$, …, where $m$ is a huge number). <figure> <img src="https://andrea.corbellini.blog/images/baby-step-giant-step.gif" alt="Baby-step, giant-step" width="310" height="346"> <figcaption>The baby-step, giant-step algorithm: initially we calculate few points via small steps and store them in a hash table. Then we perform the giant steps and compare the new points with the points in the hash table. Once a match is found, calculating the discrete logarithm is a matter of rearranging terms.</figcaption> </figure> To understand why this algorithm works, forget for a moment that the points $bP$ are cached and take the equation $Q = amP + bP$. Consider what follows: <ul> <li>When $a = 0$ we are checking whether $Q$ is equal to $bP$, where $b$ is one of the integers from 0 to $m$. This way, we are comparing $Q$ against all points from $0P$ to $mP$.</li> <li>When $a = 1$ we are checking whether $Q$ is equal to $mP + bP$. We are comparing $Q$ against all points from $mP$ to $2mP$.</li> <li>When $a = 2$ we are comparing $Q$ against all the points from $2mP$ to $3mP$.</li> <li>…</li> <li>When $a = m - 1$, we are comparing $Q$ against all points from $(m - 1)mP$ to $m^2 P = nP$.</li> </ul> In conclusion, we are checking all points from $0P$ to $nP$ (that is, all the possible points) performing at most $2m$ additions and multiplications (exactly $m$ for the baby steps, at most $m$ for the giant steps). If you consider that a lookup on a hash table takes $O(1)$ time, it’s easy to see that this algorithm has both time and space complexity $O(\sqrt{n})$ (or $O(2^{k / 2})$ if you consider the bit length). It’s still exponential time, but much better than a brute-force attack. <h3 id="baby-step-giant-step-in-practice">Baby-step giant-step in practice</h3> It may make sense to see what the complexity $O(\sqrt{n})$ means in practice. Let’s take a standardized curve: <code>prime192v1</code> (aka <code>secp192r1</code>, <code>ansiX9p192r1</code>). This curve has order $n$ = 0xffffffff ffffffff ffffffff 99def836 146bc9b1 b4d22831. The square root of $n$ is approximately 7.922816251426434 · 1028 (almost eighty octillions). Now imagine storing $\sqrt{n}$ points in a hash table. Suppose that each point requires exactly 32 bytes: our hash table would need approximately 2.5 · 1030 bytes of memory. <a href="http://www.csc.com/big_data/flxwd/83638-big_data_just_beginning_to_explode_interactive_infographic">Looking on the web</a>, it seems that the total world storage capacity is in the order of the zettabyte (1021 bytes). This is almost ten orders of magnitude lower than the memory required by our hash table! Even if our points took 1 byte each, we would be still very far from being able to store all of them. This is impressive, and is even more impressive if you consider that <code>prime192v1</code> is one of the curves with the lowest order. The order of <code>secp521r1</code> (another standard curve from NIST) is approximately 6.9 · 10156! <h3 id="playing-with-baby-step-giant-step">Playing with baby-step giant-step</h3> I made <a href="https://github.com/andreacorbellini/ecc/blob/master/logs/babygiantstep.py">a Python script</a> that computes discrete logarithms using the baby-step giant-step algorithm. Obviously it only works with curves with small orders: don’t try it with <code>secp521r1</code>, unless you want to receive a <code>MemoryError</code>. It should produce an output like this: <div class="highlight"><pre><code>Curve: y^2 = (x^3 + 1x - 1) mod 10177 Curve order: 10331 p = (0x1, 0x1) q = (0x1a28, 0x8fb) 325 * p = q log(p, q) = 325 Took 105 steps </code></pre></div> <h2 id="pollards">Pollard’s ρ</h2> Pollard’s rho is another algorithm for computing discrete logarithms. It has the same asymptotic time complexity $O(\sqrt{n})$ of the baby-step giant-step algorithm, but its space complexity is just $O(1)$. If baby-step giant-step can’t solve discrete logarithms because of the huge memory requirements, will Pollard’s rho make it? Let’s see… First of all, another reminder of the discrete logarithm problem: given $P$ and $Q$ find $x$ such that $Q = xP$. With Pollard’s rho, we will solve a sightly different problem: given $P$ and $Q$, find the integers $a$, $b$, $A$ and $B$ such that $aP + bQ = AP + BQ$. Once the four integers are found, we can use the equation $Q = xP$ to find out $x$: $$\begin{align*} aP + bQ & = AP + BQ \\ aP + bxP & = AP + BxP \\ (a + bx) P & = (A + Bx) P \\ (a - A) P & = (B - b) xP \end{align*}$$ Now we can get rid of $P$. But before doing so, remember that our subgroup is cyclic with order $n$, therefore the coefficients used in point multiplication are modulo $n$: $$\begin{align*} a - A & \equiv (B - b) x \pmod{n} \\ x & = (a - A)(B - b)^{-1} \bmod{n} \end{align*}$$ The principle of operation of Pollard’s rho is simple: we generate a pseudo-random sequence of points $X_1$, $X_2$, … where each $X = a_i P + b_i Q$. The sequence can be generated using a pseudo-random function $f$ like this: $$(a_{i + 1}, b_{i + 1}) = f(X_i)$$ That is: the pseudo-random function $f$ takes the latest point $X_i$ in the sequence as the input, and gives the coefficients $a_{i + 1}$ and $b_{i + 1}$ as the output. From there, we can calculate $X_{i + 1} = a_{i + 1} P + b_{i + 1} Q$; we can then input $X_{i + 1}$ into $f$ again and repeat. It doesn’t really matter how $f$ works internally (although certain functions may yield results faster than others), what matters is that $f$ determines the next point in the sequence based on the previous one, and that all the $a_i$ and $b_i$ coefficients are known by us. By using such $f$, sooner or later we will see a loop in our sequence. That is, we will see a point $X_j = X_i$. <figure> <img src="https://andrea.corbellini.blog/images/pollard-rho.png" alt="Pollard's rho cycle visualization" width="300" height="270"> <figcaption>A visualization of what a cycle in the sequence might look like: have some initial points ($X_0$, $X_1$, $X_2$), and then the cycle itself, formed by the points $X_3$ to $X_8$. After that, $X_9 = X_3$, $X_{10} = X_4$ and so on. This picture resembles the Greek letter ρ (rho), hence the name.</figcaption> </figure> The reason why we must see the cycle is simple: the number of points is finite, hence they must repeat sooner or later. Once we see where the cycle is, we can use the equations above to figure out the discrete logarithm. The problem now is: how do we detect the cycle in an efficient way? <h3 id="tortoise-and-hare">Tortoise and Hare</h3> To detect cycles, we have an efficient method: the tortoise and hare algorithm (also known as Floyd’s cycle-finding algorithm). The picture below shows the principle of operation of the tortoise and hare method, which is at the core of Pollard’s rho. <figure> <img src="https://andrea.corbellini.blog/images/tortoise-hare.gif" alt="Tortoise and Hare" width="650" height="101"> <figcaption>We have the curve $y^2 \equiv x^3 + 2x + 3 \pmod{97}$ and the points $P = (3, 6)$ and $Q = (80, 87)$. The points belong to a cyclic subgroup of order 5. We walk a sequence of pairs at different speeds until we find two different pairs $(a, b)$ and $(A, B)$ that produce the same point. In this case, we have found the pairs $(3, 3)$ and $(2, 0)$ that allow us to calculate the logarithm as $x = (3 - 2)(0 - 3)^{-1} \bmod{5} = 3$. And in fact we correctly have $Q = 3P$.</figcaption> </figure> We take two pets, the tortoise and the hare, and make them walk our sequence of points from left to right. The tortoise (the green spot in the picture) is slow and reads each point one by one; the hare (represented in red) is fast and skips a point at every step. After some time both the tortoise and the hare will have found the same point, but with different coefficient pairs. Or, to express that with equations, the tortoise will have found a pair $(a, b)$ and the hare will have found a pair $(A, B)$ such that $aP + bQ = AP + BQ$. It’s easy to see that this algorithm requires constant memory ($O(1)$ space complexity). Calculating the asymptotic time complexity is not that easy, but we can build a probabilistic proof that shows how the time complexity is $O(\sqrt{n})$, as we have already said. The proof is based on the “<a href="https://en.wikipedia.org/wiki/Birthday_problem">birthday paradox</a>”, which is about the probability of two people having the same birthday, where here we are concerned about the probability of two $(a, b)$ pairs yielding the same point. <h3 id="playing-with-pollards">Playing with Pollard’s ρ</h3> I’ve built <a href="https://github.com/andreacorbellini/ecc/blob/master/logs/pollardsrho.py">a Python script</a> that computes discrete logarithms using Pollard’s rho. It is not the implementation of the original Pollard’s rho, but a slight variation of it (I’ve used a more efficient method for generating the pseudo-random sequence of pairs). The script contains some useful comments, so read it if you are interested in the details of the algorithm. This script, like the baby-step giant-step one, works on a tiny curve, and produces the same kind of output. <h3 id="pollards-in-practice">Pollard’s ρ in practice</h3> We said that baby-step giant-step can’t be used in practice, because of the huge memory requirements. Pollard’s rho, on the other hand, requires very few memory. So, how practical is it? Certicom launched a <a href="https://www.certicom.com/index.php/the-certicom-ecc-challenge">challenge</a> in 1998 to compute discrete logarithms on elliptic curves with bit lengths ranging from 109 to 359. As of today, only 109-bit long curves have been successfully broken. The latest successful attempt was made in 2004. Quoting <a href="http://en.wikipedia.org/wiki/Discrete_logarithm_records">Wikipedia</a>: <blockquote> The prize was awarded on 8 April 2004 to a group of about 2600 people represented by Chris Monico. They also used a version of a parallelized Pollard rho method, taking 17 months of calendar time. </blockquote> As we have already said, <code>prime192v1</code> is one of the “smallest” elliptic curves. We also said that Pollard’s rho has $O(\sqrt{n})$ time complexity. If we used the same technique as Chris Monico (the same algorithm, on the same hardware, with the same number of machines), how much would it take to compute a logarithm on <code>prime192v1</code>? $$17\ \text{months}\ \times \frac{\sqrt{2^{192}}}{\sqrt{2^{109}}} \approx 5 \cdot 10^{13}\ \text{months}$$ This number is pretty self-explanatory and gives a clear idea of how hard it can be to break a discrete logarithm using such techniques. <h2 id="pollards-vs-baby-step-giant-step">Pollard’s ρ vs Baby-step giant-step</h2> I decided to put the <a href="https://github.com/andreacorbellini/ecc/blob/master/logs/babygiantstep.py">baby-step giant-step script</a> and the <a href="https://github.com/andreacorbellini/ecc/blob/master/logs/pollardsrho.py">Pollard’s rho script</a> together with a <a href="https://github.com/andreacorbellini/ecc/blob/master/logs/bruteforce.py">brute-force script</a> into a <a href="https://github.com/andreacorbellini/ecc/blob/master/logs/comparelogs.py">fourth script</a> to compare their performances. This fourth script computes all the logarithms for all the points on the “tiny” curve using different algorithms and reports how much time it did take: <div class="highlight"><pre><code>Curve order: 10331 Using bruteforce Computing all logarithms: 100.00% done Took 2m 31s (5193 steps on average) Using babygiantstep Computing all logarithms: 100.00% done Took 0m 6s (152 steps on average) Using pollardsrho Computing all logarithms: 100.00% done Took 0m 21s (138 steps on average) </code></pre></div> As we could expect, the brute-force method is tremendously slow if compared to the others two. Baby-step giant-step is the faster, while Pollard’s rho is more than three times slower than baby-step giant-step (although it uses far less memory and fewer number of steps on average). Also look at the number of steps: brute force used 5193 steps on average for computing each logarithm. 5193 is very near to 10331 / 2 (half the curve order). Baby-step giant-steps and Pollard’s rho used 152 steps and 138 steps respectively, two numbers very close to the square root of 10331 (101.64). <h2 id="final-consideration">Final consideration</h2> While discussing these algorithms, I have presented many numbers. It’s important to be cautious when reading them: algorithms can be greatly optimized in many ways. Hardware can improve. Specialized hardware can be built. The fact that an approach today seems impractical, does not imply that the approach can’t be improved. It also does not imply that other, better approaches exist (remember, once again, that we have no proofs for the complexity of the discrete logarithm problem). <h1 id="shors-algorithm">Shor’s algorithm</h1> If today’s techniques are unsuitable, what about tomorrow’s techniques? Well, things are a bit more worrisome: there exist a <a href="https://en.wikipedia.org/wiki/Quantum_algorithm">quantum algorithm</a> capable of computing discrete logarithms in polynomial time: <a href="https://en.wikipedia.org/wiki/Shor%27s_algorithm">Shor’s algorithm</a>, which has time complexity $O((\log n)^3)$ and space complexity $O(\log n)$. Quantum computers are still far from becoming sophisticated enough to run algorithms like Shor’s, still the need for <a href="https://en.wikipedia.org/wiki/Post-quantum_cryptography">quantum-resistant algorithms</a> may be something worth investigating now. What we encrypt today might not be safe tomorrow. <h1 id="ecc-and-rsa">ECC and RSA</h1> Now let’s forget about quantum computing, which is still far from being a serious problem. The question I’ll answer now is: why bothering with elliptic curves if RSA works well? A quick answer is given by NIST, which provides with <a href="https://www.nsa.gov/business/programs/elliptic_curve.shtml">a table that compares RSA and ECC key sizes</a> required to achieve the same level of security. <table class="table"> <thead> <tr><th>RSA key size (bits)</th><th>ECC key size (bits)</th></tr> </thead> <tbody> <tr><td>1024</td><td>160</td></tr> <tr><td>2048</td><td>224</td></tr> <tr><td>3072</td><td>256</td></tr> <tr><td>7680</td><td>384</td></tr> <tr><td>15360</td><td>521</td></tr> </tbody> </table> Note that there is no linear relationship between the RSA key sizes and the ECC key sizes (in other words: if we double the RSA key size, we don’t have to double the ECC key size). This table tells us not only that ECC uses less memory, but also that key generation and signing are considerably faster. But why is it so? The answer is that the faster algorithms for computing discrete logarithms over elliptic curves are Pollard’s rho and baby-step giant-step, while in the case of RSA we have faster algorithms. One in particular is the <a href="https://en.wikipedia.org/wiki/General_number_field_sieve">general number field sieve</a>: an algorithm for integer factorization that can be used to compute discrete logarithms. The general number field sieve is the fastest algorithm for integer factorization to date. All of this applies to other cryptosystems based on modular arithmetic as well, including DSA, D-H and ElGamal. <h1 id="hidden-threats-of-nsa">Hidden threats of NSA</h1> An now the hard part. So far we have discussed algorithms and mathematics. Now it’s time to discuss people, and things get more complicated. If you remember, in the last post we said that certain classes of elliptic curves are weak, and to solve the problem of trusting curves from dubious sources we added a random seed to our domain parameters. And if we look at standard curves from NIST we can see that they are all verifiably random. If we read the Wikipedia page for “<a href="http://en.wikipedia.org/wiki/Nothing_up_my_sleeve_number">nothing up my sleeve</a>”, we can see that: <ul> <li>The random numbers for MD5 come from the sine of integers.</li> <li>The random numbers for Blowfish come from the first digits of $\pi$.</li> <li>The random numbers for RC5 come from both $e$ and the golden ratio.</li> </ul> These numbers are random because their digits are uniformly distributed. And they are also unsuspicious, because they have a justification. Now the question is: where do the random seeds for NIST curves come from? The answer is, sadly: we don’t know. Those seeds have no justification at all. Is it possible that NIST has discovered a “sufficiently large” class of weak elliptic curves and has tried many possible seeds until they found a vulnerable curve? I can’t answer this question, but this is a legit and important question. We know that NIST has succeeded in standardizing at least a <a href="http://en.wikipedia.org/wiki/Dual_EC_DRBG">vulnerable random number generator</a> (a generator which, oddly enough, is based on elliptic curves). Perhaps they also succeeded in standardizing a set of weak elliptic curves. How do we know? We can’t. What’s important to understand is that “verifiably random” and “secure” are not synonyms. And it doesn’t matter how hard the logarithm problem is, or how long our keys are, if our algorithms are broken, there’s nothing we can do. With respect to this, RSA wins, as it does not require special domain parameters that can be tampered. RSA (as well as other modular arithmetic systems) may be a good alternative if we can’t trust authorities and if we can’t construct our own domain parameters. And in case you are asking: yes, TLS may use NIST curves. If you check <a href="https://google.com/">https://google.com</a>, you’ll see that the connection is using ECDHE and ECDSA, with a certificate based on <code>prime256v1</code> (aka <code>secp256p1</code>). <h1 id="thats-all">That’s all!</h1> I hope you have enjoyed this series. My aim was to give you the basic knowledge, terminology and conventions to understand what elliptic curve cryptography today is. If I reached my aim, you should now be able to understand existing ECC-based cryptosystems and to expand your knowledge by reading “not so gentle” documentation. When writing this series, I could have skipped over many details and use a simpler terminology, but I felt that by doing so you would have not been able to understand what the web has to offer. I believe I have found a good compromise between simplicity and completeness. Note though that by reading just this series, you are not able to implement secure ECC cryptosystems: security requires us to know many subtle but important details. Remember the <a href="https://andrea.corbellini.blog/2015/05/30/elliptic-curve-cryptography-ecdh-and-ecdsa/#random-curves">requirements for Smart’s attack</a> and <a href="https://andrea.corbellini.blog/2015/05/30/elliptic-curve-cryptography-ecdh-and-ecdsa/#ecdsa-k">Sony’s mistake</a> — these are just two examples that should teach you how easy is to produce insecure algorithms and how easy it is to exploit them. So, if you are interested in diving deeper into the world of ECC, where to go from here? First off, so far we have seen Weierstrass curves over prime fields, but you must know that there exist other kinds of curve and fields, in particular: <ul> <li>Koblitz curves over binary fields. Those are elliptic curves in the form $y^2 + xy = x^3 + ax^2 + 1$ (where $a$ is either 0 or 1) over finite fields containing $2^m$ elements (where $m$ is a prime). They allow particularly efficient point additions and scalar multiplications. Examples of standardized Koblitz curves are <code>nistk163</code>, <code>nistk283</code> and <code>nistk571</code> (three curves defined over a field of 163, 283 and 571 bits).</li> <li>Binary curves. They are very similar to Koblitz curves and are in the form $x^2 + xy = x^3 + x^2 + b$ (where $b$ is an integer often generated from a random seed). As the name suggests, binary curves are restricted to binary fields too. Examples of standardized curves are <code>nistb163</code>, <code>nistb283</code> and <code>nistb571</code>. It must be said that there are growing concerns that both Koblitz and Binary curves may not be as safe as prime curves.</li> <li>Edwards curves, in the form $x^2 + y^2 = 1 + d x^2 y^2$ (where $d$ is either 0 or 1). These are particularly interesting not only because point addition and scalar multiplication are fast, but also because the formula for point addition is always the same, in any case ($P \ne Q$, $P = Q$, $P = -Q$, …). This feature leverages the possibility of side-channel attacks, where you measure the time used for scalar multiplication and try to guess the scalar coefficient based on the time it took to compute. Edwards curves are relatively new (they were presented in 2007) and no authority such as Certicom or NIST have yet standardized any of them.</li> <li>Curve25519 and Ed25519 are two particular elliptic curves designed for ECDH and a variant of ECDSA respectively. Like Edwards curves, these two curves are fast and help preventing side-channel attacks. And like Edwards curves, these two curves have not been standardized yet and we can’t find them in any popular software (except OpenSSH, that supports Ed25519 key pairs since 2014).</li> </ul> If you are interested in the implementation details of ECC, then I suggest you read the sources of OpenSSL and GnuTLS. Finally, if you are interested in the mathematical details, rather than the security and efficiency of the algorithms, you must know that: <ul> <li>Elliptic curves are algebraic varieties with genus one.</li> <li>Points at infinity are studied in projective geometry and can be represented using homogeneous coordinates (although most of the features of projective geometry are not needed for elliptic curve cryptography).</li> </ul> And don’t forget to study finite fields and field theory. These are the keywords that you should look up if you’re interested in the topics. Now the series is officially concluded. Thank you for all your friendly comments, tweets and mails. Many have asked me if I’m going to write other series on other closely related topics. The answer is: maybe. I accept suggestions, but I can’t promise anything. Thanks for reading and see you next time!andreacorbelliniMon, 08 Jun 2015 13:28:00 +0000tag:andrea.corbellini.blog,2015-06-08:/2015/06/08/elliptic-curve-cryptography-breaking-security-and-a-comparison-with-rsa/cryptographydhdsaeccecdhecdheecdsarsasecurityElliptic Curve Cryptography: ECDH and ECDSAhttps://andrea.corbellini.blog/2015/05/30/elliptic-curve-cryptography-ecdh-and-ecdsa/This post is the third in the series <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/">ECC: a gentle introduction</a>. In the previous posts, we have seen <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/#elliptic-curves">what an elliptic curve is</a> and we have defined a <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/#group-law">group law</a> in order to do some math with the points of elliptic curves. Then we have <a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/">restricted elliptic curves to finite fields of integers modulo a prime</a>. With this restriction, we have seen that the points of elliptic curves generate <a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/#scalar-multiplication">cyclic subgroups</a> and we have introduced the terms <a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/#base-point">base point</a>, <a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/#subgroup-order">order</a> and <a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/#cofactor">cofactor</a>. Finally, we have seen that <a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/#scalar-multiplication">scalar multiplication in finite fields</a> is an “easy” problem, while the <a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/#discrete-logarithm">discrete logarithm problem</a> seems to be “hard”. Now we’ll see how all of this applies to cryptography. <h1 id="domain-parameters">Domain parameters</h1> Our elliptic curve algorithms will work in a cyclic subgroup of an elliptic curve over a finite field. Therefore, our algorithms will need the following parameters: <ul> <li>The prime $p$ that specifies the size of the finite field.</li> <li>The coefficients $a$ and $b$ of the elliptic curve equation.</li> <li>The base point $G$ that generates our subgroup.</li> <li>The order $n$ of the subgroup.</li> <li>The cofactor $h$ of the subgroup.</li> </ul> In conclusion, the domain parameters for our algorithms are the sextuple $(p, a, b, G, n, h)$. <h2 id="random-curves">Random curves</h2> When I said that the discrete logarithm problem was “hard”, I wasn’t entirely right. There are some classes of elliptic curves that are particularly weak and allow the use of special purpose algorithms to solve the discrete logarithm problem efficiently. For example, all the curves that have $p = hn$ (that is, the order of the finite field is equal to the order of the elliptic curve) are vulnerable to <a href="http://interact.sagemath.org/edu/2010/414/projects/novotney.pdf">Smart’s attack</a>, which can be used to solve discrete logarithms in polynomial time on a classical computer. Now, suppose that I give you the domain parameters of a curve. There’s the possibility that I’ve discovered a new class of weak curves that nobody knows, and probably I have built a “fast” algorithm for computing discrete logarithms on the curve I gave you. How can I convince you of the contrary, i.e. that I’m not aware of any vulnerability? How can I assure you that the curve is “safe” (in the sense that it can’t be used for special purpose attacks by me)? In an attempt to solve this kind of problem, sometimes we have an additional domain parameter: the seed $S$. This is a random number used to generate the coefficients $a$ and $b$, or the base point $G$, or both. These parameters are generated by computing the hash of the seed $S$. Hashes, as we know, are “easy” to compute, but “hard” to reverse. <figure> <img src="https://andrea.corbellini.blog/images/random-parameters-generation.png" alt="Random curve generation" width="500" height="74"> <figcaption>A simple sketch of how a random curve is generated from a seed: the hash of a random number is used to calculate different parameters of the curve.</figcaption> </figure> <figure> <img src="https://andrea.corbellini.blog/images/seed-inversion.png" alt="Building a seed from a hash" width="359" height="76"> <figcaption>If we wanted to cheat and try to construct a seed from the domain parameters, we would have to solve a "hard" problem: hash inversion.</figcaption> </figure> A curve generated through a seed is said to be verifiably random. The principle of using hashes to generate parameters is known as “<a href="http://en.wikipedia.org/wiki/Nothing_up_my_sleeve_number">nothing up my sleeve</a>”, and is commonly used in cryptography. This trick should give some sort of assurance that the curve has not been specially crafted to expose vulnerabilities known to the author. In fact, if I give you a curve together with a seed, it means I was not free to arbitrarily choose the parameters $a$ and $b$, and you should be relatively sure that the curve cannot be used for special purpose attacks by me. The reason why I say “relatively” will be explained in the next post. A standardized algorithm for generating and checking random curves is described in ANSI X9.62 and is based on <a href="https://en.wikipedia.org/wiki/SHA-1">SHA-1</a>. If you are curious, you can read the algorithms for generating verifiable random curves on <a href="http://www.secg.org/sec1-v2.pdf">a specification by SECG</a> (look for “Verifiably Random Curves and Base Point Generators”). I’ve created a <a href="https://github.com/andreacorbellini/ecc/blob/master/scripts/verifyrandom.py">tiny Python script</a> that verifies all the random curves currently <a href="https://github.com/openssl/openssl/blob/81fc390/crypto/ec/ec_curve.c">shipped with OpenSSL</a>. I strongly recommend you to check it out! <h1 id="elliptic-curve-cryptography">Elliptic Curve Cryptography</h1> It took us a long time, but finally here we are! Therefore, pure and simple: <ol> <li>The private key is a random integer $d$ chosen from $\{1, \dots, n - 1\}$ (where $n$ is the order of the subgroup).</li> <li>The public key is the point $H = dG$ (where $G$ is the base point of the subgroup).</li> </ol> You see? If we know $d$ and $G$ (along with the other domain parameters), finding $H$ is “easy”. But if we know $H$ and $G$, finding the private key $d$ is “hard”, because it requires us to solve the discrete logarithm problem. Now we are going to describe two public-key algorithms based on that: ECDH (Elliptic curve Diffie-Hellman), which is used for encryption, and ECDSA (Elliptic Curve Digital Signature Algorithm), used for digital signing. <h2 id="encryption-with-ecdh">Encryption with ECDH</h2> ECDH is a variant of the <a href="https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange">Diffie-Hellman algorithm</a> for elliptic curves. It is actually a <a href="https://en.wikipedia.org/wiki/Key-agreement_protocol">key-agreement protocol</a>, more than an encryption algorithm. This basically means that ECDH defines (to some extent) how keys should be generated and exchanged between parties. How to actually encrypt data using such keys is up to us. The problem it solves is the following: two parties (the usual <a href="http://en.wikipedia.org/wiki/Alice_and_Bob">Alice and Bob</a>) want to exchange information securely, so that a third party (the <a href="http://en.wikipedia.org/wiki/Man-in-the-middle_attack">Man In the Middle</a>) may intercept them, but may not decode them. This is one of the principles behind TLS, just to give you an example. Here’s how it works: <ol> <li> First, Alice and Bob generate their own private and public keys. We have the private key $d_A$ and the public key $H_A = d_AG$ for Alice, and the keys $d_B$ and $H_B = d_BG$ for Bob. Note that both Alice and Bob are using the same domain parameters: the same base point $G$ on the same elliptic curve on the same finite field. </li> <li> Alice and Bob exchange their public keys $H_A$ and $H_B$ over an insecure channel. The Man In the Middle would intercept $H_A$ and $H_B$, but won’t be able to find out neither $d_A$ nor $d_B$ without solving the discrete logarithm problem. </li> <li> Alice calculates $S = d_A H_B$ (using her own private key and Bob’s public key), and Bob calculates $S = d_B H_A$ (using his own private key and Alice’s public key). Note that $S$ is the same for both Alice and Bob, in fact: $$S = d_A H_B = d_A (d_B G) = d_B (d_A G) = d_B H_A$$ </li> </ol> The Man In the Middle, however, only knows $H_A$ and $H_B$ (together with the other domain parameters) and would not be able to find out the shared secret $S$. This is known as the Diffie-Hellman problem, which can be stated as follows: <blockquote> Given three points $P$, $aP$ and $bP$, what is the result of $abP$? </blockquote> Or, equivalently: <blockquote> Given three integers $k$, $k^x$ and $k^y$, what is the result of $k^{xy}$? </blockquote> (The latter form is used in the original Diffie-Hellman algorithm, based on modular arithmetic.) <figure> <img src="https://andrea.corbellini.blog/images/ecdh.png" alt="ECDH" width="468" height="196"> <figcaption>The Diffie-Hellman key exchange: Alice and Bob can "easily" calculate the shared secret, the Man in the Middle has to solve a "hard" problem.</figcaption> </figure> The principle behind the Diffie-Hellman problem is also explained in a great <a href="https://www.youtube.com/watch?v=YEBfamv-_do#t=02m37s">YouTube video by Khan Academy</a>, which later explains the Diffie-Hellman algorithm applied to modular arithmetic (not to elliptic curves). The Diffie-Hellman problem for elliptic curves is assumed to be a “hard” problem. It is believed to be as “hard” as the discrete logarithm problem, although no mathematical proofs are available. What we can tell for sure is that it can’t be “harder”, because solving the logarithm problem is a way of solving the Diffie-Hellman problem. Now that Alice and Bob have obtained the shared secret, they can exchange data with symmetric encryption. For example, they can use the $x$ coordinate of $S$ as the key to encrypt messages using secure ciphers like <a href="https://en.wikipedia.org/wiki/Advanced_Encryption_Standard">AES</a> or <a href="https://en.wikipedia.org/wiki/Triple_DES">3DES</a>. This is more or less what TLS does, the difference is that TLS concatenates the $x$ coordinate with other numbers relative to the connection and then computes a hash of the resulting byte string. <h3 id="playing-with-ecdh">Playing with ECDH</h3> I’ve created <a href="https://github.com/andreacorbellini/ecc/blob/master/scripts/ecdhe.py">another Python script</a> for computing public/private keys and shared secrets over an elliptic curve. Unlike all the examples we have seen till now, this script makes use of a standardized curve, rather than a simple curve on a small field. The curve I’ve chosen is <code>secp256k1</code>, from <a href="http://www.secg.org/">SECG</a> (the “Standards for Efficient Cryptography Group”, founded by <a href="https://www.certicom.com/">Certicom</a>). <a href="https://en.bitcoin.it/wiki/Secp256k1">This same curve is also used by Bitcoin</a> for digital signatures. Here are the domain parameters: <ul> <li>$p$ = 0xffffffff ffffffff ffffffff ffffffff ffffffff ffffffff fffffffe fffffc2f</li> <li>$a$ = 0</li> <li>$b$ = 7</li> <li>$x_G$ = 0x79be667e f9dcbbac 55a06295 ce870b07 029bfcdb 2dce28d9 59f2815b 16f81798</li> <li>$y_G$ = 0x483ada77 26a3c465 5da4fbfc 0e1108a8 fd17b448 a6855419 9c47d08f fb10d4b8</li> <li>$n$ = 0xffffffff ffffffff ffffffff fffffffe baaedce6 af48a03b bfd25e8c d0364141</li> <li>$h$ = 1</li> </ul> (These numbers were taken from <a href="https://github.com/openssl/openssl/blob/81fc390/crypto/ec/ec_curve.c#L766">OpenSSL source code</a>.) Of course, you are free to modify the script to use other curves and domain parameters, just be sure to use prime fields and curves Weierstrass normal form, otherwise the script won’t work. The script is really simple and includes some of the algorithms we have described so far: point addition, double and add, ECDH. I recommend you to read and run it. It will produce an output like this: <div class="highlight"><pre><code>Curve: secp256k1 Alice's private key: 0xe32868331fa8ef0138de0de85478346aec5e3912b6029ae71691c384237a3eeb Alice's public key: (0x86b1aa5120f079594348c67647679e7ac4c365b2c01330db782b0ba611c1d677, 0x5f4376a23eed633657a90f385ba21068ed7e29859a7fab09e953cc5b3e89beba) Bob's private key: 0xcef147652aa90162e1fff9cf07f2605ea05529ca215a04350a98ecc24aa34342 Bob's public key: (0x4034127647bb7fdab7f1526c7d10be8b28174e2bba35b06ffd8a26fc2c20134a, 0x9e773199edc1ea792b150270ea3317689286c9fe239dd5b9c5cfd9e81b4b632) Shared secret: (0x3e2ffbc3aa8a2836c1689e55cd169ba638b58a3a18803fcf7de153525b28c3cd, 0x43ca148c92af58ebdb525542488a4fe6397809200fe8c61b41a105449507083) </code></pre></div> <h3 id="ephemeral-ecdh">Ephemeral ECDH</h3> Some of you may have heard of ECDHE instead of ECDH. The “E” in ECDHE stands for “Ephemeral” and refers to the fact that the keys exchanged are temporary, rather than static. ECDHE is used, for example, in TLS, where both the client and the server generate their public-private key pair on the fly, when the connection is established. The keys are then signed with the TLS certificate (for authentication) and exchanged between the parties. <h2 id="signing-with-ecdsa">Signing with ECDSA</h2> The scenario is the following: Alice wants to sign a message with her private key ($d_A$), and Bob wants to validate the signature using Alice’s public key ($H_A$). Nobody but Alice should be able to produce valid signatures. Everyone should be able to check signatures. Again, Alice and Bob are using the same domain parameters. The algorithm we are going to see is ECDSA, a variant of the <a href="https://en.wikipedia.org/wiki/Digital_Signature_Algorithm">Digital Signature Algorithm</a> applied to elliptic curves. ECDSA works on the hash of the message, rather than on the message itself. The choice of the hash function is up to us, but it should be obvious that a <a href="http://en.wikipedia.org/wiki/Cryptographic_hash_function">cryptographically-secure hash function</a> should be chosen. The hash of the message ought to be truncated so that the bit length of the hash is the same as the bit length of $n$ (the order of the subgroup). The truncated hash is an integer and will be denoted as $z$. The algorithm performed by Alice to sign the message works as follows: <ol> <li>Take a random integer $k$ chosen from $\{1, \dots, n - 1\}$ (where $n$ is still the subgroup order).</li> <li>Calculate the point $P = kG$ (where $G$ is the base point of the subgroup).</li> <li>Calculate the number $r = x_P \bmod{n}$ (where $x_P$ is the $x$ coordinate of $P$).</li> <li>If $r = 0$, then choose another $k$ and try again.</li> <li>Calculate $s = k^{-1} (z + rd_A) \bmod{n}$ (where $d_A$ is Alice’s private key and $k^{-1}$ is the multiplicative inverse of $k$ modulo $n$).</li> <li>If $s = 0$, then choose another $k$ and try again.</li> </ol> The pair $(r, s)$ is the signature. <figure> <img src="https://andrea.corbellini.blog/images/ecdsa.png" alt="ECDSA" width="514" height="255"> <figcaption>Alice signs the hash $z$ using her private key $d_A$ and a random $k$. Bob verifies that the message has been correctly signed using Alice's public key $H_A$.</figcaption> </figure> In plain words, this algorithm first generates a secret ($k$). This secret is hidden in $r$ thanks to point multiplication (that, as we know, is “easy” one way, and “hard” the other way round). $r$ is then bound to the message hash by the equation $s = k^{-1} (z + rd_A) \bmod{n}$. Note that in order to calculate $s$, we have computed the inverse of $k$ modulo $n$. We have <a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/#p-must-be-prime">already said in the previous post</a> that this is guaranteed to work only if $n$ is a prime number. If a subgroup has a non-prime order, ECDSA can’t be used. It’s not by chance that almost all standardized curves have a prime order, and those that have a non-prime order are unsuitable for ECDSA. <h3 id="verifying-signatures">Verifying signatures</h3> In order to verify the signature we’ll need Alice’s public key $H_A$, the (truncated) hash $z$ and, obviously, the signature $(r, s)$. <ol> <li>Calculate the integer $u_1 = s^{-1} z \bmod{n}$.</li> <li>Calculate the integer $u_2 = s^{-1} r \bmod{n}$.</li> <li>Calculate the point $P = u_1 G + u_2 H_A$.</li> </ol> The signature is valid only if $r = x_P \bmod{n}$. <h2 id="correctness-of-the-algorithm">Correctness of the algorithm</h2> The logic behind this algorithm may not seem obvious at a first sight, however if we put together all the equations we have written so far, things will be clearer. Let’s start from $P = u_1 G + u_2 H_A$. We know, from the definition of public key, that $H_A = d_A G$ (where $d_A$ is the private key). We can write: $$\begin{align*} P & = u_1 G + u_2 H_A \\ & = u_1 G + u_2 d_A G \\ & = (u_1 + u_2 d_A) G \end{align*}$$ Using the definitions of $u_1$ and $u_2$, we can write: $$\begin{align*} P & = (u_1 + u_2 d_A) G \\ & = (s^{-1} z + s^{-1} r d_A) G \\ & = s^{-1} (z + r d_A) G \end{align*}$$ Here we have omitted “$\text{mod}\ n$” both for brevity, and because the cyclic subgroup generated by $G$ has order $n$, hence “$\text{mod}\ n$” is superfluous. Previously, we defined $s = k^{-1} (z + rd_A) \bmod{n}$. Multiplying each side of the equation by $k$ and dividing by $s$, we get: $k = s^{-1} (z + rd_A) \bmod{n}$. Substituting this result in our equation for $P$, we get: $$\begin{align*} P & = s^{-1} (z + r d_A) G \\ & = k G \end{align*}$$ This is the same equation for $P$ we had at step 2 of the signature generation algorithm! When generating signatures and when verifying them, we are calculating the same point $P$, just with a different set of equations. This is why the algorithm works. <h3 id="playing-with-ecdsa">Playing with ECDSA</h3> Of course, I’ve created <a href="https://github.com/andreacorbellini/ecc/blob/master/scripts/ecdsa.py">a Python script</a> for signature generation and verification. The code shares some parts with the ECDH script, in particular the domain parameters and the public/private key pair generation algorithm. Here is the kind of output produced by the script: <div class="highlight"><pre><code>Curve: secp256k1 Private key: 0x9f4c9eb899bd86e0e83ecca659602a15b2edb648e2ae4ee4a256b17bb29a1a1e Public key: (0xabd9791437093d377ca25ea974ddc099eafa3d97c7250d2ea32af6a1556f92a, 0x3fe60f6150b6d87ae8d64b78199b13f26977407c801f233288c97ddc4acca326) Message: b'Hello!' Signature: (0xddcb8b5abfe46902f2ac54ab9cd5cf205e359c03fdf66ead1130826f79d45478, 0x551a5b2cd8465db43254df998ba577cb28e1ee73c5530430395e4fba96610151) Verification: signature matches Message: b'Hi there!' Verification: invalid signature Message: b'Hello!' Public key: (0xc40572bb38dec72b82b3efb1efc8552588b8774149a32e546fb703021cf3b78a, 0x8c6e5c5a9c1ea4cad778072fe955ed1c6a2a92f516f02cab57e0ba7d0765f8bb) Verification: invalid signature </code></pre></div> As you can see, the script first signs a message (the byte string “Hello!”), then verifies the signature. Afterwards, it tries to verify the same signature against another message (“Hi there!”) and verification fails. Lastly, it tries to verify the signature against the correct message, but using another random public key and verification fails again. <h2 id="ecdsa-k">The importance of k</h2> When generating ECDSA signatures, it is important to keep the secret $k$ really secret. If we used the same $k$ for all signatures, or if our random number generator were somewhat predictable, an attacker would be able to find out the private key! <a href="http://www.bbc.com/news/technology-12116051">This is the kind of mistake made by Sony a few years ago.</a> Basically, the PlayStation 3 game console can run only games signed by Sony with ECDSA. This way, if I wanted to create a new game for PlayStation 3, I couldn’t distribute it to the public without a signature from Sony. The problem is: all the signatures made by Sony were generated using a static $k$. (Apparently, Sony’s random number generator was inspired by either <a href="http://xkcd.com/221/">XKCD</a> or <a href="http://dilbert.com/strip/2001-10-25">Dilbert</a>.) In this situation, we could easily recover Sony’s private key $d_S$ by buying just two signed games, extracting their hashes ($z_1$ and $z_2$) and their signatures ($(r_1, s_1)$ and $(r_2, s_2)$), together with the domain parameters. Here’s how: <ul> <li>First off, note that $r_1 = r_2$ (because $r = x_P \bmod{n}$ and $P = kG$ is the same for both signatures).</li> <li>Consider that $(s_1 - s_2) \bmod{n} = k^{-1} (z_1 - z_2) \bmod{n}$ (this result comes directly from the equation for $s$).</li> <li>Now multiply each side of the equation by $k$: $k (s_1 - s_2) \bmod{n} = (z_1 - z_2) \bmod{n}$.</li> <li>Divide by $(s_1 - s_2)$ to get $k = (z_1 - z_2)(s_1 - s_2)^{-1} \bmod{n}$.</li> </ul> The last equation lets us calculate $k$ using only two hashes and their corresponding signatures. Now we can extract the private key using the equation for $s$: $$s = k^{-1}(z + rd_S) \bmod{n}\ \ \Rightarrow\ \ d_S = r^{-1} (sk - z) \bmod{n}$$ Similar techniques may be employed if $k$ is not static but predictable in some way. <h1 id="have-a-great-weekend">Have a great weekend</h1> I really hope you enjoyed what I’ve written here. As usual, don’t hesitate to leave a comment or send me a poke if you need help with something. Next week I’ll publish the fourth and last article of this series. It’ll be about techniques for solving discrete logarithms, some important problems of Elliptic Curve cryptography, and how ECC compares with RSA. Don’t miss it! <a href="https://andrea.corbellini.blog/2015/06/08/elliptic-curve-cryptography-breaking-security-and-a-comparison-with-rsa/">Read the next post of the series »</a>andreacorbelliniSat, 30 May 2015 19:23:00 +0000tag:andrea.corbellini.blog,2015-05-30:/2015/05/30/elliptic-curve-cryptography-ecdh-and-ecdsa/cryptographydhdsaeccecdhecdheecdsasecuritytlsElliptic Curve Cryptography: finite fields and discrete logarithmshttps://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/This post is the second in the series <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/">ECC: a gentle introduction</a>. In the <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/">previous post</a>, we have seen how elliptic curves over the real numbers can be used to define a group. Specifically, we have defined a rule for <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/#group-law">point addition</a>: given three aligned points, their sum is zero ($P + Q + R = 0$). We have derived a <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/#geometric-addition">geometric method</a> and an <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/#algebraic-addition">algebraic method</a> for computing point additions. We then introduced <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/#scalar-multiplication">scalar multiplication</a> ($nP = P + P + \cdots + P$) and we found out an “easy” algorithm for computing scalar multiplication: <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/#double-and-add">double and add</a>. Now we will restrict our elliptic curves to finite fields, rather than the set of real numbers, and see how things change. <h1 id="the-field-of-integers-modulo-p">The field of integers modulo p</h1> A finite field is, first of all, a set with a finite number of elements. An example of finite field is the set of integers modulo $p$, where $p$ is a prime number. It is generally denoted as $\mathbb{Z}/p$, $GF(p)$ or $\mathbb{F}_p$. We will use the latter notation. In fields we have two binary operations: addition (+) and multiplication (·). Both are closed, associative and commutative. For both operations, there exist a unique identity element, and for every element there’s a unique inverse element. Finally, multiplication is distributive over the addition: $x \cdot (y + z) = x \cdot y + x \cdot z$. The set of integers modulo $p$ consists of all the integers from 0 to $p - 1$. Addition and multiplication work as in <a href="http://en.wikipedia.org/wiki/Modular_arithmetic">modular arithmetic</a> (also known as “clock arithmetic”). Here are a few examples of operations in $\mathbb{F}_{23}$: <ul> <li>Addition: $(18 + 9) \bmod{23} = 4$</li> <li>Subtraction: $(7 - 14) \bmod{23} = 16$</li> <li>Multiplication: $4 \cdot 7 \bmod{23} = 5$</li> <li> Additive inverse: $-5 \bmod{23} = 18$ Indeed: $(5 + (-5)) \bmod{23} = (5 + 18) \bmod{23} = 0$ </li> <li> Multiplicative inverse: $9^{-1} \bmod{23} = 18$ Indeed: $9 \cdot 9^{-1} \bmod{23} = 9 \cdot 18 \bmod{23} = 1$ </li> </ul> If these equations don’t look familiar to you and you need a primer on modular arithmetic, check out <a href="https://www.khanacademy.org/computing/computer-science/cryptography/modarithmetic/a/what-is-modular-arithmetic">Khan Academy</a>. As we already said, the integers modulo $p$ are a field, and therefore all the properties listed above hold. Note that the requirement for $p$ to be prime is important! The set of integers modulo 4 is not a field: 2 has no multiplicative inverse (i.e. the equation $2 \cdot x \bmod{4} = 1$ has no solutions). <h2 id="division-modulo-p">Division modulo p</h2> We will soon define elliptic curves over $\mathbb{F}_p$, but before doing so we need a clear idea of what $x / y$ means in $\mathbb{F}_p$. Simply put: $x / y = x \cdot y^{-1}$, or, in plain words, $x$ over $y$ is equal to $x$ times the multiplicative inverse of $y$. This fact is not surprising, but gives us a basic method to perform division: find the multiplicative inverse of a number and then perform a single multiplication. Computing the multiplicative inverse can be “easily” done with the <a href="http://en.wikipedia.org/wiki/Extended_Euclidean_algorithm">extended Euclidean algorithm</a>, which is $O(\log p)$ (or $O(k)$ if we consider the bit length) in the worst case. We won’t enter the details of the extended Euclidean algorithm, as it is off-topic, however here’s a working Python implementation: <div class="highlight"><pre><code>def extended_euclidean_algorithm(a, b): """ Returns a three-tuple (gcd, x, y) such that a * x + b * y == gcd, where gcd is the greatest common divisor of a and b. This function implements the extended Euclidean algorithm and runs in O(log b) in the worst case. """ s, old_s = 0, 1 t, old_t = 1, 0 r, old_r = b, a while r != 0: quotient = old_r // r old_r, r = r, old_r - quotient * r old_s, s = s, old_s - quotient * s old_t, t = t, old_t - quotient * t return old_r, old_s, old_t def inverse_of(n, p): """ Returns the multiplicative inverse of n modulo p. This function returns an integer m such that (n * m) % p == 1. """ gcd, x, y = extended_euclidean_algorithm(n, p) assert (n * x + p * y) % p == gcd if gcd != 1: # Either n is 0, or p is not a prime number. raise ValueError( '{} has no multiplicative inverse ' 'modulo {}'.format(n, p)) else: return x % p </code></pre></div> <h1 id="elliptic-curves-in-mathbbf_p">Elliptic curves in $\mathbb{F}_p$</h1> Now we have all the necessary elements to restrict elliptic curves over $\mathbb{F}_p$. The set of points, that in the <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/#elliptic-curves">previous post</a> was: $$\begin{array}{rcl} \left\{(x, y) \in \mathbb{R}^2 \right. & \left. | \right. & \left. y^2 = x^3 + ax + b, \right. \\ & & \left. 4a^3 + 27b^2 \ne 0\right\}\ \cup\ \left\{0\right\} \end{array}$$ now becomes: $$\begin{array}{rcl} \left\{(x, y) \in (\mathbb{F}_p)^2 \right. & \left. | \right. & \left. y^2 \equiv x^3 + ax + b \pmod{p}, \right. \\ & & \left. 4a^3 + 27b^2 \not\equiv 0 \pmod{p}\right\}\ \cup\ \left\{0\right\} \end{array}$$ where 0 is still the point at infinity, and $a$ and $b$ are two integers in $\mathbb{F}_p$. <figure> <img src="https://andrea.corbellini.blog/images/elliptic-curves-mod-p.png" alt="Elliptic curves in Fp" width="608" height="608"> <figcaption>The curve $y^2 \equiv x^3 - 7x + 10 \pmod{p}$ with $p = 19, 97, 127, 487$. Note that, for every $x$, there are at most two points. Also note the symmetry about $y = p / 2$.</figcaption> </figure> <figure> <img src="https://andrea.corbellini.blog/images/singular-mod-p.png" alt="Singular curve in Fp" width="300" height="300"> <figcaption>The curve $y^2 \equiv x^3 \pmod{29}$ is singular and has a triple point in $(0, 0)$. It is not a valid elliptic curve.</figcaption> </figure> What previously was a continuous curve is now a set of disjoint points in the $xy$-plane. But we can prove that, even if we have restricted our domain, elliptic curves in $\mathbb{F}_p$ still form an abelian group. <h1 id="point-addition">Point addition</h1> Clearly, we need to change a bit our definition of addition in order to make it work in $\mathbb{F}_p$. With reals, we said that the sum of three aligned points was zero ($P + Q + R = 0$). We can keep this definition, but what does it mean for three points to be aligned in $\mathbb{F}_p$? We can say that three points are aligned if there’s a line that connects all of them. Now, of course, lines in $\mathbb{F}_p$ are not the same as lines in $\mathbb{R}$. We can say, informally, that a line in $\mathbb{F}_p$ is the set of points $(x, y)$ that satisfy the equation $ax + by + c \equiv 0 \pmod{p}$ (this is the standard line equation, with the addition of “$(\text{mod}\ p)$”). <figure> <img src="https://andrea.corbellini.blog/images/point-addition-mod-p.png" alt="Point addition for elliptic curves in Z/p" width="523" height="528"> <figcaption>Point addition over the curve $y^2 \equiv x^3 - x + 3 \pmod{127}$, with $P = (16, 20)$ and $Q = (41, 120)$. Note how the line $y \equiv 4x + 83 \pmod{127}$ that connects the points "repeats" itself in the plane.</figcaption> </figure> Given that we are in a group, point addition retains the properties we already know: <ul> <li>$Q + 0 = 0 + Q = Q$ (from the definition of identity element).</li> <li>Given a non-zero point $Q$, the inverse $-Q$ is the point having the same abscissa but opposite ordinate. Or, if you prefer, $-Q = (x_Q, -y_Q \bmod{p})$. For example, if a curve in $\mathbb{F}_{29}$ has a point $Q = (2, 5)$, the inverse is $-Q = (2, -5 \bmod{29}) = (2, 24)$.</li> <li>Also, $P + (-P) = 0$ (from the definition of inverse element).</li> </ul> <h1 id="algebraic-sum">Algebraic sum</h1> The equations for calculating point additions are exactly the same as in the previous post, except for the fact that we need to add “$\text{mod}\ p$” at the end of every expression. Therefore, given $P = (x_P, y_P)$, $Q = (x_Q, y_Q)$ and $R = (x_R, y_R)$, we can calculate $P + Q = -R$ as follows: $$\begin{align*} x_R & = (m^2 - x_P - x_Q) \bmod{p} \\ y_R & = [y_P + m(x_R - x_P)] \bmod{p} \\ & = [y_Q + m(x_R - x_Q)] \bmod{p} \end{align*}$$ If $P \ne Q$, the the slope $m$ assumes the form: $$m = (y_P - y_Q)(x_P - x_Q)^{-1} \bmod{p}$$ Else, if $P = Q$, we have: $$m = (3 x_P^2 + a)(2 y_P)^{-1} \bmod{p}$$ It’s not a coincidence that the equations have not changed: in fact, these equations work in every field, finite or infinite (with the exception of $\mathbb{F}_2$ and $\mathbb{F}_3$, which are special cased). Now I feel I have to provide a justification for this fact. The problem is: proofs for the group law generally involve complex mathematical concepts. However, I found a <a href="https://arxiv.org/pdf/1710.00214">proof from Stefan Friedl</a> that uses only elementary concepts. Read it if you are interested in why these equations work in (almost) every field. Back to us — we won’t define a geometric method: in fact, there are a few problems with that. For example, in the previous post, we said that to compute $P + P$ we needed to take the tangent to the curve in $P$. But without continuity, the word “tangent” does not make any sense. We can workaround this and other problems, however a pure geometric method would just be too complicated and not practical at all. Instead, you can play with the <a href="https://andrea.corbellini.blog/ecc/interactive/modk-add.html">interactive tool</a> I’ve written for computing point additions. <h1 id="the-order-of-an-elliptic-curve-group">The order of an elliptic curve group</h1> We said that an elliptic curve defined over a finite field has a finite number of points. An important question that we need to answer is: how many points are there exactly? Firstly, let’s say that the number of points in a group is called the order of the group. Trying all the possible values for $x$ from 0 to $p - 1$ is not a feasible way to count the points, as it would require $O(p)$ steps, and this is “hard” if $p$ is a large prime. Luckily, there’s a faster algorithm for computing the order: <a href="https://en.wikipedia.org/wiki/Schoof%27s_algorithm">Schoof’s algorithm</a>. I won’t enter the details of the algorithm — what matters is that it runs in polynomial time, and this is what we need. <h1 id="scalar-multiplication">Scalar multiplication and cyclic subgroups</h1> As with reals, multiplication can be defined as: $$n P = \underbrace{P + P + \cdots + P}_{n\ \text{times}}$$ And, again, we can use the <a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/#double-and-add">double and add algorithm</a> to perform multiplication in $O(\log n)$ steps (or $O(k)$, where $k$ is the number of bits of $n$). I’ve written an <a href="https://andrea.corbellini.blog/ecc/interactive/modk-mul.html">interactive tool</a> for scalar multiplication too. Multiplication over points for elliptic curves in $\mathbb{F}_p$ has an interesting property. Take the curve $y^2 \equiv x^3 + 2x + 3 \pmod{97}$ and the point $P = (3, 6)$. Now <a href="https://andrea.corbellini.blog/ecc/interactive/modk-mul.html">calculate</a> all the multiples of $P$: <figure> <img src="https://andrea.corbellini.blog/images/cyclic-subgroup.png" alt="Cyclic subgroup" width="322" height="255"> <figcaption>The multiples of $P = (3, 6)$ are just five distinct points ($0$, $P$, $2P$, $3P$, $4P$) and they are repeating cyclically. It's easy to spot the similarity between scalar multiplication on elliptic curves and addition in modular arithmetic.</figcaption> </figure> <ul> <li>$0P = 0$</li> <li>$1P = (3, 6)$</li> <li>$2P = (80, 10)$</li> <li>$3P = (80, 87)$</li> <li>$4P = (3, 91)$</li> <li>$5P = 0$</li> <li>$6P = (3, 6)$</li> <li>$7P = (80, 10)$</li> <li>$8P = (80, 87)$</li> <li>$9P = (3, 91)$</li> <li>…</li> </ul> Here we can immediately spot two things: firstly, the multiples of $P$ are just five: the other points of the elliptic curve never appear. Secondly, they are repeating cyclically. We can write: <ul> <li>$5kP = 0$</li> <li>$(5k + 1)P = P$</li> <li>$(5k + 2)P = 2P$</li> <li>$(5k + 3)P = 3P$</li> <li>$(5k + 4)P = 4P$</li> </ul> for every integer $k$. Note that these five equations can be “compressed” into a single one, thanks to the modulo operator: $kP = (k \bmod{5})P$. Not only that, but we can immediately verify that these five points are closed under addition. Which means: however I add $0$, $P$, $2P$, $3P$ or $4P$, the result is always one of these five points. Again, the other points of the elliptic curve never appear in the results. The same holds for every point, not just for $P = (3, 6)$. In fact, if we take a generic $P$: $$nP + mP = \underbrace{P + \cdots + P}_{n\ \text{times}} + \underbrace{P + \cdots + P}_{m\ \text{times}} = (n + m)P$$ Which means: if we add two multiples of $P$, we obtain a multiple of $P$ (i.e. multiples of $P$ are closed under addition). This is enough to <a href="https://en.wikipedia.org/wiki/Subgroup#Basic_properties_of_subgroups">prove</a> that the set of the multiples of $P$ is a cyclic subgroup of the group formed by the elliptic curve. A “subgroup” is a group which is a subset of another group. A “cyclic subgroup” is a subgroup which elements are repeating cyclically, like we have shown in the previous example. The point $P$ is called generator or base point of the cyclic subgroup. Cyclic subgroups are the foundations of ECC and other cryptosystems. We will see why in the next post. <h2 id="subgroup-order">Subgroup order</h2> We can ask ourselves what the order of a subgroup generated by a point $P$ is (or, equivalently, what the order of $P$ is). To answer this question we can’t use Schoof’s algorithm, because that algorithm only works on whole elliptic curves, not on subgroups. Before approaching the problem, we need a few more bits: <ul> <li>So far, we have the defined the order as the number of points of a group. This definition is still valid, but within a cyclic subgroup we can give a new, equivalent definition: the order of $P$ is the smallest positive integer $n$ such that $nP = 0$. In fact, if you look at the previous example, our subgroup contained five points, and we had $5P = 0$.</li> <li>The order of $P$ is linked to the order of the elliptic curve by <a href="https://en.wikipedia.org/wiki/Lagrange%27s_theorem_(group_theory)">Lagrange’s theorem</a>, which states that the order of a subgroup is a divisor of the order of the parent group. In other words, if an elliptic curve contains $N$ points and one of its subgroups contains $n$ points, then $n$ is a divisor of $N$.</li> </ul> These two information together give us a way to find out the order of a subgroup with base point $P$: <ol> <li>Calculate the elliptic curve’s order $N$ using Schoof’s algorithm.</li> <li>Find out all the divisors of $N$.</li> <li>For every divisor $n$ of $N$, compute $nP$.</li> <li>The smallest $n$ such that $nP = 0$ is the order of the subgroup.</li> </ol> For example, the curve $y^2 = x^3 - x + 3$ over the field $\mathbb{F}_{37}$ has order $N = 42$. Its subgroups may have order $n = 1$, $2$, $3$, $6$, $7$, $14$, $21$ or $42$. If <a href="https://andrea.corbellini.blog/ecc/interactive/modk-mul.html?a=-1&b=3&p=37&px=2&py=3">we try $P = (2, 3)$</a> we can see that $P \ne 0$, $2P \ne 0$, …, $7P = 0$, hence the order of $P$ is $n = 7$. Note that it’s important to take the smallest divisor, not a random one. If we proceeded randomly, we could have taken $n = 14$, which is not the order of the subgroup, but one of its multiples. Another example: the elliptic curve defined by the equation $y^2 = x^3 - x + 1$ over the field $\mathbb{F}_{29}$ has order $N = 37$, which is a prime. Its subgroups may only have order $n = 1$ or $37$. As you can easily guess, when $n = 1$, the subgroup contains only the point at infinity; when $n = N$, the subgroup contains all the points of the elliptic curve. <h2 id="finding-a-base-point">Finding a base point</h2> For our ECC algorithms, we want subgroups with a high order. So in general we will choose an elliptic curve, calculate its order ($N$), choose a high divisor as the subgroup order ($n$) and eventually find a suitable base point. That is: we won’t choose a base point and then calculate its order, but we’ll do the opposite: we will first choose an order that looks good enough and then we will hunt for a suitable base point. How do we do that? Firstly, we need to introduce one more term. Lagrange’s theorem implies that the number $h = N / n$ is always an integer (because $n$ is a divisor of $N$). The number $h$ has a name: it’s the cofactor of the subgroup. Now consider that for every point of an elliptic curve we have $NP = 0$. This happens because $N$ is a multiple of any candidate $n$. Using the definition of cofactor, we can write: $$n(hP) = 0$$ Now suppose that $n$ is a prime number (for reason that will be explained in the next post, we prefer prime orders). This equation, written in this form, is telling us that the point $G = hP$ generates a subgroup of order $n$ (except when $G = hP = 0$, in which case the subgroup has order 1). In the light of this, we can outline the following algorithm: <ol> <li>Calculate the order $N$ of the elliptic curve.</li> <li>Choose the order $n$ of the subgroup. For the algorithm to work, this number must be prime and must be a divisor of $N$.</li> <li>Compute the cofactor $h = N / n$.</li> <li>Choose a random point $P$ on the curve.</li> <li>Compute $G = hP$.</li> <li>If $G$ is 0, then go back to step 4. Otherwise we have found a generator of a subgroup with order $n$ and cofactor $h$.</li> </ol> Note that this algorithm only works if $n$ is a prime. If $n$ wasn’t a prime, then the order of $G$ could be one of the divisors of $n$. <h1 id="discrete-logarithm">Discrete logarithm</h1> As we did when working with continuous elliptic curves, we are now going to discuss the question: if we know $P$ and $Q$, what is $k$ such that $Q = kP$? This problem, which is known as the discrete logarithm problem for elliptic curves, is believed to be a “hard” problem, in that there is no known polynomial time algorithm that can run on a classical computer. There are, however, no mathematical proofs for this belief. This problem is also analogous to the discrete logarithm problem used with other cryptosystems such as the Digital Signature Algorithm (DSA), the Diffie-Hellman key exchange (D-H) and the ElGamal algorithm — it’s not a coincidence that they have the same name. The difference is that, with those algorithms, we use modulo exponentiation instead of scalar multiplication. Their discrete logarithm problem can be stated as follows: if we know $a$ and $b$, what’s $k$ such that $b = a^k \bmod{p}$? Both these problems are “discrete” because they involve finite sets (more precisely, cyclic subgroups). And they are “logarithms” because they are analogous to ordinary logarithms. What makes ECC interesting is that, as of today, the discrete logarithm problem for elliptic curves seems to be “harder” if compared to other similar problems used in cryptography. This implies that we need fewer bits for the integer $k$ in order to achieve the same level of security as with other cryptosystems, as we will see in details in the fourth and last post of this series. <h1 id="more-next-week">More next week!</h1> Enough for today! I really hope you enjoyed this post. Leave a comment if you didn’t. Next week’s post will be the third in this series and will be about ECC algorithms: key pair generation, ECDH and ECDSA. That will be one of the most interesting parts of this series. Don’t miss it! <a href="https://andrea.corbellini.blog/2015/05/30/elliptic-curve-cryptography-ecdh-and-ecdsa/">Read the next post of the series »</a>andreacorbelliniSat, 23 May 2015 14:08:00 +0000tag:andrea.corbellini.blog,2015-05-23:/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/cryptographyeccmathsecurityElliptic Curve Cryptography: a gentle introductionhttps://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/Those of you who know what public-key cryptography is may have already heard of ECC, ECDH or ECDSA. The first is an acronym for Elliptic Curve Cryptography, the others are names for algorithms based on it. Today, we can find elliptic curves cryptosystems in <a href="https://tools.ietf.org/html/rfc4492">TLS</a>, <a href="https://tools.ietf.org/html/rfc6637">PGP</a> and <a href="https://tools.ietf.org/html/rfc5656">SSH</a>, which are just three of the main technologies on which the modern web and IT world are based. Not to mention <a href="https://en.bitcoin.it/wiki/Secp256k1">Bitcoin</a> and other cryptocurrencies. Before ECC become popular, almost all public-key algorithms were based on RSA, DSA, and DH, alternative cryptosystems based on modular arithmetic. RSA and friends are still very important today, and often are used alongside ECC. However, while the magic behind RSA and friends can be easily explained, is widely understood, and <a href="http://code.activestate.com/recipes/578838-rsa-a-simple-and-easy-to-read-implementation/">rough implementations can be written quite easily</a>, the foundations of ECC are still a mystery to most. With a series of blog posts I’m going to give you a gentle introduction to the world of elliptic curve cryptography. My aim is not to provide a complete and detailed guide to ECC (the web is full of information on the subject), but to provide a simple overview of what ECC is and why it is considered secure, without losing time on long mathematical proofs or boring implementation details. I will also give helpful examples together with visual interactive tools and scripts to play with. Specifically, here are the topics I’ll touch: <ol> <li><a href="https://andrea.corbellini.blog/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/">Elliptic curves over real numbers and the group law</a> (covered in this blog post)</li> <li><a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/">Elliptic curves over finite fields and the discrete logarithm problem</a></li> <li><a href="https://andrea.corbellini.blog/2015/05/30/elliptic-curve-cryptography-ecdh-and-ecdsa/">Key pair generation and two ECC algorithms: ECDH and ECDSA</a></li> <li><a href="https://andrea.corbellini.blog/2015/06/08/elliptic-curve-cryptography-breaking-security-and-a-comparison-with-rsa/">Algorithms for breaking ECC security, and a comparison with RSA</a></li> </ol> In order to understand what’s written here, you’ll need to know some basic stuff of set theory, geometry and modular arithmetic, and have familiarity with symmetric and asymmetric cryptography. Lastly, you need to have a clear idea of what an “easy” problem is, what a “hard” problem is, and their roles in cryptography. Ready? Let’s start! <h1 id="elliptic-curves">Elliptic Curves</h1> First of all: what is an elliptic curve? Wolfram MathWorld gives an excellent and complete <a href="http://mathworld.wolfram.com/EllipticCurve.html">definition</a>. But for our aims, an elliptic curve will simply be the set of points described by the equation: $$y^2 = x^3 + ax + b$$ where $4a^3 + 27b^2 \ne 0$ (this is required to exclude <a href="https://en.wikipedia.org/wiki/Singularity_(mathematics)">singular curves</a>). The equation above is what is called Weierstrass normal form for elliptic curves. <figure> <img src="https://andrea.corbellini.blog/images/curves.png" alt="Different shapes for different elliptic curves" width="440" height="450"> <figcaption>Different shapes for different elliptic curves ($b = 1$, $a$ varying from 2 to -3).</figcaption> </figure> <figure> <img src="https://andrea.corbellini.blog/images/singularities.png" alt="Types of singularities" width="300" height="220"> <figcaption>Types of singularities: on the left, a curve with a cusp ($y^2 = x^3$). On the right, a curve with a self-intersection ($y^2 = x^3 - 3x + 2$). None of them is a valid elliptic curve.</figcaption> </figure> Depending on the value of $a$ and $b$, elliptic curves may assume different shapes on the plane. As it can be easily seen and verified, elliptic curves are symmetric about the $x$-axis. For our aims, we will also need a <a href="https://en.wikipedia.org/wiki/Point_at_infinity">point at infinity</a> (also known as ideal point) to be part of our curve. From now on, we will denote our point at infinity with the symbol 0 (zero). If we want to explicitly take into account the point at infinity, we can refine our definition of elliptic curve as follows: $$\left\{ (x, y) \in \mathbb{R}^2\ |\ y^2 = x^3 + ax + b,\ 4 a^3 + 27 b^2 \ne 0 \right\}\ \cup\ \left\{ 0 \right\}$$ <h1 id="groups">Groups</h1> A group in mathematics is a set for which we have defined a binary operation that we call “addition” and indicate with the symbol +. In order for the set $\mathbb{G}$ to be a group, addition must defined so that it respects the following four properties: <ol> <li>closure: if $a$ and $b$ are members of $\mathbb{G}$, then $a + b$ is a member of $\mathbb{G}$;</li> <li>associativity: $(a + b) + c = a + (b + c)$;</li> <li>there exists an identity element 0 such that $a + 0 = 0 + a = a$;</li> <li>every element has an inverse, that is: for every $a$ there exists $b$ such that $a + b = 0$.</li> </ol> If we add a fifth requirement: <ol start="5"> <li>commutativity: $a + b = b + a$,</li> </ol> then the group is called abelian group. With the usual notion of addition, the set of integer numbers $\mathbb{Z}$ is a group (moreover, it’s an abelian group). The set of natural numbers $\mathbb{N}$ however is not a group, as the fourth property can’t be satisfied. Groups are nice because, if we can demonstrate that those four properties hold, we get some other properties for free. For example: the identity element is unique; also the inverses are unique, that is: for every $a$ there exists only one $b$ such that $a + b = 0$ (and we can write $b$ as $-a$). Either directly or indirectly, these and other facts about groups will be very important for us later. <h1 id="group-law">The group law for elliptic curves</h1> We can define a group over elliptic curves. Specifically: <ul> <li>the elements of the group are the points of an elliptic curve;</li> <li>the identity element is the point at infinity 0;</li> <li>the inverse of a point $P$ is the one symmetric about the $x$-axis;</li> <li>addition is given by the following rule: given three aligned, non-zero points $P$, $Q$ and $R$, their sum is $P + Q + R = 0$.</li> </ul> <figure> <img src="https://andrea.corbellini.blog/images/three-aligned-points.png" alt="Three aligned points" width="300" height="300"> <figcaption>The sum of three aligned point is 0.</figcaption> </figure> Note that with the last rule, we only require three aligned points, and three points are aligned without respect to order. This means that, if $P$, $Q$ and $R$ are aligned, then $P + (Q + R) = Q + (P + R) = R + (P + Q) = \cdots = 0$. This way, we have intuitively proved that our + operator is both associative and commutative: we are in an abelian group. So far, so great. But how do we actually compute the sum of two arbitrary points? <h1 id="geometric-addition">Geometric addition</h1> Thanks to the fact that we are in an abelian group, we can write $P + Q + R = 0$ as $P + Q = -R$. This equation, in this form, lets us derive a geometric method to compute the sum between two points $P$ and $Q$: if we draw a line passing through $P$ and $Q$, this line will intersect a third point on the curve, $R$ (this is implied by the fact that $P$, $Q$ and $R$ are aligned). If we take the inverse of this point, $-R$, we have found the result of $P + Q$. <figure> <img src="https://andrea.corbellini.blog/images/point-addition.png" alt="Point addition" width="287" height="300"> <figcaption>Draw the line through $P$ and $Q$. The line intersects a third point $R$. The point symmetric to it, $-R$, is the result of $P + Q$.</figcaption> </figure> This geometric method works but needs some refinement. Particularly, we need to answer a few questions: <ul> <li>What if $P = 0$ or $Q = 0$? Certainly, we can’t draw any line (0 is not on the $xy$-plane). But given that we have defined 0 as the identity element, $P + 0 = P$ and $0 + Q = Q$, for any $P$ and for any $Q$.</li> <li>What if $P = -Q$? In this case, the line going through the two points is vertical, and does not intersect any third point. But if $P$ is the inverse of $Q$, then we have $P + Q = P + (-P) = 0$ from the definition of inverse.</li> <li>What if $P = Q$? In this case, there are infinitely many lines passing through the point. Here things start getting a bit more complicated. But consider a point $Q’ \ne P$. What happens if we make $Q’$ approach $P$, getting closer and closer to it? <figure> <img src="https://andrea.corbellini.blog/images/animation-point-doubling.gif" width="300" height="300" alt="The result of P + Q as Q is approaching P"> <figcaption>As the two points become closer together, the line passing through them becomes tangent to the curve.</figcaption> </figure></li> </ul> As $Q’$ tends towards $P$, the line passing through $P$ and $Q’$ becomes tangent to the curve. In the light of this we can say that $P + P = -R$, where $R$ is the point of intersection between the curve and the line tangent to the curve in $P$. * What if $P \ne Q$, but there is no third point $R$? We are in a case very similar to the previous one. In fact, we are in the case where the line passing through $P$ and $Q$ is tangent to the curve. <figure> <img src="https://andrea.corbellini.blog/images/animation-tangent-line.gif" alt="The result of P + Q as Q is approaching P" width="300" height="300"> <figcaption>If our line intersects just two points, then it means that it's tangent to the curve. It's easy to see how the result of the sum becomes symmetric to one of the two points.</figcaption> </figure> Let’s assume that $P$ is the tangency point. In the previous case, we would have written $P + P = -Q$. That equation now becomes $P + Q = -P$. If, on the other hand, $Q$ were the tangency point, the correct equation would have been $P + Q = -Q$. The geometric method is now complete and covers all cases. With a pencil and a ruler we are able to perform addition involving every point of any elliptic curve. If you want to try, take a look at the <a href="https://andrea.corbellini.blog/ecc/interactive/reals-add.html">HTML5/JavaScript visual tool</a> I’ve built for computing sums on elliptic curves! <h1 id="algebraic-addition">Algebraic addition</h1> If we want a computer to perform point addition, we need to turn the geometric method into an algebraic method. Transforming the rules described above into a set of equations may seem straightforward, but actually it can be really tedious because it requires solving cubic equations. For this reason, here I will report only the results. First, let’s get get rid of the most annoying corner cases. We already know that $P + (-P) = 0$, and we also know that $P + 0 = 0 + P = P$. So, in our equations, we will avoid these two cases and we will only consider two non-zero, non-symmetric points $P = (x_P, y_P)$ and $Q = (x_Q, y_Q)$. If $P$ and $Q$ are distinct ($x_P \ne x_Q$), the line through them has slope: $$m = \frac{y_P - y_Q}{x_P - x_Q}$$ The intersection of this line with the elliptic curve is a third point $R = (x_R, y_R)$: $$\begin{align*} x_R & = m^2 - x_P - x_Q \\ y_R & = y_P + m(x_R - x_P) \end{align*}$$ or, equivalently: $$y_R = y_Q + m(x_R - x_Q)$$ Hence $(x_P, y_P) + (x_Q, y_Q) = (x_R, -y_R)$ (pay attention at the signs and remember that $P + Q = -R$). If we wanted to check whether this result is right, we would have had to check whether $R$ belongs to the curve and whether $P$, $Q$ and $R$ are aligned. Checking whether the points are aligned is trivial, checking that $R$ belongs to the curve is not, as we would need to solve a cubic equation, which is not fun at all. Instead, let’s play with an example: according to our <a href="https://andrea.corbellini.blog/ecc/interactive/reals-add.html">visual tool</a>, given $P = (1, 2)$ and $Q = (3, 4)$ over the curve $y^2 = x^3 - 7x + 10$, their sum is $P + Q = -R = (-3, 2)$. Let’s see if our equations agree: $$\begin{align*} m & = \frac{y_P - y_Q}{x_P - x_Q} = \frac{2 - 4}{1 - 3} = 1 \\ x_R & = m^2 - x_P - x_Q = 1^2 - 1 - 3 = -3 \\ y_R & = y_P + m(x_R - x_P) = 2 + 1 \cdot (-3 - 1) = -2 \\ & = y_Q + m(x_R - x_Q) = 4 + 1 \cdot (-3 - 3) = -2 \end{align*}$$ Yes, this is correct! Note that these equations work even if one of $P$ or $Q$ is a tangency point. Let’s try with $P = (-1, 4)$ and $Q = (1, 2)$. $$\begin{align*} m & = \frac{y_P - y_Q}{x_P - x_Q} = \frac{4 - 2}{-1 - 1} = -1 \\ x_R & = m^2 - x_P - x_Q = (-1)^2 - (-1) - 1 = 1 \\ y_R & = y_P + m(x_R - x_P) = 4 + -1 \cdot (1 - (-1)) = 2 \end{align*}$$ We get the result $P + Q = (1, -2)$, which is the same result given by the <a href="https://andrea.corbellini.blog/ecc/interactive/reals-add.html?px=-1&py=4&qx=1&qy=2">visual tool</a>. The case $P = Q$ needs to be treated a bit differently: the equations for $x_R$ and $y_R$ are the same, but given that $x_P = x_Q$, we must use a different equation for the slope: $$m = \frac{3 x_P^2 + a}{2 y_P}$$ Note that, as we would expect, this expression for $m$ is the first derivative of: $$y_P = \pm \sqrt{x_P^3 + ax_P + b}$$ To prove the validity of this result it is enough to check that $R$ belongs to the curve and that the line passing through $P$ and $R$ has only two intersections with the curve. But again, we don’t prove this fact, and instead try with an example: $P = Q = (1, 2)$. $$\begin{align*} m & = \frac{3x_P^2 + a}{2 y_P} = \frac{3 \cdot 1^2 - 7}{2 \cdot 2} = -1 \\ x_R & = m^2 - x_P - x_Q = (-1)^2 - 1 - 1 = -1 \\ y_R & = y_P + m(x_R - x_P) = 2 + (-1) \cdot (-1 - 1) = 4 \end{align*}$$ Which gives us $P + P = -R = (-1, -4)$. <a href="https://andrea.corbellini.blog/ecc/interactive/reals-add.html?px=1&py=2&qx=1&qy=2">Correct</a>! Although the procedure to derive them can be really tedious, our equations are pretty compact. This is thanks to Weierstrass normal form: without it, these equations could have been really long and complicated! <h1 id="scalar-multiplication">Scalar multiplication</h1> Other than addition, we can define another operation: scalar multiplication, that is: $$nP = \underbrace{P + P + \cdots + P}_{n\ \text{times}}$$ where $n$ is a natural number. I’ve written a <a href="https://andrea.corbellini.blog/ecc/interactive/reals-mul.html">visual tool</a> for scalar multiplication too, if you want to play with that. Written in that form, it may seem that computing $nP$ requires $n$ additions. If $n$ has $k$ binary digits, then our algorithm would be $O(2^k)$, which is not really good. But there exist faster algorithms. One of them is the double and add algorithm. Its principle of operation can be better explained with an example. Take $n = 151$. Its binary representation is $10010111_2$. This binary representation can be turned into a sum of powers of two: $$\begin{align*} 151 & = 1 \cdot 2^7 + 0 \cdot 2^6 + 0 \cdot 2^5 + 1 \cdot 2^4 + 0 \cdot 2^3 + 1 \cdot 2^2 + 1 \cdot 2^1 + 1 \cdot 2^0 \\ & = 2^7 + 2^4 + 2^2 + 2^1 + 2^0 \end{align*}$$ (We have taken each binary digit of $n$ and multiplied it by a power of two.) In view of this, we can write: $$151 \cdot P = 2^7 P + 2^4 P + 2^2 P + 2^1 P + 2^0 P$$ What the double and add algorithm tells us to do is: <ul> <li>Take $P$.</li> <li>Double it, so that we get $2P$.</li> <li>Add $2P$ to $P$ (in order to get the result of $2^1P + 2^0P$).</li> <li>Double $2P$, so that we get $2^2P$.</li> <li>Add it to our result (so that we get $2^2P + 2^1P + 2^0P$).</li> <li>Double $2^2P$ to get $2^3P$.</li> <li>Don’t perform any addition involving $2^3P$.</li> <li>Double $2^3P$ to get $2^4P$.</li> <li>Add it to our result (so that we get $2^4P + 2^2P + 2^1P + 2^0P$).</li> <li>…</li> </ul> In the end, we can compute $151 \cdot P$ performing just seven doublings and four additions. If this is not clear enough, here’s a Python script that implements the algorithm: <div class="highlight"><pre><code>def bits(n): """ Generates the binary digits of n, starting from the least significant bit. bits(151) -> 1, 1, 1, 0, 1, 0, 0, 1 """ while n: yield n & 1 n >>= 1 def double_and_add(n, x): """ Returns the result of n * x, computed using the double and add algorithm. """ result = 0 addend = x for bit in bits(n): if bit == 1: result += addend addend *= 2 return result </code></pre></div> If doubling and adding are both $O(1)$ operations, then this algorithm is $O(\log n)$ (or $O(k)$ if we consider the bit length), which is pretty good. Surely much better than the initial $O(n)$ algorithm! <h1 id="logarithm">Logarithm</h1> Given $n$ and $P$, we now have at least one polynomial time algorithm for computing $Q = nP$. But what about the other way round? What if we know $Q$ and $P$ and need to find out $n$? This problem is known as the logarithm problem. We call it “logarithm” instead of “division” for conformity with other cryptosystems (where instead of multiplication we have exponentiation). I don’t know of any “easy” algorithm for the logarithm problem, however <a href="https://andrea.corbellini.blog/ecc/interactive/reals-mul.html?a=-3&b=1&px=0&py=1">playing with multiplication</a> it’s easy to see some patterns. For example, take the curve $y^2 = x^3 - 3x + 1$ and the point $P = (0, 1)$. We can immediately verify that, if $n$ is odd, $nP$ is on the curve on the left semiplane; if $n$ is even, $nP$ is on the curve on the right semiplane. If we experimented more, we could probably find more patterns that eventually could lead us to write an algorithm for computing the logarithm on that curve efficiently. But there’s a variant of the logarithm problem: the discrete logarithm problem. As we will see in the next post, if we reduce the domain of our elliptic curves, scalar multiplication remains “easy”, while the discrete logarithm becomes a “hard” problem. This duality is the key brick of elliptic curve cryptography. <h1 id="see-you-next-week">See you next week</h1> That’s all for today, I hope you enjoyed this post! Next week we will discover finite fields and the discrete logarithm problem, along with examples and tools to play with. If this stuff sounds interesting to you, then stay tuned! <a href="https://andrea.corbellini.blog/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/">Read the next post of the series »</a>andreacorbelliniSun, 17 May 2015 11:24:00 +0000tag:andrea.corbellini.blog,2015-05-17:/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/cryptographybitcoindhdsaeccmathpgprsasecuritysshtlswebLet's Encrypt: the road towards a better web?https://andrea.corbellini.blog/2015/04/12/lets-encrypt-the-road-towards-a-better-web/I’ve always dreamed of a encrypted web, where HTTPS is the standard and plain HTTP is no more. A web where eavesdropping or manipulating information is not possible, or at least much harder than today. I remember that I got excited when I first heard of <a href="http://www.cacert.org/">CAcert</a>: “a community-driven Certificate Authority that issues certificates to the public at large for free”. Unfortunately, CAcert’s root certificate never made it into the major web browsers and operating systems. Whatever the reasons, the result is that visiting a HTTPS website with a certificate released by CAcert produces nothing but a <a href="https://cacert.org/">scary warning with a call to leave the site</a>, making CAcert unsuitable for most. <a href="https://www.startssl.com/">StarCom</a>, on the other hand, has made it into the major browsers. But despite its certificates are released for free, it has never become much widespread. Also, StarCom <a href="https://news.ycombinator.com/item?id=7557764">has</a> <a href="https://www.techdirt.com/articles/20140409/11442426859/shameful-security-startcom-charges-people-to-revoke-ssl-certs-vulnerable-to-heartbleed.shtml">been</a> <a href="https://twitter.com/startssl/status/453631038883758080">heavily</a> <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=994033">criticized</a> for how the Heartbleed vulnerability was handled, and AFAIK this has led many customers away. <h1 id="lets-encrypt">Let’s Encrypt</h1> Recently, I learned about <a href="https://letsencrypt.org/">Let’s Encrypt</a>: a “free, automated, and open” Certificate Authority arriving in mid-2015. There are many important facts that make Let’s Encrypt different and better from all the other Certificate Authorities out there. I’ll let you discover all of them. Probably, the most important fact is that Let’s Encrypt has <a href="https://letsencrypt.org/sponsors/">important sponsors</a>, including Mozilla. And this is what matters today, because it gives Let’s Encrypt a chance to be included in at least one major browser. <figure> <a href="https://letsencrypt.org/"><img src="https://andrea.corbellini.blog/images/letsencrypt-logo-horizontal.png" alt="Let's Encrypt" width="519" height="124"></a> <figcaption>Let's Encrypt logo.</figcaption> </figure> Another interesting fact about Let’s Encrypt is that its certificates are released in <a href="https://letsencrypt.org/howitworks/technology/">a way that is both secure and automated</a> at the same time. This gives the opportunity for other (potential) Certificate Authorities to adopt the same automated system. If Let’s Encrypt wins, then everyone will have an easy way to obtain a free HTTPS certificate for their website. The next big step would be making Let’s Encrypt increase in adoption and the final step would be deprecating plain HTTP. There are however a few open questions: <ul> <li>What will be the answer from Google, Apple, Microsoft and other major browser/operating systems makers?</li> <li>What will be the reaction of Verisign and Comodo? (That together hold <a href="http://w3techs.com/technologies/overview/ssl_certificate/all">more than 50%</a> of all the certificates currently used on the web.)</li> <li>Will they declare war to Let’s Encrypt or will they consolidate their efforts on customer services and Extended Validation?</li> <li>Will the technology behind Let’s Encrypt allow the creation of a new model for certificate management? Will we see web servers and providers with built-in support for it?</li> </ul> I do not have an answer to these questions, time will tell. However I really hope my dream to become a reality soon. If you, like me, want Let’s Encrypt to be a success, then please share and discuss about it. Perhaps, one day, we will find ourselves teaching juniors that HTTPS has not always been the standard… :)andreacorbelliniSun, 12 Apr 2015 16:07:00 +0000tag:andrea.corbellini.blog,2015-04-12:/2015/04/12/lets-encrypt-the-road-towards-a-better-web/information-technologysecuritytlsweblet's encryptRunning Ubuntu Snappy inside Dockerhttps://andrea.corbellini.blog/2015/03/25/running-ubuntu-snappy-inside-docker/Many of you may have already heard of <a href="https://developer.ubuntu.com/en/snappy/">Ubuntu Core</a>. For those who haven’t, it’s a minimal Ubuntu version, running only a few essential services and ships with a new package manager (snappy) that provides transactional updates. Ubuntu Core provides a lightweight base operating system which is fast to deploy and easy to maintain up to date. It also uses a nice <a href="https://wiki.ubuntu.com/SecurityTeam/Specifications/SnappyConfinement">security model</a>. All these characteristics make it particularly appealing for the cloud. And, in fact, people are starting considering it for building their (micro)services architectures. Some weeks ago, a user on Ask Ubuntu asked: <a href="http://askubuntu.com/questions/566736/can-i-run-snappy-ubuntu-core-as-a-guest-inside-docker/577248">Can I run Snappy Ubuntu Core as a guest inside Docker?</a> The problem is that Ubuntu Core does not ship with an official Docker image that we can pull, so we are forced to set it up manually. Here’s how. <h1 id="creating-the-docker-image">Creating the Docker image</h1> <h2 id="step-1-get-the-latest-ubuntu-core">Step 1: get the latest Ubuntu Core</h2> As of writing, the latest Ubuntu Core image is alpha 3 and can be downloaded with: <div class="highlight"><pre><code>$ wget http://cdimage.ubuntu.com/ubuntu-core/releases/alpha-3/ubuntu-core-WEBDM-alpha-03_amd64-generic.img.xz </code></pre></div> (If you browse to <a href="http://cdimage.ubuntu.com/ubuntu-core/releases/alpha-3/">cdimage.ubuntu.com</a>, you can also find the signed hashsums.) The downloaded image is XZ-compressed and we need to extract it: <div class="highlight"><pre><code>$ unxz ubuntu-core-WEBDM-alpha-03_amd64-generic.img.xz </code></pre></div> <h2 id="step-2-connect-the-image-using-qemu-nbd">Step 2: connect the image using qemu-nbd</h2> The file we have just downloaded and extracted is a filesystem dump. The previous version of the image (Alpha 2) was a QCOW2 image (the format used by QEMU). In order to access its contents, we have a few options. Here I’ll show one that works with both filesystem dumps and QCOW2 images. The trick consists in using <code>qemu-nbd</code> (a tool from the <a href="https://apps.ubuntu.com/cat/applications/qemu-utils/">qemu-utils</a> package): <div class="highlight"><pre><code># qemu-nbd -rc /dev/nbd0 ubuntu-core-WEBDM-alpha-03_amd64-generic.img </code></pre></div> This command will create a virtual device named <code>/dev/nbd0</code>, with virtual partitions named <code>/dev/nbd0p1</code>, <code>/dev/nbd0p2</code>, … Use <code>fdisk -l /dev/nbd0</code> to get an idea of what partitions are inside the QCOW2 image. <h2 id="step-3-mount-the-filesystem">Step 3: mount the filesystem</h2> The partition we are interested in is <code>/dev/nbd0p3</code>, so we need to mount it: <div class="highlight"><pre><code># mkdir nbd0p3 # mount -r /dev/nbd0p3 nbd0p3 </code></pre></div> <h2 id="step-4-create-a-base-docker-image">Step 4: create a base Docker image</h2> As suggested on the <a href="https://docs.docker.com/articles/baseimages/">Docker documentation</a>, creating a base Docker image from a directory is pretty straightforward: <div class="highlight"><pre><code># tar -C nbd0p3 -c . | docker import - ubuntu-core alpha-3 </code></pre></div> Our newly created image will now appear when running <code>docker images</code>: <div class="highlight"><pre><code># docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE ubuntu-core alpha-3 f6df3c0e2d74 5 seconds ago 543.5 MB </code></pre></div> Let’s verify if we did a good job: <div class="highlight"><pre><code># docker run ubuntu-core:alpha-3 snappy Usage:snappy [-h] [-v] {info,versions,search,update-versions,update,rollback,install,uninstall,tags,config,build,booted,chroot,framework,fake-version,nap} ... </code></pre></div> Yes! We have successfully added Ubuntu Core to the available Docker images and we have run our first snappy container! <h1 id="installing-and-running-software">Installing and running software</h1> Without wasting too many words, here’s how to install and run the <code>xkcd-webserver</code> snappy package inside docker: <div class="highlight"><pre><code># docker run -p 8000:80 ubuntu-core:alpha-3 /bin/sh -c 'snappy install xkcd-webserver && cd /apps/xkcd-webserver/0.3.1 && ./bin/xkcd-webserver' WARN: AppArmor not available when processing AppArmor hook Failed to get D-Bus connection: Operation not permitted Failed to get D-Bus connection: Operation not permitted ** (process:13): WARNING **: user.vala:637: Can not connect to logind xkcd-webserver 21 kB [======================================] OK WARNING: failed to connect to dbus: org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory Part Tag Installed Available Fingerprint Active xkcd-webserver edge 0.3.1 - 3a9152b8bff494 * </code></pre></div> Now, if you visit http://localhost:8000/ you should see a random XKCD comic. If you have payed attention, you may have noticed a few warnings about AppArmor, DBus and logind. The reason why you are seeing these warnings is pretty simple: we did not start neither AppArmor nor DBus nor logind. Now, generally speaking, we could run init inside Docker and fix these and other warnings. However that’s not what Docker is meant for. So if you want to run AppArmor or similar stuff from inside Docker or LXC, then probably you should consider virtualization. <h1 id="dockerfile">Dockerfile</h1> Once you have created the base Docker image, you can start creating some <code>Dockerfile</code>s, if you need to. Here’s an example: <div class="highlight"><pre><code>FROM ubuntu-core:alpha-3 RUN snappy install xkcd-webserver EXPOSE 8000:80 CMD cd /apps/xkcd-webserver/0.3.1 && ./bin/xkcd-webserver </code></pre></div> This <code>Dockerfile</code> does the same job as the previous command: it installs and runs <code>xkcd-webserver</code> on port 8000. In order to use it, first build it: <div class="highlight"><pre><code># docker build -t xkcd-webserver . </code></pre></div> Check that it has been correctly installed: <div class="highlight"><pre><code># docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE xkcd-webserver latest 260e0116e9e3 3 minutes ago 543.5 MB ubuntu-core alpha-3 f6df3c0e2d74 About an hour ago 543.5 MB </code></pre></div> Then run it: <div class="highlight"><pre><code># docker run xkcd-webserver </code></pre></div> Again, you should see a random XKCD comic on <a href="http://localhost:8000/">http://localhost:8000/</a>. <h1 id="conclusion">Conclusion</h1> That’s all folks! I hope you enjoyed this tiny guide, and if you need help, please ask a question on Ask Ubuntu with the <a href="http://askubuntu.com/questions/tagged/ubuntu-core">ubuntu-core tag</a>, which I’m subscribed to.andreacorbelliniWed, 25 Mar 2015 20:46:00 +0000tag:andrea.corbellini.blog,2015-03-25:/2015/03/25/running-ubuntu-snappy-inside-docker/ubuntudockersnappyubuntuubuntu corexkcdAre LXC and Docker secure?https://andrea.corbellini.blog/2015/02/20/are-lxc-and-docker-secure/Since its initial release in 2008, LXC has become widespread among servers. Today, it is becoming the preferred deployment strategy in many contexts, also thanks to Docker and, more recently, LXD. LXC and Docker are used not only to achieve modular architecture design, but also as a way to run untrusted code in an isolated environment. We can agree that the LXC and Docker ecosystems are great and work well, but there’s an important question that I believe everyone should ask, but too few people are asking: are LXC and Docker secure? <figure> <img src="https://andrea.corbellini.blog/images/broken-chain.jpg" alt="Broken Chain"> <figcaption>A system is as safe as its weakest component.</figcaption> </figure> In order to answer this question, I won’t go deep into the details of what LXC and Docker are. The web is full of information on <a href="http://en.wikipedia.org/wiki/Cgroups#NAMESPACE-ISOLATION">namespaces</a> and <a href="http://en.wikipedia.org/wiki/Cgroups">cgroups</a>. Rather, I’d like to show what LXC and Docker can do, what they cannot do, and what their default configuration allows them to do. My hope is to provide a quick checklist for those who want to go with LXC/Docker, but are unsure on what they need to pay attention to. <h1 id="what-lxc-and-docker-can-do">What LXC and Docker can do</h1> As we all know, LXC confines processes mainly thanks to two Linux kernel features: namespaces and cgroups. These provide ways to control and limit access to resource such as memory or filesystem. So, for example, you can limit the bandwidth used by processes inside a container, you can limit the priority of the CPU scheduler, and so on. As it is well known, processes inside a LXC guest cannot: <ul> <li>directly interact with the host processes, or with other LXC containers;</li> <li>access the root filesystem, unless configured otherwise;</li> <li>access special devices (block devices, network interfaces, …), unless configured otherwise;</li> <li>mount arbitrary filesystems;</li> <li>execute special <code>ioctl</code>s, special syscalls or special interrupts, that would affect the behavior host.</li> </ul> And at the same time, processes inside an LXC guest can find an environment that is perfectly suitable to run a working operating system: I can run init, I can read from <code>/proc</code>, I can access the internet. This is most of what LXC can do, and it’s also what you get by default. Docker (when used with the LXC backend) is a wrapper around LXC that provides utilities for easy deployment and management of the containers, so everything that applies to LXC, applies to Docker too. If this sounds great, then beware that there are the things you should know… <h1 id="you-need-a-security-context">You need a security context</h1> LXC is somewhat incomplete. What I mean is that some parts of special filesystems like procfs or sysfs are not faked. For example, as of now, I can successfully change the value of host’s <code>/proc/sys/kernel/panic</code> or <code>/sys/class/thermal/cooling_device0/cur_state</code>. The reason why LXC is “incomplete” doesn’t really matter (it’s actually the kernel to be incomplete, but anyhow…). What matters is that certain nasty actions can be forbade, not by LXC itself, but by an AppArmor/SELinux profile that blocks read and write access certain <code>/proc</code> and <code>/sys</code> components. The AppArmor rules were shipped in Ubuntu since 12.10 (Quantal), and have been included upstream since early 2014, together with the SELinux rules. Therefore, a security context like AppArmor or SELinux is required to run LXC safely. Without it, the root user inside a guest can take control of the host. Check that AppArmor or SELinux are running and are configured properly. If you want to go with Grsecurity, then remember to configure it manually. <h1 id="limit-resource-consumption">Limit resource consumption</h1> LXC offers ways to limit resource usage, but no special restrictions are put in place by default. You have to configure them by yourself. With the default configuration, I can run fork-bombs, request huge memory maps, keep all CPUs busy, doing high loads of I/O. All of this without special privileges. Remember this when running untrusted code. <figure> <img src="https://andrea.corbellini.blog/images/memory-usage.png" alt="Uncontrolled memory consumption"> </figure> To limit resource consumption in LXC, open the configuration file for your container and set the <code>lxc.cgroup.<system></code> values you need. For example, if you want to limit the container memory usage to 512 MiB, set <code>lxc.cgroup.memory.limit_in_bytes = 512M</code>. Note that the container with that option, once it exceeds the 512 MiB cap, will start using the swap without limits. If this is not what you want, then set <code>lxc.cgroup.memory.memsw.max_usage_in_bytes = 512M</code>. Note that to use both options you may need to add <code>cgroup_enable=memory</code> and <code>swapaccount=1</code> to the kernel command line. To have an overview of all possible options, check out <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch-Subsystems_and_Tunable_Parameters.html">Red Hat’s documentation</a> or the <a href="https://www.kernel.org/doc/Documentation/cgroups/">Kernel documentation</a>. With Docker, the story is similar: just use <code>--lxc-conf</code> from the command line to set LXC’s options. <h1 id="limit-disk-usage">Limit disk usage</h1> Something that LXC cannot do is limiting mass storage usage. Luckily, <a href="https://www.stgraber.org/2013/12/27/lxc-1-0-container-storage/">LXC integrates nicely with LVM</a> (and brtfs, and zfs, and overlayfs), and you can use that for easily limiting disk usage. You can, for example, create a logical volume for each of your guests, and give that volume a limited size, so that space usage inside a guest cannot grow indefinitely. The same <a href="http://developerblog.redhat.com/2014/09/30/overview-storage-scalability-docker/">holds for Docker</a>. <h1 id="pay-attention-at-devrandom">Pay attention at <code>/dev/random</code></h1> Processes inside LXC guests, by default, can read from <code>/dev/random</code> and can consume the entropy of the host. This may cause troubles if you need big amounts of randomness (to generate keys or whatever). If this is something that you don’t want, then configure LXC so that it <a href="https://wiki.archlinux.org/index.php/Linux_Containers#Cgroups_device_configuration">denies access to the character devices</a> <code>1:8</code> (random) and <code>1:9</code> (urandom). Denying access to the path <code>/dev/random</code> is not enough, as <code>mknod</code> is allowed inside guests. Note however that doing so may break many applications inside the LXC guest that need randomness. Maybe consider using a different machine for processes that require randomness for security purposes. <h1 id="use-unprivileged-containers">Use unprivileged containers</h1> Containers can be <a href="https://www.stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/">run from an unprivileged user</a>. This means UID 0 of the guest can’t match UID 0 of the host, and many potential security holes can’t simply be exploited. Unfortunately, <a href="https://github.com/docker/docker/issues/2918">Docker has not support for unprivileged containers</a> yet. However, if Docker is not a requirement and you can do well with LXC, start experimenting with unprivileged containers and consider using them in production. Programs like Apache will complain that it’s unable to change its ulimit (because setting the ulimit is a privilege of the real root user). If you need to run programs that require special privileges, either configure them so that they do not complain, or consider using <a href="http://linux.die.net/man/7/capabilities">capabilities</a> (but do not abuse them, and be cautious, or you risk introducing more problems than the ones your are trying to solve!) <h1 id="conclusion">Conclusion</h1> LXC, Docker and the entire ecosystem around them can be considered quite mature and stable. They’re surely production ready, and, if the right configuration is put in place, it can be pretty difficult to cause troubles to the host. However, whether they can be considered secure or not is up to you: what are you using containers for? Who are you giving access to? What privileges are you giving, what actions are you restricting? Always remember what LXC and Docker do by default, and what they do not do, especially when you use them to run untrusted code. Those that I have listed may only be a few of the problems that LXC, Docker and friends may expose. Remember to carefully review your configuration before opening the doors to others. <h1 id="further-reading">Further reading</h1> If you liked this article, you’ll find these ones interesting too: <ul> <li><a href="http://blog.docker.com/2013/08/containers-docker-how-secure-are-they/">Containers & Docker: how secure are they?</a>, from the Docker blog.</li> <li>Stéphane Graber’s <a href="https://www.stgraber.org/2014/01/01/lxc-1-0-security-features/">Security features</a> from his <a href="https://www.stgraber.org/2013/12/20/lxc-1-0-blog-post-series/">LXC 1.0: Blog post series</a>.</li> </ul>andreacorbelliniFri, 20 Feb 2015 16:36:00 +0000tag:andrea.corbellini.blog,2015-02-20:/2015/02/20/are-lxc-and-docker-secure/information-technologydockerlxcsecurityPrime numbers and universe factorieshttps://andrea.corbellini.blog/2015/02/15/prime-numbers-and-universe-factories/I’m a XKCD fan, and I look it up regularly. There’s a comic that I particularly enjoyed: <a href="http://xkcd.com/10/">Pi Equals</a>. <figure> <a href="http://xkcd.com/10/"><img src="http://imgs.xkcd.com/comics/pi.jpg" width="469" height="247" alt="Pi Equals"></a> <figcaption>The comic <a href="http://xkcd.com/10/" title="Pi Equals">Pi Equals</a>, from XKCD.com (CC-BY-NC 2.5).</figcaption> </figure> Well, it appears that Randall was right in that there’s a help message hidden somewhere. And I just found it in a prime number: <div class="highlight"><pre><code>245178888024581899558766786108789912235672909204719666025638877624752119760547413887830514281649480308707369249 </code></pre></div> That number corresponds to the ASCII encoding of this message: <div class="highlight"><pre><code>help!! i'm trapped in a universe factory!!!!!! </code></pre></div> Apparently, universe factory workers speak English and write ASCII. Nice coincidence, huh? <h1 id="the-discovery">The discovery</h1> Yesterday I was playing with the two <a href="https://en.wikipedia.org/wiki/Illegal_prime">illegal primes</a> listed on Wikipedia. I was already aware of them, but I had never decoded them till yesterday. While doing so I wondered: how many prime numbers can be directly mapped to an executable file? Also, how many prime numbers can be directly mapped to plain English texts? Perhaps, while digging prime numbers, could we find something like the Iliad or a fully working operating system? Well, while asking myself those highly philosophical questions, Randall’s comic quickly came to my mind, and I decided to start looking for help requests hidden in primes. You can’t imagine how many of them I found! At first I tried looking for all prime numbers corresponding to strings starting with <code>HELP! I'M TRAPPED IN A UNIVERSE FACTORY!</code>, with an arbitrary suffix. I found many of them, but I wasn’t satisfied with the result: I wanted something that was purely English/ASCII, without any garbage. Therefore I tried appending hashtags like <code>#help</code> or <code>#universe</code>, but could not find any interesting combination that was also a prime number (apparently, use of Twitter is forbidden inside universe factories). So I decided to change approach: I looked for all primes corresponding to <code>HELP</code>, followed by a variable number of exclamation marks, followed by <code>I'M TRAPPED IN A UNIVERSE FACTORY</code>, followed by other exclamation marks. I could not find anything. But then I tried with a lower case string, and… I found lots of such primes! <div class="highlight"><pre><code>help i'm trapped in a universe factory!!!!!!! help! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!! i'm trapped in a universe factory!!!!!! help!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!!!!! i'm trapped in a universe factory!!!! help!!!!!!!!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!!!!!!!!! i'm trapped in a universe factory! help!!!!!!!!!!!!!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!!!!!!!!!!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!!!!!!!!!!!!! i'm trapped in a universe factory!!!!!!!!!! help!!!!!!!!!!!!!!!!!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!!!!!!!!!!!!!!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!!!!!!!!!!!!!!!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!!!!!!!!!!!!!!!!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!!!!!!!!!!!!!!!!!!!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!!!!!!!!!!!!!!!!!!!!!! i'm trapped in a universe factory!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! help!!!!!!!!!!!!!!!!!!!!!!!!!!! i'm trapped in a universe factory!!!!!!! ... </code></pre></div> I picked the one I liked most and verified its primality with <a href="http://www.wolframalpha.com/input/?i=is+245178888024581899558766786108789912235672909204719666025638877624752119760547413887830514281649480308707369249+prime%3F">Wolfram|Alpha</a> and <a href="http://www.numberempire.com/primenumbers.php">numberempire.com</a>. I’m not 100% sure that all the others are primes, as I used <a href="https://en.wikipedia.org/wiki/Fermat_primality_test">Fermat primality test</a>. However I’m impressed by what I found. Now I can’t stop wondering how much literature, physics or technology could be hidden in prime numbers, in plain English and UTF-8 encoded. :D (Obviously, I’m perfectly conscious on what’s happening here, but I though this was a nice fact to share. It could also be a nice number to print on a shirt.) Dear universe factory worker, I’m going to rescue you, sooner or later. Just tell me how.andreacorbelliniSun, 15 Feb 2015 16:54:00 +0000tag:andrea.corbellini.blog,2015-02-15:/2015/02/15/prime-numbers-and-universe-factories/numbersfunmathNew blog, againhttps://andrea.corbellini.blog/2015/02/15/new-blog-again/This must be the third blog I start from scratch. But this time, I’m taking a serious commitment: I’m going to write here regularly. Wish me luck!andreacorbelliniSun, 15 Feb 2015 12:23:00 +0000tag:andrea.corbellini.blog,2015-02-15:/2015/02/15/new-blog-again/blogmiscblog