NCrunch Blog

NCrunch MCP Server

Mon, 20 Jul 2026 03:47:00 GMT

The life of a software developer during what seems like the unfolding singularity is definitely not boring at the moment. Agentic AI tooling is having a big impact on the way we now write code, and v5.22 is focused on bringing NCrunch up to speed.

Agentic Software Development on the NCrunch Solution

I've been receiving a lot of requests from users that want their AI agents to be able to interact with NCrunch through an MCP server. I've been doing more and more work with agents myself, and the feature gap is real. Over the past few months, I've been experimenting with ways to enable AI integration with NCrunch by allowing the engine to be steered and queried from running agents both inside and outside the IDE.

It's been an interesting journey, involving a lot of excitement, confusion, and more than a little frustation fused with occasional moments of disbelief. I'm amazed at what agents can do. It's also so spectacular just how wrong they can go without the right guidance.

One of the first things that was clear to me with this project was that constraints were needed. Frontier AI models are very well aligned and it seems we can trust their intentions, and hallucination seems to be less of a problem than it was two years ago. Unfortunately, they suffer from a lack of domain knowledge and still seem to struggle somewhat with complex real-world scenarios. So for me, the first stage of this project was almost entirely about infrastructure.

The vision was to build a constellation of Debian and Windows based VMs that could support the environment needed for an agent to perform operations not only on a single repository, but over a network containing a range of different services, such as a CI server, Git Repo, file server, NCrunch grid nodes, test environments, etc. In every major software project I've worked on, infrastructure was always a big time sink, and I wanted to be able to automate as much of it as possible.

The eventual result is something that I feel strikes the right balance between paranoia and agentic madness. An OpenClaw-based server hosts a range of different agents with different specialties (i.e. infrastructure management, NCrunch solution coding, conceptual design), with gated/firewalled access to critical systems that allows them to run scripts with human approval. Agents have fairly broad read access and are able to draft their own changesets which must be reviewed before commit. In this way, I can direct agents to fix infrastructure problems, troubleshoot bugs and add features without risking them manically installing external software or rewiring systems without authorisation.

The critical piece of all this, unsurprisingly, was NCrunch itself. For a range of reasons, NCrunch itself is the only tool that can run its own tests, so without a working MCP server, it was often impossible for agents to verify their own work on the NCrunch codebase. This led to a bizarre mash-up of human directed effort with unhinged agents optimistically churning out MCP scaffolding, only to result in hours of human-driven untangling. The critical lesson: AI self-verification of work is extremely important.

Once the MCP server was operational in its most basic state (allowing agents to start sessions, run tests and read results), everything got easier. It became possible to direct agents to extend the server themselves, adding functions as directed and designing signatures in ways they could easily operate. I still needed to be quite involved around integration with the engine itself, particularly around threading constraints, but the rate of acceleration was quite remarkable.

I'm quite happy with the result. It's probably of similar quality to what I could have eventually done on my own, but required much less effort and was built in a tighter timeframe than I expected. I thought I would be releasing the MCP server in v5.23, but instead it's ready for v5.22! It's still a bit experimental though.

IDE Integration

By popular request, NCrunch's MCP server has been designed to integrate with GitHub Copilot under Visual Studio and the JetBrains AI Assistant under Rider. This fills the most obvious feature gap of allowing IDE-hosted agents to use NCrunch to run tests and query results without needing to rely on other tooling (i.e. dotnet test).

My preference was to set this up so that NCrunch's MCP server would auto register with the IDE hosted agent on startup or installation, so there'd be nothing you would need to enable to make use of it. Reality eventually scuttled this plan in the following ways:

Security considerations make it a bad idea. The standing convention around agents is that they should be distrusted until proven otherwise, and giving them full power over NCrunch by default seems like a decision that a user should make deliberately.
Auto registration is somewhat ropey. At present there isn't a single API call that will do this and anything that stands a chance of doing it reliably involves a number of steps and decisions (i.e. should it be global or per solution? what should it run?) that a user should be involved in. Broadly, it's complicated, and complicated things tend to break.
There is massive change ongoing in this area. Both Microsoft and JetBrains are investing a lot of time in building and extending these tools. That makes it hard to rely on any integration point being future-proof. It wouldn't be a good experience if auto-registration only resulted in failure and confusion as soon as the tooling was updated.

So for the time being, you'll need to set up NCrunch's MCP server manually using your chosen IDE. This should be as simple as just pointing the IDE to NCrunch.MCP.exe and authorising the various functions you want to have available to your agent.

When running under the IDE, the MCP server behaves a bit differently to how it does in fully headless mode. Under VS, it uses an environment variable to identify the running engine, which it then hooks up to. Under Rider, it uses Win32 APIs to traverse the process tree to find a common ancestor (Rider64.exe). It's important that your NCrunch engine is running when you let an agent try to steer it, otherwise you'll get complaints about timeouts.

See here for more details on how to set things up.

Headless Mode via Console Tool

When installed through the NCrunch console tool and run directly via the command line, NCrunch.MCP.exe enters detached/headless mode for use by fully autonomous agents such as OpenClaw and Hermes. This is the mode I mostly use myself and it's the one that I'm personally the most excited about.

The headless MCP server acts as a front-end for what is able to be multiple NCrunch engine sessions. It's possible for an agent to use the HTTP interface to start the engine for a specific solution, run tests, extract test coverage data, run builds, read results, etc. Sessions can be left running, and they use all of NCrunch's file synchronisation systems. This means that an agent (or human) can modify source code on disk and the session will automatically load the changes, run builds, apply impact detection logic and automatically run tests according to selected engine mode.

In a manner of speaking, this turns NCrunch from an IDE-integrated toolset into an agent facing IDE all on its own, though minus a lot of the features modern IDEs currently give us, like source navigation, find usages, and so on.

To enable headless mode, you'll need to specify the /http parameter with an address, listening port and secret. See here for more details.

Regarding RDI

As of v5.22, RDI features are not yet surfaced to agents. Sessions can have RDI enabled and will record data as usual, but there aren't any functions exposed for agents to read it. I think that RDI has massive potential for agents because it will allow them to efficiently debug code without actually needing to use a debugger. It's going to take some experimentation to get this right though, as RDI involves massive amounts of data that agents need to be able to efficiently sift through.

I do also expect it to be tricky to guide agents through working with RDI data. The existing functions for running tests and querying results involved quite a bit of fine tuning to stop agents from trying to treat NCrunch like a static runner where they'd just target a specific test and wait for a result. It's clear that the models are better tuned for static end-to-end testing than they are for working with a fully continuous tool, so I've found that careful documentation around the functions is very important.

Other Things in v5.22

NCrunch v5.22 isn't limited to just MCP work. It's also a full maintenance release containing a range of fixes and performance improvements, along with support for JetBrains Rider 2026.2 (released today!). See here for the full list of changes, or go download it now!

Core Control for Execution, Lots of Fixes

Sun, 22 Mar 2026 23:28:00 GMT

I'm happy to announce the release of NCrunch v5.21.

While mostly a maintenance release, this new version also includes a niche feature area that may appeal to people running large tests in resource constrained environments.

Control Cores Used For Test Execution

The Control Cores Used For Test Execution setting defines a new mode of operation for the NCrunch engine, in which the engine will attempt to control the CPU cores on which test code will run. In this way, tests are allocated fixed CPU resources for their execution, rather than relying on the O/S to pool CPU resources across all runner processes.

This approach comes with both advantages and disadvantages.

A test that has been allocated a CPU core will be limited by it and all threads underneath the test will be scheduled to run on the same core. This helps to prevent overstepping, where a test would otherwise be able to take over all CPU resources on the system, causing resource starvation and timeouts.

The main drawback of this approach is that is can cause underutilisation. Tests that might have all their threads in a waiting state won't use the entire core they've been allocated, which might otherwise have been available for other tests that could make use of it.

So enabling this setting could be expected to improve the predicability of testing cycles, while also reducing the throughput of the engine. It may also offer some potential for performance improvement in NUMA based environments, though this is largely theoretical and will likely be extremely context-specific.

This setting is relevant in the global and grid node levels only. I realise this is somewhat limiting, as it would be more useful to also be able to configure it at solution level. However, I couldn't find a practical way to make it work in this way due to it being impossible to design a grid node that could cater for clients connecting with different values for this setting. If the setting proves popular, maybe I can find a way to have a solution-level option that isn't used for distributed processing.

I'll note that NCrunch's implementation of this setting is best effort only. The design of the Windows operating system makes it impossible to fully control all processing activity on the system, and the APIs used to control core affinity have serious limitations that require some degree of compromise.

I strongly recommend having a read of the documentation for this setting before you put it to use.

JetBrains Rider 2026.1 Support

NCrunch v5.21 introduces support for Rider 2026.1. This was a relatively straightforward upgrade with minimal changes necessary, so I'm hopeful there won't be any unexpected issues. As usual, note that NCrunch only supports the latest version of Rider at the time of its release, so v5.21 will work only on Rider 2026.1.

Discontinued Support for Visual Studio 2010-2015

Those paying close attention to the download options for v5.21 will notice that this is the first release of NCrunch that does not have support for versions of VS prior to 2017.

Although I'm not particularly happy about discontinuing support for these older IDEs (it took a lot of effort to introduce it!), I've eventually come to accept that the cost of maintaining integration with them is no longer worth the effort.

Very few people still use these older IDEs, and the ones that do can still be well serviced by v5.20 of NCrunch, which will remain accessible on the download page of this site.

Maintaining support for these IDEs complicates many areas of NCrunch, particularly around the management of test environments. Microsoft no longer support these IDEs or platforms they run on, so the only way to test for them is to lean on decaying environments running ancient (and often insecure) versions of software. Pulling the plug has allowed me to upgrade NCrunch's test infrastructure to improve reliability and performance. It also means that I won't need to implement complex workarounds for features and fixes that might need to work differently for older VS versions.

I apologise to anyone impacted by this, but please understand that eventually things need to move on.

Lots of Fixes

A lot of fixes have been rolled into the new version. Most of these are targeted to Rider, but there are also some important improvements that are relevant to Visual Studio.

See here for a full list of changes. V5.21 is available here.

Visual Studio 2026

Sat, 13 Sep 2025 10:05:00 GMT

The first preview (or 'Insiders') build of Visual Studio 2026 was made public this week. With a 4 year gap from the last major release (2022), this is the longest wait we've had between major versions of Visual Studio.

As is typical for a major release of VS, the release notes detail a lot of extensive improvements to the IDE. Most of the changes involve adding extra features to Copilot to enable more AI tooling integrated with the IDE.

Patrick Smacchia has a good analysis of the current state of MS's migration of VS to the newer .NET platform and away from .NET Framework. Sensibly, they have been moving individual components over to netstandard so that they can eventually flick the switch over to .NET 10 (or whatever will be the latest .NET at the time). It looks like they still have a long way to go, but it doesn't look like a small task.

Since adding support for Rider, NCrunch has been able to host its engine under newer versions of .NET, so I'm hopeful that when VS leaves .NET Framework the process of switching over should be simple for NCrunch. However, for the time being, we're still on .NET Framework, which has made adding support for VS2026 much easier. MS have also done a remarkable job with backwards compatibility with the first 2026 build.

Consequently, I'm happy to say that NCrunch now has full support for VS2026 just 4 days after it was released. The v5.18 release includes a single item in the release notes: Visual Studio 2026 Support. It's now available for download!

Rider 2025.1, MSTest v3.8, More fixes

Thu, 10 Apr 2025 04:23:00 GMT

NCrunch v5.13 is now available. This release is actually quite a big one in terms of the sheer volume of fixes and compatibility improvements.

JetBrains Rider 2025.1 Support

V5.13 introduces support for the newest version of Rider, and is timed to be released at more or less the same time (while writing this, 2025.1 has just hit RC state and RTM is likely only hours away). There were a few changes in 2025.1 that required adjustments in NCrunch, so v5.13 will not be able to run on versions prior to 2025.1 and v5.12 is incompatible. As usual, this means that the version of NCrunch you have installed needs to be aligned with the version of Rider you are using.

MSTest v3.8 Support

The recently released MSTest v3.8 introduced a number of improvements to MSTest, particularly around dynamic data tests. Of note also is the introduction of RetryAttribute, which can be used to force the runner to retry a test a number of times to suppress intermittent behaviour.

NCrunch's support for MSTest is emulated rather than integrated. This means that when new features are introduced in MSTest, NCrunch needs to have the same features added to its emulator before they become available for use. The reason for this is largely historical. Early versions of MSTest were so difficult to integrate with that it proved much simpler to just reproduce its most commonly used features inside an adapter that was better tuned to NCrunch's architecture. This approach has proven a wise choice over the multiple rewrites MSTest has undergone over the last 15 years, in which surgical changes to the adapter were easier than building a whole new one every few years. It also provides more opportunity for performance optimisation.

The downside is that NCrunch's emulator is not 100% aligned with the standard MSTest runners, so you might notice some small differences, particularly around edge cases. It also hasn't proven to be practical to implement all features of MSTest, since many of these don't seem to be widely used and some are very expensive to fully align with (such as coded UI tests and data driven tests).

Anyway, V5.13 introduces support for a number of the headline features in MSTest v3.8, more specifically:

DataRow tests using generics
TestDataRow-enabled dynamic data tests
Ignore attributes on data rows
Auto-detection of property vs method based dynamic data tests
RetryAttribute

Note that support for RetryAttribute does not extend to custom implementations of this attribute. I do also recommend using NCrunch's RDI and Churn Mode on tests that intermittently fail to try and stabilise them properly before considering the use of this attribute, as using it extensively can significantly increase the time taken to run tests.

Support for .SLNX Files

It's great to see the old .sln solution files getting cleaned up, and I'm happy to say that NCrunch now supports the new (and much cleaner) .slnx files. I expect that these files will become more widespread as the overall .NET toolset expands to support them better.

More Fixes and Improvements

As is typical with every release, v5.13 brings a range of general fixes and improvements. I had hoped that it would be possible to remove the 'BETA' tag for Rider support in this release, but I feel that we aren't quite there yet (though we're getting close!).

See here for the full list of changes.

Fixes Fixes Fixes

Thu, 20 Feb 2025 08:18:00 GMT

NCrunch v5.12 is now available. This release focuses entirely on stability and performance fixes, with a few of these deserving of note.

Xunit 3 1.1.0 Compatibility Fix

Version 1.1.0 of Xunit was released to Nuget about two weeks ago, including a change that checks to ensure that the Xunit test project is compiled as an executable with AppHost options enabled.

Under NCrunch v5.11, this check will fail every time, because NCrunch forces the UseAppHost build property to 'false', since the app host normally isn't required by NCrunch and there are a few downstream problems that can sometimes appear from having it enabled (mostly impacting synchronisation with grid nodes).

The check makes sense in the context of Xunit v3, as this version of Xunit relies on the test project itself being invokable as an EXE - an important change in Xunit as the test framework can then rely on the platform building the test environment.

However, NCrunch uses its own hosting infrastructure that is independent of this, so the app host isn't needed. NCrunch v5.12 thus disables the check and everything should work as normal.

File Synchronisation Changes

NCrunch relies heavily on FileSystemWatcher instances to keep track of files that change in the foreground solution, so that the contents of these files can then be synchronised with the engine. This is important because not all files are open in the IDE at a given point in time, and we still need to be able to push their contents into workspaces (perhaps even uploading to remote grid nodes) to keep everything in alignment. The previous approach for this has been to use a single FileSystemWatcher per project, with its target directory being set to the point of the lowest file in the directory structure being used by the project.

The cost of this can be significant in solutions with hundreds of projects making use of files several levels above their project directory, as it can cause a situation where we have several hundred watchers all assessing file system updates for the entire solution tree. When a large number of updates happen on disk, the cost of tracking everything quickly becomes overwhelming. The nature of this cost also has a heavy impact, as the FileSystemWatcher internally relies on I/O completion ports that are tied into the operating system with strict resource limitations. When a large number of updates are issued, these ports block up and actually start slowing down operations on the file system itself.

There have also been problems in the past related to directory locks being held by watchers, which can be annoying when making structural changes to a solution with NCrunch enabled.

So, under NCrunch v5.12, this approach has been revised. The watching of solution paths is now managed centrally to exclude redundant watchers. So in a situation where 100 projects all watching for changes at the base solution path, we will now have just one FileSystemWatcher that broadcasts updates internally.

In practice, this resolves the issue with I/O completion ports being jammed up and removes the overhead of O/S resources being overutilised. However, this does move the performance issue into the NCrunch engine itself, where it may (at times) take the engine a while to catch up if there are a massive number of operations on the file system in a short period of time.

I'm hopeful no further optimisation will be needed here, but I'm open to feedback on whether people are still having problems with NCrunch when moving large numbers of files inside their solution (i.e. on foreground builds that copy tens of thousands of files).

Console Support for Dotnet Build

It's now theoretically possible to use the NCrunch console tool without Visual Studio installed on the machine, using the .NET SDK for building projects instead.

Previously, when running the console tool with VS, you'd use something like the command line: ncrunch.exe /VS 2022

Now, it's possible to use: ncrunch.exe /VS dotnet

When this is specified, NCrunch will attempt to use installed versions of dotnet instead of VS. If you need to specify a version of the SDK to use, you can make use of the BuildSdk configuration setting, for example: ncrunch.exe /VS dotnet -BuildSdk 9.0.200

Note that some MS toolsets may still have dependencies on Visual Studio, so it's possible some solutions may still require VS to be installed on the machine.

Jetbrains Rider Fixes

v5.12 contains a disproportionate number of fixes for NCrunch under Rider. With Rider integration still being very new, there's naturally a lot of issues in this area still being worked through. I'm hopeful around getting Rider support out of Beta sometime in the next few months. Using Rider under v5.12 is feeling quite stable now and I'm personally finding issues much less often than before.

Rider 2024.3 Support, VS and Xunit fixes

Thu, 14 Nov 2024 05:24:00 GMT

NCrunch v5.11 is now available for download.

JetBrains Rider 2024.3 Support

This build extends support to the Rider 2024.3 release that went out a few hours ago. 2024.3 contained several breaking changes and it took a bit of effort to bring us into alignment, so there's no way of running this version of Rider without an NCrunch update. Note that each NCrunch build is only intended to target a single version of Rider, though in some cases we are able to support a range of versions depending on what has changed between them.

When running with Rider, you should always install the version of NCrunch that has been marked as tested against that particular version (this is noted on the NCrunch download page). If you are installing NCrunch via the JetBrains Market, this should be resolved automatically.

Visual Studio 2022 17.12 Support

This week also saw the release of another version of VS, which includes the introduction of .NET 9.0. We've been forwards compatible with .NET 9.0 for a while now, though prior to VS 17.12 hitting RTM, it was necessary to make a configuration adjustment (Build SDK) for this to work reliably with NCrunch. Now that 17.12 is properly released, this adjustment should no longer be needed.

17.12 saw some internal changes in how previously opened tool windows are initialised during IDE startup, which required a fix from NCrunch's side. Earlier versions of NCrunch should also work with 17.12, but you'll need to close and reopen your tool windows if you open a solution directly without going through the IDE's startup solution selection screen.

Xunit v3 Adjustments

NCrunch v5.11 saw some changes in how we handle integration with Xunit v3. Previously, we were packaging the Xunit runner binaries with NCrunch and relying on these to interact with the Xunit binaries being referenced from open test project(s). This created a problem when the packaged binaries were not the same version as the version of Xunit being targeted, which could cause instability inside Xunit, usually resulting in weird errors or tests not being discovered.

Under v5.11, we no longer distribute Xunit binaries. Instead, we fish them out of the Nuget package cache directory on the machine responsible for executing tests, according to the version being referenced by the test project. In this way, the versions are always consistent and we shouldn't have instability. The drawback of this approach is that it will only work if the Xunit runner binaries are installed in the package cache. To mitigate this issue, our change has been matched on the Xunit side where the runner binaries should now always be resolved and installed along side the xunit.v3.core package.

It's important to note that if the above setup doesn't work for you, it's still possible to force NCrunch to resolve Xunit v3 binaries by simply adding an assembly reference to them. Assembly references will always take precedence over package cache resolution.

NCrunch's integration with Xunit v3 is late bound and is not statically tied to any version. This means that as long as the Xunit v3 APIs stay stable (which they generally do), NCrunch should be both forwards and backwards compatible with all versions of Xunit v3 from the upcoming RTM release onward.

Xunit v3 Support and continued fixes

Thu, 26 Sep 2024 00:46:00 GMT

I've just published the release of NCrunch v5.10, consisting of another round of fixes to address recently reported and discovered issues.

We've had a number of releases since the big push of v5 earlier this year, and the goal continues to be to improve stability and move our Rider support out of a beta state.

Xunit V3 Support

Worth mentioning in v5.10 is the release of our new Xunit test adapter. This is a whole new adapter built from the ground up to target Xunit v3, which is getting closer to its RTM release.

I'd like to thank Brad Wilson for his support in developing this adapter. Test framework integration is often a complex and messy process, but his efforts on the side of Xunit made this smooth and relatively painless. I'm hopeful that the new adapter should be forwards compatible with new versions of Xunit. If not, I'm confident that we can quickly address any issues.

v5.10 is now available on the download page. Full release notes are also available.

Fixes and Improvements for V5 Release

Thu, 28 Mar 2024 03:50:00 GMT

It makes me proud to look back on our launch of V5 of NCrunch to see that we've managed to make many people happy with this release. A release of such a size always brings with it a massive feeling of anxiety as it approaches. There are countless things that can go wrong when 3 years of development goes out the door all at once. Although we've seen a range of issues come in since the big push, we've been able to promptly fix the serious ones, and the major features (JetBrains Rider support and RDI) seem to be doing rather well.

Another Fix Release

Today, we've released 5.4. Like the three other minor versions we've released before it, this build is all about fixing issues and making improvements to RDI and our Rider support.

Included in v5.4 we have a range of fixes targeting some of the more niche areas of our Rider plugin, such as DPI scaling and UI responsiveness. Our Rider plugin is rapidly solidifying, with performance and reliability improving with every update.

We've also introduced some fixes for the handling of async methods in RDI. Async/await is an extremely challenging construction for us to support with RDI, as it involves multiple threads coordinating over a single method and needing to be stitched together to be shown in one coherent RDI frame. This involves a lot of tracking down oddities in async that can be caused by race conditions or custom logic in the threading context. I'm hopeful that we've now managed to pin these issues down, but if you are able to build an async method that RDI can't handle correctly, please do take the time to share it with us.

Please see here for the full list of changes in v5.4.

Future Plans

At the moment, our focus is still on improving the stability and performance of V5 so that it lives up to its promises. This means we'll continue to push small, regular updates over the next few weeks to address reported issues as they come in.

Beyond this, we hope to eventually start working more on addressing some of the larger gaps around the new V5 features.

We are also finalising our submission to the JetBrains Marketplace to make our Rider plugin more accessible and hopefully open up NCrunch to a wider range of users.

I can't express how happy I am to have V5 be received as warmly as it has been. I want to thank everyone that's helped to make it a success.

NCrunch V5 Now Available

Tue, 27 Feb 2024 23:39:00 GMT

I'm happy to announce that NCrunch version 5 is now available for download.

This is the first major feature release we've had in a bit over 4 years, and in terms of both effort and feature-set, it's the largest release in the history of the product.

V5 contains two new feature areas.

JetBrains Rider Support

By enthusiastic user request, we now have a plugin that supports JetBrains Rider.

This wasn't easy to do, as the frontend for Rider is a JVM based application, while NCrunch is developed in .NET. We host the NCrunch engine outside the IDE in our own .NET process, then pipe much of the UI into Rider through shared memory.

I recommend reading my previous blog post about our support for Rider, particularly if you're interested in how the plugin works and some of the history behind it.

Windows Only

Please note that even though we now have support for Rider, we are still limited to the Windows platform and cannot run on Linux or MacOS. Our UI is still heavily dependent on Windows and we make heavy use of Win32 for internal optimisations, so porting over to another OS would be a serious project.

BETA State

I will also stress that as of now, our Rider plugin is most definitely in a BETA state. It hasn't yet seen the sort of rigorous real-world testing that we've had with our VS plugin, and this is the first time it's been exposed to a wide user base. Stuff will break. There will likely be performance issues we haven't yet found or addressed. Unfortunately, there isn't a way for us to ship a highly reliable plugin on the first try, and it will take us a while to work through the issues and solidify what we have. Please take the time to report problems you encounter using the bug report option or by going through the support forum. If you are able to share clear steps for how to reproduce a problem, this will help us to provide a speedy fix.

Limitations

The Rider plugin has two limitations that presently do not exist with our VS plugin:

No Support for Gallio/MbUnit: Because of the need to host our engine under .NET6 to work with the Rider backend, our integration with Gallio (and MbUnit) could not be easily ported over, as this framework is now very old and has been out of maintenance for quite a while. We examined options for sandboxing it, but decided that with so few people still using it, the effort to maintain this integration point was simply not worth it.

No Support for Xunit v1: Our Xunit v1 support goes via Gallio, so this is being removed for the same reason. Xunit v2 still works fine, so you have a clear migration path if you want to use NCrunch with Rider and Xunit.

It's expected that the next version of Visual Studio (2024?) will be running on .NET 8 or .NET 9, so we'll need to similarly host our engine in a way that includes the above constraints. This means that our Rider support has feature parity with the next version of VS.

The UI performance of RDI under Rider is also something we are still working on. Presently, the plugin has trouble rendering efficiently when dealing with large source files. This is not an issue we expect to have to deal with long term as we have many ideas on how to optimise it. RDI support for Rider was our last work item prior to release, so naturally it is our least 'finished' item on the board right now.

A Huge Thank You to JetBrains

We've been working with the team at JetBrains over the last 3 years to make this possible. I'm not sure how we would have achieved it without their help. They helped us greatly with the ability to 'wormhole' our UI through IPC and directly into Rider. Without this, we would have needed to rebuild our UI from scratch, which would have made the project infeasible.

Over all, I've been extremely impressed with the helpfulness of everyone at JetBrains, throughout their technical teams, management and marketing. It isn't easy to build an eco-system of plugins on a new IDE, but I think they are doing a stellar job of it. I'm grateful for the chance to bring NCrunch to this platform.

I do also want to extend a huge thank-you to Michael Kroes for his work on integrating NCrunch with Rider. Michael has been the architect of this work and was responsible for most of its development. Without him, we would not have Rider support.

Runtime Data Inspection (RDI)

RDI is the second headline feature of the V5 release. This is a feature that exists in both the VS and Rider plugins.

RDI is an analysis system that lets you examine the behaviour of your code as it was run by your tests, with no debugger necessary.

I've written a blog post that describes RDI in detail. I recommend reading it as I think that RDI has the potential to change the way we develop software.

RDI is the result of 15 years of R&D. It is reliant on the entirety of NCrunch's infrastructure and is fully integrated with all areas of the product (test runners, engine, UI, build system, distributed processing, IDEs, etc).

V5 includes a special new wizard showcasing RDI and giving you options to configure it the first time you open a solution. I've been working toward shipping this feature for a significant portion of my life and I really hope you try it out.

State of RDI

We've been using RDI internally on the NCrunch solution for over a year now, and it's already been through several rounds of controlled testing. I'm tempted to tag a BETA warning on it, simply because I expect that people will discover things we haven't thought of or tested for, but generally it's felt quite solid for us for a while now. There are some feature gaps and issues that I still want to address, including:

Collection of complex types: Presently we just call ToString() on complex types and we make no effort to capture their fields or properties. We have plans to do better in this area, as I think capturing full object trees (data limits permitting) would be very useful.

Collection of array/list contents: For arrays and lists, right now we're just capturing and reporting the number of items. Future plans include capturing and reporting all items in the array/list up to a configurable limit.

Navigation: The navigation between frames isn't perfect. Right now we don't have any handling for control flow diversions caused by the JIT when it loads assemblies or calls static constructors, so when this happens, navigating out of a frame can sometimes land you in an odd place. Forward navigation also doesn't yet properly handle situations where multiple navigation points exist, which is quite noticeable when enumerating using lambdas.

Image collection: At the moment we only support collection of WPF UIElements and images with a render size set. I'd like to expand this to other UI platforms too, and maybe even include options for customised image collection from objects.

Exception reporting: We don't yet report exceptions in RDI. You'll simply see the control flow hit a point in the method, then stop. I have some cool ideas on how we can visualise exceptions and I'm looking forward to having a play with this.

Data limit reporting: We do have some limited reporting for when data limits are reached when showing frames and values, but I think we can do better in this area. Ideally, it should be extremely visible if RDI has capped collection due to a data limit.

Debugger access: Sometimes it's still useful to be able to run a debugger into an open frame to examine the state of memory. There are some limitations here on deterministic behaviour around the process (for example, if a method is called a variable number of times), but I think it would be nice to have this.

Extra navigation options: It would be nice to have a top-down view on a whole navigation tree, something like a virtual call stack.

Optimisation: I'm not yet happy with RDI's engine memory consumption or its impact on engine startup/shutdown times when working with very large solutions. There is work still to be done in this area.

A Bit of History

The concept of RDI is something that actually predates the work I did on the first NCrunch prototype back in early 2009. My original idea was to have a tool that could visually show the execution path alongside the code, giving useful metrics that were collected passively. It occurred to me that for this to be working at its best, it would need to run the tests continually and report this data back while the code was being written. Some experimentation quickly showed that the concept was almost hilariously impractical. My poor dual-core laptop quickly collapsed under even the most trivial sets of code, and no amount of optimisation or creativity was going to save it. I eventually resigned to simply aggregating the coverage results instead, and NCrunch's inline coverage was born.

Years went by and hardware improved. As the concept of NCrunch proved to be widely successful (inspiring a number of similar tools) and the engine's capabilities expanded, RDI started to seem like more than just an insane dream. Unfortunately, to fully enable it, the NCrunch engine needed to be able to operate at a very high capacity. It needed a high performance instrumentation system. Meanwhile, Microsoft was busy rebuilding .NET from scratch at platform level, keeping us extremely busy just maintaining the existing features of NCrunch. I find it hard to express how frustrating these years were. RDI seemed achievable. I had a plan for how to reach it .. I just didn't have the time to actually implement it.

Then things changed. Michael Kroes joined me on the project. Microsoft solidified the new platform. We got on top of the workload. COVID-19 happened, and everything slowed down. There was finally a gap for more development work.

It came just in time. For several years, we hadn't been able to visibly break much new ground with NCrunch. Most of the work we'd done revolved around internal improvements and compatibility for more toolsets. We've always had a very committed user base with NCrunch, but people weren't talking about the product anymore, making it harder for us to reach new users. A decision had to be made. Either we needed to implement something big (like RDI), or let things wind down and fade into obscurity.

I didn't want NCrunch to fade away. But there was also a problem. The emergence of JetBrains Rider meant that the .NET developer base was becoming split between two different IDEs, and it was getting really hard to justify not having support for Rider. It wasn't enough to just implement support for Rider, and not have another big feature with it. We also couldn't ship just RDI without making a lot of Rider users very upset. V5 had to include both Rider support and RDI.

In hindsight, this was a logical decision. But also, an extremely risky one. There was no timeframe on how long it would take to complete either of these two features. They might have even proved to be completely impractical. We couldn't deliver either of them incrementally, so there would be no way to get user feedback or keep people excited in the project.

It was like stepping off a cliff with a blindfold on. We didn't know how far we'd fall, what we'd land on, or even if anyone would still be around to witness the possibly giant mess at the end.

Meanwhile, the next version of Visual Studio was always just around the corner, built on a whole different variant of .NET and including unforeseen changes we would need to deal with.

I had to adopt a similar mindset to the one I used when working on NCrunch version 1. Every day I told myself that it didn't matter if anyone would ever use what I was building, as long as I got to build something cool. I accepted the very real possibility of spending several years building something for no reason other than being unable to forgive myself if I hadn't at least tried to build it.

And now, 3 years later, we have something to ship. Is it perfect? No. But so far, it's exceeded my best expectations. We're in Rider! And now we can do things never done on such a scale before.

As I write this and look back on my life over the last 15 years, I feel myself getting quite emotional. When I started on this project, I never imagined we would be here right now.

Onward

Over the next few weeks, our primary focus is the stability of RDI and our Rider integration. With these features only just out of the oven, there are going to new problems discovered as more people start using them. I expect we'll likely need to make a number of small releases shortly after v5.0 to address these issues.

When we're confident everything is working well with these features, we'll be focusing more on performance improvements and addressing feature gaps.

I fully expect that we'll receive a new barrage of feature requests during this time. If there's anything you want to request, please do make the time to submit it through uservoice. I know that I don't always respond to every request, but I do read them all. We are also a team of just two people, so please understand that we need to prioritise our work carefully.

I want to stress that with both RDI and the Rider plugin being very new, these areas of the product haven't had the same level of optimisation as we've had for other areas of the product. Please be understanding of this. Performance optimisation and improving resource utilisation are an ongoing exercise on this project, and you can expect this to improve in the coming months.

You can find V5 available now on the Download page. Happy crunching!

Introducing Runtime Data Inspection

Fri, 23 Feb 2024 04:51:00 GMT

With the v5.0 release just around the corner, I'd like to share some detail about the second major feature area we're introducing in this release (the first being support for JetBrains Rider).

We call it Runtime Data Inspection, or 'RDI' for short.

RDI is an analysis system. It could be called a debugger replacement, but I prefer to think of it as a different way of understanding code.

With a debugger, we step through our code usually one statement at a time. The tools allow us to inspect the contents of a process's memory at a specific point in time that we advance generally in a forward direction. If we step past a critical point, we usually need to restart the session to get back to it. The debugger is built around trying to understand a program while it is running.

RDI works on a different principle. When the code runs, NCrunch captures a wealth of data from its execution (such as variable values, parameters, fields, method control flow, etc). This data then gets stored in a compacted binary log that is written to disk as code executes. After the test has finished its run, this data is then available to NCrunch for analysis. We can then mine it to understand how the code behaved during its execution.

This isn't a new concept, as we've all been using logging systems since forever. What is new however, is the scale on which we do this with RDI, and the way in which we analyse the data.

The RDI Overlay

RDI presents captured data using an overlay that can be opened over your source code while the NCrunch engine is running.

The system underneath the overlay extracts data relevant to the method you're working in, then restructures it and maps it to your code. RDI can then show the full path of execution through your code for any of the captured method calls, along with any performance bottlenecks, and the values of any variables, parameters, fields or methods that were accessed while the code ran.

Because a method can be called any number of times, the overlay lets you choose which of these frames you want to view. Subframes caused by loops can also be enumerated, so you can inspect the behaviour of every iteration of a loop.

Navigation

When using a debugger, navigation has always been restricted by program execution sequence. RDI knows no such boundaries. It's possible to navigate into any method call captured in the log.

You can also navigate backward to the caller of the opened frame.

In this way, you can traverse your code any way you like. It's possible to work backward from a point of failure, then sideways into related code to find the true cause of the failure in an upstream component. Troubleshooting in this manner usually would require multiple debug sessions. With RDI, there's no need to wait for a debug session to start or for the code to run, because the binary log already contains the data we need. All we need to do is open the frame to show it.

Search and Filter

One of the best things about working with a dataset instead of a live process is the ability to sift through data. When using a debugger, we're heavily dependent on breakpoints to get us to the point of execution we're interested in. Sometimes this involves the use of conditional breakpoints, which can be remarkably fiddly, and in my experience have historically been somewhat unreliable. RDI's data sifting makes this so much easier. When the overlay is open, you can choose any data point to show all of the values that were captured for it.

Double-clicking a value in the list will add a filter to the overlay, so it will only show frames that matched your chosen value when they hit the data point. Filters can be stacked, so you can slice your way down through the dataset.

Aggregations

While I expect most of RDI's use will be in understanding why a given test is failing, it can also aggregate data across multiple tests. It's possible to open a special aggregation overlay that will load frames from all the tests that cover a specific method. By clicking on a data point, you can then search and filter on all collected values for all covering tests that interacted with that data point. I've used this approach myself quite often when trying to understand how a given piece of code is being tested, then finding the tests responsible.

Aggregations open up some pretty exciting ways to troubleshoot problems with tests. One of my favourite ones is in dealing with sequence related test failures. For example, you might have a test that messes up something in the test environment that causes later tests to fail. In a big test suite, finding the test responsible for this can be very time consuming, but aggregations make it easy. All you need to do is aggregate the frames in the related code, and you can see what every test did with it.

Image Capture

RDI can capture and render image types too. You can pin these to the overlay while you work on the code producing them, so it can be like Live Reload for desktop development (without the need to save your code).

Right now we only support the capture and rendering of WPF elements that have an established render size, and there are some limitations around collecting these from grid nodes without a rendering GUI. This feature is still pretty experimental, but we want to do more in this area.

Other Features

Although RDI has a relatively compact UI and hopefully shouldn't be too hard to understand, there's a lot of work that's been done to make it as ready for the real world as we can make it.

Live Coding: All RDI overlays get updated dynamically from code as it changes. You can write code with the overlay open and watch how the data changes.

Fully Distributed: RDI is fully integrated with NCrunch's distributed processing. Logs are stored on the machine responsible for test execution. You can even aggregate test results across your entire grid.

Multi-Threaded and Multi-Process: RDI data collection won't go wrong when multiple threads are involved. The collection session can also span multiple processes if those processes are started from your test code. You can RDI entire process trees.

Async/Await: Handling this with RDI was hard, but we do it. If you have a method where the physical flow of execution is split using async/await, RDI will reconnect this and show it visually as a single frame.

Configurable: RDI is as configurable as the rest of NCrunch. There are dozens of settings and overrides to control how it behaves, so you can tune it to fit your unique needs.

Large Logs and Resource Consumption

Depending on how it's used, RDI can produce some pretty massive log sets. On the NCrunch solution with fairly aggressive collection settings, the logs for our 25k tests generally sit at around half a terabyte spread over our grid.

There is a clear cost associated with this in terms of both CPU and storage. Writing logs causes tests to consume more CPU and take longer to run. In our situation, it proved worthwhile to have a dedicated SSD set aside on each grid node just for RDI storage.

RDI uses a range of different limits to prevent overzealous collection of data. This prevents a situation where you might have a particularly loopy test that would otherwise capture 100GB of data by itself. Naturally, these limits are all configurable, so if you have the hardware available, it's possible to ramp things up and capture an insane amount of data. We haven't yet found the upper limit on what the engine can handle, so it should be interesting to see what people can do with it.

I'll add that I would consider the NCrunch solution to be fairly extreme in terms of both its number of tests and its coverage density. Most of our test solutions collect less than 5GB of data for a full test run using RDI, and this could be further reduced by setting the method data limit lower.

The Impact of RDI

As we dogfood everything we build, the first priority with RDI was always getting it to the point where we could use it ourselves. Given the clear challenges in building a system that can do what RDI can, and with the NCrunch solution becoming bigger and more complex by the day, this was no walk in the park. Using unstable software to fix unstable software can get pretty frustrating, but as RDI matured and started working properly, everything got very exciting.

It isn't easy for me to put a figure on how much more productive RDI has made us in working with code, but I think that a 50-100% increase is probably not an exaggeration. I have no idea what it will be like for other teams. I think the biggest boost we've had from RDI is when working with long running tests. A debug session on a test that takes 3 minutes to run isn't just a tedious wait - it's a full break in concentration span. Being able to just open an overlay and see what happened with a single click has really changed things for us.

Being able to see a rendering of WPF UIElements without needing to build and run a whole application has also been a game changer for us. All of the detailed RDI popup dialogs were built using this feature, and even though it can be a bit picky about render sizes, it has saved us a lot of time.

I can't wait to ship this and see what people think of it.