I recently purchased a new Zoncn NZ100 Variable Frequency Drive to drive a three phase motor used on a CNC. My intention is to integrate it into the controller that controls the CNC over Modbus, which the VFD supports using RS-485.

However, to communicate with it, one must know the registers of the VFD. This isn’t outlined, or even mentioned, in its manual.

Luckily, there are other VFDs which have documented their protocols better, for instance the SKI-780, Salicru CV-50, GS20 and NZ100’s big brother Z2000. The NZ100 implements the Modbus interface as a bit of a mix. Not quite like the Z2000, which surprised me. I thought I had it when I found its manual. However, when I commanded it to start by setting register 0x2000 to 1 as documented on p.176 in the Z2000 manual, nothing happened. The VFD didn’t do anything if I set 0x1000 to a percentage of the desired maximum speed and starting it from the VFD’s control panel, either.

I started looking at the available Modbus documentation for similar VFDs. I tried writing to registers suggested by the documentation of those. All these documents gave me enough hints to be able to figure out how the NZ100 is controlled. Here’s my findings:

The desired output frequency is set in units of 1/10 Hz (instead of 1/100 Hz as is customary) in register 0x2001. An output frequency of 50.0Hz is thus written as 500. The VFD will limit the output frequency according to the values set in parameter P105 (max frequency) and P106 (min frequency).

On the topic of the documented parameters Pxxx: Parameter P105 is available at register address 0xF105. The register P212 is located at 0xF20C. The entire set of parameters outlined in the manual is located according to this pattern. To write a new value without persisting it to the unit’s EEPROM i.e. writing it to RAM, change the F into a 0. Note that reads are invalid at this address. It may be desirable to write to RAM instead of EEPROM, as the write cycles on the EEPROM is limited.

In practice, register 0x2000 and 0x2001 are used to control the frequency and operation of the drive, according to the following table:

Example conversations

Modbus RTU frame format is quite straight-forward. It’s a multi-drop bus, so there’s one client device that initiates communication and one or more servers that each carry a unique address and responds to the requests. In this example, the VFD acts as a Modbus Server on address 1.

The first byte of each frame is the destination address. The second byte is a function code. These VFDs only implements the Modbus function code 3: read holding register and function code 6: write holding register. As a proprietary addition, a temporary write i.e. write to RAM can be performed by issuing a write request using function code 7.

Read P215 Rated Motor Frequency

Read P212 Rated RPM @ Rated Motor Frequency

Together with P215, P212 helps us figure out what RPM a specific frequency results in. By dividing P212 with P215 (in Hz), we get the number of RPMs for each Hz. My spindle outputs 3000 RPM @ 50.0Hz, which translates to 60RPM/Hz. At the spindle’s maximum frequency, 400Hz, it rotates at 60*400 = 24000 RPM.

Read P105 Maximum Frequency

Read P106 Minimum Frequency

Read 0x3000 (status)

Unconfirmed, but following the Z2000’s description of the status register: 1 – Running (Forward), 2 – Running (Reverse), 3 – Stopped

Read 0x2000 – Desired Output Frequency

Write 0x2000 Operating Mode

Write 0x2001 Desired Output Frequency

This message tells the VFD that the desired output frequency is 150Hz. At 60RPM/Hz, that is 9000 RPM.

The VFD confirms by echoing the write request.

Write 0x2000 Operating Mode

This is the write that starts the spindle. As outlined above, the written value 0x0012 tells the VFD it should Run Clockwise.

The VFD confirms by echoing the write request.

In Episerver, the output cache is enabled in the applicationSettings element’s httpCacheExpiration attribute in web.config. That sets the default Duration for a page.

Note that requests are not saved to the output cache if any of the following criteria are true:

• Current user is authenticated
• Duration (httpCacheExpiration in web.config) is zero.
• The HTTP method is not GET

In addition, requests are not served from the output cache if the DataFactoryCache.Version has changed since it was cached.

Please note that Output Caching makes Visitor Groups not work for anonymous visitors, which might be a deal-breaker for some.

Content publishing

When new content is published in Episerver, the item saved is clearly known so the cache for that item is dropped from the Object Instance cache. However, which of the output caches that needs to be re-rendered cannot be known. This is because any type of querying can be done to retrieve a content item or data from another content item. In the end, the generated HTML for a page is almost always a mix of many content items; menus and their labels are generated from the names of the pages they link to, content lists, children listings and many other elements provides summary information from other pages.

Thus, when any content item is saved, the DataFactoryCache.Version is updated and this causes all output caches to be invalidated.

Cache Stampede

As long as the page is stored in the output cache, the servers easily handles incoming load. When the page expires, the server goes through the process of rendering it from start to finish. Under very high load, the number of requests that ‘falls through’ the cache and simultaneously starts a process to render the same page may grow large enough to cause excessive delays in rendering and high loads on the supporting infrastructure such as databases and services. This is called a cache stampede.

Enter the Burst Cache

The idea of the burst cache is to limit the impact of dropping the output caches when they time out (Duration) or them all being invalidated at the same time when new content is published. This works by storing the version of the item in the cache and, in the case when a request was let through to re-render it, the version being rendered.

Only one thread is let through to re-render each output cache, while the other requests hitting one of those will continue to serve the (now slightly stale) cache.

Clone it from GitHub: EPiServer.BurstCache

If you use a thread or ThreadPool.QueueUserWorkItem to run code asynchronously and don’t catch your exceptions inside your WaitCallback, you will bring your entire application down when it breaks.

// This code snippet kills your web app process.
System.Threading.ThreadPool.QueueUserWorkItem(delegate
{
    var ctx = System.Web.HttpContext.Current;
    var u = ctx.User;
});

On the highlighted line, a NullReferenceException is thrown (because a background thread has no HttpContext).

If you have a JIT debugger such as Visual Studio installed on the machine, you may also see this:

This is what you will find in the event log:

1. One event with source ASP.NET 4.0.30319.0:

   An unhandled exception occurred and the process was terminated.
   
   Application ID: /LM/W3SVC/7/ROOT
   
   Process ID: 14232
   
   Exception: System.NullReferenceException
   
   Message: Object reference not set to an instance of an object.
   
   StackTrace:    at YourSite.Web.Controllers.Pages.StartPageController.<>c.<Index>b__2_0(Object <state>) in E:\Source\Repos\YourSite\src\YourSite.Web\wwwroot\Controllers\Pages\StartPageController.cs:line 37
      at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
      at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
      at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
      at System.Threading.ThreadPoolWorkQueue.Dispatch()

    

2. Followed by a .NET Runtime log message:

   Application: w3wp.exe
   Framework Version: v4.0.30319
   Description: The process was terminated due to an unhandled exception.
   Exception Info: System.NullReferenceException
      at YourSite.Web.Controllers.Pages.StartPageController+<>c.<Index>b__2_0(System.Object)
      at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
      at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
      at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
      at System.Threading.ThreadPoolWorkQueue.Dispatch()
   

    

3. Followed by yet another one, this time from unmanaged error reporting having the source “Application Error”:

   Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
   Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
   Exception code: 0xc0000005
   Fault offset: 0x00007ffbd9521b5c
   Faulting process id: 0x3798
   Faulting application start time: 0x01d180f9f39a4977
   Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
   Faulting module path: unknown
   Report Id: 378450e1-ecf2-11e5-beda-10604b64d92d
   Faulting package full name: 
   Faulting package-relative application ID:

   When a managed exception is reported to Windows, it is translated into an unmanaged exception. The managed exception NullReferenceException is translated to Exception Code 0xc0000005  which is “Access Violation” (“Segmentation Fault” for you Unix hackers).

   If you read yesterday’s post, a StackOverflowException is 0xc00000fd  and virtually every other managed exception type goes under the CLR unmanaged exception code 0xe0434352 .

4. And last, a message on Information level from Windows Error Reporting:

   Fault bucket , type 0
   Event Name: CLR20r3
   Response: Not available
   Cab Id: 0
   
   Problem signature:
   P1: w3wp.exe
   P2: 8.5.9600.16384
   P3: 5215df96
   P4: YourSite.Web
   P5: 1.0.0.0
   P6: 56ebd4d1
   P7: 186c
   P8: 12
   P9: System.NullReferenceException
   P10: 
   
   Attached files:
   
   These files may be available here:
   
   
   Analysis symbol: 
   Rechecking for solution: 0
   Report Id: 378450e1-ecf2-11e5-beda-10604b64d92d
   Report Status: 0
   Hashed bucket:

    

Recognizing it

This one is relatively easy to spot, if you know to look in the Windows Event Viewer or monitor your w3wp.exe’s PIDs over time. You will experience a very poor performance as the web site is continuously restarting and dropping its caches, causing additional load on the database and increased wait times for your users while your application tries to start back up.

Death by Stack Overflow

This one is interesting, yet it has nothing to do with StackOverflow.com.

The StackOverflowException is one of very few exception classes in .NET that are not catch-able. Scratch that, it’s not possible to “swallow” it in a try-catch clause, because even if you don’t re-throw it, .NET will:

Starting with the .NET Framework 2.0, you can’t catch a StackOverflowException object with a try/catch block, and the corresponding process is terminated by default.

Some time back, I was involved in a case where the w3wp.exe’s started crashing. No recent deployments had been made. This message could be found in the event log:

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: clr.dll, version: 4.0.30319.18034, time stamp: 0x50b5a783
Exception code: 0xc00000fd Fault offset: 0x000000000001a840
Faulting process id: 0xd50
Faulting application start time: 0x01ce97fe076d27b4
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll Report Id: e0c90a5f-0455-11e3-8f0e-005056891553

The important piece in this message is 0xc00000fd , that’s the exception code meaning “Stack Overflow”.

Needless to say, the restarts were detrimental to the site’s performance.

It was found that this exception was caused by a code snippet like this:

// StackOverflowException-causing example code -- Do Not Use!
string FindUniqueName(string suggestedName, string ext, int counter)
{
    string candidateName = string.Format("{0}{1}.{2}", suggestedName, counter, ext);
    if (HostingEnvironment.VirtualPathProvider.FileExists(candidateName))
    {
        return FindUniqueName(suggestedName, ext, counter+1);
    }
    return candidateName;
}

If the method is called with the same suggested name repeatedly (turns out most mobile devices name their uploads “image.jpg”) and then image0.jpg .. image6842.jpg, after a few months “image6843.jpg” would be churned out as the unique name for the upload. Unfortunately when the function recursed that deep, it ran out of stack space and so the StackOverflowException was thrown by the .NET runtime.

This blog post covers the tools needed to troubleshoot issues of this kind quite well and also explains steps required to find which piece of your application causes it to die.

Here’s a quick patch that fixes the issue for sites running this code:

using EPiServer.Framework;
using EPiServer.Framework.Timers;
using EPiServer.ServiceLocation;
using System;
using EPiEvents = EPiServer.Events;

namespace Hotfix.CMS1405.EPiServer.Events
{
    [InitializableModule]
    public class PatchInitializer : IInitializableModule
    {
        public void Initialize(global::EPiServer.Framework.Initialization.InitializationEngine context)
        {
            context.InitComplete += context_InitComplete;
        }

        void context_InitComplete(object sender, EventArgs e)
        {

            var eps = EPiEvents.Providers.EventProviderService.Instance;
            var emvField = eps.GetType().GetField("_messageValidator", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
            if (emvField == null)
            {
                throw new InvalidOperationException("Could not locate _messageValidator variable; this patch might not be valid for the current application.");
            }

            var emv = emvField.GetValue(eps);
            var timerField = emv.GetType().GetField("_sequenceCheckTimer", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
            if (timerField == null)
            {
                throw new InvalidOperationException("Could not locate _sequenceCheckTimer variable to patch; this patch might not be valid for the current application.");
            }

            var timer = (ITimer)timerField.GetValue(emv);
            if (timer.Enabled)
            {
                throw new InvalidOperationException("Timer already enabled; this patch might not be valid for the current application.");
            }
            timer.Enabled = true;
        }

        public void Preload(string[] parameters)
        {
        }

        public void Uninitialize(global::EPiServer.Framework.Initialization.InitializationEngine context)
        {
        }

    }
}

Recognizing it

After leaving your application running for some time, take a full user dump of the application (using through Task Manager/Right Click w3wp.exe/Create Dump File, ProcDump -ma and the like).
Open it up in WinDBG, and load the SOS extension. Being lazy, I first let the .loadby sos clr attempt loading and fail. It fails because the dump I analyze was taken off an Azure server, which has the system loaded on the D:-drive. Copy the path, sans the first character which I change to a C, and the SOS extension loads properly.

0:000> .loadby sos clr      
The call to LoadLibrary(D:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos) failed, Win32 error 0n126       
 "The specified module could not be found."      
Please check your debugger configuration and/or network access.      
0:000> .load c:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos      

Then I call !dumpheap -stat . After crunching for a while, it will write out a table with all the different types of objects in your application, sorted by the biggest space-takers in the bottom:

0:000> !dumpheap -stat
Statistics:
              MT    Count    TotalSize Class Name
... and close to the bottom, you'll find ...
000007f8f37e98a0  3204485    102543520 EPiServer.Events.Providers.EventSequence
000007f8f37ed210  3204485    153815280 System.Collections.Generic.LinkedListNode`1[[EPiServer.Events.Providers.EventSequence, EPiServer.Events]]

In this dump, I found more than 3 million instances of each of these types. Together, they take up 102+153 MB of memory. You would typically not expect any more instances than the number of remote events that could be emitted within a 5 second time span. That number is well below 3 million! That’s 255 MB wasted!

Downloads

Death by IIS Ping

IIS’ built-in health monitoring is quite simple, but despite it having the word “ping” in its name, it has nothing to do with the network/ICMP type ping that we are used to.

Instead, what it does is that it queues a “ping” work item into the thread pool of the monitored application pool, and waits for it to be executed. Now consider the case where the application has restarted for whatever reason during peak hours and your users are lining up long waiting to get the first pages from the freshly started application. The IIS ping health check has to wait in the same line and if it doesn’t get its check executed within the time span, the application pool is deemed unhealthy and restarted. This doesn’t make the situation any better even by a tiny bit unfortunately, and you may be stuck having a seemingly never-starting process.

For every application, there’s a threshold where the inflow of new requests are low enough to allow the IIS ping check to be executed within the time frame given. This explains why you will be able to get the application up and running if you block incoming requests in the firewall for the duration of the startup, and then let the load back on.

Or, just disable IIS ping.