Yesterday I was asked to convert a Dynamics CRM 2013 installation from an internal to an Internet Facing Deployment (IFD). It is a bit fiddly, but I have done this before so I was confident.

The installation in question is only for test; the company has its production CRM 2011 on another server. Because it is for test though, it is a small deployment on a single server.

I got to work running the Claims Based Authentication wizard in the CRM Deployment Manager but also noticed something odd about the server. WSUS (Windows Server Update Services) was installed though it was not in use. This seems a bad idea so I asked if I could remove it. Sure, it was just a quick experiment. I removed WSUS and got on with the next steps of configuring IFD.

Unfortunately ADFS 2.0 (in this case) would not play ball. It could not communicate with CRM. I quickly saw why: attempting to browse to the special FederationMetadata.xml URL raised a 500 error.

I tried a few things. There are plenty of odd things that can go wrong: permissions on the private keys of the certificate used for the CRM web site, Service Principal Names, incorrect DNS entries and so on. All seemed fine. Still the error.

I decided to backtrack and temporarily disable Claims Based Authentication. Unfortunately it appeared that I had broken CRM completely. All access to the site raised the same 500.19 IIS error.

The web page IIS delivers says that the most likely causes are that the worker process is unable to read the ApplicationHost.config or web.config file, or malformed XML in the applicationhost.config or web.config file, or incorrect NTFS permissions.

I did a repair install on CRM. I reapplied the rollups. No difference.

I ran Process Monitor to try to figure out what configuration file was causing the problem. It was not a great help, but did point me in the right direction to the extent that it seemed that ASP.NET was not working properly at all. I now focused on this rather than CRM itself, observing also that there were not many CRM-related errors in the event log and I would expect more if it was really broken.

I created a hello world ASP.NET application and installed it in a separate site on a different port. Same error.

Searching for help on this particular error was not particularly helpful. In the context of CRM, the few users that encountered something similar had reinstalled everything from scratch. However, now at least I knew that IIS rather than CRM was broken. This helpful MSDN article actually includes a hint to the solution:

For above specific error (mentioned in this example), DynamicCompressionModule module is causing the trouble. This is because of the XPress compression scheme module (suscomp.dll) which gets installed with WSUS. Since Compression schemes are defined globally and try to load in every application Pool, it will result in this error when 64bit version of suscomp.dll attempts to load in an application pool which is running in 32bit mode.

which is also referenced here. These refer to WSUS breaking 32-bit applications, but in my case after removing WSUS neither 64-bit nor 32-bit apps were running.

Let me put it more clearly. If you remove WSUS using the role wizard in Server Manager, a number of bits get left behind, including a setting in ApplicationHost.config (in /System32/Inetsrv) that breaks IIS.

So it was my attempt to clean up the server that had made it worse.

That said, this is also a Windows Server failure. Adding and removing a role should result as far as possible in no change.

Once identified, the problem is easy to fix (this is often true). Still, several hours wasted and more evidence for Martin Fowler’s assertion that you should automate server configuration and spin up a new one from scratch when you want to make a change, to avoid configuration drift. There is a more detailed post on the same theme – Phoenix servers that rise from the ashes, not snowflake servers that are unique and ugly – here.

In a small business context this perhaps is harder to achieve – though the cost of entry gets lower all the time, through either cloud computing or internal virtualization platforms.