Stephen King fans like me for sure know Night Shift, a collection of 19 short stories I first read almost (oh, my…) almost 20 years ago. In Italian the title of this collection has been translated as “A volte ritornano” which literally means Sometimes they come back. This is exactly what I thought yesterday when I got yet another version of the “503 Service Unavailable” error: after I have already written about this error here and here, I’m writing the third post about it. I’m wondering if I’ll end up with a 19 post collection as Night Shift has… 😏
Anyway, here’s the new story: the application pools of this web server were failing with the following error
(eventid 281) The worker process for application pool ‘MSExchangeOWAAppPool’ encountered an error ‘Unrecognized attribute ‘precondition’ trying to read global module configuration data from file ‘\\?\C:\inetpub\temp\apppools\MSExchangeOWAAppPool.config’, line number ‘281’. Worker process startup aborted.
As you can see the precondition keyword is mistyped (it should be preCondition, capital “C”) and if we changed the .config file mentioned in the error message we were able to start our application pool but randomly (most of the time after a reboot, though) the problem came back. Interesting to note is the path of the .config file mentioned in the error: C:\inetpub\temp\AppPools\xxxxxx.config 🤓
This is where Configuration Isolation stores the application pool specific configuration files:
IIS worker processes do not have Read access to applicationHost.config. How, then, are they able to read any of the configuration set in this file?
The answer lies in the configuration isolation feature provided by IIS 7.0, which is always on by default. Instead of enabling IIS worker processes to read applicationHost.config directly when reading the configuration file hierarchy, IIS generates filtered copies of this file and uses these copies as a replacement of applicationHost.config when configuration is read inside the IIS worker process.
The reason for doing this is to prevent IIS worker processes from application pool A to be able to read configuration information in applicationHost.config that is intended for application pool B. Because applicationHost.config may contain sensitive information, such as the user name and password for custom application pool identities, as well as user name and password for virtual directories, allowing all application pools to access applicationHost.config would break application pool isolation.
WAS is responsible for generating the temporary application pool configuration files that each IIS worker process uses as a replacement of applicationHost.config. These files are placed by default in the %SystemDrive%\Inetpub\Temp\Apppools directory and are named AppPoolName.config. As mentioned earlier, these files are configured to allow access only to the IIS worker processes in the corresponding application pool, by using the IIS APPPOOL\AppPoolName Application Pool SID.
Note This process occurs automatically each time applicationHost.config is changed and therefore does not require any manual action from the administrator outside of normal configuration procedures.
Each application pool configuration file contains the configuration in applicationHost.config, with the following information removed:
- All application pool definitions in the system.applicationHost/applicationPools configuration section. Only WAS is required to read this configuration section.
- Any Web site definitions in the system.applicationHost/sites configuration section for sites that do not have applications in the current application pool.
- Any configuration in location tags for specific Web sites, applications, or URLs that do not reside inside the applications in the current application pool.
Caution All application definitions (and their virtual directory definitions, possibly containing user name and password credentials) for any site that has at least one application in the current application pool will be present in the application pool configuration file. To disable this behavior and include only the application definitions for applications in the application pool, set the IsolationWholeSiteInclude DWORD value to 0 in the HKLM\System\CurrentControlSet\Services\WAS\Parameters key and perform an IISRESET. This may break applications in sites with applications in multiple application pools when they attempt to map physical paths for URLs in other applications.
Keep in mind that global configuration settings set in the applicationHost.config (without using location tags to apply them to specific Web sites, applications, or URLs) are not filtered. Each application pool configuration file will contain all of these settings.
Configuration isolation is a key part of the application pool isolation strategy in IIS 7.0. It is enabled by default to provide configuration isolation for server-level configuration in applicationHost.config. For strategies on achieving proper application pool isolation, see the section titled “Isolating Applications” earlier in this chapter.
Caution Configuration stored in .NET Framework machine.config and root web.config files is not isolated. Only configuration stored in applicationHost.config is isolated.
So, this means the mistyped entry very likely originates from applicationHost.config… We do not know what (or who 😉) changed this since I have the same roles and features installed on my machine as the customer had on his server, but after fixing the typo in the centralized configuration file the problem disappeared.
Carlo
Quote of the day:
To avoid situations in which you might make mistakes may be the biggest mistake of all. – Peter McWilliams