Every now and then we got a call for an application which, randomly and without a specific pattern or apparent reason, shuts down all user’s sessions. Since ASP.NET 2.0 we can use Web Events to have more information on what is happening at the runtime level and this time the message we had in the event log was pretty clear:
Event code: 1002
Event message: Application is shutting down. Reason: Configuration changed.
Event time:
Event time (UTC):
Event ID:
Event sequence:
Event occurrence: 1
Event detail code:Application information:
Application domain: /LM/W3SVC/1045621189/Root-2-1742334186915428
Trust level: Full
Application Virtual Path: /
Application Path: C:\Inetpub\wwwroot\
Machine name: <machinename>Process information:
Process ID: 1234
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE
We used Process Monitor to track access to the relevant .config files which are usually the cause for this kind of problem, but with no luck.
To make a long story short, we had a look at the events around the time when the session was lost and we noticed it usually followed some exception tracking logic the customer had used, so we had a look at the source code and found something interesting: the customer had some code to write and maintain a custom log on a text file and this work was done on a background thread. This code was using the ConfigurationManager class to read some settings from the web.config.
Now the interesting part: the background thread was executed in a different AppDomain than the ASP.NET application and it needed to initialize this new AppDomain to use the ConfigurationManager; as part of this initialization we create a new File Change Notification object to monitor machine.config and web.config files for changes. When the background thread exited after his job was done, the File Change Notification object encountered this error:
0x800703E3 ( -2147023901 ) = ERROR_OPERATION_ABORTED
The I/O operation has been aborted because of either a thread exit or an application request
Note, that is not an exception; but this change notification event was “head” by all ASP.NET applications which is the expected behavior, and all AppDomains were restarted: the end users were therefore redirected to the login page ?.
How do you avoid this situation? Well, we made a small change to the exception handling code to not use ConfigurationManager in the background thread.
Carlo
Quote of the day:
When a person can no longer laugh at himself, it is time for others to laugh at him. – Thomas Szasz