Here’s another interesting crash problem I worked on a few days ago: the application pool was randomly crashing and the following errors were logged: Event Type: Error Event Source: .NET Runtime 2.0 Error Reporting Event Category: None Event ID: 1000 Date: 21/08/2009 Time: 12:23:12 User: N/A Computer: <computername> Description: Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d691cc, faulting module kernel32.dll, version 5.2.3790.4062, stamp 462643a7, debug? 0, fault address 0x0000000000027d8d. Event Type: Warning Event Source: W3SVC Event Category: None Event ID: 1011 Date: 21/08/2009 Time: 12:23:14 User: N/A Computer: <computername> Description: A process serving application pool <AppPoolName> suffered a fatal communication error with the World Wide Web Publishing Service. The process id was xxx. The data field contains the error number. Even if with some difficulties (we had to disable Error Reporting as described here) we managed to capture a coupe of dumps for this crash; dumping the exception was not very helpful: 0:027> !gle LastErrorValue: (Win32) 0x3f0 (1008) – An attempt was made to reference a token that does not exist. LastStatusValue: (NTSTATUS) 0xc000007c – An attempt was made to reference a token that doesn’t exist. This is typically done by referencing the token associated with a thread when the thread…
-
-
Logparser automated within Windbg
The .shell command in Windbg allows to pipe the output of a debugger command to an external process and automatically print its output back inside the debugger window; a useful example is the command FIND, for example if we want to parse the stack for every thread and find every call where the word “isapi” is involved: 0:036> .shell -ci "~*kpL1000" find /i "isapi"013cff78 1004f94e ISAPI_Rewrite!TerminateFilter+0x3cef013cffb0 1004f9f3 ISAPI_Rewrite!TerminateFilter+0x4462e013cffec 00000000 ISAPI_Rewrite!TerminateFilter+0x446d30144ff78 1004f94e ISAPI_Rewrite+0x77630144ffb0 1004f9f3 ISAPI_Rewrite!TerminateFilter+0x4462e0144ffec 00000000 ISAPI_Rewrite!TerminateFilter+0x446d3.shell: Process exited Incidentally also LogParser (one of my favorite debugging tools) can accept data to be parsed from the input stream using the STDIN keyword, so for example refactoring a script I posted some time ago we can find out if there are any duplicated assemblies in our application pool that should be moved to the GAC: 0:000> .shell -ci "!peb" logparser "select extract_filename(text) as Duplicated_Assemblies, count(Duplicated_Assemblies) as Hits from STDIN where index_of(text, 'temporary asp.net files') > 0 group by Duplicated_Assemblies having count(Duplicated_Assemblies) > 1" -i:textline -o:nat -rtp:-1Duplicated_Assemblies Hits ---------------------------- ----errormanager.dll 2winformsui.dll 2externallibraryinterface.dll 2ptshopengine.dll 2schemas.dll 2dbengine.dll 2flowservice.dll 2Statistics:-----------Elements processed: 182Elements output: 7Execution time: 0.02 seconds.shell: Process exited Following the same principle, we can find out if there are strong named assemblies in our…
-
StackOverflowException and DataBind()
The application pool for this site was getting disabled quite frequently and we found quite a few entries like the following in the event log: Event Type: Error Event Source: W3SVC Event Category: None Event ID: 1002 Date: 19/11/2008 Time: 15:20:23 User: N/A Computer: <computername> Description: Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool In this case the customer already had some debugging skills so when he called CSS he already had a few dumps available to analyze; they were some process shutdown dumps (taken on Kernel32!TerminateProcess), strangely we had no second chance dumps (maybe because of one of these reasons?) but they’ve been a good point to start from anyway. First, let’s have a look at the exceptions: 0:000> !dumpheap -type Exception -stat [...]Statistics: MT Count TotalSize Class Name79333ed4 1 12 System.Text.DecoderExceptionFallback79333e90 1 12 System.Text.EncoderExceptionFallback79330d44 1 72 System.ExecutionEngineException79330cb4 1 72 System.StackOverflowException79330c24 1 72 System.OutOfMemoryException7931ffd4 1 72 System.NullReferenceException6610c7fc 1 84 System.Web.HttpUnhandledException79330dd4 2 144 System.Threading.ThreadAbortException79318afc 2 144 System.InvalidOperationException7931740c 5 160 System.UnhandledExceptionEventHandlerTotal 16 objects Remember, a few special exceptions are loaded when the AppDomain is first created (see here) so let’s try to see if there is still something significant…
-
Where’s my dump gone?!?
Have you ever found yourself patiently waiting for a problem to reproduce with your debugger ready, and when it happens you just find there are no signs of your dump anywhere? If you are using adplus you likely had the text logs, but nothing more… In such situations it is possible that the OS is terminating the process before the dump is fully written. If you are tying to dump an IIS process and you want to save some time and headaches, try this small cunning: temporarily disable the “Enable pinging” and “Enable rapid-fail protection” flags in the Health tab for your application pool: Sometimes can be useful to tell IIS to not kill a failing worker process but rather leave it orphaned: Features of Worker Process Isolation Mode Orphaning Worker Processes You can configure worker process isolation mode to orphan a worker process that the WWW service deems to be failing. The WWW service usually terminates a failing worker process and replaces it. If you enable orphaning, the WWW service allows a failing worker process to continue running, but separates it from the application pool (making it an orphan) and starts a new worker process in its place. You…
-
ViewState validation troubles?
If you work with web sites in a complex/multi server environment you might be familiar with this error and as the error message itself suggest, the first thing to check if the machineKey value which must be the same across all of the involved server. Anyway every now and then it happens that you might get this same error even if you are not in a NBL/Cluter environment (Tess and Tom have blogged about this here, here and here) or if you are absolutely sure you have properly configured the machineKey value. So what else can lease you to this state? If you want some quick background you have a look at my post here; basically we have the following (very raw) steps: Begin processing the new request Load the “old” state (ViewState and ControlState) received with the posted values Process the posted data (the actual form values) and raise the “change” events to change control state and do the real page work Save the new state (again, ViewState and ControlState) Send the result back to the client It is important to note that if you dynamically add controls at runtime on your page (i.e. you do not only rely…
-
How much garbage to you still have on the managed heap?
When you’re focused on a specific problem, you’re sunk into it and you’re at a dead end, you have to take a step back to see the whole picture and breath again before being able to continue; often discussing with a colleague whom still has a “fresh mind” and doesn’t know anything about the specific problem and the troubleshooting you’ve already done can give nice suggestions and tips about next steps and a different approach to the problem. This is what happened a couple of days ago with Doug while we were discussing a memory leak issue I’m currently working on. This seems to be a complex and a bit confusing one (I do not want to dig into the details for post, I’ll likely talk again in details when the problem will be resolved 🤓); anyway a question that should always come to mind dealing with memory leaks is: how much of the memory I see still allocated in a dump is really valid and how much is garbage which our friendly Garbage Collector will remove next time it will come into the game? Well… a nice solution is to have a very simple “service” page to trigger a…
-
Troubles downloading public symbols?
Imagine you are debugging your dump, are already looking forward for the output of the command you just typed (say a kpn to get the stack of a thread) and you get this message instead: *** ERROR: Symbol file could not be found. Defaulted to export symbols for mscorwks.dll WARNING: Frame IP not in any known module. Following frames may be wrong. As you can understand, not having the symbols for mscorwks in a .NET dump can make your analysis harder and above all without matching symbols you cannot really trust what you see in your debugger (despite the title of my blog! ?) and ultimately it can make the dump almost useless. I actually talked with a customer a few days ago whom got this exact situation and he was quite frustrated and upset with Microsoft for not hiving the symbol for the core .NET dll on the public symbol server. Anyway, without losing his temper that developer used the !sym noisy command to have a look at the symbol loading output, here is what he got: SYMSRV: The operation timed out SYMSRV: d:\symbols\mscorwks.pdb\62286AFFFC674D5198883B98518936FF2\mscorwks.pdb not found SYMSRV: http://msdl.microsoft.com/download/symbols/mscorwks.pdb/62286AFFFC674D5198883B98518936FF2/mscorwks.pdb not found The pdb was actually on the server, but for…
-
Invalid length for a Base-64 char array
There are various causes for this error, this is one we had some time ago when paginating through a DataGrid (for this case we’re talking about ASP.NET 1.1); the error appeared systematically under a specific user pattern: Invalid length for a Base-64 char array. Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.FormatException: Invalid length for a Base-64 char array. Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. Stack Trace: [FormatException: Invalid length for a Base-64 char array.] System.Convert.FromBase64String(String s) +0 System.Web.UI.LosFormatter.Deserialize(String input) +25 System.Web.UI.Page.LoadPageStateFromPersistenceMedium() +101 [HttpException (0x80004005): Invalid_Viewstate Client IP: 213.199.128.155 Port: 24949 User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506; .NET CLR 1.1.4322; InfoPath.2; MS-RTC LM 8) ViewState: dDw2Mjg2Njc3OTM7dDw7bDxpPDA+Oz47bDx0PDtsPGk8MT47aTwzPjs+O2w8dDxwPHA8bDxUZXh0Oz47bDxcZTs+Pjs+Ozs+O3Q8QDA8cDxwP Gw8V2ViR3JpZF9Tb3J0RXhwcmVzc2lvbjtTaG93Rm9vdGVyO1Zpc2libGU7XyFTQjtQYWdlQ291bnQ7UGFnZVNpemU7V2ViR3JpZF9MYWJlbHM7 V2ViR3JpZF9Dc3NDbGFzc0J1dHRvbnM7Q3NzQ2xhc3M7XyFJdGVtQ291bnQ7XyFEYXRhU291cmNlSXRlbUNvdW50O1dlYkdyaWRfU29ydFR5cGU7 V2ViR3JpZF9Dc3NDbGFzcztEYXRhS2V5czs+O2w8TnVtZXJvO288Zj47bzx0PjtpPDI+O2k8MT47aTwxMD47YjxBQUVBQUFELy8vLy9BUUFBQUFBQ UFBQU1BZ0FBQUU1SmJuWmhMbGRsWWtOdmJuUnliMnh6TENCV1pYSnphVzl1UFRFdU1DNHlORGMzTGpFMk16UTVMQ0JEZFd4MGRYSmxQVzV sZFhSeVlXd3NJRkIxWW14cFkwdGxlVlJ2YTJWdVBXNTFiR3dGQVFBQUFCNUpiblpoTGxkbFlrTnZiblJ5YjJ4ekxsZGxZa2R5YVdSTVlXSmxiSE1UQUF BQUIxSnZkMFZrYVhRSlVtOTNTVzV6WlhKMENWSnZkMVZ3WkdGMFpRbFNiM2RFWld4bGRHVUpVbTkzUTJGdVkyVnNFVkp2ZDBSbGJHVjBaVkY xWlhOMGFXOXVDMUp2ZDFSbGVIUkdiM0p0Q1ZKdmQxUmxlSFJVYnc1U2IzZFVaWGgwVUdWeVVHRm5aUTlRWVdkbFZHVjRkRU4xY25KbGJuUU tVR0ZuWlZSbGVIUlBaZzVTYjNkVFpXeGxZM1JsWkZScGNBNVNiM2RVYjFObGJHVmpkRlJwY0F4UVlXZGxSbWx5YzNSVWFYQUxVR0ZuWlZCeVpY WlVhWEFMVUdGblpVNWxlSFJVYVhBTFVHRm5aVXhoYzNSVWFYQUlVR0ZuWlVkdlZHOExVR0ZuWlVkdlZHOVVhWEFCQVFFQkFRRUJBUUVCQVFF QkFRRUJBUUVCQWdBQUFBWURBQUFBQ0UxdlpHbG1hV05oQmdRQUFBQUZUblZ2ZG04R0JRQUFBQVZUWVd4MllRWUdBQUFBQjBWc2FXMXBi bUVHQndBQUFBZEJibTUxYkd4aEJnZ0FBQUF4UTI5dVptVnliV2tnYkdFZ1kyRnVZMlZzYkdGNmFXOXVaU0JrWld4c1lTQnlhV2RoSUhObGJHVjZhV zl1WVhSaFB3WUpBQUFBQmxKcFoyaGxPZ1lLQUFBQUFtUnBCZ3NBQUFBUVVtbG5hR1VnY0dWeUlIQmhaMmx1WVFZTUFBQUFCMUJoWjJsdVlU b0pDZ0FBQUFZT0FBQUFFRkpwWjJFZ2MyVnNaWHBwYjI1aGRHRUdEd0FBQUJWVFpXeGxlbWx2Ym1FZ2NYVmxjM1JoSUhKcFoyRUdFQUFBQUF 4UWNtbHRZU0J3WVdkcGJtRUdFUUFBQUJGUVlXZHBibUVnY0hKbFkyVmtaVzUwWlFZU0FBQUFFVkJoWjJsdVlTQnpkV05qWlhOemFYWmhCaE 1BQUFBTlZXeDBhVzFoSUhCaFoybHVZUVlVQUFBQUExWmhhUVlWQUFBQUQxWmhhU0JoYkd4aElIQmhaMmx1WVFzPT47V2ViR3JpZEdyaWRC dXR0b247V2ViR3JpZDtpPDEwPjtpPDEwPjtJbnZhLldlYkNvbnRyb2xzLldlYkdyaWQrZVNvcnRUeXBlLCBJbnZhLldlYkNvbnRyb2xzLCBWZXJzaW9 uPTEuMC4yNDc3LjE2MzQ5LCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2VuPW51bGw8QXNjZW5kaW5nPjtXZWJHcmlkO2w8aTwxPjtpP DI+O2k8Mz47aTw0PjtpPDU+O2k8Nj47aTw3PjtpPDg+O2k8OT47aTwxMD47Pjs+Pjs+O0AwPEAwPHA8bDxIZWFkZXJUZXh0O1Zpc2libGU7PjtsPFx lO288Zj47Pj47cDxsPENzc0NsYXNzO18hU0I7PjtsPFdlYkdyaWRIZWFkZXI7aTwyPjs+Pjs7Oz47QDA8cDxsPEhlYWRlclRleHQ7PjtsPFxlOz4+Ozs7Oz47 QDA8cDxsPEhlYWRlclRleHQ7PjtsPE51bS4gwqsgOz4+Ozs7Oz47QDA8cDxsPEhlYWRlclRleHQ7PjtsPERhdGE7Pj47Ozs7PjtAMDxwPGw8SGVhZGV yVGV4dDs+O2w8T2dnZXR0bzs+Pjs7Ozs+O0AwPHA8bDxIZWFkZXJUZXh0Oz47bDxBdHRvOz4+Ozs7Oz47QDA8cDxsPEhlYWRlclRleHQ7PjtsPENv bnRlbnV0bzs+Pjs7Ozs+Oz47cDxsPENzc0NsYXNzO18hU0I7PjtsPFdlYkdyaWRQYWdlcjtpPDI+Oz4+O3A8bDxDc3NDbGFzcztfIVNCOz47bDxXZWJH cmlkSGVhZGVyO2k8Mj47Pj47cDxsPENzc0NsYXNzO18hU0I7PjtsPFdlYkdyaWRGb290ZXI7aTwyPjs+PjtwPGw8Q3NzQ2xhc3M7XyFTQjs+O2w8V2 ViR3JpZEl0ZW07aTwyPjs+PjtwPGw8Q3NzQ2xhc3M7XyFTQjs+O2w8V2ViR3JpZEFsdGVybmF0ZUl0ZW07aTwyPjs+PjtwPGw8Q3NzQ2xhc3M7Xy FTQjs+O2w8V2ViR3JpZFNlbGVjdGVkSXRlbTtpPDI+Oz4+Ozs7PjtsPGk8MD47PjtsPHQ8O2w8aTwwPjtpPDI+O2k8Mz47aTw0PjtpPDU+O2k8Nj47a Tw3PjtpPDg+O2k8OT47aTwxMD47aTwxMT47aTwxMz47PjtsPHQ8O2w8aTwwPjs+O2w8dDw7bDxpPDA+Oz47bDx0PDtsPGk8MD47PjtsPHQ8O2 w8aTwwPjs+O2w8dDw7bDxpPDY+Oz47bDx0PHQ8OztsPGk8MD47Pj47Oz47Pj47Pj47Pj47Pj47Pj47dDw7bDxpPDE+O2k8Mj47aTwzPjtpPDQ+O2k 8NT47aTw2Pjs+O2w8dDxwPHA8bDxUZXh0Oz47bDwxOz4+Oz47Oz47dDxwPHA8bDxUZXh0Oz47bDwxOz4+Oz47Oz47dDxwPHA8bDxUZXh0Oz4 7bDwwOC8wMS8yMDA3Oz4+Oz47Oz47dDxwPHA8bDxUZXh0Oz47bDxcPGEgc3R5bGU9InRleHQtZGVjb3JhdGlvbjogbm9uZVw7IGZvbnQtd2Vp Z2h0OiBib2xkXDsiIGhyZWY9IjUzMDAxIlw+QVJFQSBOLiA3IC0gVVJCQU5JU1RJQ0EgLSBFU1BST1BSSSAtIExBVk9SSSBESSBDT1NUUlVaSU9ORSB ERUxMQSBTRURFIERFTCBESVNUQUNDQU1FTlRPIERFSSBWSUdJTEkgREVMIEZVT0NPIFZPTE9OVEFSSSBESSBTQUlOVCBNQVJUSU4gSU4gTE9D QUxJVMOAIFBMRU9ELiBJTVBFR05PIERFTExFIFNPTU1FIE5FQ0VTU0FSSUUgUEVSIExBIFBST0NFRFVSQSBESSBFU1BST1BSSU8uIExJUVVJREFaS…
-
Visual Studio 2008 crashes in “split view”
“Split view” is one of the new features in Visual Studio 2008 web designer: this is the possibility to have Design View and Source View of your page at the same time (see What’s New in ASP.NET and Web Development, “Visual Web Developer Enhancements” paragraph). A few days ago a customer called in to report a problem with his Visual Studio and split view: when trying to view a file (even a new one) in split view or design view the IDE either crashed or frozen. As you can imagine we took a crash dump with adplus, but a first look at the threads, stack, exceptions etc… (the usual stuff) did not show anything interesting. Then I thought to some external process (like an antivirus, third parties add-ons etc…) that might meddle in and had a look at the list of modules loaded within the process with lmf (list loaded modules with full path). There where only a couple of odd dlls loaded, from Office 2003: 33f20000 34118000 FPCUTL (deferred) Image path: C:\Program Files\Microsoft Office\OFFICE11\FPCUTL.DLL Image name: FPCUTL.DLL Timestamp: Wed Jun 06 19:44:55 2007 (4666F297) CheckSum: 001EC356 ImageSize: 001F8000 File version: 11.0.8170.0 Product version: 11.0.8170.0 File flags: 0 (Mask 3F)…
-
An interesting fusion cache lock: that’s what the GAC is meant for
This has been an interesting case where we had an ASP.NET 2.0 application which under load was completely blocked after a few minutes. Since we were talking about a hang/deadlock (as reported from the customer), the !critlist command (you can find the command within the SieExtPub.dll extension) is a good start: 0:021> !critlist CritSec at 7a393800. Owned by thread 21. Waiting Threads: 6 7 8 9 10 11 12 13 42 72 104 107 108 109 110 111 112 113 114 115 116 CritSec at e194c. Owned by thread 32. Waiting Threads: 21 Thread 32 holds a critical section and thread 21 is waiting on it, but thread 21 is also owning another critical section, and about 20 other threads are waiting there… so apparently this is not a real deadlock (21 is waiting on 32 but 32 is not waiting on 21); but what is thread 32 doing? 0:032> kpL2000 ChildEBP RetAddr 0327fd30 7c822124 ntdll!KiFastSystemCallRet 0327fd34 77e6baa8 ntdll!NtWaitForSingleObject+0xc 0327fda4 79e718fd kernel32!WaitForSingleObjectEx+0xac 0327fde8 79e718c6 mscorwks!PEImage::LoadImage+0x199 0327fe38 79e7187c mscorwks!CLREvent::WaitEx+0x117 0327fe48 7a0e288e mscorwks!CLREvent::Wait+0x17 0327fe9c 7a086e76 mscorwks!Thread::SysSuspendForGC+0x52a 0327ff88 7a0d867b mscorwks!SVR::GCHeap::SuspendEE+0x16c 0327ffa8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x3b 0327ffb8 77e66063 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b 0327ffec 00000000 kernel32!BaseThreadStart+0x34 It is waiting to start GC but it cannot, because thread 95 has PreEmptive…