It may appear as a contradiction after my previous post, but the first thing to do to start analyzing a memory dump is ask yourself: do I really need a dump?!? ? Let me explain: when you need to troubleshoot an error there are a number of things to do before really going down the dump path, simply because not all problems can be resolved in that way… keep in mind that a dump is nothing more than a snapshot of a process at a certain point in time, we can try to understand what happened in the past but with some limitations (as long as the details we are looking for are still in memory), and of course we can’t know what happened to the process after the dump has been taken; exactly like a picture you can take with your digital camera. For example if you are having problems to remotely debug your application, I hardly think a dump can add any value to your troubleshooting… while in case of a memory leak a dump is one of the first things I ask the customer to provide me (but again there are some things before this step). What’s…
-
-
New to debugging? How it all begun (and how could begin for you, too…)
When I joint Microsoft and the EMEA Internet Dev Support Team in late 2004, I soon realized that I had to build a new skillset to have a future in my new role; before that I was a kind of “self made” developer, in the sense that almost everything I learnt I did in “the hard way”, buying and readings my own books, building a lot of samples, testing, making a lot of mistakes etc… I’ve not had a real mentor, I simply tried to learn from my own experiences, testing things “in real life” and watching the results. And for me in that period the word “debugging” meant only the Visual Studio debugger, run the application and inspect my code hunting for bugs; I never thought to WinDbg, memory dumps and deep system internals (well, to he honest I knew something about memory dumps, but I thought it was too hard for me and never really tried to get my hands dirty with that stuff). Then I join Microsoft and the iDev team, and my horizons expanded. Not too much time had to pass before I realized the kind of cases we were managing required some different (and deeper)…
-
Start to play with LogParser
A few months ago (back at the beginning of March) at the annual offsite meeting my virtual team had in Lisbon, my colleague Doug held an hour session about LogParser and some of the cool features it has. Doug promised to blog about it with more details, so I don’t want to steal him an argument (and he for sure knows more than me on the subject), but I started play with it to analyze some logs I had, and among the others I wrote a couple of scripts to automate some tasks I previously had to do manually, and just wanted to share with you (and post on my blog as a future reference). So, here’s the deal: you may know from Tess, strong named assemblies should not be deployed in the /bin folder; and if you ever had to troubleshoot a managed memory leak, you probably know that if you have an assembly which is loaded multiple times in memory because you deployed it in the /bin folder of every application you have, it’s a good idea to install it in the GAC to avoid wasting server resources. So far so good, but how to find those assemblies?…
-
I have an OutOfMemory exception in my dump! Am I leaking memory?
Well… the short answer is: not necessarily ? I had this discussion a few days ago with a customer who called CSS for a crash they were having, and since he was also starting to learn some debugging and dump analysis basics, he was interested to have some deeper details about my findings in the crash dump they sent in. In particular, he thought the crash was due so an OutOfMemoryException because checking the exceptions with WinDBG, he saw something like the following: 0:000> !dumpallexceptions Going to dump the .NET Exceptions found in the heap. Number of exceptions of this type: 1 Exception 0cf500bc in MT 79b94ee4: System.ExecutionEngineException —————– Number of exceptions of this type: 1 Exception 0cf5007c in MT 79b94dac: System.StackOverflowException —————– Number of exceptions of this type: 1 Exception 0cf5003c in MT 79b94c74: System.OutOfMemoryException —————– The point is that we are pre-loading a few exception objects on the heap when starting an AppDomain, and the reason is quite simple if you think about it for a moment. Let’s pretend we don’t have exception objects preloaded: if an application ever reaches a condition where needs to allocate some more memory but there is not enough memory available, then the…
-
Visual Studio debugger conflicts on port 80
I personally got this problem on my laptop a while ago where IIS and Skype where involved, and even if this time the customer reported two different applications involved (Visual Studio 2005 and Cisco IP Communicator), the symptoms where the same: with Visual Studio 2005 and Cisco IP Communicator running ad the same time, he was unable to debug his web application with the message “Unable to start debugging on the web server. Could not start ASP.NET or ATL Server debugging. Verify that ASP.NET or ATL Server is correctly installed on the server“. If he stopped the IP Communicator the problem went away; clearly a conflict between IIS and Cisco Communicator. Just to be sure we had a look at “netstat -a -b”, and found (as expected) that both applications were listening on the HTTP port TCP computername:http computername.domain.com:0 LISTENING 492 [inetinfo.exe] TCP computername:http computername.domain.com:0 LISTENING 3872 [Communicator.exe] Since I don’t know how Cisco IP Communicator works (and we can’t support third party apps anyway), we decided to change the IIS port to 81, but this time the customer was getting “Unable to start debugging on the web server. Unable to connect to the web server. Verify that the web…
-
Debugger not debugging, and Response.Redirect() not redirecting…
The customer was working on a WinXP machine joined to a domain, but had logged in using a local account (I’ve not tested if this may have contributed to the problem so this may be a useless information). Anyway to make things easy I connected to the faulting machine using EasyAssist to have a better look at the configuration: everything looked fine both in IIS and Visual Studio. Interestingly browsing the application from IE directly we found that even if he was using “http://localhost/app” the page was shown in the “Internet” zone (hence the debugger error). This usually happens when the application is created using either the IP address or the FQDN (Fully Qualified Domain Name) of the web server; to verify, we created a new test project as “http://localhost/test” but got the same debugging problem, and again the page belongs to the Internet zone. As a test we enabled “Automatic logon with current username and password” for the Internet zone and then the debugging started working fine. Talking about this with my team colleagues they suggested uncheck “Automatically detect intranet network” in the “Local Intranet” settings and enable the three remaining checkboxes; this because under some circumstances IE may…
-
ActiveX component can’t create object… a tough one!
This week I’ve been working on a case from a customer who was migrating his web application to Windows 2003, and while on the old servers (Windows 2000) and on his development machine everything was working fine (as usual 😊), after deployment on the production server he got a nasty “ActiveX component can’t create object” error message, and was not able to fix it. The troubleshooting started from the usual and maybe most obvious possible causes (component registration, permissions etc…), but we soon realized the problem was not that obvious as we expected. The application was made by a custom ActiveX VB6 object which internally referenced other custom dlls and msrdo20.dll; this was then embedded into a .aspx page to show some use interface in IE to the end user. The customer sent me his application to try to reproduce the problem on my machine, and at the first run I got exactly the same error message he reported! Ah Ah! I got you! 😊 On my machine I got rid of it registering all the involved components, something we already tried on the phone, but worth another check. I came back to the customer and set up an EasyAssist session to carefully check his…
-
Are you profiling your application and now you can’t debug anymore?
This is an interesting case I got very recently and that made me scratch my head a lot, wondering what was going wrong on customer’s machine for a while… until I called Doug on the rescue and we (he) drove the call to an happy ending 😊 I got this case from a colleague on the Visual Studio team; the customer reported a weird problem with the debugging of their ASP.NET 1.1 application (with Visual Studio 2003): when trying to step into the code line by line with F11, the debugger actually stopped only on the signature of every method skipping completely the body of the method, and to add something more, this happened only when they were using a specific namespace (e.g. MyCompany.MyNamespace.Group); if they used the same exact code but in a different namespace, everything worked fine 😲. Moreover, this was happening on that very specific web server only, if they moved the code to another machine the debugger worked like charm (but moving to a different machine was not an option for them). My first thought went to the .pdb files, some kind of mismatch between the between the code esecuted in memory and the symbols used…
-
Find which w3wp.exe instance is hosting your application
Just a quick and easy one for this Friday evening: let’s say you have multiple w3wp.exe instances running on your web server (maybe you have multiple application pools, or you enabled Web Garden etc…), one of the ASP.NET applications you have on that server has some problems and you need to capture a memory dump to troubleshoot it: what can you do? Easy one: install the Debugging Tools for Windows and run adplus with something like the following: cscript adplus.vbs -hang -pn w3wp.exe But, hey, you don’t want to end up with a full dump for every w3wp.exe loaded in memory, you just need one of them, the one which is hosting your application… 🙄 Well… here is where iisapp.vbs comes to the rescue! 😊 You can find it under C:\Windows\System32 folder on your Windows 2003 machine, and if you simply run it at the command line, it’ll print to the console the PID of all w3wp.exe processes running and the Application Pool ID that particular process is hosting; here is a screenshot I just took in my machine: Since you are supposed to know under which application pool your application is running (and if you don’t know you can check…
-
A story of gflags and the crashing Notepad
As you can imagine, being able to reproduce a problem on a machine you can put your hands on is the best thing to troubleshoot and resolve it, so as I guess almost all my colleagues in CSS I quite often install on one of my machines the test apps I got from customers, or make experiments to confirm/deny a theory I have… this is how this story begun. A while ago, on my Vista x64 machine notepad.exe started to intermittently fail to load and the only “visible” symptom (of course apart the fact that Notepad was not showing up) was a very quick command prompt but unfortunately the content was impossible to read, and I had nothing in the Event Log or any other “obvious” logs. I noticed this problem because the CRM tool we use to track our work on support cases, offer the possibility to review the entire log of a call in notepad, and every now and then I happen to use that function if I want to have a quick recap of what has been done so far, specially if I got the case from another Engineer and I’m not fully aware of all the…