In my case IIS 7.5 was hosting a faulty application (framework 4) that was causing it's application pool to crash. Developers were unable to find out what was causing the application pool to crash. Two error event messages were logged, one in application and one in system log.
Application Log error event with event id 1000 from source Application Error :
System Log warning event with event id 5011 from source WAS:
After the error event 1000 in application log there is information event containing minidump files for debugging from source Windows error reporting:
To debug with wndbg x64 download and install windows sdk from http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=8442
After installing windbg, open the crash dump file. First, you can setup symbol path files (local and internet):
.sympath SRV*C:\localsymbols*http://msdl.microsoft.com/download/symbols
Load SOS debugging extension
.loadby sos clr
----------------------------------------------------------------------------
The user dump currently examined is a minidump. Consequently, only a subset
of sos.dll functionality will be available. If needed, attaching to the live
process or debugging a full dump will allow access to sos.dll's full feature
set.
To create a full user dump use the command: .dump /ma <filename>
----------------------------------------------------------------------------
Few commands are available when debugging minidump. CLRstack and PrintException are available for executing. After running !CLRstack in windbg there were couple of functions that were executing in loop. Two of them were :
From the event log, faulting module is oci.dll, suggesting that application is trying to run something using installed oracle client. To get clearer picture what has happened I have run !PrintException -nested. There were multiple nested exceptions from function to send email when something is wrong with application :
and, one exception that triggered send email function :
After running !PrintException 00000001c0923aa0 :
So, application pool crashing was caused by badly written recursion function to send notification email message to the developer when something was wrong with IIS application. SQL statement Insert Into using oracle client has failed, and called send email function to notify developer, but email server was unreachable. Email sending function has entered into endless loop causing IIS application to crash.
Application Log error event with event id 1000 from source Application Error :
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2Faulting module name: oci.dll, version: 11.2.0.1, time stamp: 0x4bb1da76
Exception code: 0xc00000fd
Fault offset: 0x000000000006f837
Faulting process id: 0x11a4
Faulting application start time: 0x01cd02af9bce5a0e
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: c:\<oracle client installation path>\oci.dll
Report Id: d8925b5a-6ea9-11e1-906a-00215e63edd4
System Log warning event with event id 5011 from source WAS:
A process serving application pool 'name of the application pool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4084'. The data field contains the error number.
After the error event 1000 in application log there is information event containing minidump files for debugging from source Windows error reporting:
Fault bucket , type 0Event Name: APPCRASH
Response: Not available
Cab Id: 0
Problem signature:
P1: w3wp.exe
P2: 7.5.7601.17514
P3: 4ce7afa2
P4: oci.dll
P5: 11.2.0.1
P6: 4bb1da76
P7: c00000fd
P8: 000000000006f837
P9:
P10:
Attached files:
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_....
To debug with wndbg x64 download and install windows sdk from http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=8442
After installing windbg, open the crash dump file. First, you can setup symbol path files (local and internet):
.sympath SRV*C:\localsymbols*http://msdl.microsoft.com/download/symbols
Load SOS debugging extension
.loadby sos clr
----------------------------------------------------------------------------
The user dump currently examined is a minidump. Consequently, only a subset
of sos.dll functionality will be available. If needed, attaching to the live
process or debugging a full dump will allow access to sos.dll's full feature
set.
To create a full user dump use the command: .dump /ma <filename>
----------------------------------------------------------------------------
Few commands are available when debugging minidump. CLRstack and PrintException are available for executing. After running !CLRstack in windbg there were couple of functions that were executing in loop. Two of them were :
System.Net.Mail.SmtpConnection.GetConnection(System.Net.ServicePoint)
System.Net.Mail.SmtpClient.Send(System.Net.Mail.MailMessage)
From the event log, faulting module is oci.dll, suggesting that application is trying to run something using installed oracle client. To get clearer picture what has happened I have run !PrintException -nested. There were multiple nested exceptions from function to send email when something is wrong with application :
Exception type: System.Net.Mail.SmtpExceptionMessage: Failure sending mail.
InnerException: System.Net.WebException
and, one exception that triggered send email function :
Exception object: 00000001c0925e70Exception type: <Unknown>
Message: could not insert : [....][SQL: INSERT INTO ...]InnerException: System.Data.OracleClient.OracleException, Use !PrintException 00000001c0923aa0 to see more
After running !PrintException 00000001c0923aa0 :
Exception object: 00000001c0923aa0Exception type: System.Data.OracleClient.OracleException
Message: ORA-00001: unique constraint (....) violated
So, application pool crashing was caused by badly written recursion function to send notification email message to the developer when something was wrong with IIS application. SQL statement Insert Into using oracle client has failed, and called send email function to notify developer, but email server was unreachable. Email sending function has entered into endless loop causing IIS application to crash.
what version of windows server were you using? I'm not getting the dump files in Windows 2008 R2 SP1
ReplyDeleteServer version is the same Windows 2008 r2 SP1. Have you checked http://blogs.msdn.com/b/webtopics/archive/2009/11/25/how-to-collect-a-crash-dump-of-an-iis-worker-process-w3wp-exe-on-iis-7-0-and-above.aspx .
ReplyDeleteThanks for the quick reply, it's pointed me in the right direction, I've found http://blogs.msdn.com/b/tess/archive/2010/08/23/getting-full-user-mode-dumps-automatically-when-your-process-crashes.aspx which hopefully with help.
ReplyDeleteWonderful illustrated information. I thank you about that. No doubt it will be very useful for my future projects. Would like to see some other posts on the same subject! spicewood custom pool builder
ReplyDelete