Debugging Windows 2000 COM Applications - Documents (Page 3 of 4 )
Server II Not Responding
I. Configuration Server I and Server II are configured to serve website pages in a coordinated effort. The servers are active and achieve balanced delivery of web pages to users under Windows 2000 Advanced Server Load Balancing (WLB).
II. Situation Server I is the primary server in WLB and delivering 100% uptime. Server II, a secondary server in WLB, is giving 60% uptime and daily requires a staff administrator to periodically resolve Server Error errors manually. During office hours, errors cause a disruption of services to staff personnel. Over weekends, services are not provided to external users.
NOTE: Servers I and II are equal in hardware, OS configuration, IIS Configuration and Application code.
III. Problem Server II consistently stops delivery of services to users.
IV. Error Page cannot be displayed, 500 internal server error. System Log No messages. Application Log No messages. IIS Log Server busy message. Performance Monitor No activity greater then 50% CPU or RAM for longer than 45 seconds. System is not locked-up. There are no visible performance issues.
V. Resolution * Unacceptable Reboot will restore use to users. Stop and start of IIS out of process component will restore use to users.
VI. Server Hardware (Server I and II) Compaq Proliant 6400R- Model 6/500 (1024k cache) 4- Pentium III Xeon tm 500-mhz processors 2- gigabyte EDO 50ns memory Smart array 3200 Controller 2- NC3122 PCI Dual 10/100 4- 9.1 GB Wide Ultra2 Universal 10k drives 3- 18.2 GB Wide Ultra2 Universal 10k drives
VII. Server Software Compaq Network Teaming and Configuration Compaq Load Balancing Team Compaq Smart Start Version 5.2 Windows 2000 Advanced Server Service Pack 2 (sp2) OS Build Service Pack Service Pack 2 (sp2) Hot Fixes None Windows 2000 IIS Server Configure Window Load Balancing (Advanced Server) for Server I and Server II Affinity setting on primary (Server I): Port 80: None, Port 443: Class C Server II software has been reloaded according current Server I documentation 2 rebuilds, QA, Passed both times.
VIII. TESTING (To reproduce and correct error) Status o Error can not be induced manually 1. Error can not be manually reproduced
o Server II- System and Application Information 1. System Auditing Server II& I are configured the same 2. Application Components/website copied and setup from Server I to II
o Server I& II Compaq Diagnostic testing, Version 5.175 All tests passed successfully 2. CPU 3. Memory 4. Printing Devices 5. Parallel Interface 6. Graphics Controller 7. Smart Array Controller 8. Serial Interface 9. CDROM
Scenario 1 Reproduce error conditions.
o Server II Test process 1. Directly access Server II by IP 2. Execute lengthy processes, reports, logins, and thoroughly navigate website. 3. Review the Windows Performance Monitor of CPU and RAM during daily usage. Activate and monitor Profile Trace in supporting SQL server for unduly large queries or memory requirements.
Scenario 2 The idea of scenario 2 testing was to activate Server II (the secondary WLB server) as a single/standalone primary server providing website services and test its responsiveness. o Server II Test process 4. Turn off Server I leave off 5. Uninstall the WLB from Server II 6. Change IP address of Server II to 159.XXX.X.XXX in Compaq team, TCP/IP properties. 7. Server IIIP is 159.XXX.X.XXX normally. Server IIP is 159.XXX.XX.XXX, IP of registered domain name is 159.XXX.X.XXX 8. Reset Server IIIP to 159.XXX.X.XXX 9. Reboot Server II 10. Test access to website (http://XXX.state.XX.us/) from the inside network and outside the network. 11. Test Failed. Server II is unavailable. 12. Server I was brought back on line, and Server II was reconfigured as WLB secondary server. 13. All services restored to pre-test status.
o Result 1. Server responds as standalone server
Resolution o Error cannot be reproduced or corrected IX. WLB Document Login as Local Administrator Stop website Go to Network Properties Compaq Team Properties Check mark in WNLB Properties Virtual IP 10.XX.X.XX (Prod Staff View) Test 10.XXX.X.XXX Full Internet name: XXX.XX.us Host properties Priority ID = 1 1 for first server on team 2 for second team server Check mark in Initial Cluster Act Dedicated Address Physical address of the server (NIC card) Port Rules 80 80 TCP Multiple EQUAL NONE 443 443 TCP Multiple - EQUAL Class C OK TCP Properties Advanced IP listed should be: 1 Physical IP of server 1 Virtual IP of team Example: Platz 10.XXX.X.XXX 10.XX.X.XX (Production Virtual) No Reboot Required X. END No Resolution
Server II Debug Process Please execute items 1, 2 and 3 to prepare for collection of information when the server crashes. Additionally I will need the information from 1 and 2 as soon as you generate it for analysis. Item 4 should be executed again (now) to make sure it functions properly. After IIS fails, accomplish items 4, 5, and 6.
Please do Now 1) Machine Information. Please extract the files from the IISTools.zip into a directory and run the IISTools.bat file on Server II. This will generate an iistools.txt file. a) <<iistools.zip>> b) Put file in USERS/MBURNE1/HANG 2) Application Information. Please run the following applications on Server II (so that we can see if the VB DLLs are crashing) and put file in USERS/USER1/HANG. a) Windows 2000 i) <<vbchkw2k.zip>> a. You can run this as follows: vbchkw2k /sc > DLLoutput.txt b. Put file in USERS/USER1/HANG. 3) Before the hang occurs. Copy the folder named: DEBUG2 from the directory (Technetz) USERS/USER1 to the destination server. The files will work on your server after you have done this without rebooting the server. a) Open Task Manager and get the PID for Inetinfo (Write it down). b) Get the PID for every instance of DLLHOST.EXE (Write them down). c) From the Command Prompt execute this command within the DEBUG2 folder for each PID: userdump <PID> d) This will put a file with *.dmp into the folder where userdump.exe resides. After each command, rename the resulting file uniquely. e) Item a to d above are testing. To insure the process works, delete all *.dmp files from the folder. With ERROR
Once the hang has occurred 4) The following steps will allow us to get a memory dump of the IIS process as well as a debug log that will directly correspond to the memory dump. a) Open Task Manager and get the PID for Inetinfo (Write it down). b) Get the PID for every instance of DLLHOST.EXE (Write them down). c) From the Command Prompt execute this command within the DEBUG2 folder for each PID: userdump <PID>. RENAME the result file every time the command is executed. d) This will put a file with *.dmp into the folder where userdump.exe resides. After each command, rename the resulting file uniquely. e) Get a Perfmon with the following (with all available counters for all of them): (1) ASP (2) Processor (3) Process (4) WebService (5) Memory (6) Thread ii) Put All *.dmp files and Perfmon files in USERS/USER1/HANG.
5) Restore Services by Start and Stop of OUT-OF-PROCESS Service. 6) Put all files in Users/USER1/hang