Prev
Next
Forums S3 Systems Administration

CLOSE_WAIT Clogging port and shutting down WAS app Server

Sort:
You are not authorized to post a reply.
Author
Messages
John Cunningham
Advanced Member
Posts: 31
Advanced Member
7/30/2010 6:05 PM
    Ever since we upgraded to ENV 9.0.1.5 we have been having trouble with CLOSE_WAIT threads backing up and killing our application server. We have a remote install of 2 HTTPServers, 2 AppServers Vertically scaled. They are WAS 7.0 FP 7. It seems like once a week someone calls with issues of portal locking up. I do a netstat -Aan | grep 9080 / 9082 (APP Server ports) and one of them will have 4-500 processes in close wait status. I have to shut down the corresponding app server until we reboot. Can anyone assist me with this issue, has anyone had to deal with it. I suspect that Lawson is not correctly closing its connections, but have had no luck with them. Any help with this would be greatly appreciated.
    John Cunningham
    Advanced Member
    Posts: 31
    Advanced Member
    8/23/2010 12:44 PM
      Posting an update, because it was such a pain to figure this out, hopefully it helps someone:
      At lawsons instruction, performed a thread dump on application server when CLOSE_WAIT started to accumulate. Lawson suggested this was an issue with communication between WAS and LDAP. It also seemed to occur when we were getting hit pretty heavily. I looked at the $DB2HOME/$INSTANCE/logs/ibmslapd.log and noticed this error:

      08/16/10 17:38:42 GLPSRV204W The server has temporarily suspended reading client
      requests from the network 5 times. There are 0 of 15 worker threads attempting
      to write results.

      After some research on IBM found this:

      Explanation:
      The server has temporarily suspended reading client requests from the network because there are no free worker threads for the server to process additional requests. The number of database connections may be too small for the current workload.

      Operator response:
      The server will start reading client request from the network as soon as a worker thread becomes available. Increase the number of worker threads in the server configuration by increasing the number of database connections.

      Found the dbconnections are set in $DB2HOME/$INSTANCE/etc/ibmslapd.conf and correspond to the ibm-slapdDbConnections which for us were set to 15. Bumped this value to the max (50) and restarted the LDAP and we have not had a problem for over a week. Lawson suggested some other tuning parameters which we implemented yesterday.
      You are not authorized to post a reply.