Hung Threads - Lawson crashes daily during business hours

Sort:
You are not authorized to post a reply.
Author
Messages
Ron Maskal
Basic Member
Posts: 5
Basic Member

    We are IBM i, and run S3 Financials (AR, AP, AM, AC and GL).  We just upgraded Lawson from 9.0.0 to 9.0.1 and also upgraded our hardware, including moving from V5R4 to 7.1.  We use all 3 UIs (Portal, LID and LSO) and have MS Add Ins.

    Every since the upgrade, every day Lawson goes down due to "hung threads".  I am wondering if anyone has any experience with this and/or know someone who does.  Currently we have a Priority 1 ticket open with Lawson support, but this is 2 weeks now where we are constantly restarting Lawson during business hours.

    I can provide more technical details if you need them, any help is appreciated.

    Greg Moeller
    Veteran Member
    Posts: 1498
    Veteran Member
      We had this happen to us... Check to make sure that all of your Element Groups transferred/upgraded appropriately. The tools (lsdump/lsload) back when we did our upgrade did not transfer these across. *May have been fixed since then.
      allbusinessgomab
      Advanced Member
      Posts: 31
      Advanced Member
        I believe I can help you with that, but I would need to look at your SystemOut.log and accesslogs.  If interested , send those to me at cadixon@corvondix.com.
        Ron Maskal
        Basic Member
        Posts: 5
        Basic Member
          Thank you Greg.  I have confirmed the Element Groups exist.
          TammyT
          Basic Member
          Posts: 5
          Basic Member
            We've been dealing with this on/off since we moved to LSF 9.0.1 almost 2 years ago.  Currently we are on LSF 9.0.1.8.  In our most recent case with Lawson on this topic, they had me apply the tech patch for JT-336259 (that was early August 2012).  The system was stable for a while, but it's recently started hanging again occasionally although we haven't noticed the hung threads in the log files anymore.
            One other thing I should mention is sometimes when our system seems "slow", simply stopping/restarting lase seems to help vs completely restarting Websphere.  That doesn't always work, but sometimes it does.
            Good luck... I'd be interested to know if you ever get a lasting resolution.  We still have not.
            Brian Allen
            Veteran Member
            Posts: 104
            Veteran Member
              We had similiar issues where lase was the culprit. This became much better once we vertically scaled (3 JVMs) and set min / max JVM performance parameters as recommended by Lawson Prof. Services.  Heavy Lawson Security activity seemed to be the issue for us. 
              Greg Moeller
              Veteran Member
              Posts: 1498
              Veteran Member
                We have also scaled vertically to 3 App servers (3 JVMs). Also, if you are finding issues with lase and are using TDS, this link helped us quite a bit. Once we added indexes to some of the most heavily used searches, things straightened out.
                http://www.ibm.com/develo.../library/t-tds-perf/
                mark.cook
                Veteran Member
                Posts: 444
                Veteran Member
                  The JVM issue seems to be common. We too have mulitple JVM's. What I have seen talking to a number of customers is that they scaled to mulitple JVM's based on the needs of the business. So one hospital in our area scaled to accomodate Add-in calls, another LBI and we scaled our to accomodate MSCM calls. In any case, the scaling to mutliple app servers seems to have taken care of the issues. We also recycle websphere monthly during a standard downtime window that seems to help as well.

                  How many JVm's are you running currently?
                  Ron Maskal
                  Basic Member
                  Posts: 5
                  Basic Member
                    Thanks everyone. We use TDS, are running 1 JVM, use LAUA security still. Lawson has provided a config change that has kept Lawson up and running longer (made it 2 weeks before another crash), but in the last handful of days we've experienced 2 more outages. Lawson's response to that was to install some logging programs they've provided to help pinpoint the root cause. Still no permanent solution, I'll keep you updated.
                    Brian Allen
                    Veteran Member
                    Posts: 104
                    Veteran Member
                      I understand that Lawson is recommending 2 or more JVMs on any new installations if you run Lawson Security. 1 JVM can sometimes struggle with a busy ERP system under Lawson Security.

                      When you asked about heavy Lawson Security activity, we have correlated issues to running larger security reports or making many changes in classes, especially when multiple users are in the security application (i.e. during our security implementation).
                      TJ Mann
                      Veteran Member
                      Posts: 44
                      Veteran Member
                        very similar issue about a few yrs back. i created additional cluster servers, and moved MSCM onto its WAS, as 2nd instance. since then, no issue for me.
                        Zeke
                        Basic Member
                        Posts: 6
                        Basic Member
                          Posted By Brian Allen on 09/25/2012 04:37 PM
                          We had similiar issues where lase was the culprit. This became much better once we vertically scaled (3 JVMs) and set min / max JVM performance parameters as recommended by Lawson Prof. Services.  Heavy Lawson Security activity seemed to be the issue for us. 

                          So what did they have you set the min/max to?
                          Brian Allen
                          Veteran Member
                          Posts: 104
                          Veteran Member
                            LPS had us set the following JVM settings.  These recommendations were from their discussions with Redmond / IBM that best works with how Lawson runs.  They recommended NOT going with the max settings due to how garbage collection works.

                            Sorry for the delayed response... too many projects lately.
                            min 384
                            max 1586

                            Generic JVM Arguments:
                            -Xgcpolicy:gencon
                            Ron Maskal
                            Basic Member
                            Posts: 5
                            Basic Member
                              We've had no issue with Lawson hangs since late October, so my team and Lawson worked together to determine why/how it was fixed.
                              Memory to Memory replication was on... we turned it off (Lawson recommends this anyway). A new LADB and service programs were provided by Lawson as well. We created a lower environment, prepared it as we had Prod at go live and successfully replicated the hang (Excel spreadsheet designed to pump in transactions mimicking a custom process that creates AP invoices, AP vendors and AR comments real time). Then tried different versions of LADB and service programs that were made available and found that a particular version of the service programs allowed semaphores to open/close correctly and we are now ok.
                              Lawson is preparing a technology patch.
                              You are not authorized to post a reply.