Severe Performance Problems Running 9.01 on Windows -

Name	Points
Greg Moeller	4184
David Williams	3349
JonA	3288
Kat V	2984
Woozy	1973
Jimmy Chiu	1883
Kwane McNeal	1437
Ragu Raghavan	1348
Roger French	1311
mark.cook	1244

Rob Steedle

Basic Member

Posts: 8

2/9/2011 2:11 AM

We are experiencing severe, but intermittent, performance problems with our new Lawson implementation and are wondering if any of you have encountered something similar. The system performance is acceptable most of the time, but there are times when it becomes excruciatingly slow. During those periods, which can list for 30 minutes or more, it can take minutes for screen refreshes to take place. During some of these epoisodes, CPU utilization can be very high, at other times CPU utilization is low and yet the machine appears to be waiting for work.

We opened a case with Lawson, who examined our Websphere and Lawson logs and found nothing. We opened a case with Microsoft, and they too found nothing.

We are running version 9.01 on Windows 2003 SP2 Enterprise Edition with Physical Address Extension enabled. The database server is Microsoft SqlServer 2005. The database server hosts one other database, and the application that uses that database has no performance issues.

The server is virtual , with 10.5 gig of RAM and 4 virtual 2.83 GHz processors. The host machine is running Hyper-V within a Windows Server 2008 R2 Datacenter with two four-core Xeon 2.82 Ghz processors and 32 gig of RAM.

Network statistics indicate that we have plenty of available bandwidth.

We are perplexed, to say the least. Any insights welcomed.We are experiencing severe, but intermittent, performance problems with our new Lawson implementation and are wondering if any of you have encountered something similar. The system performance is acceptable most of the time, but there are times when it becomes excruciatingly slow. During those periods, which can list for 30 minutes or more, it can take minutes for screen refreshes to take place. During some of these epoisodes, CPU utilization can be very high, at other times CPU utilization is low and the machine appears to be waiting for work. We have discerned no pattern for thsi behavior, although it is more likley to occur in the afternoon. We reboot the server every night.

We opened a case with Lawson, who examined our Websphere and Lawson logs and found nothing. We opened a case with Microsoft, and they too found nothing.

We are running version 9.01 on Windows 2003 SP2 Enterprise Edition with Physical Address Extension enabled. The database server is Microsoft SqlServer 2005. The database server hosts one other database, and the application that uses that database has no performance issues.

The server is virtual , with 10.5 gig of RAM and 4 virtual 2.83 GHz processors. The host machine is running Windows Server 2008 R2 Datacenter with two four-core Xeon 2.82 Ghz processors.

Network statistics indicate that we have plenty of available bandwidth.

We are perplexed, to say the least. Any insights or suggestions welcomed!

Jimmy Chiu

Veteran Member

Posts: 641

2/9/2011 4:48 PM

***The server is virtual , with 10.5 gig of RAM and 4 virtual 2.83 GHz processors. The host machine is running Hyper-V within a Windows Server 2008 R2 Datacenter with two four-core Xeon 2.82 Ghz processors and 32 gig of RAM.***

With Windows 2003 SP2 as guest OS, you should lower the CPU count to 2. Microsoft Hyper-V supports up to 2 virtual processors for WIN2003, 4 virtual processors for WIN2008.

You should also disable Time synchronization in your VM settings under Integration Services when the guest OS is WIN2003.

John Cunningham

Advanced Member

Posts: 31

2/9/2011 5:29 PM

Rob,
We experienced performance issues on our Lawson 9.0.1 install as well. We are on UNIX. I dont know much about Windows, but next time you have the slow down try to stop and start the LASE process. That has helped us get through our slow response times. A couple of questions.. Are you using ADAM? What fixpack of WAS/Java are you on? Are you integrated with LBI? We are still having some issues with performance and it would be nice to start to eliminate some of the causes. I would be very interested to know if the startlase etc.. fixes the slowness.

John Henley

Senior Member

Posts: 3348

2/9/2011 5:44 PM

Are you running the database server on the same server?
How are backups of the VM and the host performed--is it possible that they are running at the same time as the poor performance?
What else is running on that hyper-v host?

Rob Steedle

Basic Member

Posts: 8

2/10/2011 1:03 AM

We are using ADAM. WAS is 6.1.0.23 and Java is 1.5.0_17. We are integrated with LBI, but we are not yet using LBI; that is, the plumbing is in place but we don't have the reports and dashboards yet. (THis is a new installation of Lawson. We have HR live and are trying to bring up financials and procurement.) We have stopped and started LASE in the past without any effect, and we have enve tried turning security offf altogether for a period with no performance improvement.

Rob Steedle

Basic Member

Posts: 8

2/10/2011 1:08 AM

Good catch on the # of processors, thanks, we'll fix that tonight.

Rob Steedle

Basic Member

Posts: 8

2/10/2011 1:13 AM

The database server is on another host. The backups of the VM and host occur in the wee hours when no one is on the machine. After they finish, the machine is restarted. The host is running a couple of light-weight machines (a coupel of XP machines and a 2008 virtual domain controller). A few weeks ago, we had the Lawson VM running on a host with no other guest machines. Because of the performance problems, we moved it tho the current host because it's a little beefier than the previous host, but the performance problems persisted.

George Graham

Veteran Member

Posts: 201

2/10/2011 1:45 AM

Have you looked at Antivirus?

Rob Steedle

Basic Member

Posts: 8

2/10/2011 1:58 AM

Yes, I should have mentioned that. We took the draconian step of removing the antivirus software. This did nothing to improve the situation.

John Henley

Senior Member

Posts: 3348

2/10/2011 2:22 AM

Also look at DB server and see if compute statistics is on. =

EricS

Veteran Member

Posts: 80

2/10/2011 12:26 PM

We had a similar problem on an MSCM vritual server and on a Websphere server for another application. It turned out to be a "ballon memory setting" according to my VM administrators. The way I understood it, this is a way that VM will fool the server farm into reserving more memory than its using. Sorry I can't be more specific, I'm a UNIX guy and don't really understand the VM settings. I can tell you that once the found and removed the ballon memory the server performance went back to normal. If I remember correctly, this setting is turned on by default and must be turned off.

Jimmy Chiu

Veteran Member

Posts: 641

2/10/2011 3:42 PM

The host server, is it AMD or Intel processors?

Rob Steedle

Basic Member

Posts: 8

2/10/2011 4:12 PM

Intel Xeon

Ari

Veteran Member

Posts: 49

2/10/2011 4:51 PM

John, what would be the benefit of turning on compute statistics?

John Henley

Senior Member

Posts: 3348

2/10/2011 5:31 PM

There are two facets to my question.

First, if statistics are set to automatically be recomputed, and the database is very large, that can often bring other database processes to a standstill, and might explain why your performance suffers intermittently. So, I first wanted to know if they were turned on. If statistics are not being recomputed automatically, then that eliminates that as a potential cause for your performance issues being intermittent.

Second, if statistics are not used, that can cause non-Lawson processes (i.e. Crystal reports) to suffer. Lawson itself doesn't use statistics, since it really doesn't use complicated SQL queries with joins, etc. and elaborate query plans (a topic for another day). If you have non-Lawson processes, statistics should be turned used, but ideally should be run only as part of a scheduled job during off-peak hours.

Your first step toward resolving your issues is to understand why performance problems are intermittent, and what is coinciding with them.

Jimmy Chiu

Veteran Member

Posts: 641

2/10/2011 6:11 PM

Do this with WIN2003 guest OS on Microsoft Hyper-V:

1) Disable Time Synchronization Integration Services in your VM settings.

2) Add the parameter, /usepmtimer, in your guest os' boot.ini

3) if you are running Xeon 5500 (Nehalem)

http://support.microsoft.com/kb/2000977

hiepster

New Member

Posts: 2

2/16/2011 11:33 PM

We have the same issue as Rod almost same set up except ours is windows 2008 64 bit. 4 CPU 16GB of RAM. CPU always 90% and higher. We uninstalled Sophos. Any help would be much appreciated.

Thanks.

Rob Steedle

Basic Member

Posts: 8

2/17/2011 12:53 AM

hiepster, are you running this on Hyper-V?

Jimmy, thanks for your suggestions. We already had timesync disabled, and our processors are Xeon 5400s

hiepster

New Member

Posts: 2

2/17/2011 10:35 PM

Nope, we're on ESX 4.0. My previous jobs we run LS on Unix and never had any problems.

Robert

Basic Member

Posts: 5

3/1/2011 3:51 PM

We are also running Windows Server 2008 R2 hyper-v and Win2kR2 64bit on Lawson 9.01. We dont have issues with slowness but we have issues with running Microsoft Office Addin queries. The queries will timeout with a internal server error 500 error. It also happens randomly and GSC has not been any help.

Rob Steedle

Basic Member

Posts: 8

3/7/2011 10:07 PM

As of the end of last week, we resolved our performance problems. There were several factors which contributed to those problems. In summary:

The LS cache was increased from a value of 10 sec to 900 sec (resulted in a much faster response times from our domain controllers)
A memory pool tag issue identified and corrected. Server 03’ was failing to release a memory tag which would gobble up memory and force the system into higher disk I/O due to page faulting. Poolmon showed that the MMST was the culprit. Refer to Microsoft KB Article 324446
The virtual instance of the application server was moved to a dedicated drive resulting in less overall disk I/O on the SAN.
Additional memory was allocated to the application server. It had 10gig, and we doubled it.
Adjustments to WebSphere (heapsize, garbage collection) resulted in better memory handling
Additional JVM’s were added which resulted in a marked decrease of processor load. We had just one initially. The single JVM was spending most of its time performing garbage collection. We added two more on the same server.
Our local antivirus policy was incompletely configured, and the Lawson servers were not white-listed. Rectifying this resulted in marked decrease of time during drillarounds.

Thanks to all who contributed their thoughts and expertise, and many thanks to the good folks at Lawson who helped us in the tuning of WebSphere, including the vertical scaling of the JVMs.