Upgrade 9.0.0 to 9.0.1 - Slow MS Addins Performance on PROD only

Name	Points
Greg Moeller	4184
David Williams	3349
JonA	3288
Kat V	2984
Woozy	1973
Jimmy Chiu	1883
Kwane McNeal	1437
Ragu Raghavan	1351
Roger French	1311
mark.cook	1244

Scott Whitfield

Basic Member

Posts: 7

11/1/2011 3:02 PM

This is my first post and I thank everyone in advance for their time.

We recently did a fresh new install of Lawson 9.0.1 along with the latest websphere, LUU etc...data was exported from our exissting data from a 9.0.0 env

At this point, everything is fine except with Addins performance.

The new environment is Virtual with Windows 2008 Server std x64 and SQL 2008 Server R2 x64 with separate PROD and TEST environments.

The environments are equal except for PROD is using Raw Data Maps for dedicated space.

Performance numbers for a 1000 record update from Excel 2010 using the latest MS Addins for Lawson ----

TEST - 1 min

PROD - 20 min

If we do a direct upload/change/add query to a temporary table in SQL on PROD we get the 1 min result

Any thoughts, ideas, suggestions are welcomed.

Thank you,

The addins error log is as follows:

"DATE: 10/6/2011 TIME: 3:32:44 PM
"
"ACTUAL CALL STRING THAT GENERATED THE ERROR:
"
"http://fhcplawprod.fhcp.local/servlet/Router/Transaction/Erp?_PDL=PROD&_TKN=GL40.1&_EVT=CHG&_RTN=DATA&_TDS=IGNORE&FC=Change&_ADDINS=TRUE&_f1=C&_f39=10&_f44=2011&_f45=9&_f46=GL&_f48=N&_f49=1000&_f50=&_f67r0=&_f68r0=A&_f69r0=10&_f70r0=10600200&_f71r0=77000&_f75r0=0&_OUT=XML&_EOT=TRUE"
"
---------------
"
"ERROR KEY = TRANSACTION_ERROR
"
"ERROR MESSAGE = Error occurred while performing transaction

"
"ERROR DETAILS:
"
" com.lawson.tesla.agent.TransactionException: TRANSACTION_ERROR
at com.lawson.tesla.exec.Tesla.invoke(Tesla.java:374)
at com.lawson.tesla.agent.ErpTransactionAgent.processRequest(ErpTransactionAgent.java:184)
at com.lawson.ios.agent.container.AgentContainerImpl$AgentWrapper.processRequest(AgentContainerImpl.java:479)
at com.lawson.tesla.broker.TransactionBroker.processRequest(TransactionBroker.java:78)
at com.lawson.ios.agent.container.AgentContainerImpl$AgentWrapper.processRequest(AgentContainerImpl.java:479)
at com.lawson.ios.agent.container.AgentContainerImpl.processRequest(AgentContainerImpl.java:255)
at com.lawson.ios.servlet.Router.doGet(Router.java:212)
at com.lawson.ios.servlet.Router.doPost(Router.java:293)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at com.lawson.servlet.TransformFilter.doFilter(TransformFilter.java:106)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.lawson.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:96)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.lawson.servlet.XSSValidatorFilter.doFilter(XSSValidatorFilter.java:99)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:445)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:504)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:301)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Caused by: java.lang.IllegalArgumentException: Whole value too large
at java.lang.Throwable.(Throwable.java:67)
at com.lawson.rdtech.crtio.CrtioNumber.set(Unknown Source)
at com.lawson.rdtech.crtio.CrtioNumber.set(Unknown Source)
at com.lawson.rdtech.crtio.CrtioServiceMessageBuffer.setTypeNumericInt(Unknown Source)
at com.lawson.rdtech.crtio.DynamicCrtioMsg.setFldValue(Unknown Source)
at com.lawson.rdtech.crtio.DynamicCrtioMsg.setFlds(Unknown Source)
at com.lawson.tesla.exec.Tesla.createCrtioBuffer(Tesla.java:1077)
at com.lawson.tesla.exec.Tesla.invoke(Tesla.java:314)
... 42 more
"
"
"

Alex Tsekhansky

Veteran Member

Posts: 92

11/2/2011 1:24 PM

USUALLY such slow performance is caused by WebSphere setup issues, lack of RAM, or missing indexes in the database. I would check the latter first. To check whether it's WAS or the database, try exporting data out using command line tools (e.g. rngdbdump) and import them back into a table where addins are slow. If rngdbdump/importdb is fast, then the issue is on a web side of things.

SP

Veteran Member

Posts: 122

11/2/2011 2:37 PM

Have you ran the "verifyxxxx" util (where "xxxx" equals appropriate db
platform)?

I would start there. Could be corrupt/missing indexes.

SP

Veteran Member

Posts: 122

11/2/2011 2:37 PM

How as the data exported/imported? Was this with native db tools, Lawson
tools, etc...??

SP

Veteran Member

Posts: 122

11/2/2011 2:42 PM

Scott,

How is performance with 1 record on PROD? 5 records? 50 records?
100?..???

Does anything appear in the ladb.log?

Also, after running verifymsf2000.exe, you might want to look at your
maintenance plans on the new db and make sure that weekly, they include a
db integrity check, index reorg, and update stats.

Scott Whitfield

Basic Member

Posts: 7

11/3/2011 7:59 PM

Thank you for the quick responses... I will have the team take each of your recommendations and pull the requested metrics.

Scott Whitfield

Basic Member

Posts: 7

11/8/2011 12:48 PM

I moved the virtual test environment to the same host and blade center as prod ... this move did not affect addins in TEST

I ran bldmsf2000ddl -URIq PROD and after completion ran addins ... 1000 records still takes 20 minutes in PROD

I'm thinking maybe it's a setting/config within websphere?

John Henley

Senior Member

Posts: 3348

11/8/2011 1:11 PM

Is query wizard slow, or just upload wizard?

John Henley

Senior Member

Posts: 3348

11/8/2011 1:14 PM

Also, make sure you don't have tracing turned on in WAS; DO NOT look at the settings via WAS console (I don't trust it)...look for the traces/log files in the WAS directory tree...if they are huge, you have tracing turned on in WAS.

Jimmy Chiu

Veteran Member

Posts: 641

11/8/2011 1:41 PM

The same portal server is hosting both your PROD and TEST portal right?

SP

Veteran Member

Posts: 122

11/8/2011 4:54 PM

What were the results of the verifymsf2000?

If you run verifysmf2000 on both TEST and PROD, do they report the same
numbers for tables, indexes, etc...?

Scott Whitfield

Basic Member

Posts: 7

11/8/2011 7:31 PM

yes... same number of indexes, tables, and no inconsistencies found.

John Henley

Senior Member

Posts: 3348

11/8/2011 8:57 PM

Are your test and prod running in same websphere instance and lawson environment ?

SP

Veteran Member

Posts: 122

11/8/2011 9:48 PM

Do you have multiple productlines in the production environment (e.g. Train, test, etc...)?

Did you recompile PROD (cobcmp) after the LSF upgrade?

Scott Whitfield

Basic Member

Posts: 7

11/9/2011 12:08 PM

there is a test and a prod websphere instance

yes, a cobcmp was run post LSF upgrade

John Henley

Senior Member

Posts: 3348

11/9/2011 1:41 PM

Going back to the beginning, you included an error message from the Addins. Are you getting that for each row that you are trying to upload? If you're getting the error, are you sure the addins upload is actually working?

Greg Moeller

Veteran Member

Posts: 1498

11/9/2011 4:26 PM

We are noticing some of the same issues just with job performance after our MSP 5/ESP 7 upgrade. Weird thing is TEST and PROD are now very close to the same levels, but TEST can run a GL190 importing 27,000 JE in about 3 minutes and in PROD the same job takes 2+ hours.

John Henley

Senior Member

Posts: 3348

11/9/2011 5:14 PM

That's interesting since batch is not using websphere. Are they in the same environment?

Karen Ploof

Veteran Member

Posts: 118

11/9/2011 5:33 PM

Are the security rules in TEST the same as those in PROD? I know that my TEST environments tend to have less complex security since access to that environment is limited. If security is more complex in PROD, and an update job runs faster in TEST than PROD, could it be because the PROD job (batch or addins) has to read through and resolve all the security rules, to be sure the user who submitted the task has proper authority, before processing the action?

Greg Moeller

Veteran Member

Posts: 1498

11/9/2011 6:07 PM

No, different environments, different servers, but the exact same security (if the lsdump/lsload utilities worked correctly). No guarantees there.

Maybe this should have been a separate topic, but I thought it fit here, too.

SP

Veteran Member

Posts: 122

11/9/2011 9:17 PM

Greg,

If you are running LSF901, you should look at the following: JT-241140,
JT-241140, JT-264451

Examine the joblogs and check to see if the jobs are spending an inordinate
amount of time actually moving from active to completed status.

On Wed, Nov 9, 2011 at 1:15 PM, <forums-lsf-s3-sys-admin@lawsonguru.com>wrote:

> [image: LawsonGuru.com Logo] <https://www.lawsonguru.com/> S3 Systems
> Administration Forum Notification A message was posted to a thread you
> are tracking. *RE: Upgrade 9.0.0 to 9.0.1 - Slow MS Addins Performance
> on PROD only* Posted by: *Greg Moeller*
> 11/09/2011 01:07 PM No, different environments, different servers, but
> the exact same security (if the lsdump/lsload utilities worked correctly).
> No guarantees there.
>
> Maybe this should have been a separate topic, but I thought it fit here,
> too.
> ------------------------------
>
> To view the complete thread and reply via your browser, please visit:
>
> https://www.lawsonguru.co...rmance-on-prod-only/
>
> You were sent this email because you opted to receive email notifications
> when someone posted and/or responded to a message on this forum.
> To unsubscribe to this thread please visit your user profile page and
> change your subscription options.
>
> Thank you,
> LawsonGuru.com
>

Scott Whitfield

Basic Member

Posts: 7

12/9/2011 7:43 PM

The users who are running the addins query are using LAUA ... we even created a 2nd websphere instance to rule out a 'bad' install ... we have the same results. PROD is 10x slower than TEST 1027 records in our control file takes 2m in TEST and 20m in PROD

Greg Moeller

Veteran Member

Posts: 1498

12/12/2011 3:31 PM

Scott: Did you ever achieve resolution to your slow add-ins experience? If so, would you mind sharing what it was?

Scott Whitfield

Basic Member

Posts: 7

6/28/2012 10:43 AM

My apologies for the long delay...

Ultimately, between a certified lawson vendor and lawson support no solution could be found. We rebuilt our app server and migrated the data over and addins performance was up to par.

Our theory is that the original upgrade used the 'cygwin' utility which may have caused some unknown issue during the migration. We did not use this tool with the 2nd app server... this is the only difference between our upgrade process.

Regards