PrevPrev Go to previous topic
NextNext Go to next topic
Last Post 01/19/2015 1:19 PM by  Peter O
Mass transaction upload - TM to S3 interface
 9 Replies
Sort:
You are not authorized to post a reply.
Author Messages
bearcat32
Hr Apps Admin
TriHealth
Basic Member
(22 points)
Basic Member
Posts:12


Send Message:

--
09/11/2014 7:24 PM
    What is your approach to uploading mass transactions (ie Merit or pay rate changes) into TM and having them reflect in S3? After a few tests it seems the delivered flows take hours/days to simply interface the data to the LT holding tables. There has to be a more efficient solution.

    Thanks for any advice.
    Srini Rao
    Private
    Private
    Veteran Member
    (287 points)
    Veteran Member
    Posts:133


    Send Message:

    --
    09/11/2014 9:13 PM
    Hello, Last week we processed 9700 pay rate change transactions and it took 53 minutes to export to LT11.

    You check couple of things...
    1. Change process flow's log levels to "No Debug"
    2. Check out if you have any user exists on Lt11.
    Woozy
    Private
    Private
    Veteran Member
    (3469 points)
    Veteran Member
    Posts:701


    Send Message:

    --
    09/12/2014 9:11 AM
    The interface is really the best and only option that won't cause headaches down the road.

    Srini Rao is absolutely correct. I'm guessing you have logging turned on, which greatly decreases performance. We keep logging completely off for all of our batch flows unless there is something very specific we are troubleshooting - and then only with a very small batch.

    We do turn on some logging for short duration single record flows - i.e. Action flows - so that we track what happened in each event.

    Good Luck!
    Kelly Meade
    J. R. Simplot Company
    Boise, ID
    bearcat32
    Hr Apps Admin
    TriHealth
    Basic Member
    (22 points)
    Basic Member
    Posts:12


    Send Message:

    --
    09/12/2014 11:32 AM
    Thank you both very much for your responses.
    If I may ask a few additional questions in terms of your process.

    *Do you use spreadsheet designer to load the actual pay rates to TM or another utility?

    *Do you commit the data to TM in batches (e.g 1,000 at a time) or in a single load (e.g. all 9,700 transactions at once)?

    *Do you use process immediately for these or do you run the export flows to interface all the records under one work unit?

    Thanks again!
    Woozy
    Private
    Private
    Veteran Member
    (3469 points)
    Veteran Member
    Posts:701


    Send Message:

    --
    09/12/2014 1:11 PM
    In our case, we generally use LSD when we need to do these loads. Sometimes we do them all at once, and sometimes we break them up - but this is usually because of the data source or calculation differences rather than performance reasons. It isn't unusual for us to upload 5000+ records at once with LSD, and we frequenly do tens of thousands - though it is very slow.

    Our TM to S3 interface runs on a schedule twice-a-day, and we generally don't run it manually, even for these types of processes. We just let the interface process pick up whatever is available when it runs.
    Kelly Meade
    J. R. Simplot Company
    Boise, ID
    Srini Rao
    Private
    Private
    Veteran Member
    (287 points)
    Veteran Member
    Posts:133


    Send Message:

    --
    09/12/2014 1:44 PM
    We uploaded 9700 records in two batches. After uploading we schedule LT101 and export flow.

    bearcat32
    Hr Apps Admin
    TriHealth
    Basic Member
    (22 points)
    Basic Member
    Posts:12


    Send Message:

    --
    09/12/2014 1:52 PM
    Thanks Kelly.

    In terms of benchmarking could you give a ballpark figure on how long 10,000 records would take you to upload via LSD
    as well as how long it would take for 10,000 records to interface to S3 from TM via the export flows? We are getting varying definitions of "slow"
    so trying to ascertain what other clients are experiencing.

    Hope to see you @ Inforum!
    Woozy
    Private
    Private
    Veteran Member
    (3469 points)
    Veteran Member
    Posts:701


    Send Message:

    --
    09/12/2014 2:06 PM
    Unfortunately, I really don't know. I'm on the development side, so I'm not the one doing the uploads or running the interface.

    I know the upload speed really depends on how many fields, which business classes, etc. I think we generally see LSD uploads of 1-5 records per second, but it could be significantly slower if the server is busy or if the upload is "wide" or complicated. By "slow" I mean slow in comparison to real batch processing engines like SQL.

    On the TM to S3 interface side, when we do a mass pay change it may cause the interface to run for several hours instead of a few minutes. Again, I can't really give you a good estimate. Sorry.
    Kelly Meade
    J. R. Simplot Company
    Boise, ID
    Woozy
    Private
    Private
    Veteran Member
    (3469 points)
    Veteran Member
    Posts:701


    Send Message:

    --
    09/12/2014 2:09 PM
    By the way - unfortunately we won't be at Inforum this year. We were planning to be going live with our upgrade to LMRK/TM 10 this weekend. However, that has now been bumped out to next month. Lots to do.

    Have fun anyway! Say "hi" to everyone for me.
    Kelly Meade
    J. R. Simplot Company
    Boise, ID
    Peter O
    Systems Analyst
    Independent
    Veteran Member
    (205 points)
    Veteran Member
    Posts:69


    Send Message:

    --
    01/19/2015 1:19 PM
    I see this thread is old, but thought I would add my two cents anyways regarding how we perform merits/uploads:

    -We use LSD for transactions that are approx 10,000 records which will take a few hours. It depends on if we're doing an upload during peak utilization as well.
    -Merits are generally done with LSD as well, but anything that is done semi-regularly we develop into an IPA flow and run it which generally gives us better time (depending on the transactions).
    -Our Exports generally take only take a few minutes and we schedule them to run daily (anywhere from a few hundred ~1000 transactions)
    -We had the issue with exports taking a very long time as well (hours to days), but found that the issue was that our debugging was wide open (as stated above) - once we turned that off, our exports would run in a few hours for mass exports and a few minutes for the daily transactions.
    You are not authorized to post a reply.