Mass transaction upload - TM to S3 interface

Hello, Last week we processed 9700 pay rate change transactions and it took 53 minutes to export to LT11.

You check couple of things...
1. Change process flow's log levels to "No Debug"
2. Check out if you have any user exists on Lt11.

The interface is really the best and only option that won't cause headaches down the road.

Srini Rao is absolutely correct. I'm guessing you have logging turned on, which greatly decreases performance. We keep logging completely off for all of our batch flows unless there is something very specific we are troubleshooting - and then only with a very small batch.

We do turn on some logging for short duration single record flows - i.e. Action flows - so that we track what happened in each event.

Good Luck!

Thank you both very much for your responses.
If I may ask a few additional questions in terms of your process.

*Do you use spreadsheet designer to load the actual pay rates to TM or another utility?

*Do you commit the data to TM in batches (e.g 1,000 at a time) or in a single load (e.g. all 9,700 transactions at once)?

*Do you use process immediately for these or do you run the export flows to interface all the records under one work unit?

Thanks again!

In our case, we generally use LSD when we need to do these loads. Sometimes we do them all at once, and sometimes we break them up - but this is usually because of the data source or calculation differences rather than performance reasons. It isn't unusual for us to upload 5000+ records at once with LSD, and we frequenly do tens of thousands - though it is very slow.

Our TM to S3 interface runs on a schedule twice-a-day, and we generally don't run it manually, even for these types of processes. We just let the interface process pick up whatever is available when it runs.

We uploaded 9700 records in two batches. After uploading we schedule LT101 and export flow.

Thanks Kelly.

In terms of benchmarking could you give a ballpark figure on how long 10,000 records would take you to upload via LSD
as well as how long it would take for 10,000 records to interface to S3 from TM via the export flows? We are getting varying definitions of "slow"
so trying to ascertain what other clients are experiencing.

Hope to see you @ Inforum!

Unfortunately, I really don't know. I'm on the development side, so I'm not the one doing the uploads or running the interface.

I know the upload speed really depends on how many fields, which business classes, etc. I think we generally see LSD uploads of 1-5 records per second, but it could be significantly slower if the server is busy or if the upload is "wide" or complicated. By "slow" I mean slow in comparison to real batch processing engines like SQL.

On the TM to S3 interface side, when we do a mass pay change it may cause the interface to run for several hours instead of a few minutes. Again, I can't really give you a good estimate. Sorry.

By the way - unfortunately we won't be at Inforum this year. We were planning to be going live with our upgrade to LMRK/TM 10 this weekend. However, that has now been bumped out to next month. Lots to do.

Have fun anyway! Say "hi" to everyone for me.

I see this thread is old, but thought I would add my two cents anyways regarding how we perform merits/uploads:

-We use LSD for transactions that are approx 10,000 records which will take a few hours. It depends on if we're doing an upload during peak utilization as well.
-Merits are generally done with LSD as well, but anything that is done semi-regularly we develop into an IPA flow and run it which generally gives us better time (depending on the transactions).
-Our Exports generally take only take a few minutes and we schedule them to run daily (anywhere from a few hundred ~1000 transactions)
-We had the issue with exports taking a very long time as well (hours to days), but found that the issue was that our debugging was wide open (as stated above) - once we turned that off, our exports would run in a few hours for mass exports and a few minutes for the daily transactions.