Tweet
Hi all and ex-Glog friends,
I am looking for feedback and input on concurrent integration posting volume (WMServlet).
The issue is.. integration is posting like a dream.
Then suddenly integration starts posting very slowly.. as in only a few posts per minute.. or even a few minutes between posts..
*Posting via BizTalk
I found that an RDP application on the app/web server (vino-server) was eating 99.9% CPU.
When integration slowed to a crawl, vino-server showed for the first time on TOP, at the top with 99.9 CPU.
*Direct correlation between integration slowing and vino-server taking 99.9 CPU
Killing vino-server restored idle CPU. but integration never picked back up..
Biztalk still posted very slowly.. as described above.. so I am just not clear here..
I was wondering if maybe there were too many hung connections wanting to timeout.. but wasn't sure.
I tried to make sense of netstat.. but again wasn't clear exactly..
I checked google and this is a known issue with vino-server.. taking 100% CPU - it's a bug.
I am fairly convinced that the vino-server app is the cause of the issue.. but noone seems to want to believe me.
THus, I am looking to the big guns for some input.
Quick specs on Test system being used:
OTM 5.5 CU02 #4
App and Web are on the same box
*** App is OAS
OS = Red Hat Linux
4 CPU's, 4 GB ram
DB: 10g 10.2.0.3
Win 2003
4 CPU's, 4 GB ram
SAN storage
Not high-end specs.. but a Test instance.
Assuming for a second nothing is wrong with this instance and everything is working ok,
can someone please provide input on the following:
My first question is about concurrent posts to WMServlet.
What actually controls this limit ?
I see tomcat config shows maxthreads 500
*CLient uses BizTalk to HTTPPOST and Biztalk can grow to hundreds of concurrent threads.
I found the following:
1: Not running out of CPU on web/app
2: memory is not swapping at all per vmstat
3: both JVM's appear to be ok as far as memory in the log
Tomcat: INFO | jvm 1 | 2008/01/14 13:00:10 | 621345K->414715K(1023424K), 0.1054570 secs]
OAS: [GC 647391K->433253K(1023424K), 0.0487000 secs]
4: No errors in any logs including Apache access logs
5: No backlog appearing in Event Diag
At the highest volume:
OAS JVM was using 1.6 GB ram (per TOP)
and Tomcat JVM was using 900 MB ram (per TOP)
*Again.. zero swapping was taking place and both JVM logs showed ok memory as noted in #3 above.
If system resources looked ok, can you pump 100's of concurrent posts without causing a problem ?
For example, the client must upload 100,000 location UPDATES into OTM.
FOr reasons.. this cannot be via .CSV or DirLoadServlet since they location loads are all updates to existing locations. (IU)
My 2 goals here are:
1: How many concurrent threads are realistic to be posting location updates to WMServlet (or any inbound integration to OTM)
2: Other than the vino-server issue, what else would/could cause integration to just slow to a crawl for no obvious reason ?
Lastly, any idea what type of expectation there should be on the time to load 100K location updates via WMServlet ?
Thanks for reading this.
I greatly appreciate any suggestions or input.
Best Regards,
Brian R
I am looking for feedback and input on concurrent integration posting volume (WMServlet).
The issue is.. integration is posting like a dream.
Then suddenly integration starts posting very slowly.. as in only a few posts per minute.. or even a few minutes between posts..
*Posting via BizTalk
I found that an RDP application on the app/web server (vino-server) was eating 99.9% CPU.
When integration slowed to a crawl, vino-server showed for the first time on TOP, at the top with 99.9 CPU.
*Direct correlation between integration slowing and vino-server taking 99.9 CPU
Killing vino-server restored idle CPU. but integration never picked back up..
Biztalk still posted very slowly.. as described above.. so I am just not clear here..
I was wondering if maybe there were too many hung connections wanting to timeout.. but wasn't sure.
I tried to make sense of netstat.. but again wasn't clear exactly..
I checked google and this is a known issue with vino-server.. taking 100% CPU - it's a bug.
I am fairly convinced that the vino-server app is the cause of the issue.. but noone seems to want to believe me.
THus, I am looking to the big guns for some input.
Quick specs on Test system being used:
OTM 5.5 CU02 #4
App and Web are on the same box
*** App is OAS
OS = Red Hat Linux
4 CPU's, 4 GB ram
DB: 10g 10.2.0.3
Win 2003
4 CPU's, 4 GB ram
SAN storage
Not high-end specs.. but a Test instance.
Assuming for a second nothing is wrong with this instance and everything is working ok,
can someone please provide input on the following:
My first question is about concurrent posts to WMServlet.
What actually controls this limit ?
I see tomcat config shows maxthreads 500
*CLient uses BizTalk to HTTPPOST and Biztalk can grow to hundreds of concurrent threads.
I found the following:
1: Not running out of CPU on web/app
2: memory is not swapping at all per vmstat
3: both JVM's appear to be ok as far as memory in the log
Tomcat: INFO | jvm 1 | 2008/01/14 13:00:10 | 621345K->414715K(1023424K), 0.1054570 secs]
OAS: [GC 647391K->433253K(1023424K), 0.0487000 secs]
4: No errors in any logs including Apache access logs
5: No backlog appearing in Event Diag
At the highest volume:
OAS JVM was using 1.6 GB ram (per TOP)
and Tomcat JVM was using 900 MB ram (per TOP)
*Again.. zero swapping was taking place and both JVM logs showed ok memory as noted in #3 above.
If system resources looked ok, can you pump 100's of concurrent posts without causing a problem ?
For example, the client must upload 100,000 location UPDATES into OTM.
FOr reasons.. this cannot be via .CSV or DirLoadServlet since they location loads are all updates to existing locations. (IU)
My 2 goals here are:
1: How many concurrent threads are realistic to be posting location updates to WMServlet (or any inbound integration to OTM)
2: Other than the vino-server issue, what else would/could cause integration to just slow to a crawl for no obvious reason ?
Lastly, any idea what type of expectation there should be on the time to load 100K location updates via WMServlet ?
Thanks for reading this.
I greatly appreciate any suggestions or input.
Best Regards,
Brian R
Comment