Optimizing GC3 v5.0 Performance - Part 1

[chrisplough]

During the GC3 v5.0 release, there was strong continued development - for additional features, as well as performance optimization. As a result, multiple installers were released between GA (General Availability - the initial release) and now. The downside to this tactic is that some of the early GC3 v5.0 installations are not optimized for the best performance.

In this post, I'll cover how to update your GC3 v5.0 on Linux installation for improved performance and scalability.

IMPORTANT NOTE: The use-cases of GC3 are so widely varied that no one configuration will work for all GC3 instances. It is crucial that you make any applicable changes only to your test/pre-production environments and only promote them to production after they have proven beneficial and stable in your environment. Also, ALWAYS MAKE A BACKUP prior to making any configuration changes or applying any patches. Trust me

First, check the version of JRockit (the Java JVM used on Linux) that you have installed with GC3 using the following commands:

cd <gc3_install_dir>/jdk/bin/
./java -version

It should spit out a line containing either "1.4.2_04" or "1.4.2_05". "1.4.2_05" is a newer version of JRockit that introduced several performance and, more importantly, stability fixes. If it says "1.4.2_05", then you're installation used one of the later installers an you may already be optimized - you can reference the steps below to to verify your configuration. If it says "1.4.2_04", then you'll need to contact Oracle support and ask for the patch for the 1.4.2_05 JDK on Linux. This is distributed in quick-patch form, so installation is dead easy.

Once you have the patch, copy it to /tmp, shutdown GC3 and then do the following. I've used the name gc3v50-jdk1.4.2_05-linux.zip for the quickpatch name - this should be correct unless Oracle has recently changed it. Note: You'll need to run the following steps on all of you web and app servers to ensure that they are all running the same JDK version.

cd <gc3_home_dir>
mv jdk jdk.old     #Make a backup of the existing JDK directory
cd /tmp
unzip gc3v50-jdk1.4.2_05-linux.zip
cd gc3v50-jdk1.4.2_05-linux/
cp -r * <gc3_home_dir>     #This will copy the patch into your GC3 directory
cd <gc3_home_dir>/jdk/bin/
./java -version     #Ensure the version says "1.4.2_05" - if so, it is installed correctly.

Once that is complete, startup GC3 and run through your normal screens, just to ensure that it works. I wouldn't expect any problems here, though it is always good to test - just in case. Once you're done, we can how update the configuration files to better optimize the Java processes.

On your GC3 Web server(s), edit the <gc3_home_dir>/tomcat/bin/tomcat.conf file and find the following line:

wrapper.java.additional.3=-Xmx1025m   #Your memory value may differ from this one, the -Xmx is the important part.

Then add the following lines after it:

wrapper.java.additional.4=-Xgcprio:throughput
wrapper.java.additional.5=-Xverbose:memory

If you see either of the following lines, you can remove them:

wrapper.java.additional.4=-Xgcpolicy:optavgpause
wrapper.java.additional.4=-Xgcpolicy:parallel

Note: the lines in this section must run in numerical order (i.e. wrapper.java.additional.1=..., wrapper.java.additional.2=..., etc) - so you'll have to renumber the remaining lines in the section after inserting these two.

On your GC3 App server(s), edit the <gc3_home_dir>/weblogic/config/gc3domain/weblogic.conf file and find the following line:

wrapper.java.additional.3=-Xmx1025m   #Your memory value may differ from this one, the -Xmx is the important part.

Then add the following lines after it:

wrapper.java.additional.4=-Xgcprio:throughput
wrapper.java.additional.5=-Xverbose:memory

If you see either of the following lines, you can remove them:

wrapper.java.additional.4=-Xgcpolicy:optavgpause
wrapper.java.additional.4=-Xgcpolicy:parallel

Note: the lines in this section must run in numerical order (i.e. wrapper.java.additional.1=..., wrapper.java.additional.2=..., etc) - so you'll have to renumber the remaining lines in the section after inserting these two.

Another crucial aspect of performance is the amount of memory to allocate to the JVM. This is a fairly involved topic, so I'll write more about it later. One quick note, though - one of the reasons that JRockit is so fast is that it uses a lot of memory outside of the allocated heap size for optimization. As such, if you allocate a 1.5GB heap for JRockit, it can use as much as 2.5GB of memory. Keep this in mind when setting your initial memory sizes.

In fact, many of these same pricipals and configurations apply to GC3 v4.5 on Linux. If you're interested, let me know and i'll post instructions on how to make the same changes to GC3 v4.5.

In later parts of this series, I'll cover additional topics on optimizing your GC3 instance's performance - including JVM memory allocation, operating system tuning, GC3 thread tuning, etc. Stay tuned!

[adowla]

Re: Optimizing GC3 v5.0 Performance - Part 1

Chris,

We have done some performance settings for the app server and set the memory usage to 2 GB. However, we have a 8 GB box but when we tried setting the memory in the config files to 4 GB, we couldn't boot the app server. Any ideas on how to get past this limitation? We running on OTM on OAS.

Thanks for any help you can provide.
Arham

[chrisplough]

Re: Optimizing GC3 v5.0 Performance - Part 1

Arham,

Can you tell me what platform you're running on? There are memory limitations with the current 32-bit JDKs and OTM doesn't yet support (or work with) the newer 64-bit JDKs. Each platform has different memory limitations, and I believe the current limits are:

• AIX - 3GB
• HP-UX - 3GB
• Linux - 2.5 GB
• Solaris - 2GB
• Win32 - 1.5GB

Of these, JRockit on Linux is by FAR the best performing, even though it doesn't support the largest heap. This is partially due to the massive optimizations that JRockit performs -- in fact a JRockit instance with a 2.5GB heap can consume nearly 4GB of total memory (heap + jvm process).

Another point - you're one of the first clients to use OTM with OAS and I caution you to test diligently. While OAS both performs and functions better than OTM on WebSphere, it doesn't support all of the functions, nor perform as well as OTM on WebLogic. Also, since it's a newer platform, it is considerably more buggy.

Let me know what platform you're running on and what volumes you're intending to run through and I can provide you with more detailed info.

Thanks!
--Chris

[adowla]

Re: Optimizing GC3 v5.0 Performance - Part 1

Chris,

Thanks for the quick reply. We are using Linux with OAS. The amount of volume will quite substantial with somewhere around 500,000 order releases. We are looking into partioning the orders and running with fixed itineraries so are looking at all avenues of optimization.

Thanks for your help,
Arham

[chrisplough]

Re: Optimizing GC3 v5.0 Performance - Part 1

Arham,

I'm assuming that the 500K OR volume is an annual number? If so, then I would estimate between 1400 - 2500 ORs per day, depending on the weekly cycles of your company. While this is high, it's not within the upper tier of OTM installations. I would definitely recommend using new Linux servers with Intel 5160 or E5355 CPUs and a 2.5GB Java heap. Actual sizing will depend on a variety of factors, including solution complexity, line items, bulk plan size, concurrent users, number of reports, etc.

With high volumes, you'll want to split out the OTM tiers onto separate servers (web, app, rpt, db). Also, your DB server and storage will be critical, as most of OTM's performance issues usually boil down to that tier.

I would also ask you to consider WebLogic as your OTM App server. Oracle's OAS was just supported as of OTM v5.5 CU2, hasn't been production proven, and doesn't support all of the features of OTM (SCA, etc). I was involved with the port of OTM to WebSphere and helped lay the groundwork for the port to OAS before leaving last year. OTM has been running under WebLogic since late 1999 and enjoys a considerable lead over the other app servers with respect to stability, ease of support, performance, etc. I'd look at OAS again with OTM v6.0 or later, once they've had a chance to work out all of the kinks.

Thanks!
--Chris

[miks]

Re: Optimizing GC3 v5.0 Performance - Part 1

Chris,

Do you have further info on Optimizing Performance. I'm primarily interested in Tuning an OTM 5.5 CU#3 version running on Window 2003 Server. We are seeing sparatic performance problems during the day, which might be attributed to GC and the like.

Thanks in advance.

- Mike

[chrisplough]

Re: Optimizing GC3 v5.0 Performance - Part 1

Mike,

The areas I'd look at next on your system are Threads (including Queue backups)

Threads are a very involved topic and requires too much info to go into detail here - though you can view them and their respective queues using the EventDiag servlet: http://otmweb.company.com/GC3/glog.w...entDiagServlet

You can also view the current garbage collection graph via the WebLogic Console: http://otmapp.company.com:7001/console

The biggest issue you're likely to hit is around the maximum heap size available on the Windows platform. You can only use heap sizes of 1.2 - 1.4GB, whereas you can use heaps of 2GB and higher on other platforms. This is a tremendous issue for an application like OTM, which is very memory intensive.

If you are indeed hitting memory limits and aren't able to move off of the Windows platform, then you may have to look towards workarounds, like modifying when Bulk Plans and other events flow into the system or throttling integration in order to keep the system usable.

Thanks,
Chris