Announcement

Collapse
No announcement yet.

Oracle Application Server running out of memory

Collapse
X
 
  • Time
  • Show
Clear All
new posts

  • Oracle Application Server running out of memory

    We have this OAS running Out of Memory issue in Production on a daily basis and we had to restart the Server everytime. OTM becomes slow to the point that the application was not responding.

    A check across the system showed the OAS JVM was using 100% of 1 CPU
    And the OAS heap was maxed out on memory:

    [Full GC 1024447K->1024447K(1024448K), 4.0970930 secs]
    [Full GC 1024447K->1024447K(1024448K), 4.0963020 secs]
    [Full GC 1024447K->1024447K(1024448K), 4.0915010 secs]

    Garbage Collection (GC) was taking place, but little or no un-used memory was being Unloaded.

    Scanning the OAS console.log, a pattern was seen where memory began rising.

    10:47 AM – OK at this point
    [GC 665769K->456278K(1023424K), 0.0744990 secs]

    As we approach 11:21 AM, OAS starts using more total memory and releases less memory via GC.
    The GC times increase also indicating more work is needed to find memory to Unload.

    11:21 AM
    [GC 875287K->672253K(1024448K), 3.7772500 secs]

    By 11:26 AM, OAS has begun maxing the heap before GC runs,
    and OAS begins performing FULL GC”s.

    [Full GC 1024447K->815040K(1024448K), 3.4803020 secs]

    At this point, GC is still Unloading memory and the application is still responding.

    As time goes on, OAS is able to Unload less and less memory.

    As of 11:38 AM, OAS is getting little or no memory back via GC.
    At this point, the application has already stopped responding.

    Please help.

  • #2
    Re: Oracle Application Server running out of memory

    JayVal,

    The biggest issue is that you don't have enough memory allocated to the OTM App server. Allocating 1GB is the bare minimum for OTM and doesn't leave enough room for memory-intensive OTM processes.

    If you're running on Win32, then you'll be limited to 1.2 or 1.4GB, but if you're running on any other platform, you should bump up to 2GB. This is done within your $OTM_HOME/oas/bin/oc4j.sh script.

    Keep in mind that you'll need enough physical memory on your server to accommodate a heap of this size. If you're only running OAS on this server, and the heap is 2GB, then you should have 4GB of physical memory to allow room for the heap, Java native memory and OS. If you're running more, then you'll need more memory.

    Hope this helps!

    --Chris
    Chris Plough
    twitter.com/chrisplough
    MavenWire

    Comment


    • #3
      Re: Oracle Application Server running out of memory

      Thanks Chris.

      Here is some information I got. Also I am giving here the contents of the oc4j.sh. I am not an Admin guy, so I may not be able to put the pieces together.

      1) The Oracle App server and Apache Web Server runs on the same box.
      2) The OS is Linux 32bit
      3) The System is 64bit.

      **************************************
      oc4j.sh contents
      **************************************
      #!/bin/sh
      #
      # oc4j - shell for invoking OC4J basic operations.
      #
      # Usage: oc4j [Options]
      #
      # Options:
      # -start : start OC4J
      # -shutdown -port <ORMI port> -password <password>
      # : stop OC4J
      # -version : display the version
      # -help : display this message
      #
      # Copyright (c) 2004, 2005, Oracle. All rights reserved.
      #
      ################################################## #######
      ########## START CONFIGURATION SECTION ##################
      ################################################## #######
      GLOG_HOME=/u02/otmPROD
      JAVA_HOME=/u02/otmPROD/jdk
      ORACLE_HOME=$GLOG_HOME/oas
      J2EE_HOME=$ORACLE_HOME/j2ee/home
      VERBOSE=on
      MEM_ARGS="-XX:PermSize=256m -XX:MaxPermSize=256m -Xms2048m -Xmx2048m -XX:NewSize=256m -XX:MaxNewSize=256m -verbose:gc"
      GC3_OPTIONS="-Duser.home=/u02/otmPROD/glog/config -Dglog.properties=glog.properties"
      OC4J_JVM_ARGS="$MEM_ARGS $GC3_OPTIONS"
      echo $JVMARGS
      # Set DASH variables
      DASH_HOME=$GLOG_HOME/dash; export DASH_HOME
      XPRESS=$DASH_HOME/bin; export XPRESS
      LD_LIBRARY_PATH=$DASH_HOME/lib:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH
      PATH=$DASH_HOME/lib:$PATH; export PATH
      #Any persistent arguments to specify at the JVM level can be set here
      #By default this will be read from the operating system environment
      if [ "$OC4J_JVM_ARGS" ]
      then
      JVMARGS=$OC4J_JVM_ARGS
      else
      JVMARGS=
      fi
      CMDARGS=-userThreads
      if [ "$VERBOSE" ]
      then
      VERBOSE=$VERBOSE
      else
      VERBOSE=off
      fi
      ORMI_URL=ormi://localhost
      ORMI_USER=oc4jadmin
      OC4J_JAR=$J2EE_HOME/oc4j.jar
      ADMIN_JAR=$J2EE_HOME/admin.jar
      SERVER_XML=$J2EE_HOME/config/server.xml
      ################################################## #######
      ########## END CONFIGURATION SECTION ##################
      ################################################## #######
      check_oc4j()
      {
      EXIT=0
      if [ "$JAVA_HOME" = "" ]
      then
      echo "Error: JAVA_HOME environment variable is not defined."
      check_msg="correct JAVA_HOME environment variable."
      EXIT=2
      elif [ ! -x $JAVA_HOME/bin/java ]
      then
      echo "Error: Can not find java executable in $JAVA_HOME/bin."
      check_msg="correct java executable."
      EXIT=2
      elif [ "$ORACLE_HOME" = "" ]
      then
      echo "Error: The ORACLE_HOME environment variable must be set before executing this script. Set this to the directory into which you unzipped oc4j_extended.zip."
      check_msg="correct ORACLE_HOME environment variable."
      EXIT=2
      elif [ ! -r "$OC4J_JAR" ]
      then
      check_msg="readable $OC4J_JAR."
      EXIT=2
      elif [ ! -w "$SERVER_XML" ]
      then
      check_msg="writable $SERVER_XML."
      EXIT=2
      elif [ ! -r "$ADMIN_JAR" ]
      then
      check_msg="readable $ADMIN_JAR."
      EXIT=2
      fi
      }
      echo_check_msg()
      {
      echo "Error: The command can not be run without $1"
      }
      start_oc4j()
      {
      echo "Starting OC4J from $J2EE_HOME ..."
      CMDARGS="-userThreads -config $SERVER_XML"
      while [ $# -ge 1 ]
      do
      case $1 in
      *)
      echo ""
      echo "Error: The optoin \"$1\" was not recognized."
      EXIT=1
      return
      ;;
      esac
      done
      if [ "$VERBOSE" = "on" ]
      then
      echo "Executing: $JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS"
      fi
      $JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS
      }
      shutdown_oc4j()
      {
      while [ $# -ge 1 ]
      do
      case $1 in
      -port)
      shift
      if [ "$1" = "" ]
      then
      echo ""
      echo "Error: You must specify the ORMI port value."
      EXIT=1
      return
      else
      ORMI_PORT=$1
      shift
      fi
      ;;
      -password)
      shift
      if [ "$1" = "" ]
      then
      echo ""
      echo "Error: You must specify the password value."
      EXIT=1
      return
      else
      ORMI_PASSWORD=$1
      shift
      fi
      ;;
      *)
      echo ""
      echo "Error: The option \"$1\" was not recognized."
      EXIT=1
      return
      ;;
      esac
      done
      if [ "$ORMI_PORT" = "" ]
      then
      echo ""
      echo "Error: You must specify the ORMI port using the -port switch."
      echo " The port value can be found in $J2EE_HOME/config/rmi.xml."
      EXIT=1
      return
      fi
      if [ "$ORMI_PASSWORD" = "" ]
      then
      echo ""
      echo "Error: You must specify the ORMI password using the -password switch."
      EXIT=1
      return
      fi
      echo "Shutdown OC4J instance..."
      CMDARGS="$ORMI_URL:$ORMI_PORT $ORMI_USER $ORMI_PASSWORD -shutdown"
      if [ "$VERBOSE" = "on" ]
      then
      echo "Executing: $JAVA_HOME/bin/java $JVMARGS -jar $ADMIN_JAR $CMDARGS"
      fi
      $JAVA_HOME/bin/java $JVMARGS -jar $ADMIN_JAR $CMDARGS
      }
      version()
      {
      echo "Getting the version of OC4J instance..."
      CMDARGS="-version"
      if [ "$VERBOSE" = "on" ]
      then
      echo "Executing: $JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS"
      fi
      $JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS
      }
      help()
      {
      cat <<EOF
      Usage: oc4j [Options]
      Options:
      -start : start OC4J
      -shutdown -port <ORMI port> -password <password>
      : stop OC4J
      -version : display the version
      -help : display this message
      EOF
      }
      ########################################
      ### Start main function section ###
      ########################################
      check_oc4j
      if [ ! "$EXIT" -eq 0 ]
      then
      echo_check_msg "$check_msg"
      else
      if [ $# -eq 0 ]
      then
      help
      else
      # the first argument
      CMDARG="$1"
      # decrement number of arguments
      shift
      # get the rest of the command line
      case $CMDARG in
      -start)
      start_oc4j "$@"
      ;;
      -shutdown)
      shutdown_oc4j "$@"
      ;;
      -version)
      if [ $# -gt 0 ] ; then
      echo ""
      echo "Error: The option \"$CMDARG\" does not take any argument."
      EXIT=1
      else
      version
      fi
      ;;
      -help)
      help
      EXIT=0
      ;;
      *)
      echo ""
      echo "Error: The option \"$CMDARG\" was not recognized."
      EXIT=1
      ;;
      esac
      if [ $EXIT -eq 1 ]
      then
      help
      fi
      fi
      fi

      Comment


      • #4
        Re: Oracle Application Server running out of memory

        Thanks for the details - based on the script, the memory values have been edited correctly (now 2048MB). Just make sure that you've got enough memory on this server. If running web and app on the same box, you should have 8GB of physical memory.

        --Chris
        Chris Plough
        twitter.com/chrisplough
        MavenWire

        Comment


        • #5
          Re: Oracle Application Server running out of memory

          Thanks again.

          While working on this, I found something. The App and Web Servers run on UNIX environment. I find that the $OTM_HOME is not created. I am wondering if the memory setting change done to oc4j.sh will be picked up by the Oracle application server.

          This is just an observation. Correct me if I am wrong.

          Thanks,

          Comment


          • #6
            Re: Oracle Application Server running out of memory

            You're welcome.

            I use the notation $OTM_HOME, simply to reference the OTM install directory. This variable doesn't need to be defined within your environment for OTM to startup. So, you should be fine.

            After restarting OTM, you can verify that the memory values have increased by watching the $OTM_HOME/logs/oas/console.log file. The Java GC statements should look like this:
            Code:
            [FONT=Arial][SIZE=2][COLOR=navy][COLOR=navy][FONT=Arial][Full GC 1024447K->815040K(2048896K), 3.4803020 secs][/FONT][/COLOR][/COLOR][/SIZE][/FONT]
            Where 2048896K is the amount of memory assigned to OAS.

            --Chris
            Chris Plough
            twitter.com/chrisplough
            MavenWire

            Comment

            Working...
            X
            😀
            🥰
            🤢
            😎
            😡
            👍
            👎