Home
How to troubleshoot Spring Boot in OpenShift

How to troubleshoot Spring Boot in OpenShift

Admin | March 9, 2025 |



Spring Boot application deployment in Red Hat OpenShift is beneficial for having stand-alone Spring applications as microservices, given that it provides features such as externalized configuration, health check, circuit breaker, and failover. It is also very simple to containerize it.

Red Hat OpenShift support for Spring Boot tests and verifies various popular Java development frameworks and technologies running on the Red Hat OpenShift Container Platform. 

Use case deployment

Let’s deploy a Spring Boot application in OpenShift, following the application in Alexander Barbosa’s article Integrate a Spring Boot application with Red Hat Data Grid. First, run the following:

$ git clone -b openshift https://github.com/alexbarbosa1989/hotrodspringboot
$ oc project springboot-test
$ mvn clean fabric8:deploy -Popenshift

This will create a BuildConfiguration file, such as the following:

  strategy:
    type: Source
    sourceStrategy:
      from:
        kind: DockerImage
        name: 'registry.redhat.io/openjdk/openjdk-11-rhel8:latest' <-- attention*

This will result in three pods: builder, deployer, and application, as follows:

$ oc get pod
NAME                           READY   STATUS      RESTARTS   AGE
hotrodspringboot-1-deploy      0/1     Completed   0          11m
hotrodspringboot-1-ln5q2       1/1     Running     0          11m
hotrodspringboot-s2i-1-build   0/1     Completed   0          12m

Let’s look at the Spring Boot application logs:

Starting the Java application using /opt/jboss/container/java/run/run-java.sh ...
INFO exec java -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=50.0 -XX:MaxRAMPercentage=80.0 -cp "." -jar /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jolokia.util.ClassUtil (file:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar) to constructor sun.security.x509.X500Name(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String)
WARNING: Please consider reporting this to the maintainers of org.jolokia.util.ClassUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
I> No access restrictor found, access to any MBean is allowed
Jolokia: Agent started with URL https://10.129.0.64:8778/jolokia/
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.8.BUILD-SNAPSHOT) <--- Spring Boot 2.3.8

As you can see, this is starting the Java Virtual Machine (JVM) using run-java.sh entry point and should get the updated MaxRAMPercentage value and be a container-aware feature. The application is just a jar called hotrodspringboot-0.0.1-SNAPSHOT.jar located on the deployments directory.

  • In case run-java.sh is used: The percentages of heap/off-heap will be denoted by the Red Hat OpenJDK’s images settings, at 80%/20% (current) or 50%/50% (before). The user can set JAVA_OPTS and JAVA_OPTS_APPEND environment variables, given those are set on the initial scripts.
  • In case run-java.sh is not used: The percentages of heap/off-heap will be denoted by OpenJDK’s default settings at 25% heap/75% off-heap. The user cannot set JAVA_OPTS and JAVA_OPTS_APPEND environment variables, because Java is initiated without the scripts.

Now, let’s see its threads. We can see that the Spring Boot application will have the garbage collection (GC) threads, the Catalina engine threads, and a few more threads up to 200 threads on the default:

$ oc exec hotrodspringboot-POD -- jcmd 1 Thread.print > Threads
...
Full thread dump OpenJDK 64-Bit Server VM (11.0.8+10-LTS mixed mode, sharing):
Threads class SMR info:
_java_thread_list=0x00007f9038001ee0, length=30, elements={
0x00007f9088122800, 0x00007f9088124800, 0x00007f9088136000, 0x00007f9088138800,
0x00007f908813a800, 0x00007f908813c800, 0x00007f9088175800, 0x00007f908826e000,
0x00007f9020031000, 0x00007f9000001800, 0x00007f90201bb800, 0x00007f9088c02800,
0x00007f900c0b6000, 0x00007f9088c10800, 0x00007f9088d4c800, 0x00007f90888fa800,
0x00007f90888e7800, 0x00007f9088de9000, 0x00007f9088dea000, 0x00007f9088deb800,
0x00007f9088ded800, 0x00007f9088e0c800, 0x00007f9088e0e800, 0x00007f9088e5a000,
0x00007f9088e5c000, 0x00007f9088e5e000, 0x00007f9088acb000, 0x00007f9088aca000,
0x00007f9088016800, 0x00007f9038001000
}
"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=2.50ms elapsed=480.76s tid=0x00007f9088122800 nid=0x8a waiting on condition  [0x00007f906cb11000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList([email protected]/Native Method)
	at java.lang.ref.Reference.processPendingReferences([email protected]/Reference.java:241)
	at java.lang.ref.Reference$ReferenceHandler.run([email protected]/Reference.java:213)

Now let’s generate a VM.info, which will provide several details (including cgroups) of the JVM in this container:

$ oc exec hotrodspringboot-POD -- jcmd 1 VM.info > VM.info
...
# JRE version: OpenJDK Runtime Environment 18.9 (11.0.8+10) (build 11.0.8+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM 18.9 (11.0.8+10-LTS, mixed mode, sharing, tiered, compressed oops, parallel gc, linux-amd64)
---------------  S U M M A R Y ------------
Command Line: -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar
...

The process details will tell the exact size of the heap, which brings the value to 4GB as shown here:

---------------  P R O C E S S  ---------------
Heap address: 0x00000004ef000000, size: 3926 MB <--- ~4GB
Narrow klass base: 0x0000000800000000, Narrow klass shift: 3

This will have 4GB for heap as an arbitrary value, given that it’s hard-coded from the OpenJDK script. However, that’s not adequate for my deployment. In this case, I want the heap to be 1.6GB exactly. 

Let’s set the heap to 80% of the container size. To do that, let’s add some Java settings for container awareness, as follows:

$ oc get dc -o yaml | grep -a1  JAVA_OPTS_APPEND
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+ <- OCP 4.14
          - name: JAVA_OPTS_APPEND
            value: ' -XX:MinRAMPercentage=50.0 -XX:MaxRAMPercentage=80.0'

Note: I’m not using/setting CATALINA_OPTS /CATALINA_OPTS_APPEND, which is the Tomcat JVM option. Instead, I’m using Red Hat build of OpenJDK JAVA_OPTS_APPEND to add/control the percentage of heap/off-heap.

This should result in the following VM.info:

# JRE version: OpenJDK Runtime Environment 18.9 (11.0.8+10) (build 11.0.8+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM 18.9 (11.0.8+10-LTS, mixed mode, sharing, tiered, compressed oops, parallel gc, linux-amd64)
...
Host: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, 4 cores, 15G, Red Hat Enterprise Linux release 8.2 (Ootpa)
Time: Fri Jul 12 23:31:24 2024 UTC elapsed time: 133 seconds (0d 0h 2m 13s)
---------------  P R O C E S S  ---------------
Heap address: 0x00000004ef000000, size: 12.560 MB, Compressed Oops mode: Zero based, Oop shift amount: 3 <--- that's 12GB with 15GB host
Narrow klass base: 0x0000000800000000, Narrow klass shift: 3
Compressed class space size: 96468992 Address: 0x0000000840000000
...
VM Arguments:
jvm_args: -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=50.0 -XX:MaxRAMPercentage=80.0

Now the value of the heap goes to 12GB. That’s because the container does not have limits. So as I explained in my previous article, How to use Java container awareness in OpenShift 4, the limit used by Java will be the host:

    spec:
      containers:
        - resources: {} <--- container does not have limits

Given that the container does not have limits, it will use the host limit. So let’s add a limit on the container for 2GB (80% of that should be the 1.6GB heap that my application was intended to be):

$ oc get dc -o yaml | grep -a5  resources
...
          resources:
            limits:
              memory: 2Gi
            requests:
              memory: 2Gi

Then the max heap will be 80% of the 2GB, which is 1.6GB. But this does not work:

###  Except it won't work 
###
Command Line: -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=50.0 -XX:MaxRAMPercentage=80.0 /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar
Host: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, 4 cores, 15G, Red Hat Enterprise Linux release 8.2 (Ootpa)
Time: Sat Jul 13 00:36:28 2024 UTC elapsed time: 63 seconds (0d 0h 1m 3s)
---------------  P R O C E S S  ---------------
Heap address: 0x00000004ef000000, size: 12560 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000800000000, Narrow klass shift: 3
Compressed class space size: 96468992 Address: 0x0000000840000000
...
VM Arguments:
jvm_args: -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=50.0 -XX:MaxRAMPercentage=80.0

That’s because the container is coming from Red Hat OpenShift Container Platform 4.14, which has cgroups v2 (cgv2) as the default. So collecting the VM.info, we see the cgroups details (below, example cgroupsv1) are not present, therefore there are no cgroups limits:

container (cgroup) information:
container_type: cgroupv1 <--- cgv1
cpu_cpuset_cpus: 0-15
cpu_memory_nodes: 0
active_processor_count: 2
cpu_quota: 200000
cpu_period: 100000
cpu_shares: 819
memory_limit_in_bytes: 4194304000
memory_and_swap_limit_in_bytes: 4194304000
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 1012035584
memory_max_usage_in_bytes: 1016967168

Verify the cgroups version. We confirm it is cgv2:

$ stat -fc %T /sys/fs/cgroup/
cgroup2fs <--- Shows this is cgv2

Wondering why the container is not calculating 80% from 2GB and instead keeping 80% of 15GB, which is the host? The answer: cgroups v2. It is only detectable on JDK 11.0.16+ or JDK 1.8.372. But any JDK 17 (or later) and the image deployed here is no other than: registry.redhat.io/openjdk/openjdk-11-rhel8:latest:

sh-4.4$ java -version
openjdk version "11.0.8" 2020-07-14 LTS

To fix this, we need to edit the BuildConfig used above to use the latest OpenJDK image and make it pull an image that is cgv2-compatible.

  1. Pull a cgv2-compatible image, like  registry.access.redhat.com/ubi8/openjdk-11:1.20-1.
  2. Then change the BuildConfig for using it accordingly.

In case you face error building at STEP "RUN /usr/local/s2i/assemble": exit status 23, see this solution for alternatives. 

Given that I don’t want to face this problem, and I have the binary at hand; I will create a new BuildConfig following the several alternatives presented in the article 4 ways to deploy Quarkus applications in OpenShift Container Platform 4:

### First import the JDK 11 latest:
$ oc import-image ubi8/openjdk-11:1.20-1 --from=registry.access.redhat.com/ubi8/openjdk-11:1.20-1 --confirm
###
### Now build:
$ oc new-app springboot-test/openjdk-11:1.20-1~/tmp/nocontent --name=springboot-jdk11-latest
###
### Confirm the BC:
$ oc get bc
NAME                      TYPE     FROM     LATEST
springboot-jdk11-latest   Source   Binary   1 <---
### Get the IS created:
$ oc get is
NAME                      IMAGE REPOSITORY                                                                           TAGS     UPDATED
springboot-jdk11-latest   image-registry.openshift-image-registry.svc:5000/springboot-test/springboot-jdk11-latest   latest   2 minutes ago

The previous build started will create a deployment with the image automatically. This new image will have the JDK 11 latest (OpenJDK 11.0.23):

$ oc get deployment
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
springboot-jdk11-latest   1/1     1            1           10m
...
...
$ oc get deployment -o yaml | grep image
        - image: image-registry.openshift-image-registry.svc:5000/springboot-test/springboot-jdk11-latest@sha256:..
### Confirm the Red Hat OpenJDK version
$ oc rsh springboot-jdk11-latest-78bfcbcd8b-2wxrl
sh-4.4$ java -version
openjdk version "11.0.23" 2024-04-16 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.23.0.9-2) (build 11.0.23+9-LTS)

Then we have it:

# JRE version: OpenJDK Runtime Environment (Red_Hat-11.0.23.0.9-2) 
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-11.0.23.0.9-2)
---------------  S U M M A R Y ------------
Command Line: -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar
Host: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz, 4 cores, 2G, Red Hat Enterprise Linux release 8.10 (Ootpa)
Time: Sat Jul 13 02:00:26 2024 UTC elapsed time: 44.937451 seconds (0d 0h 0m 44s)
---------------  P R O C E S S  ---------------
Heap address: 0x0000000099800000, size: 1640 MB <---- 1.6GB

Collecting the VM.info, we correctly see the cgroups v2 detected:

container (cgroup) information:
container_type: cgroupv2
cpu_cpuset_cpus: not supported
cpu_memory_nodes: not supported
active_processor_count: 4
cpu_quota: no quota
cpu_period: 100000
cpu_shares: 28
memory_limit_in_bytes: 2097152 k
memory_and_swap_limit_in_bytes: 2097152 k
memory_soft_limit_in_bytes: unlimited
memory_usage_in_bytes: 184028 k
memory_max_usage_in_bytes: not supported
memory_swap_current_in_bytes: unlimited
memory_swap_max_limit_in_bytes: unlimited
maximum number of tasks: 98809
current number of tasks: 56

That is exactly the expected value with 80% of 2GB, 1.6GB. Q.E.D.

Is the container bounded?

One question that might be relevant is how to determine whether a container is bounded or not by container (cgroup) limits and how much are those limits from inside a container? 

The answer is, if the container is limited (bounded), then cat /sys/fs/cgroup/memory.max will return the size of the container:

## bounded
$ cat /sys/fs/cgroup/memory.max 
2147483648 <-- that's in bytes, converted that's 2.147483648 GB

If the container does not have limits (i.e., it is unbounded) then cat /sys/fs/cgroup/memory.max will return max, as shown here: 

## the cgropus is unbounded, so then that file will see:
$ cat /sys/fs/cgroup/memory.max 
max

Problematic scenarios from the use case

Using the previous example, which shows how straightforward Spring Boot is, let’s explore a few scenarios.

Decoupling the heap vs. the container size

Issue: Although that’s not an immediate problem, unless manifested as an OOMKill or OOME, the decoupling between the heap and container sizes can be a problem, as explained in the article How to use Java container awareness in OpenShift 4. This can be problematic and eventually cause the OOMKill in the following block. 

Worse than decoupling from the container size at runtime would be embedding the image with the Java heap size hard-coded at build time such that it cannot be changed at runtime. For example, in the previous use case, if the built application jar is built into the image with Dockerfile, I embed the image with Xmx/Xms hard-coded as follows:

-javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError Xmx2Gb /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar

Hint: Verify if the image is container-aware and has the heap values set at runtime, not build time. Verifying the container logs will show the max heap (Xmx) and if that’s not hard-coded. If the value is hard coded and decoupled from the container size, so changing the respective container (size) won’t impact on the heap.

Fix: Avoid hard-coding the heap size in the image. This considerably restricts the idea of a container and might cause problems if the container/heap is not adequate. This can be generalized for anything that can be set at runtime and/or detected on the container, such as threads (calculated from the cores).

Cgroups OutOfMemory Kill (OOM KILL)

Let’s say that after deploying the previous Spring Boot, the user forgot to use a cgv2-compatible image, so the cgroups won’t be detected. 

Issue: Given that Java won’t detect cgroups boundaries, it will use the host boundaries. However, the JVM is still deployed inside a container so eventually when the memory starts to balloon, it will cross the container boundaries—not incurring in OOME—but will incur in OutOfMemory Kill by the OCP Node kernel. 

Hints: 

  • The heap won’t be bounded by the container size.
  •  The VM.info will not present cgroups details at all. The section will just be absent. This is the hint: cgroups are not being used as a limit. Even using percentage flags, the percentages will match the host, not the container.
  • Sos report of the OpenShift Container Platform Node will present several OOMKills by the cgroups.

Fix: Use a Java base image in the application image that is compatible container-aware and compatible with cgroup v2 in the case of Red Hat OpenShift Container Platform, 4.14.

Immediately kill/exit

Issue: There is another scenario in which when the user realizes the MaxRAM is not being used and will try to impose a Xmx value manually, which is an advanced setting without extensive benchmarking. However, the flag ExitOnOutOfMemory was used, so the JVM won’t create a heap or a crash file. Instead, it will exit on the spot.

Hints: 

  • Verify if the application using all the memory it has available.  
  • No OOMKills will be present on the OpenShift Container Platform node on which it was deployed.
  • In the case of removing ExitOnOutOfMemory, an OutOfMemory exception will be triggered.

Fix: Usage of container-awareness is prefered rather than setting Xmx manually to avoid the application exploding this value and incurring an automatic container exit. Use heap percentage settings and increase the container size in case needed accordingly. 

OutOfMemoryException (OOME)

Issue: In a scenario similar to the above, the user sets Xmx and removes ExitOnOutOfMemory JVM flag, but the application will still use more memory than was allocated for its heap, so it will incur an OOME Exception. The container total memory usage is not crossed, so no OOMKill happens.

Hints: 

  • Verify if the application uses all the memory it has available.  
  • No OOMKills will be present on the OpenShift Container Platform node on which it was deployed.
  • Given no ExitOnOutOfMemory, the OutOfMemory Exception will be triggered and the heap can be saved with HeapDumpOnOutOfMemoryError and setting the path for the heap as HeapDumpPath.

Fix: Again, avoid setting Xmx manually to avoid the application exploding this value and incurring an automatic exit. Use percentage settings and increase the container size in case needed. Collect and verify the heap dump in case needed.

Kubelet’s SIGTERM

Issue: In this scenario, the application is deployed not using container limits, and now it has container limits. So the CPU used to calculate the threads was reduced, and the application was not benchmarked after this critical change. Now the application is facing SIGTERMs from Kubelet’s probes. In this scenario, the probes are triggering the pod kills.

Hints: 

  • Verify if the application using all the memory it has available.  
  • No OOMKills will be present on the OpenShift Container Platform node it was deployed with no specific exit code and no heap generated.
  • The probes are failing on the pod logs.

Fix: Deploy the application with the adequate set of CPUs and re-benchmark the application behavior, and, if needed, tune the probes and/or distribute the load, in case it is possible. 

Latencies

Issue: Spring Boot will have by default 200 threads as max values for the pool (see above). This can impact multiple simultaneous (or quasi-simultaneous) access. So in the scenario of the application being properly deployed and benchmarked, however, it still experiences latencies. 

Hints: 

  • Verify how much memory the application is using.
  • Verify if crashes or OOM Exceptions were triggered.
  • The probes are failing on the pod logs.
  • Increasing the CPU won’t directly mean more Tomcat threads to be used by Spring Boot. The pool size can be changed directly via MaxThreads settings. 

Fix: Generically speaking, given adequate memory and CPU, collecting the thread dumps and verifying what the threads are doing/behaving should suffice for most use cases. In case container awareness was not used, then increasing the container memory and CPU won’t help. 

Communication issues

Issue: Let’s say the application cannot be accessed from within the OpenShift Container Platform cluster or from outside via browser, for example.

Hints: 

  • Verify the access using other methods such as cURL access.
  • If the browser is the only means of accessing: verify the certificate and validations. 
  • If the the container is inaccessible: verify the access from inside the OpenShift Container Platform cluster. Track networking details in case it is necessary.

Fix: Generically speaking, given adequate memory and CPU, collecting the thread dumps and verifying what the threads are doing/behaving should suffice for most use cases. In case container awareness was not used, then increasing the container memory and CPU won’t help. 

Garbage collection issues

Issue: Given JVM usage, the memory will be automatically collected by the JVM itself. This is an asset but must be combined with adequate settings. Not specifically discussing tuning, but usage of the proper settings in general.

To exemplify above, let’s say the application receives large quantities of random requests (i.e., at no pre-defined period) for the Data Grid application, and for some unknown reason, the garbage collector was set as JAVA_OPTS_APPEND=-XX:UseShenandoah. This means using a non-generational collection with an application with random and generational tendencies, which might later implicate extensive cleaning not staggered in generational phases, eventually leading to Degenerated States and then Full GCs, as explained on A beginner’s guide to the Shenandoah garbage collector

Hints: 

  • Verify the Garbage collection logs and its patterns and behaviors. For Shenandoah lots of Degraded/Full GCs sub-sequentially hint the application is likely more generational biased.
  • Container resources: CPU usage and memory usage.
  • Existence or absence of OOMKills (from the Openshift Node side).

Fix: Usage of an adequate garbage collector, such as latency collector (G1GC), throughput collector (Parallel GC), or Shenandoah/GC can avoid this scenario. For instance, usage of a non-generational collector (either Shenandoah or ZGC) in a generational workload should suffice. Also, benchmarking/comparing the application behavior with two or more collectors can also be an asset. 

Troubleshooting steps

For most use cases the following should suffice: 

  • Inspect: oc adm inspect ns/namespace
  • GC logs: JDK 11+ example usage JAVA_OPTS_APPEND=-Xlog:gc*. I wrote an extensive overview of JDK 11 Xlog on this solution, where Xlog consolidates all the GC logging. Sometimes it could be useful to keep it as standard output (so goes to pod log), and others might be useful to dump it into a persisted file. 
  • VM.info: jcmd PID VM.info.
  • Thread dumps: jcmd PID Thread.print usually collects several in a row for investigations.
  • SOS report: sos report, can be used for detecting OOMKills or VMWare utilization/overcommitment. 

Recap and additional resources

This Spring Boot use case is straightforward and complements the article Java container awareness in several aspects, but also presents scenarios that can happen on Java/ Spring Boot deployments. It also builds it is a real use case with BuildConfigs presented in this article: 4 ways to deploy Quarkus applications in OpenShift Container Platform 4. Furthermore, it presents several problematic scenarios and alternatives to address them. 

As a disclaimer, the image tags might be outdated depending on the publication date of this article. The purpose of the article is not to discuss tag/specific images but to present scenarios built on Java images that don’t comply with cgroupsv2 and other issues that can happen.

In this article, the file VM.info is discussed in some detail and presents a great tool to detect JVM settings and container details, but also has many other utilizations. 

To learn more, read Java 17: What’s new in OpenJDK’s container awareness. For any other specific inquiries, please open a case with Red Hat support. Our global team of experts can help you with any issues.

Special thanks to Alexander Barbosa, Ryan Howe, and Will Russell for their input throughout those six years working with OpenShift.



Source link

Related Posts

About The Author

Admin
More From This Author

Add Comment