Munin JVM Monitoring on Multiple Docker Containers

Munin JVM Monitoring on Multiple Docker Containers

JVM monitoring is crucial to ensure that the Java applications are running in good health especially in the aspect of the heap usage. JMX is one of the way to query the stats of the Java application. In this post, I will be explaining the setup in order to monitor these dockerized Java applications.

Pre-requisites

  • Munin master installed
  • Munin node installed
  • Dockerized Java applications

Brief

So, we have 3 docker containers, each running a Java application. Each docker container is exposing a different port for JMX client connectivity where in this case the client is the Munin-node process running in the same VM instance.

Connection Overview

Munin master collects data regularly from the Munin node. We will be seeing important JVM stats e.g. heap, CPU and threads, etc. You can customize the plugin to query some other attributes from the MBeans exposed.

Enable JMX

In order to query some stats from the Java application, we first need to enable JMX, so that JMX client can connect to the app and get some JVM stats.

We can do this in the entrypoint.sh file (this is where we start our Java application), where we will configure the listing port for JMX connection. You can see that the JMX_PORT is a variable, that this allows us to create multiple instances with different JMX listening ports.

#!/bin/bash

exec java -server \
  -Xms$APP_HEAP_MIN -Xmx$APP_HEAP_MAX \
  -Dcom.sun.management.jmxremote.port=$JMX_PORT \
  -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT \
  -Dcom.sun.management.jmxremote=true \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.local.only=false \
  -Djava.rmi.server.hostname=0.0.0.0 \
  -jar app.jar

Expose the JMX port, so that the Munin node can connect to the JMX listening port.

...
# Expose ports
EXPOSE $JMX_PORT
...

After spinning up the docker containers, check if the JMX port is opened on each of them. You should see connection refused error if the port is not configured correctly.

[email protected]:~# telnet localhost 9000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

Munin JMX plugin

This plugin is 2 years old but it’s working fine with the JDK 8 we’re running on. The installation guide is pretty much straight-forward, like installing JMX query jar into Munin, etc. Once that’s done, create a new config file named zzz-munin-node.

Note: this file is commonly used to override or declare custom plugins’ parameters.

[email protected]:/etc/munin/plugin-conf.d# cat zzz-munin-node 
[jmx_alpha_*]
  env.jmxurl service:jmx:rmi:///jndi/rmi://localhost:9002/jmxrmi

[jmx_beta_*]
  env.jmxurl service:jmx:rmi:///jndi/rmi://localhost:9001/jmxrmi

[jmx_stable_*]
  env.jmxurl service:jmx:rmi:///jndi/rmi://localhost:9000/jmxrmi

Create the config files. There should be total nine of them, for three instances. The list below shows all alpha instance’s Munin JMX configs. You can create a custom config if you need to monitor some custom application-specific attributes.

[email protected]:/etc/munin/plugins# ls -al /usr/share/munin/plugins/alpha*
-rw-r--r-- 1 root root  727 Apr 24  2019 /usr/share/munin/plugins/alpha_cpu.conf
-rw-r--r-- 1 root root 1953 Apr 24  2019 /usr/share/munin/plugins/alpha_process_memory.conf
-rw-r--r-- 1 root root  435 Apr 24  2019 /usr/share/munin/plugins/alpha_threads.conf

Symlinks to the jmx_ file with appropriate names should be named after the respective config filename created earlier.

[email protected]:/etc/munin/plugins# ls -l jmx*
lrwxrwxrwx 1 root root 29 Apr 24  2019 jmx_alpha_cpu -> /usr/share/munin/plugins/jmx_
lrwxrwxrwx 1 root root 29 Apr 24  2019 jmx_alpha_process_memory -> /usr/share/munin/plugins/jmx_
lrwxrwxrwx 1 root root 29 Apr 24  2019 jmx_alpha_threads -> /usr/share/munin/plugins/jmx_
lrwxrwxrwx 1 root root 29 Apr 24  2019 jmx_beta_cpu -> /usr/share/munin/plugins/jmx_
lrwxrwxrwx 1 root root 29 Apr 24  2019 jmx_beta_process_memory -> /usr/share/munin/plugins/jmx_
lrwxrwxrwx 1 root root 29 Apr 24  2019 jmx_beta_threads -> /usr/share/munin/plugins/jmx_
lrwxrwxrwx 1 root root 29 Apr 24  2019 jmx_stable_cpu -> /usr/share/munin/plugins/jmx_
lrwxrwxrwx 1 root root 29 Apr 24  2019 jmx_stable_process_memory -> /usr/share/munin/plugins/jmx_
lrwxrwxrwx 1 root root 29 Apr 24  2019 jmx_stable_threads -> /usr/share/munin/plugins/jmx_

See if the configuration works

You should see similar results if things are configured correctly for Munin JMX plugin. YAY!

[email protected]:/etc/munin/plugins# munin-run jmx_alpha_process_memory
java_memory_nonheap_committed.value 327127040
java_memory_nonheap_max.value -1
java_memory_nonheap_used.value 313941864
java_memory_heap_committed.value 2027110400
java_memory_heap_max.value 3758096384
java_memory_heap_used.value 1028654920
os_memory_physical.value 2530406400
os_memory_vm.value 6409654272

Complete the connection

Assume that the config installation is done at the Munin node host. Now we need to expose the Munin node to the Munin master.

Doing this, nothing is easier than allowing the connection from the master (say the IP is 35.242.107.61) to the node.

# /etc/munin/munin-node.conf

# A list of addresses that are allowed to connect.  This must be a
# regular expression, since Net::Server does not understand CIDR-style
# network notation unless the perl module Net::CIDR is installed.  You
# may repeat the allow line as many times as you'd like

allow ^35\.242\.107\.61$

NOTE: you may also need to configure the local firewall to allow such inbound connection to the Munin node.

Now restart the Munin node process

systemctl restart munin-node

Next up, we gonna let Munin master know who to connect to.

# /etc/munin/munin.conf

[java_app]
    address 101.203.100.81
    use_node_name yes

Now restart the Munin master process too. Done! Now you can browse to the Munin address in your browser and see that the JVM stats are collected from these 3 Java instances.

Leave a Reply

Your email address will not be published. Required fields are marked *