JVM monitoring is crucial to ensure that the Java applications are running in good health especially in the aspect of the heap usage. JMX is one of the way to query the stats of the Java application. In this post, I will be explaining the setup in order to monitor these dockerized Java applications.
- Munin master installed
- Munin node installed
- Dockerized Java applications
So, we have 3 docker containers, each running a Java application. Each docker container is exposing a different port for JMX client connectivity where in this case the client is the Munin-node process running in the same VM instance.
Munin master collects data regularly from the Munin node. We will be seeing important JVM stats e.g. heap, CPU and threads, etc. You can customize the plugin to query some other attributes from the MBeans exposed.
In order to query some stats from the Java application, we first need to enable JMX, so that JMX client can connect to the app and get some JVM stats.
We can do this in the entrypoint.sh file (this is where we start our Java application), where we will configure the listing port for JMX connection. You can see that the JMX_PORT is a variable, that this allows us to create multiple instances with different JMX listening ports.
#!/bin/bash exec java -server \ -Xms$APP_HEAP_MIN -Xmx$APP_HEAP_MAX \ -Dcom.sun.management.jmxremote.port=$JMX_PORT \ -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT \ -Dcom.sun.management.jmxremote=true \ -Dcom.sun.management.jmxremote.ssl=false \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.local.only=false \ -Djava.rmi.server.hostname=0.0.0.0 \ -jar app.jar
Expose the JMX port, so that the Munin node can connect to the JMX listening port.
... # Expose ports EXPOSE $JMX_PORT ...
After spinning up the docker containers, check if the JMX port is opened on each of them. You should see connection refused error if the port is not configured correctly.
[email protected]:~# telnet localhost 9000 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. ^] telnet> quit Connection closed.
Munin JMX plugin
This plugin is 2 years old but it’s working fine with the JDK 8 we’re running on. The installation guide is pretty much straight-forward, like installing JMX query jar into Munin, etc. Once that’s done, create a new config file named zzz-munin-node.
Note: this file is commonly used to override or declare custom plugins’ parameters.
[email protected]:/etc/munin/plugin-conf.d# cat zzz-munin-node [jmx_alpha_*] env.jmxurl service:jmx:rmi:///jndi/rmi://localhost:9002/jmxrmi [jmx_beta_*] env.jmxurl service:jmx:rmi:///jndi/rmi://localhost:9001/jmxrmi [jmx_stable_*] env.jmxurl service:jmx:rmi:///jndi/rmi://localhost:9000/jmxrmi
Create the config files. There should be total nine of them, for three instances. The list below shows all alpha instance’s Munin JMX configs. You can create a custom config if you need to monitor some custom application-specific attributes.
[email protected]:/etc/munin/plugins# ls -al /usr/share/munin/plugins/alpha* -rw-r--r-- 1 root root 727 Apr 24 2019 /usr/share/munin/plugins/alpha_cpu.conf -rw-r--r-- 1 root root 1953 Apr 24 2019 /usr/share/munin/plugins/alpha_process_memory.conf -rw-r--r-- 1 root root 435 Apr 24 2019 /usr/share/munin/plugins/alpha_threads.conf
Symlinks to the jmx_ file with appropriate names should be named after the respective config filename created earlier.
[email protected]:/etc/munin/plugins# ls -l jmx* lrwxrwxrwx 1 root root 29 Apr 24 2019 jmx_alpha_cpu -> /usr/share/munin/plugins/jmx_ lrwxrwxrwx 1 root root 29 Apr 24 2019 jmx_alpha_process_memory -> /usr/share/munin/plugins/jmx_ lrwxrwxrwx 1 root root 29 Apr 24 2019 jmx_alpha_threads -> /usr/share/munin/plugins/jmx_ lrwxrwxrwx 1 root root 29 Apr 24 2019 jmx_beta_cpu -> /usr/share/munin/plugins/jmx_ lrwxrwxrwx 1 root root 29 Apr 24 2019 jmx_beta_process_memory -> /usr/share/munin/plugins/jmx_ lrwxrwxrwx 1 root root 29 Apr 24 2019 jmx_beta_threads -> /usr/share/munin/plugins/jmx_ lrwxrwxrwx 1 root root 29 Apr 24 2019 jmx_stable_cpu -> /usr/share/munin/plugins/jmx_ lrwxrwxrwx 1 root root 29 Apr 24 2019 jmx_stable_process_memory -> /usr/share/munin/plugins/jmx_ lrwxrwxrwx 1 root root 29 Apr 24 2019 jmx_stable_threads -> /usr/share/munin/plugins/jmx_
See if the configuration works
You should see similar results if things are configured correctly for Munin JMX plugin. YAY!
[email protected]:/etc/munin/plugins# munin-run jmx_alpha_process_memory java_memory_nonheap_committed.value 327127040 java_memory_nonheap_max.value -1 java_memory_nonheap_used.value 313941864 java_memory_heap_committed.value 2027110400 java_memory_heap_max.value 3758096384 java_memory_heap_used.value 1028654920 os_memory_physical.value 2530406400 os_memory_vm.value 6409654272
Complete the connection
Assume that the config installation is done at the Munin node host. Now we need to expose the Munin node to the Munin master.
Doing this, nothing is easier than allowing the connection from the master (say the IP is 126.96.36.199) to the node.
# /etc/munin/munin-node.conf # A list of addresses that are allowed to connect. This must be a # regular expression, since Net::Server does not understand CIDR-style # network notation unless the perl module Net::CIDR is installed. You # may repeat the allow line as many times as you'd like allow ^35\.242\.107\.61$
NOTE: you may also need to configure the local firewall to allow such inbound connection to the Munin node.
Now restart the Munin node process
systemctl restart munin-node
Next up, we gonna let Munin master know who to connect to.
# /etc/munin/munin.conf [java_app] address 188.8.131.52 use_node_name yes
Now restart the Munin master process too. Done! Now you can browse to the Munin address in your browser and see that the JVM stats are collected from these 3 Java instances.