Remote JMX connection to Java Docker Container

Remote JMX Connection

Sometimes we need to hook up to remote Java instances for some JVM debugging work or monitoring.

Recently I come across this issue when connecting my local VisualVM to the remote instance (Java app in a docker container) hoping to do some JVM heapspace troubleshooting. The connection was never successful! *scratch head*

Here’re some things I did to narrow down the issue:

Telnet is working, meaning the JMX port is opened, also ruled out firewall issue.
Invoking JMX connection from the remote machine itself, works!

Ok this is weird… I know :S

Setting RMI hostname

Digging through the web, I saw people suggesting to put a public IP or valid domain at the RMI hostname parameter, something like this:

java -server \
  ...
  -Dcom.sun.management.jmxremote.port=9002 \
  -Dcom.sun.management.jmxremote.rmi.port=9002 \
  -Dcom.sun.management.jmxremote=true \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.local.only=false \
  -Djava.rmi.server.hostname=101.201.33.48\
  ... 
  -jar app.jar

Ok it works now! But I don’t quite like this. I’m just being lazy because otherwise I need to figure out how to put a public IP in there with all the docker building processes.

On top of that, the Munin JMX process is now not able to query from the instance anymore, which I expected to break. Note: This Munin JMX process is running on the remote server on the same host as the Java instance I am connecting to remotely from my local.

Here’s the error I got slapped with:

[email protected]:/etc/munin/plugins# munin-run jmx_chat_alpha_threads
Unknown host: chat; nested exception is: 
        java.net.UnknownHostException: chatquerying service:jmx:rmi:///jndi/rmi://localhost:9002/jmxrmi
java.rmi.UnknownHostException: Unknown host: chat; nested exception is: 
        java.net.UnknownHostException: chat
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:620)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132)
        at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
        at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
        at com.sun.proxy.$Proxy0.newClient(Unknown Source)
        at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
        at org.munin.JMXQuery.connect(JMXQuery.java:73)
        at org.munin.JMXQuery.main(JMXQuery.java:206)
Caused by: java.net.UnknownHostException: chat
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at java.net.Socket.connect(Socket.java:556)
        at java.net.Socket.<init>(Socket.java:452)
        at java.net.Socket.<init>(Socket.java:229)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617)
        ... 11 more

Yeah, helpful… localhost != chat lol.

Trick comes true!

Then I found a trick… where I specify the RMI hostname as chat (because I was working on a chatserver).

-Djava.rmi.server.hostname=chat \

Rebuild and spin the container up. Next I added a new entry in the file /etc/host in my local, so that the hostname resolved to the remote server’s IP.

# /etc/hosts
...
101.201.33.48    chat

Now if I ping the domain ‘chat’, it will be pinging the remote server at IP 101.201.33.48. Neat 🙂 I gonna connect to the remote instance with JConsole. Good I am in!

VisualVM works too!

But how about the broken Munin JMX query?

The solution is similar to the one I did in my local, which is to setup an alias (chat) to localhost.

# /etc/hosts
...
127.0.0.1 chat

Then update the Munin plugins config file.

# /etc/munin/plugin-conf.d/zzz-munin-node
...
[jmx_chat_alpha]
  env.jmxurl service:jmx:rmi:///jndi/rmi://chat:9002/jmxrmi

Now the JMX plugin should be working again!

[email protected]:/etc/munin/plugins# munin-run jmx_chat_alpha_threads
java_thread_count.value 155
java_thread_count_peak.value 253

Extra

What’s the /etc/hosts file for?

Its primary present-day use is to bypass DNS resolution. A match found in the /etc/hosts file will be used before any DNS entry. In fact, if the name searched (like localhost) is found in the file, no DNS resolution will be performed at all.
https://unix.stackexchange.com/questions/421491/what-is-the-purpose-of-etc-hosts

solidx.dev