Remote JMX Connection
Sometimes we need to hook up to remote Java instances for some JVM debugging work or monitoring.
Recently I come across this issue when connecting my local VisualVM to the remote instance (Java app in a docker container) hoping to do some JVM heapspace troubleshooting. The connection was never successful! *scratch head*
Here’re some things I did to narrow down the issue:
- Telnet is working, meaning the JMX port is opened, also ruled out firewall issue.
- Invoking JMX connection from the remote machine itself, works!
Ok this is weird… I know :S
Setting RMI hostname
Digging through the web, I saw people suggesting to put a public IP or valid domain at the RMI hostname parameter, something like this:
java -server \
...
-Dcom.sun.management.jmxremote.port=9002 \
-Dcom.sun.management.jmxremote.rmi.port=9002 \
-Dcom.sun.management.jmxremote=true \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.local.only=false \
-Djava.rmi.server.hostname=101.201.33.48\
...
-jar app.jar
Ok it works now! But I don’t quite like this. I’m just being lazy because otherwise I need to figure out how to put a public IP in there with all the docker building processes.
On top of that, the Munin JMX process is now not able to query from the instance anymore, which I expected to break. Note: This Munin JMX process is running on the remote server on the same host as the Java instance I am connecting to remotely from my local.
Here’s the error I got slapped with:
[email protected]:/etc/munin/plugins# munin-run jmx_chat_alpha_threads
Unknown host: chat; nested exception is:
java.net.UnknownHostException: chatquerying service:jmx:rmi:///jndi/rmi://localhost:9002/jmxrmi
java.rmi.UnknownHostException: Unknown host: chat; nested exception is:
java.net.UnknownHostException: chat
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:620)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
at com.sun.proxy.$Proxy0.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at org.munin.JMXQuery.connect(JMXQuery.java:73)
at org.munin.JMXQuery.main(JMXQuery.java:206)
Caused by: java.net.UnknownHostException: chat
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at java.net.Socket.connect(Socket.java:556)
at java.net.Socket.<init>(Socket.java:452)
at java.net.Socket.<init>(Socket.java:229)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617)
... 11 more
Yeah, helpful… localhost != chat lol.
Trick comes true!
Then I found a trick… where I specify the RMI hostname as chat (because I was working on a chatserver).
-Djava.rmi.server.hostname=chat \
Rebuild and spin the container up. Next I added a new entry in the file /etc/host in my local, so that the hostname resolved to the remote server’s IP.
# /etc/hosts
...
101.201.33.48 chat
Now if I ping the domain ‘chat’, it will be pinging the remote server at IP 101.201.33.48. Neat 🙂 I gonna connect to the remote instance with JConsole. Good I am in!

VisualVM works too!

But how about the broken Munin JMX query?
The solution is similar to the one I did in my local, which is to setup an alias (chat) to localhost.
# /etc/hosts
...
127.0.0.1 chat
Then update the Munin plugins config file.
# /etc/munin/plugin-conf.d/zzz-munin-node
...
[jmx_chat_alpha]
env.jmxurl service:jmx:rmi:///jndi/rmi://chat:9002/jmxrmi
Now the JMX plugin should be working again!
[email protected]:/etc/munin/plugins# munin-run jmx_chat_alpha_threads
java_thread_count.value 155
java_thread_count_peak.value 253
Extra
What’s the /etc/hosts file for?
Its primary present-day use is to bypass DNS resolution. A match found in the
https://unix.stackexchange.com/questions/421491/what-is-the-purpose-of-etc-hosts/etc/hosts
file will be used before any DNS entry. In fact, if the name searched (likelocalhost
) is found in the file, no DNS resolution will be performed at all.