Remote JMX Connection
Sometimes we need to hook up to remote Java instances for some JVM debugging work or monitoring.
Recently I come across this issue when connecting my local VisualVM to the remote instance (Java app in a docker container) hoping to do some JVM heapspace troubleshooting. The connection was never successful! *scratch head*
Here’re some things I did to narrow down the issue:
- Telnet is working, meaning the JMX port is opened, also ruled out firewall issue.
- Invoking JMX connection from the remote machine itself, works!
Ok this is weird… I know :S
Setting RMI hostname
Digging through the web, I saw people suggesting to put a public IP or valid domain at the RMI hostname parameter, something like this:
java -server \ ... -Dcom.sun.management.jmxremote.port=9002 \ -Dcom.sun.management.jmxremote.rmi.port=9002 \ -Dcom.sun.management.jmxremote=true \ -Dcom.sun.management.jmxremote.ssl=false \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.local.only=false \ -Djava.rmi.server.hostname=126.96.36.199\ ... -jar app.jar
Ok it works now! But I don’t quite like this. I’m just being lazy because otherwise I need to figure out how to put a public IP in there with all the docker building processes.
On top of that, the Munin JMX process is now not able to query from the instance anymore, which I expected to break. Note: This Munin JMX process is running on the remote server on the same host as the Java instance I am connecting to remotely from my local.
Here’s the error I got slapped with:
[email protected]:/etc/munin/plugins# munin-run jmx_chat_alpha_threads Unknown host: chat; nested exception is: java.net.UnknownHostException: chatquerying service:jmx:rmi:///jndi/rmi://localhost:9002/jmxrmi java.rmi.UnknownHostException: Unknown host: chat; nested exception is: java.net.UnknownHostException: chat at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:620) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180) at com.sun.proxy.$Proxy0.newClient(Unknown Source) at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at org.munin.JMXQuery.connect(JMXQuery.java:73) at org.munin.JMXQuery.main(JMXQuery.java:206) Caused by: java.net.UnknownHostException: chat at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at java.net.Socket.connect(Socket.java:556) at java.net.Socket.<init>(Socket.java:452) at java.net.Socket.<init>(Socket.java:229) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617) ... 11 more
Yeah, helpful… localhost != chat lol.
Trick comes true!
Then I found a trick… where I specify the RMI hostname as chat (because I was working on a chatserver).
Rebuild and spin the container up. Next I added a new entry in the file /etc/host in my local, so that the hostname resolved to the remote server’s IP.
# /etc/hosts ... 188.8.131.52 chat
Now if I ping the domain ‘chat’, it will be pinging the remote server at IP 184.108.40.206. Neat 🙂 I gonna connect to the remote instance with JConsole. Good I am in!
VisualVM works too!
But how about the broken Munin JMX query?
The solution is similar to the one I did in my local, which is to setup an alias (chat) to localhost.
# /etc/hosts ... 127.0.0.1 chat
Then update the Munin plugins config file.
# /etc/munin/plugin-conf.d/zzz-munin-node ... [jmx_chat_alpha] env.jmxurl service:jmx:rmi:///jndi/rmi://chat:9002/jmxrmi
Now the JMX plugin should be working again!
[email protected]:/etc/munin/plugins# munin-run jmx_chat_alpha_threads java_thread_count.value 155 java_thread_count_peak.value 253
What’s the /etc/hosts file for?
Its primary present-day use is to bypass DNS resolution. A match found in thehttps://unix.stackexchange.com/questions/421491/what-is-the-purpose-of-etc-hosts
/etc/hostsfile will be used before any DNS entry. In fact, if the name searched (like
localhost) is found in the file, no DNS resolution will be performed at all.