File-based persistent store state mismatch

Sometimes things change within a corporate network. And sometimes these changes also impact a GlassFish domain. How much does relocating a server impact the GlassFish Domain? That depends; When the glassfish installation was performed using proper DNS / hostnames then there should not be very much impact. When the GlassFish configuration itself (like JDBC resources or resource adapters) are also spared of IP-address usages then the impact should be diminished even further.

Unfortunately GlassFish uses some internal components which store the more physical information of the network location of the various GlassFish instances in a domain. This becomes apparent when the GlassFish domain is relocated to another IP-address range or VLAN.

Starting the domain after changing its underlying networkstructure could result in not being able to start domain or server instances within the domain. The logfiles might indicate that a connection to the IMQBroker has not been possible within the last 6000 milliseconds.

java.lang.RuntimeException: MQJMSRA_LB4001: start:Aborted:
Unable to ping Broker within 60000 millis (startTimeOut)

An exception like this will also be shown in the logfiles of the failing instance:

[C4003]: Error occurred on connection creation [localhost:37676].
- cause: java.net.ConnectException: Connection refused

The most common cause for this is network problems. Checking the correct DNS lookup and reverse lookup of the DAS and the various nodes containing in stances in the domain will provide more information whether there is a network problem.
If DNS lookup succeeds and the hosts-file of the operating system does not contain any incorrect references then other things might go wrong.

The problem could also lie within a corrupted persistent store of the IMQbroker(s). Looking at the imq log.txt files which are located in the server instance folders (and more specifically in the imq/instances/[full unique instancename]/log folder within the nodeagents folder) one might see message like:

ERROR [B3095]: Cannot join the cluster because of persistent store state mismatch. Please clear the existing persistent store with “-reset store” and try again. Broker exiting.

Clearing the persistent store can actually be done by executing the imqbroker daemon using the -reset store parameters. Issue the following command from within the [GLASSFISH-HOME]/imq/bin directory:

./imqbrokerd -varhome [GLASSFISH_HOME]/nodeagents/[NODEAGENT_NAME]/[AFFECTED_INSTANCENAME]/imq -name [CLUSTERNAME][INSTANCENAME] -reset store

Replace the variables between brackets as appropriate! Mind the -name argument which contains a concatenated cluster and instancename!

For example:

/opt/glassfish-v2.1/imq/bin/imqbrokerd -varhome /opt/glassfish-v2.1/nodeagents/domain1-nodeagent1/serverinstance1/imq -name myclusterserverinstance1 -reset store

After this has been started, stop it and start the server instance which would previously not start. The persistent store for this instance has been cleared so a rendez-vous with the DAS should now work once again.