• Task
  • Status: Closed
  • 2 Major
  • Resolution: Fixed
  • Documentation,DSO:L2
  • etsai
  • Reporter: gkeim
  • January 20, 2009
  • 0
  • Watchers: 0
  • April 10, 2009
  • March 12, 2009

Description

When a quarantined client re-connect attempt is rejected due to a server bounce sans permanent-store persistence-mode, provide better messaging as to the cause and inform of remedies: (1) manually terminate clients and keep going or, (2) change persistence-mode to permanent-store and restart cluster.

The message of concern: ConnectionID(0.8759f44cca6948718f7a66ba18bae64b): Connection refused

An example of the confusion: http://forums.terracotta.org/forums/posts/list/1709.page

Also, the online document referred to by the forum thread doesn’t make it absolutely clear that permanent-store mode is required for HA: http://www.terracotta.org/web/display/docs/Configuring+Terracotta+For+High+Availability

Background for Igal: The server stores the client connections in its database for fail over reasons. When the server instances are not configured for a persistent database, at server restart those connections are lost since the database is cleared. Now a previously connected client tries to reconnect to the newly started server instance. The server doesn’t find a match for the connection and the above message is printed to the server log. If persistence-store mode is on, the database isn’t cleared and the old client is allowed to reconnect.

Comments

Fiona OShea 2009-01-21

Please work with taylor to get a good error message.

When error message is ready, Assign this issue in unresolved state to ServerTeam

ilevy 2009-02-11

Latest build (rev11611) puts out this message:

2009-02-11 12:28:00,852 INFO - Unable to find communications stack. ConnectionID(0.686d84df57e44c6eadb221cce42a0e84) not found. Connection is being requested from 10.0.0.92:50265. This is usually caused by a client that is not connected to the cluster. While that client is being rejected, everything else should proceed as normal. Some possible reasons for this situation might be: the client is from a previous run and can’t safely join this newer run; or the client couldn’t reconnect (configurable through several TC properties: ‘l2.l1reconnect.enabled’, ‘l2.l1reconnect.timeout.millis’, …)

A more concise message is needed.

ilevy 2009-03-09

Below is the new suggested message.

2009-02-11 12:28:00,852 INFO - Client Cannot Reconnect ConnectionID(0.686d84df57e44c6eadb221cce42a0e84) not found. Connection attempts from the Terracotta client at 10.0.0.92:50265 are being rejected by the Terracotta server array. Restart the client to allow it to rejoin the cluster. Many client reconnection failures can be avoided by configuring the Terracotta server array for “permanent-store” and tuning reconnection parameters. For more information, see http://www.terracotta.org/ha

Fiona OShea 2009-03-10

Igal, can you have the link be converted to a redirect with Dave? Otherwise it is fine, so when the redirect has been added just assign back to Transparency Team.

ilevy 2009-03-10

IT reports this redirect is ready:

Tue Mar 10 11:41:08 2009: Request 1755 was acted upon. Transaction: Ticket created by [email protected] Queue: IT Subject: new redirect needed Owner: Nobody Requestors: [email protected] Status: new Ticket <URL: http://helpdesk.terracotta.lan/Ticket/Display.html?id=1755 >

please create the following redirect:

http://www.terrracotta.org/ha

to

http://www.terracotta.org/web/display/docs/Configuring+Terracotta+For+High+Availability

Fiona OShea 2009-03-10

I edited the comment Igal Levy added a comment - 09/Mar/09 02:34 AM to have the correct URL. All you need to do is cut and paste that text.

Saravanan Subbiah 2009-03-10

Change the logging to the one provided by Igal above.

Erh-Yuan Tsai 2009-03-12

Changed as suggested text.

nadeem ghani 2009-03-27

workgin on rev12356

2009-03-27 10:01:17,346 INFO - Client Cannot Reconnect ConnectionID(0.88cd9462a0864cde9c4e1d3930e2be87) not found. Connection attempts from the Terracotta client at 10.0.0.113:59325 are being rejected by the Terracotta server array. Restart the client to allow it to rejoin the cluster. Many client reconnection failures can be avoided by configuring the Terracotta server array for “permanent-store” and tuning reconnection parameters. For more information, see http://www.terracotta.org/ha