Skip to main content
Version: 6.0-SNAPSHOT

Restart a cluster

Static configuration is configuration that cannot be changed while members are running. To apply updates to static configuration, you must restart the cluster.

Restart options​

There are two options for restarting a cluster:

Where possible, you should perform a rolling restart. In a rolling restart, members are stopped one at a time. A member that shuts down gracefully first migrates its data to the remaining cluster members. This minimizes the risk of service interruptions and data loss. When the replacement member joins the cluster, data is repartitioned again. Members do not migrate data in a cluster state that does not allow migration, such as NO_MIGRATION or FROZEN.

Depending on the scope of your configuration changes, you may need to update a single member, a cluster, or multiple clusters. The process is the same in all cases.

tip

You can also use a rolling restart for a canary test: testing changes on a single member before updating the rest of the cluster.

Some configuration needs to be consistent across all members in a cluster. To update this configuration, you must restart the whole cluster simultaneously. By default, this causes all data in the cluster to be lost. To avoid data loss, you must enable persistence, which requires Enterprise Edition. To avoid downtime, you must divert traffic to another Hazelcast cluster before restarting. Hazelcast supports blue-green deployments to facilitate this.

Member processes are not reused. In all cases, members are terminated, and new members are created to replace them.

Persistence​

If persistence is enabled and the cluster enters FROZEN state, a replacement member can reuse the persistence directory of a terminated member. The replacement member then adopts the terminated member's UUID and attempts to load persisted data from disk. This avoids the repartitioning process, and the cluster recovers in less time during a rolling restart.

Persistence is required to avoid data loss during a whole cluster restart. Persistence requires Enterprise Edition. See Configuring Persistence for details.

Embedded mode​

If your cluster is operating in embedded mode, you should restart your host application instead of trying to restart the embedded Hazelcast members independently.

Perform a restart​

This section describes how to restart a cluster using a choice of common tools. Hazelcast supports a wide range of deployment options that may require other tools or allow you to automate parts of the process, but the required steps are similar.

You should perform restarts in a low utilization or maintenance window to reduce the risk of service impacts.

Deprecation Notice for the Community Edition REST API

The Community Edition REST API has been deprecated and will be removed as of Hazelcast version 7.0. An improved Enterprise version of this feature is available and actively being developed. For more information, see Enterprise REST API.

Perform a rolling restart​

You should check the cluster is healthy before and after making changes. The best way to do this is to run a Healthcheck in Management Center. For other options, see Monitoring.

To perform a rolling restart:

  1. Update the static configuration as required.

  2. Gracefully shut down the first member:

    important

    This command shuts down all running members on the local machine.

    The hz-stop command sends SIGTERM to each member. By default, the member terminates without a graceful shutdown. To shut down gracefully, add -Dhazelcast.shutdownhook.policy=GRACEFUL to JAVA_OPTS before you start the member.

    bash
    hz-stop
  3. Wait until all partition migrations are completed. You can monitor the progress of migrations on the cluster dashboard in Management Center.

  4. Start a new member.

    bash
    hz start
  5. Wait until the new member joins the cluster and all partition migrations are completed. Again, you can monitor progress using the cluster dashboard in Management Center.

  6. Repeat this process until all members have been restarted with the updated configuration.

Perform a whole cluster restart​

You should check the cluster is healthy before and after making changes. The best way to do this is to run a Healthcheck in Management Center. For other options, see Monitoring.

To perform a whole cluster restart:

  1. Update the static configuration as required.

  2. Divert traffic to another Hazelcast cluster. How to do this depends on how your Hazelcast deployment is architected. In Management Center, you can manage client connections with filtering rules.

  3. If you have enabled persistence, change the cluster state to NO_MIGRATION or FROZEN:

    bash
    hz-cluster-admin -o change-state -a <MEMBER_ADDRESS> -c <CLUSTER_NAME> -s no_migration

    You can specify the IP address of any member in the cluster.

    The script calls the Community Edition REST API. It works only if the REST API and its CLUSTER_WRITE endpoint group are enabled, and both are disabled by default. See hz-cluster-admin script.

  4. Gracefully shut down the cluster:

    important

    This command shuts down all running members on the local machine.

    The hz-stop command sends SIGTERM to each member. By default, the member terminates without a graceful shutdown. To shut down gracefully, add -Dhazelcast.shutdownhook.policy=GRACEFUL to JAVA_OPTS before you start the member.

    bash
    hz-stop
  5. Check that all members are shut down. The cluster dashboard in Management Center shows which members are in SHUT_DOWN state. If all members are shut down, Management Center disconnects from the cluster.

  6. Recreate the cluster:

    Use hz start to create each new member. The members form a cluster automatically.

    bash
    hz start
  7. Check that all members are in ACTIVE state using the cluster dashboard in Management Center or change state to ACTIVE if NO_MIGRATION or FROZEN was used for restart.

  8. Confirm the cluster is healthy, for example by running a Healthcheck.

  9. Restore traffic to the cluster.