Was this page helpful?
Caution
You're viewing documentation for a previous version. Switch to the latest stable version.
Replace a Running Node in a ScyllaDB Cluster¶
There are two methods to replace a running node in a Scylla cluster.
Note
Updating the cluster topology requires at least a quorum of nodes in a cluster to be available. If the quorum is lost, it must be restored before you change the cluster topology. See Handling Node Failures for details.
You can check the status of the nodes in the cluster using the nodetool status command.
Add a new node to the cluster and then decommission the old node¶
Adding a new node and only then decommissioning the old node allows the cluster to keep the same level of data replication throughout the process, but at the cost of more data being transferred during the procedure. When adding a new node to a Scylla cluster, existing nodes will give the new node responsibility for a subset of their vNodes, making sure that data is once again equally distributed. In the process, these nodes will stream relevant data to the new node. When decommissioning a node from a Scylla cluster, it will give its vNodes to other nodes, making sure data is once again equally distributed. In the process, this node will stream its data to the other nodes. Hence, replacing a node by adding and decommissioning redistribute the vNodes twice, streaming a node worth of data each time.
Procedure¶
Follow the procedure: Adding a New Node Into an Existing Scylla Cluster.
Decommission the old node using the Remove a Node from a Scylla Cluster procedure
Run the nodetool cleanup command on all the remaining nodes in the cluster
Verify that the node removed using nodetool status command
Replace a running node - by taking its place in the cluster¶
Stopping a node and taking its place in the cluster is not as safe as the data replication factor is temporarily reduced during the process. However, it is more efficient, as vNode distribution does not change, and only one node worth of data is streamed
Procedure¶
Run nodetool drain command (Scylla stops listening to its connections from the client and other nodes).
Stop the Scylla node you want to replace
sudo systemctl stop scylla-server
docker exec -it some-scylla supervisorctl stop scylla
(without stopping some-scylla container)
Follow the Replace a Dead Node in a Scylla Cluster procedure
Verify that the node is successfully replaced using nodetool status command