High Availability mode

This document describes the principles, preparations, and server operations of the high availability mode

1.Theory

TuGraph provides high availability mode (HA mode) through multiple server hot backups. In high availability mode, writes are synchronized to all servers so that service availability is not affected even if some servers go down.

In high availability mode, multiple TuGraph servers form a backup group. Each backup group consists of three or more TuGraph servers, one of which serves as the ‘leader’ and the other replication group servers serve as the ‘follower’. Write requests are serviced by the ‘leader’ who replicates each request to the ‘follower’ and responds to the client only after the request has been synchronized to the server. This way, if any server fails, the other servers will still have all the data that has been written so far. If the ‘leader’ server fails, other servers will automatically select a new ‘leader’.

The high availability mode is only available in the enterprise edition. The open-source Community Edition does not include this feature.

2.Preparation

To enable high availability mode, users need to:

Three or more instances of TuGraph servers.
To enable high availability mode when starting lgraph_server, the ‘enable_ha’ option can be set to ‘true’ using a configuration file or the command line.
Set the correct rpc_port through the configuration file or command line

3.Start the initial backup group

3.1.The initial data is consistent

When the data in all servers is the same or there is no data at startup, the user can specify --conf host1:port1,host2:port2 to start the server. In this way, all prepared TuGraph instances can be added to the initial backup group at one time, All servers in the backup group elect leader according to the RAFT protocol, and other servers join the backup group with the role of follower.

An example command to start an initial backup group is as follows:

$ ./lgraph_server -c lgraph.json --rpc_port 9090 --enable_ha true --conf 172.22.224.15:9090,172.22.224.16:9090,172.22.224.17:9090

After the first server is started, it will elect itself as the ‘leader’ and organize a backup group with only itself.

3.2.Inconsistent initial data

If there is already data in the first server (imported by the lgraph_import tool or transferred from a server in non-high availability mode), And it has not been used in high-availability mode before, the user should use the boostrap method to start. Start the server with data in bootstrap mode with the ha_bootstrap_role parameter as 1, and specify the machine as the leader through the conf parameter. In bootstrap mode, the server will copy its own data to the new server before adding the newly joined server to the backup group, so that the data in each server is consistent.

An example command to start a data server is as follows:

$ ./lgraph_server -c lgraph.json --rpc_port 9090 --enable_ha true --conf 172.22.224.15:9090 --ha_bootstrap_role 1

Other servers without data need to specify the ha_bootstrap_role parameter as 2, and specify the leader through the conf parameter. The command example is as follows

**$ ./lgraph_server -c lgraph.json --rpc_port 9090 --enable_ha true --conf 172.22.224.15:9090 --ha_bootstrap_role 2

4.Scale out other servers

After starting the initial backup group, if you want to scale out the backup group, add new servers to the backup group, The --conf HOST:PORT option should be used, where HOST can be the IP address of any server already in this backup group, And PORT is its RPC port. E.g:

./lgraph_server -c lgraph.json --rpc_port 9090 --enable_ha true --conf 172.22.224.15:9090

This command will start a TuGraph server in high availability mode and try to add it to the backup group containing the server 172.22.224.15:9090. Note that joining a backup group requires a server to synchronize its data with the backup group’s leader server, and this process may take a considerable amount of time, depending on the size of the data.

5.Stopping the Server

When a server goes offline via ‘CTRL-C’, it will notify the current ‘leader’ server to remove the server from the backup group. If the leader server goes offline, it will pass the leader identity permission to another server before going offline.

If a server is terminated or disconnected from other servers in the backup group, the server is considered a failed node and the leader server will remove it from the backup group after a specified time limit.

If any server leaves the backup group and wishes to rejoin, it must start with the ‘–conf {HOST:PORT}’ option, where ‘HOST’ is the IP address of a server in the current backup group.

6.Restarting the Server

Restarting the entire backup group is not recommended as it disrupts service. All servers can be shut down if desired. But on reboot, It must be ensured that at least N/2+1 servers in the backup group at shutdown can start normally, otherwise the startup will fail. and, Regardless of whether enable_bootstrap is specified as true when initially starting the replication group, restarting the server only needs to pass Specify the --conf host1:port1,host2:port2 parameter to restart all servers at once. The command example is as follows:

$ ./lgraph_server -c lgraph.json --rpc_port 9090 --enable_ha true --conf 172.22.224.15:9090,172.22.224.16:9090,172.22.224.17:9090

7.docker deploys a highly available cluster

In real business scenarios, it is likely to encounter the need to deploy high-availability clusters on multiple operating systems or architectures. Differentiated environments may cause some dependencies to be missing when compiling TuGraph. therefore, Compiling software in docker and deploying high-availability clusters is very valuable. Take the centos7 version of docker as an example, The steps to deploy a highly available cluster are as follows.

7.1.Install mirror

Use the following command to download TuGraph’s compiled docker image environment

docker pull tugraph/tugraph-compile-centos7

Then pull the TuGraph source code and compile and install

7.2.Create container

Use the following command to create a container, use --net=host to make the container run in host mode, in this mode Docker and the host machine share the network namespace, that is, they share the same IP.

docker run --net=host -itd -p -v {src_dir}:{dst_dir} --name tugraph_ha tugraph/tugraph-compile-centos7 /bin/bash

7.3.Start service

Use the following command to start the service on each server, because docker and the host share IP, so you can directly specify to start the service on the host IP

$ lgraph_server -c lgraph.json --host 172.22.224.15 --rpc_port 9090 --enable_ha true --conf 172.22.224.15:9090,172.22.224.16:9090,172.22.224.17:9090

8.Server Status

The current status of the backup group can be obtained from the TuGraph visualization tool, REST API, and Cypher query.

In the TuGraph visualization tool, you can find the list of servers and their roles in the backup group in the DBInfo section.

With the REST API, you can use GET /info/peers to request information.

In Cypher, the CALL dbms.listServers() statement is used to query the status information of the current backup group.

9.Data synchronization in high availability mode

In high availability mode, different servers in the same backup group may not always be in the same state. For performance reasons, if a request has been synchronized to more than half of the servers, the leader server will consider the request to be in the committed state. Although the rest of the servers will eventually receive the new request, the inconsistent state of the servers will persist for some time. A client may also send a request to a server that has just restarted, thus having an older state and waiting to join a backup group.

To ensure that the client sees consistently continuous data, and in particular to get rid of the ‘reverse time travel’ problem, where the client reads a state older than it has seen before, each TuGraph server keeps a monotonically increasing data version number. The mapping of the data version number to the database state in the backup group is globally consistent, meaning that if two servers have the same data version number, they must have the same data. When responding to a request, the server includes its data version number in the response. Thus, the client can tell which version it has seen. The client can choose to send this data version number along with the request. Upon receiving a request with a data version number, the server compares the data version number to its current version and rejects the request if its own version is lower than the requested version. This mechanism ensures that the client never reads a state that is older than before.