Skip to content

Arbitration Cluster

The arbitration cluster stores ECOX metadata, including node configurations and roles.

Note

Unless otherwise specified, the following operations should be performed on all distributed coordination service nodes.

Hostname Resolution

This step is optional. To enable hostname resolution without DNS, add the node IPs and names to /etc/hosts as follows:

cat <<EOF | sudo tee -a /etc/hosts
10.9.5.5    dcs01
10.9.5.6    dcs02
10.9.5.9    dcs03
EOF

Create User

Create a dedicated user and group (e.g., dcs) to run the arbitration service. You can adjust these names to align with your system policies

sudo groupadd dcs
sudo useradd -m -d /home/dcs --shell /bin/bash -g dcs dcs

ZooKeeper Cluster

Installation

ZooKeeper requires a Java environment to run, so you'll need to install Java first.

sudo dnf install -y java-1.8.0-openjdk-devel

sudo apt install -y openjdk-8-jdk

sudo apt install -y openjdk-8-jdk

Next, download the ZooKeeper binary distribution. This guide is based on version 3.8.4.

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz
sudo tar -xf apache-zookeeper-3.8.4-bin.tar.gz -C /usr/local/

Tip

To simplify future operations, we recommend adding the ZooKeeper program's path to the PATH environment variable.

cat <<'EOF' | sudo tee -a ~dcs/.bashrc
export PATH=/usr/local/apache-zookeeper-3.8.4-bin/bin:$PATH
EOF

We recommend storing ZooKeeper data in the dcs home directory. Run the commands below to create the directories, ensuring there is adequate disk space.

sudo su - dcs
mkdir -p $HOME/zook/{conf,data,logs}
  • conf Used to store ZooKeeper's configuration files.
  • data Used to store ZooKeeper's data files.
  • logs Used to store ZooKeeper's log files.

Create Configuration File

The ZooKeeper package provides a sample configuration file. Copy this template to create your initial configuration.

cp /usr/local/apache-zookeeper-3.8.4-bin/bin/conf/zoo_sample.cfg \
    $HOME/zook/conf/zoo.cfg
/home/dcs/zook/conf/zoo.cfg
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/dcs/zook/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpHost=0.0.0.0
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true

server.1=dcs01:2888:3888
server.2=dcs02:2888:3888
server.3=dcs03:2888:3888

Key configuration parameters:

  • dataDir

    The directory where ZooKeeper stores in-memory database snapshots and transaction logs (unless a separate dataLogDir is specified).

  • clientPort

    The port that the server listens on for client connections (used by ECOX). The default is 2181.

  • server.x=[hostname]:nnnn:nnnn

    Defines the nodes in the ZooKeeper cluster. The x represents the server ID. When the server starts, it identifies itself by reading the myid file in the dataDir. This file must contain an integer matching the x in this setting.

    Note: Use IP address if hostname resolution is not configured.

For a complete list of configuration parameters, please refer to the ZooKeeper Administrator's Guide.

Configure logging directory using the ZOO_LOG_DIR environment variable:

echo "export ZOO_LOG_DIR=/home/dcs/zook/logs" >>~dcs/.bashrc

Finally, create the myid file in the ZooKeeper data directory (dataDir). This file must contain a single integer that corresponds to the server.x ID defined in /home/dcs/zook/conf/zoo.cfg.

Perform the following steps on each of the three nodes:

echo 1 >$HOME/zook/data/myid

echo 2 >$HOME/zook/data/myid

echo 3 >$HOME/zook/data/myid

Setup Firewall

If the system firewall is enabled, you must configure it to allow ZooKeeper traffic. ZooKeeper utilizes three ports as defined in the configuration file. Execute the following commands to apply these rules:

sudo firewall-cmd --zone=public --add-port=2181/tcp --permanent
sudo firewall-cmd --zone=public --add-port=2888/tcp --permanent
sudo firewall-cmd --zone=public --add-port=3888/tcp --permanent
sudo firewall-cmd --reload

sudo ufw allow 2181/tcp
sudo ufw allow 2888/tcp
sudo ufw allow 3888/tcp

sudo ufw allow 2181/tcp
sudo ufw allow 2888/tcp
sudo ufw allow 3888/tcp

Testing

To start the ZooKeeper cluster, run the following command on all cluster nodes:

zkServer.sh --config $HOME/zook/conf start

Note

Switch to the dcs user. Use the full path to start the service if ZooKeeper is not added to your system path.

/usr/local/apache-zookeeper-3.8.4-bin/bin/zkServer.sh --config $HOME/zook-data/conf start

After startup, verify the cluster status with the command below:

zkServer.sh --config $HOME/zook/conf status

To stop the cluster, execute the following command:

zkServer.sh --config $HOME/zook/conf stop

Register Service

To simplify management, register ZooKeeper as a system service using the following configuration:

cat <<'EOF' | sudo tee /etc/systemd/system/zookeeper.service
[Unit]
Description=Zookeeper Service unit Configuration
After=network.target

[Service]
User=dcs
Group=dcs
Type=forking
Environment=ZOO_LOG_DIR=/home/dcs/dcs-data/logs
Environment=ZOO_HOME=/usr/local/apache-zookeeper-3.8.4-bin
ExecStart=/bin/sh -c "${ZOO_HOME}/bin/zkServer.sh --config /home/dcs/dcs-data/conf start"
ExecStop=/bin/sh -c "${ZOO_HOME}/bin/zkServer.sh --config /home/dcs/dcs-data/conf stop"
KillMode=none
Restart=on-failure
RestartSec=30s

[Install]
WantedBy=multi-user.target
EOF

Reload the configuration and enable the service for automatic startup.

sudo systemctl daemon-reload
sudo systemctl enable zookeeper.service