Load balancing JBoss Data Virtualization cluster with HAProxy

Vijay Chintalapati bio photo By Vijay Chintalapati Comment

HA Proxy - Background

What is HAProxy and when do you need it ?

In a single line HAProxy is a reliable, high performance TCP/HTTP load balancer 1.

For someone familiar with the basic concepts of load-balancing but who is interested in much deeper understanding of what HAProxy can or cannot do, I would recommend a good read of the official documentation

On a very high-level you would use HAProxy if :

  • You have a set of identical backend servers
  • Each serving the same content and/or functionality
  • You would like to provide one single address (IP/PORT combination) to end consumers/clients
  • You want to distribute or load-balance the incoming requests so that no one backend is overwhelmed dealing with all the requests
  • And in cases of one or more backends failing the expectation is that the load-balancer detects it and routes traffic to only the live backends

Obviously there is more to HAProxy than what is mentioned above but the key thing to note is, it is :

  1. Open Source
  2. Free
  3. Software based
  4. Written in C and has a repution of being fast and effient in processing and memory usage
  5. Does TCP loadbalancing too (in addition to HTTP)

The only other comparable and competetive product in this space is NGINX

Using HAProxy to front cluster of JDV servers


The following prequisites are assumed in this article:

  1. Linux host, preferably RHEL or Fedora to run the HAProxy
  2. JBoss Data Virtualization 6.3 downloaded/installed on the same host (for convenience)
  3. The JDV cluster is setup and run in/using Domain Mode

Installing HAProxy

On the Linux host (Fedora used below) check and install HAProxy using the following commands:

# Check for HAProxy package
sudo dnf search haproxy

# Install HAProxy 
sudo dnf -y install haproxy

# Check for the package details to see where the configuration file is installed
sudo rpm -ql haproxy.x86_64

Starting the JDV cluster in Domain Mode

Assuming $JDV_HOME points to the root folder (with bin as one of the subfolders) of the JDV runtime, take the following steps:

Step 1 : In folder $JDV_HOME/domain/configuration, copy domain.xml as ha-proxy-domain.xml and copy host-master.xml as ha-proxy-master.xml

cd $JDV_HOME/domain/configuration
cp domain.xml ha-proxy-domain.xml 
cp host-master.xml ha-proxy-master.xml

Step 2 : Start the JDV server in Domain Mode, remove existing server groups, add a HA server group and create server runtimes under the new server group

# Go to the bin folder of server runtime
cd $JDV_HOME/bin

# Start JDV in Domain Mode and keep an eye on the logs
./domain.sh --domain-config=ha-proxy-domain.xml --host-config=ha-proxy-master.xml

# In a seperate terminal, run the JBoss CLI command and connect to the domain controller
./jboss-cli.sh -c

# Remove the existing server groups while in the CLI mode. 
# The prompt is shown here only for convenience
[domain@localhost:9999 /] /server-group=main-server-group:remove
[domain@localhost:9999 /] /server-group=other-server-group:remove

# Add the server-configs to the master host
[domain@localhost:9999 /] /host=master/server-config=server-one:add(auto-start=true,group=ha-server-group,socket-binding-port-offset=0,socket-binding-group=ha-sockets)
[domain@localhost:9999 /] /host=master/server-config=server-two:add(auto-start=true,group=ha-server-group,socket-binding-port-offset=100,socket-binding-group=ha-sockets)
[domain@localhost:9999 /] /host=master/server-config=server-three:add(auto-start=true,group=ha-server-group,socket-binding-port-offset=200,socket-binding-group=ha-sockets)

# Add the necessary interfaces 
[domain@localhost:9999 /] /host=master/interface=public:add(inet-address="${jboss.bind.address:}")
[domain@localhost:9999 /] /host=master/interface=unsecure:add(inet-address="${jboss.bind.address.unsecure:}")

# Start the servers
[domain@localhost:9999 /] /server-group=ha-server-group:start-servers

Step 3 : Make a note of the ports where the JDBC connections are exposed

You might see a line like below in the logs that clearly points to the host and port to which a given JDV server is bound to.

[Server:server-three] 12:45:33,220 INFO  [org.teiid.RUNTIME] (MSC service thread 1-2) TEIID50012 Teiid JDBC - Name = jdbc, Host = localhost.pnq.redhat.com,  Port = 31200, SSL = OFF, security-domains = teiid-security

Alternatively run the following command to confirm the ports are accessible :

netstat -nl | grep '[000|100|200]'

# Output
tcp        0      0*               LISTEN     
tcp        0      0*               LISTEN     
tcp        0      0*               LISTEN  

Configuring HAProxy and start the service

Step 1 : Refer to the haproxy.cfg that would right away work for current setup. Place the file as /etc/haproxy/haproxy.cfg and then start the haproxy service with the following commands:

# Start the HAProxy service
sudo systemctl start haproxy

# Optionally enable the service so that it is started
# with the system boot
sudo systemctl enable haproxy

If curious about the differences between the original and the suggested HAProxy configuration, see here

Essentially what was done was, we took a configuration file that was geared more towards a HTTP load balancing and we

  • Removed the requirement for SSL and its verification
  • Removed properties and options for HTTP
  • Renamed the backend app to jdv
  • Changed the mode of the frontend and backend app from http to tcp
  • Changed the count and ports of the backend servers. Note that the ports point to JDBC interface of the three JDV servers.

Step 2 : Check the status of the service to see if it started correctly

The service should start up clean as shown in the sample output below

# Run the following command to check the status of the service 
$ sudo systemctl status haproxy 
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-09-22 13:19:00 PDT; 2s ago
 Main PID: 16971 (haproxy-systemd)
    Tasks: 3 (limit: 512)
   Memory: 2.1M
      CPU: 8ms
   CGroup: /system.slice/haproxy.service
           ├─16971 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           ├─16982 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           └─16993 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Sep 22 13:19:00 intrepid systemd[1]: Started HAProxy Load Balancer.
Sep 22 13:19:00 intrepid haproxy-systemd-wrapper[16971]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

Note: You may run into a SELinux issue where it says SELinux is preventing /usr/sbin/haproxy from name_connect access on the tcp_socket. If you did run the command: sudo setsebool -P haproxy_connect_any 1 and that error should disappear

Deploy a working VDB, connect thru HAProxy and Test

Creation of a VDB is outside the scope of this article but to deploy the VDB you must first ensure that the ha profile has all the datasources the VDB depends on by JNDI name. You must also deploy and assign any JDBC4 drivers that you might have used or referred to while setting up the datasources.

Assuming we have deployed one such VDB: dvdemo.vdb successfully and assigned it to the ha profile in the domain mode, lets begin with testing. Follow the steps in order to test your setup:

  1. Ensure that the three JDV servers are reachable on JDBC interface at ports : 31000, 31100 and 31200 respectively for the deployed dvdemo.vdb. You can do this in JBoss Developer Studio using the Database Development perspective as shown
  2. Now, reuse the same connection window and change the port of the connection to : 5000 (port to which the haproxy frontend is bound to) and establish a connection as shown
  3. Run a sample query against a deployed virtual table using the SQL Scrapbook as shown
  4. Verify to which server the query was routed to in the logs, as shown
  5. Shutdown the verified server from the JBoss Domain Mode Management Console without touching the JBoss Developer Studio and the SQL Scapbook window
  6. Re-run the same query. You will notice that the connection is still maintained and the query now is routed to a different and live server.
comments powered by Disqus