HAProxy-Plugin Manager
Configurable HAProxy Resolver Plugin for Plugin Manager
Getting Started with VNS3 Plugin System
The following HAProxy DNS Resolution functionality is deployed to VNS3 as a plugin using the container system and configured via the VNS3 Plugin Manager (available in versions 4.11.3 and above). These instructions cover customization of the container image that will be used so that customer keys and rule sets can be employed.
Please be familiar with the VNS3 Plug-In Configuration Guide.
HAProxy Resolver Plugin - What does it do?
Simply stated HAProxy Resolver Plugin acts as a layer 4 forward proxy and resolves DNS at runtime.
Many VNS3 users run application topologies in 3rd party controlled environments like public clouds where they have limited access, control and visibility to the underlying infrastructure.
In the cloud underlying IPs can changes due to reboot or normal operations of services like Load Balancers (e.g. AWS Classic Elastic Load Balancers and Application Load Balancers). Many Layer 4 - 7 Network services can be configured using DNS names but resolution is usually done at startup and the resulting IP is cached for the length of the process.
This container plugin is designed to process a server or service’s name resolution at runtime to honor shorter TTL in the case of Load Balancers and allow for infrastructure modifications at the IP level without reloading configurations to update the cached DNS resolution.
Example Architecture: Simple Cloud Application with Load Balancers
- DNS advertisement of URLs to non-RFC1918 IP addresses included in remote IPsec tunnel encryption domain “behind” VNS3 Controller.
- VNS3 Netmaps traffic to non-RFC1918 IP addresses to HA Proxy Plugin.
- HAProxy Plugin performs layer 4 proxy forward to AWS ALB DNS name. It also performs DNS Resolution at runtime (configurable but default every 10s) to handle the ALB floating private IP.
HAProxy Plugin - How does it work?
HAProxy DNS Resolver plugin performs name resolution in two places:
- Process Startup - The plugin parses the configuration and resolves any DNS name included via the configured name server.
- Runtime Success - Periodic resolutions are performed for any servers using DNS name in the configuration.
- Runtime Failed Health Check - If a health check results in a connection timeout, a resolution is triggered.
This periodic and configurable resolution allows VNS3 users to take advantage of highly available cloud servers that periodically change the underlaying IP associated with a DNS name. The most common example of such a service is AWS Elastic Load Balancers and Applications Load Balancers. This Plugin is compatible with VNS3 Peering Meshes, Highly Available IPsec VPN connections via BGP and the HA Plugin (both cloud and on-premises version) for a highly available/fault tolerant architecture.
HAProxy Plugin - What does it need?
The HAProxy configuration file can be edited via the Plugin Manager (/etc/haproxy/haproxy.cfg):
- At first run the container will populate the IP address of the container that it binds to (ex. bind 198.51.100.2:FRONTENDPORT).
- Edit FRONTENDPORT to the port number that the frontend object is listening on. You can not have multiple frontends listening on the same port. Where the base template has named the two frontends frontend1 & frontend2, you can name them anything descriptive so long as they are unique.
- Edit LBURL to be the DNS name of the load balancer (ex. internal-haproxy-test-877810498.us-east-1.elb.amazonaws.com).
- Edit the corresponding LBPORT to the port number that the load balancer is listening on. Typically this will match the port number defined for the corresponding frontend object, but is not required. You can listen on one port and send to another.
- You can create as many or few pairs of frontend and backend objects as you need. You can name your frontend and backend objects something descriptive as long as the frontend references the named backend as its default_backend. The template is just for illustrative purposes.
- Under (listen stats) you can change: stats auth stats:stats. Where stats:stats is username:password. This is for logging into the HAProxy statistics page.
Getting the HAProxy Resolver Plugin
The Linux-based (Ubuntu 18.04) HAProxy Resolver Plugin that uses the Plugin Manager to manage configuration, logs, processes and statistics is accessible at the following URL:
https://vns3-containers-read-all.s3.amazonaws.com/HAproxy_Resolver/HAProxy_Resolver_PluginManager_base_18_20201208.tar.gz
This is a read-only Amazon S3 storage location. Only Cohesive Networks can update or modify files stored in this location.
This URL can be used directly in a VNS3 Controller via the Web UI or API to import the container for use into that controller. (General screenshot walkthrough and help available in the plug-in configuration document.)
Uploading the Container Image to the VNS3 Plugin System
From the Container —> Images menu item, choose Upload Image.
To use the pre-configured plugin paste the URL into the Image File URL box.
Allocating a Container from the HAProxy Image
When the Image has imported it will say Ready in the Status Column.
To then launch a running HA On Premise container, choose Allocate from the Action menu.
Launching the HAProxy Resolver Plugin
After selecting Allocate from the Actions menu you then name your container, provide a description and the command used to execute the container.
The name and description should be something meaningful within the context of your organization and its policies.
For the HAProxy Plugin the command used to run plugin containers will be:/usr/bin/supervisord
Confirming the HAProxy Resolver Plugin is Running
After executing the Allocate operation you will be taken to the Container Display page.
You should see your HAProxy Plugin with the name you specified. The Status should be Running and it should have been given an IP address on your internal plug-in subnet (in this case 198.51.100.2).
Managing the HAProxy Resolver Plugin (Plugin Manager)
With VNS3 version 4.11.3 and above you can manage the HAProxy Plugin with the Plugin Manager from the Web Interface. From the Container page click on the Container Name in blue. This will bring you to the Container Details page. Now from the Actions button in the top right of the page you can choose Manage.
The first thing you will want to do is click on the edit button under the Configuration Files section on the right had of the screen. Here you can edit FRONTENDPORT, LBURL and LBPORT for any amount of frontends and backends that you need. Once done click Save new version. You will see that you have incremented to versions 2.
Next, under the Process section on the left hand side of the window click on start. This will open the Run command window where you will click on Run:start. You should see the output haproxy: started.
You can check the HAProxy output logs by clicking the View button under Logging. You can also check the Supervisor logs by clicking the View button for Standard Output if HAProxy is not starting due to configuration or other issues.
You can see the firewall rules that apply to the HAProxy Plugin by clicking on the Manage button under the Firewall section. Here you can create a MACRO_CUST rule allowing plugin outbound access via masquerade rule. See our docmentation for more on providing network access to your container. You can also create a port forwarding rule allowing SSH access to container SSH port. In addition you will want to add a rule to view the Stats web page
PREROUTING_CUST -i eth0 -p tcp -s 0.0.0.0/0 –dport 80 -j DNAT –to 198.51.100.2:1936
where 198.51.100.2 is the IP of this container and 0.0.0.0/0 is replaced with a secure endpoint.
If you have set up the firewall rule for the Stats web page you will see Plugin UI Open button available to click on. This will bring you to HAProxy Stats page: https://www.haproxy.com/blog/exploring-the-haproxy-stats-page/
Customizing the HAProxy Resolver Plugin
Accessing the HAProxy Plugin
With the Plugin Manager you do not need to SSH into the Plugin in order to configure it, the Plugin Manager provides a web interface for full lifecycle management of the Plugin configuration, processes, logs and statistics. If you do need to access the Plugin via SSH in order to add a logging agent or make other edits that will require additions to the inbound hypervisor firewall rules with the VNS3 Controller as well as VNS3 Firewall.
The following example shows how to access an SSH server running in a Plugin listening on port 22.
Network Firewall/Security Group Rule
Allow port 22 from your source IP or subnets.
VNS3 Firewall
Enter rules to port forward incoming traffic to the Container Network and Masquerade outgoing traffic off the VNS3 Controller’s outer network interface.
#Let the Container Subnet Access the Internet or Local Network Via the VNS3 Controller’s Outer or Public IP
POSTROUTING_CUST -o eth0 -s <HA Container Network IP> -j MASQUERADE
#Port forward port 44 to the BGP HA Plugin port 22
PREROUTING_CUST -i eth0 -p tcp -s 0.0.0.0/0 --dport 44 -j DNAT --to <HA Container Network IP>:22
Securing the HA container
By default the container has the following accounts, configured as described.
“root” - The root account is locked. The root account is not allowed to remote shell into the container. This is our recommended approach. However, if you wish to, you can use the “container_admin” account to unlock root, provide a root password, and edit /etc/ssh/sshd_config to allow remote login by root.
“container_admin” - The default password is container_admin_123!. The default demo public key is also installed in the /home/container_admin/.ssh/authorized_keys. PLEASE change this password and this key when configuring, or create a new default image as your base for future use, following your authentication procedures. The account “container_admin” has “sudo” or superuser privileges, and is allowed to remote shell into the container.
Primary files for customization - Securing SSH
There are two significant files for securing the BGP HA container:
/etc/ssh/sshd_config
Please ensure this file is configured to your organization’s best practices.
/home/container_admin/.ssh/authorized_keys
The base container comes with an example public key installed, and private key for use in VNS3 documentation. Please remove after initial use or programmatic configuration.
Primary files for customization - Supervisor
/etc/supervisor/conf.d/supervisord.conf
This file defines what services are started when the container is started. Looking at the default you will see SSH, and the first run HAProxy configuration script which will populate (/etc/haproxy/haproxy.cfg) with the IP of the container.
/etc/supervisor/conf.d/haproxy.conf
This file defines the startup and logging parameters of haproxy.
Primary file for customization - HAProxy
The HA Proxy configuration file, /etc/haproxy/haproxy.cfg, is the primary file to control both the DNS resolution periodicity and target DNS name server and port for the Layer 4 forward proxy.
The following sections provide summary information about the relevant configuration sections to setup the resolve and forwarding behavior for your use-case.
Detailed documentation for configuration of the HAProxy Server DNS Resolution can be found here:
http://cbonte.github.io/haproxy-dconv/1.9/configuration.html#5.3
haproxy.cfg - Resolvers Section
The resolvers section begins with resolvers
resolvers aws
nameserver aws1 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 10s
hold other 30s
hold refused 30s
hold nx 30s
hold timeout 30s
hold obsolete 30s
Default settings resolve DNS every 10 seconds when receiving valid responses and 30 seconds when receiving error to avoid flapping.
haproxy.cfg - Proxies
The default Proxy configuration is included in the frontend
The “frontend” section describes a set of listening sockets that accept client connections in.
The “backend” section describes a set of servers to which the proxy will connect to forward incoming connections. This is where you specify the DNS name (in our example an AWS ALB).
frontend www
bind 198.51.100.2:443
default_backend alb-backend
backend alb-backend
balance roundrobin
mode tcp
server alb-1 internal-haproxy-test-877810498.us-east-1.elb.amazonaws.com:443 check resolvers aws
Forwarding Traffic to the HAProxy Plugin
Forwarding traffic to the container uses the same technique as was shown for accessing the container via Remote Shell.
VNS3 Firewall
Add a destination NAT rule to the VNS3 firewall to take traffic coming into the VNS3 controller primary network interface. This will NAT traffic with specific destination IP and destination port to a specific listening port on the HAproxy Resolver Container’s IP.
#Destination NAT inbound traffic to a <public or private IP> on a <destination port> and send it to the <HAproxy Listening Port> on the <HAproxy Container IP>:
PREROUTING_CUST -i eth0 -p tcp -d <public or private IP>/32 --dport <destination port> -j DNAT --to <HAproxy Container IP>:<HAproxy Listening Port>
Here is an example rule:
PREROUTING_CUST -i eth0 -p tcp -d 54.232.104.168/32 --dport 443 -j DNAT --to 198.51.100.2:443
