The following HAProxy DNS Resolution functionality is deployed to VNS3 as a plugin using the container system. These instructions cover customization of the container image that will be used so that customer keys and rule sets can be employed.

Please be familiar with the VNS3 Plug-In Configuration Guide.

HAProxy Resolver Plugin - What does it do?

Simply stated HAProxy Resolver Plugin acts as a layer 4 forward proxy and resolves DNS at runtime.

Many VNS3 users run application topologies in 3rd party controlled environments like public clouds where they have limited access, control and visibility to the underlying infrastructure.

In the cloud underlying IPs can changes due to reboot or normal operations of services like Load Balancers (e.g. AWS Classic Elastic Load Balancers and Application Load Balancers). Many Layer 4 - 7 Network services can be configured using DNS names but resolution is usually done at startup and the resulting IP is cached for the length of the process.

This container plugin is designed to process a server or service’s name resolution at runtime to honor shorter TTL in the case of Load Balancers and allow for infrastructure modifications at the IP level without reloading configurations to update the cached DNS resolution.

Example Architecture: Simple Cloud Application with Load Balancers

DNS advertisement of URLs to non-RFC1918 IP addresses included in remote IPsec tunnel encryption domain “behind” VNS3 Controller.
VNS3 Netmaps traffic to non-RFC1918 IP addresses to HA Proxy Plugin.
HAProxy Plugin performs layer 4 proxy forward to AWS ALB DNS name. It also performs DNS Resolution at runtime (configurable but default every 10s) to handle the ALB floating private IP.

HAProxy Plugin - How does it work?

HAProxy DNS Resolver plugin performs name resolution in two places:

Process Startup - The plugin parses the configuration and resolves any DNS name included via the configured name server.
Runtime Success - Periodic resolutions are performed for any servers using DNS name in the configuration.
Runtime Failed Health Check - If a health check results in a connection timeout, a resolution is triggered.

This periodic and configurable resolution allows VNS3 users to take advantage of highly available cloud servers that periodically change the underlaying IP associated with a DNS name. The most common example of such a service is AWS Elastic Load Balancers and Applications Load Balancers. This Plugin is compatible with VNS3 Peering Meshes, Highly Available IPsec VPN connections via BGP and the HA Plugin (both cloud and on-premises version) for a highly available/fault tolerant architecture.

Getting the HAProxy Resolver Plugin

The Linux-based (Ubuntu 18.04) HAProxy Resolver Plugin that uses the Plugin Manager to manage configuration, logs, processes and statistics is accessible at the following URL:
https://vns3-containers-read-all.s3.amazonaws.com/HAproxy_Resolver/HAProxy_Resolver_PluginManager_base_18_20201203.tar.gz

The Linux-based (Ubuntu 18.04) HAProxy Resolver Plugin that uses Environment Variables to set a simple configuration is accessible at the following URL: https://vns3-containers-read-all.s3.amazonaws.com/HAproxy_Resolver/HAproxy_Resolver_base_18-20200304.tar.gz

This is a read-only Amazon S3 storage location. Only Cohesive Networks can update or modify files stored in this location.

This URL can be used directly in a VNS3 Controller via the Web UI or API to import the container for use into that controller. (General screenshot walkthrough and help available in the plug-in configuration document.)

Uploading the Container Image to the VNS3 Plugin System

From the Container —> Images menu item, choose Upload Image.

To use the pre-configured plugin paste the URL into the Image File URL box.

Uploading the Container Image to the VNS3 Plugin System

Allocating a Container from the HA Image

When the Image has imported it will say Ready in the Status Column.

To then launch a running HA On Premise container, choose Allocate from the Action menu.

Allocating a Container from the HA Image

Launching the HAProxy Resolver Plugin (Plugin Manager)

After selecting Allocate from the Actions menu you then name your container, provide a description and the command used to execute the container.

The name and description should be something meaningful within the context of your organization and its policies.

For the HA Plugin the command used to run plugin containers will be:
/usr/bin/supervisord

Launching the HAProxy Plugin

Launching the HAProxy Resolver Plugin (Environment Variables)

With VNS3 version 4.8.4 and above we have added the ability to pass in environment variables to containers when they are allocated via the GUI and API. This removes the need to tunnel into the container and customize the configuration file for HA Proxy. You will still want to follow directions for Securing the HA container (Password and SSH).

After selecting Allocate from the Actions menu you then name your container, provide a description and the command used to execute the container.

The name and description should be something meaningful within the context of your organization and its policies.

The three variable names are FE_PORT (Front End Port), LB_URL (Load Balancer ULR) and LB_PORT (Load Balancer Port).

This is a comma delimited list. ex: FE_PORT=443, LB_URL=internal-haproxy-test-877810498.us-east-1.elb.amazonaws.com, LB_PORT=8443

For the HA Plugin the command used to run plugin containers will be: /usr/bin/supervisord

Launching the HAProxy Plugin

Confirming the HAProxy Resolver Plugin is Running

After executing the Allocate operation you will be taken to the Container Display page.

You should see your HA Plugin with the name you specified. The Status should be Running and it should have been given an IP address on your internal plug-in subnet (in this case 198.51.100.2).

Confirming the HAProxy Plugin is Running

Customizing the HAProxy Resolver Plugin

Accessing the HAProxy Plugin

Accessing a Plugin from the Public Internet or your internal subnets will require additions to the inbound hypervisor firewall rules with the VNS3 Controller as well as VNS3 Firewall.

The following example shows how to access an SSH server running in a Plugin listening on port 22.

Network Firewall/Security Group Rule

Allow port 22 from your source IP or subnets.

VNS3 Firewall

Enter rules to port forward incoming traffic to the Container Network and Masquerade outgoing traffic off the VNS3 Controller’s outer network interface.

#Let the Container Subnet Access the Internet or Local Network Via the VNS3 Controller’s Outer or Public IP
POSTROUTING_CUST -o eth0 -s <HA Container Network IP> -j MASQUERADE
#Port forward port 44 to the BGP HA Plugin port 22
PREROUTING_CUST -i eth0 -p tcp -s 0.0.0.0/0 --dport 44 -j DNAT --to <HA Container Network IP>:22

Accessing the HAProxy Plugin

Securing the HA container

By default the container has the following accounts, configured as described.

“root” - The root account is locked. The root account is not allowed to remote shell into the container. This is our recommended approach. However, if you wish to, you can use the “container_admin” account to unlock root, provide a root password, and edit /etc/ssh/sshd_config to allow remote login by root.

“container_admin” - The default password is container_admin_123!. The default demo public key is also installed in the /home/container_admin/.ssh/authorized_keys. PLEASE change this password and this key when configuring, or create a new default image as your base for future use, following your authentication procedures. The account “container_admin” has “sudo” or superuser privileges, and is allowed to remote shell into the container.

Primary files for customization - Securing SSH

There are two significant files for securing the BGP HA container:

/etc/ssh/sshd_config
Please ensure this file is configured to your organization’s best practices.

/home/container_admin/.ssh/authorized_keys
The base container comes with an example public key installed, and private key for use in VNS3 documentation. Please remove after initial use or programmatic configuration.

Primary files for customization - Supervisor (Ubuntu 14.04)

/etc/supervisor/conf.d/supervisord.conf
This file defines what services are started when the container is started. Looking at the default you will see Rsyslogd, SSH, and the HAContainer script.

Primary file for customization - HAProxy (Ubuntu 14.04)

The HA Proxy configuration file, /etc/haproxy/haproxy.cfg, is the primary file to control both the DNS resolution periodicity and target DNS name server and port for the Layer 4 forward proxy.

The following sections provide summary information about the relevant configuration sections to setup the resolve and forwarding behavior for your use-case.

Detailed documentation for configuration of the HAProxy Server DNS Resolution can be found here:
http://cbonte.github.io/haproxy-dconv/1.9/configuration.html#5.3

Primary file for customization - HAProxy (Ubuntu 18.04)

The HA Proxy configuration file, /etc/haproxy/haproxy.cfg will get configured from the envirenment variables that get passed in. In the event that you may need to change any non variablized values the template file can be found at /opt/haproxy.cfg.template. After making changes to this file you will want to stop the container and save a new image to be allocated.

haproxy.cfg - Resolvers Section

The resolvers section begins with resolvers and includes information related to name resolution. There can be as many resolvers sections as needed but this plugin comes with a single section.

NOTE: The default configuration uses a loopback address for the name server which allows the VNS3 Plugin system provided by Docker to use the VNS3 host name server. This is the recommended approach but other name servers can be specified.

resolvers aws
 nameserver aws1 127.0.0.11:53
 resolve_retries 3
 timeout resolve 1s
 timeout retry 1s
 hold valid 30s
 hold other 30s
 hold refused 30s
 hold nx 30s
 hold timeout 30s
 hold valid 10s
 hold obsolete 30s

Default settings resolve DNS every 10 seconds when receiving valid responses and 30 seconds when receiving error to avoid flapping.

haproxy.cfg - Proxies

The default Proxy configuration is included in the frontend and backend sections.

The “frontend” section describes a set of listening sockets that accept client connections in.

The “backend” section describes a set of servers to which the proxy will connect to forward incoming connections. This is where you specify the DNS name (in our example an AWS ALB).

frontend www
 bind 198.51.100.2:443
 default_backend alb-backend

backend alb-backend
 balance roundrobin
 mode tcp
 server alb-1 <alb name>:<alb listenerport> check resolvers aws

Forwarding Traffic to the HAProxy Plugin

Forwarding traffic to the container uses the same technique as was shown for accessing the container via Remote Shell.

VNS3 Firewall

Add a destination NAT rule to the VNS3 firewall to take traffic coming into the VNS3 controller primary network interface. This will NAT traffic with specific destination IP and destination port to a specific listening port on the HAproxy Resolver Container’s IP.

#Destination NAT inbound traffic to a <public or private IP> on a <destination port> and send it to the <HAproxy Listening Port> on the <HAproxy Container IP>:

PREROUTING_CUST -i eth0 -p tcp -d <public or private IP>/32 --dport <destination port> -j DNAT --to <HAproxy Container IP>:<HAproxy Listening Port>

Here is an example rule:

PREROUTING_CUST -i eth0 -p tcp -d 54.232.104.168/32 --dport 443 -j DNAT --to 198.51.100.2:443

Forwarding Traffic to the HAProxy Plugin

Running the HAProxy Resolver Plugin (Ubuntu 14.04)

Once the container is running on your VNS3 controller, SSH in and update the /etc/haproxy/haproxy.cfg file as specified previously. Save the file. Stop and start the container via the VNS3 UI Containers page.