The following Overlay Engine functionality is deployed to VNS3 as a plugin using the container system. These instructions cover customization of the container image that will be used so that customer keys and rule sets can be employed.

Please be familiar with the VNS3 Plug-In Configuration Guide.

Overlay Engine Plugin - What does it do?

Simply stated Overlay Engines make your Overlay Network faster.

With Overlay Engines running on a VNS3 controller in a jumbo frame enabled cloud environment like AWS VPC, total throughput of 12-20 Gbps is achievable (tested on c4.2xl) depending on a tradeoff for the use-case between latency and throughput.

Overlay Engine Plugins add another Overlay Network process to a VNS3 Controller allowing more connections at higher speeds. The number of Overlay Engines allocated to a particular VNS3 controller depends on the number of plugin slots licensed. It should be noted that launching more Overlay Engines than your instance has CPU cores will not produce any benefit.

Overlay Engine Plugin - How does it work?

Overlay Engines are run as a sealed plugin (no user access via SSH) which gets access to relevant Overlay Network keys and mechanisms in the host VNS3 controller. The Overlay Engine then runs all the processes necessary to handle Overlay Network client connections. This allows the Overlay Network to utilize multithreaded capabilities of the VNS3 instance resources.

Overlay Engine Plugin - What does it need?

Overlay Engine plugins need two things:

The Container Image uploaded to the VNS3 controller needs “overlayengine” in the image name.
VNS3 firewall rules will be necessary to DNAT traffic to the Overlay Engine container(s).

Running the Overlay Engine Plugin

Getting the Overlay Engine Plugin

For access to this plugin please contact Cohesive Sales at sales@cohesive.net.

The Overlay Engine Plugin is accessible at the following restricted URLs; be sure to select the appropriate version for your version of VNS3:

VNS3 v4.4.0 - 4.8.X: https://vns3-containers-read-all.s3.amazonaws.com/Overlayengine/overlayengine_20190416.tar.gz
VNS3 v4.9.X - current: https://vns3-containers-read-all.s3.amazonaws.com/Overlayengine/overlayengine_20200510.tar.gz

These are in a read-only Amazon S3 storage location. Only Cohesive Networks can update or modify files stored in this location.

Cohesive Sales/Support can provide URLs to be used directly in a VNS3 Controller via the Web UI or API to import the container for use into that controller. (General screenshot walkthrough and help available in the plug-in configuration document.)

Uploading the Container Image to the VNS3 Plugin System

From the Container —> Images menu item, choose Upload Image.

Name the Container Image overlayengine.

This is a requirement to get the allocated plugins to receive access to the required directories on the VNS3 host.

To use the pre-configured plugin paste the URL into the Image File URL box.

Uploading the Container Image

Allocating a Container from the Overlay Engine Image

When the Image has imported it will say Ready in the Status Column.

To launch an Overlay Engine container, choose Allocate from the Action menu.

Allocating a Container from the Overlay Engine Image

Launching the Overlay Engine Plugin

After selecting Allocate from the Actions menu, you will be prompted for a container name, a description, and the command to be used to start the container.

The name should be UNIQUE among all Overlay Engine containers allocated to a VNS3 controller.

For the Overlay Engine Plugin, the start command is: /usr/bin/supervisord

Launching the BGP HA Plugin

Confirming the Overlay Engine Plugin is running

After executing the Allocate operation, you will be taken to the Container Display page.

You should see your Overlay Engine container running with the name you specified. It should have been given an IP address on your internal plug-in subnet (in this case ‘192.51.100.2’).

Confirming the BGP HA Plugin is running

Overlay Engine Firewall Rules

Overlay Engines will need two types of firewall rules added to the VNS3 controller:

DNAT rules to redirect incoming clients’ traffic to an Overlay Engine.
Depending on the number of CPU cores your instance offers, you may be able to increase your total overlay throughput further by using additional overlay engine containers. The ideal situation is having one fewer Overlay Engine containers than CPU cores. The format for these rules is as follows:
```
PREROUTING_CUST -i eth0 -p udp --dport 1194 -j HMARK --hmark-tuple src,sport --hmark-mod [N+1] --hmark-rnd 0xf1e2d3c4 --hmark-offset 0x1000
PREROUTING_CUST -p udp --dport 1194 -m mark --mark 0x1000 -j ACCEPT
PREROUTING_CUST -p udp --dport 1194 -m mark --mark 0x1001 -j DNAT --to-destination 198.51.100.2:1194
...
PREROUTING_CUST -p udp --dport 1194 -m mark --mark 0x[1000+N] -j DNAT --to-destination 198.51.100.[2+N-1]:1194
```
Where “N” is the number of Overlay Engine containers you have.
Repeat the pattern of “PREROUTING_CUST … -j DNAT” rules as appropriate. The first “PREROUTING_CUST … -j ACCEPT” rule is for traffic which will connect to VNS3’s default internal OpenVPN server; there should then follow one DNAT rule for each Overlay Engine container.
Remember if you are using more than nine Overlay Engines that you are working in hexadecimal with regards to the “–mark”; the tenth packet mark would be 0x100A in the example above.

FORWARD rules to allow OpenVPN packets to move across the VNS3 host to/from the Overlay Engine Plugin(s):

FORWARD_CUST -i plugin0 -p udp --sport 1194 -j ACCEPT
FORWARD_CUST -o plugin0 -p udp --dport 1194 -j ACCEPT

Keep in mind that you will now have overlay clients’ traffic arriving on VNS3’s plugin0 interface as well as tun0. You may need to update any firewall rules which reference the tun0 interface. You may specify interfaces as a comma separated list, such as “-i tun0,plugin0”.

Overlay Engine Firewall Rules

Overlay Engine Best Practices

Best Practices for using the Overlay Engine Plugin

Overlay Engine Names need to be unique - The Overlay Engine container name space is used to update status information on the host VNS3 controller. Overlap in naming will result in incorrect reporting with respect to connected client servers.
Not Rename Overlay Engine allocated Containers - this reduces the chance of overlapping name space in allocated containers.
Run Logging Container - Logging Container has access to all the Overlay Engine logs.
Use non-overlapping Container Network Subnets if running Overlay Engines on multiple Peered VNS3 Controller - This provides maximum flexibility and ensures no collision of routing when running Overlay Engines in a peered mesh.
Use the client configuration gateway directive. To ensure that all routes known to the network are reachable, overlay engine clients should use the “redirect-gateway def1” configuration line. This makes the connected client use the overlay engine, and the VNS3 controller as its default gateway for everything, including the cloud subnet. More information on this directive can be found here.

Restrictions/Limitations

UDP Multicast not supported
Normally client devices connected to VNS3 controllers can utilize the udp multicast protocol, despite it not being supported by the underlying cloud vendor. This capability runs over the encrypted overlay network. Client devices connected to overlay engines CANNOT send/receive multicast packets. Cohesive is working to remove this limitation.
Cohesive VNS3 Routing Agent not supported
Normally overlay client devices receive their route specifications ONLY when they connect to a VNS3 Controller. This is equally true for client devices connected to an overlay engine running on a VNS3 Controller. This creates the need for client devices to have their TLS Tunneling Agent (openvpn client) stopped/started to get new routes for the network. To prevent this administrative burden, Cohesive provides a separate piece of software to run on overlay clients, the VNS3 Routing Agent. The VNS3 Routing Agent CANNOT receive route updates when running on a client device connected to an overlay engine. Cohesive is working to remove this limitation

Updated on 16 Aug 2019