Routing to and from plugins

IN PROGRESS

The VNS3 Network Edge Plugin System provides the ability to run plugins (containers) directly in the VNS3 network edge. These plugins can can do things like DNS resolution, load balancing, network intrusion detection and more. There are effectively 3 firewall patterns for sending traffic to a plugin:

  1. Routing to plugin’s IP address with a DNAT rule
  2. Routing through a plugin by looking up plugins route table (new in v5.2.4)
  3. Copying traffic to a plugin with the -j COPY rule action

Note for all examples below, if VNS3 version is prior to VNS3 5.3 the rule tables must end in _CUST. e.g. PREROUTING_CUST.

VNS3 Packet Path Overview

First lets briefly review how a packet flows through VNS3:

VNS3 packet routing

Packets hit a VNS3 interface and then are matched against any PREROUTING rules before the first routing decision is made. The default route table that is looked up is called the main route table and can be reviewed from the /routes#os-view tab on the VNS3 Routes page. Before VNS3 version 5.2.4 all user added routes were placed in the main table. Starting with 5.2.4, users may now place routes in several different route tables.

The firewall tables [1] that are traversed depend on the destination of the packet:

  • Traffic routed through VNS3 will traverse the PREROUTING, FORWARD and POSTROUTING tables.
  • Traffic terminating at VNS3 will traverse the PREROUTING and INPUT tables
  • Traffic originating from VNS3 will traverse the OUTPUT and POSTROUTING tables

Routing to a plugin

Traffic can be route to a problem by updating the destination of packet with a DNAT rule:

PREROUTING -i eth0 -p tcp -s 0.0.0.0/0 --dport 45 -j DNAT --to 198.51.100.3:22

Here 198.51.100.3 is the IP address of the running plugin instance. This rule routes all (source 0.0.0.0) tcp traffic coming in on interface eth0 at port 45 to plugin port 22 at IP 198.51.100.3.

Starting in VNS3 v5.3 you can use firewall variables:

PREROUTING -i eth0 -p tcp -s 0.0.0.0/0 --dport 45 -j DNAT --to ${plugin_instance_1_ip}:22

Routing from a plugin

Traffic originating from a plugin needs a way out. This is useful for monitoring or alerting plugins such as Datadog. A MASQUERADE rule allows the plugin IP to talk to the internet via the VNS3 outer interface, typically eth0.

POSTROUTING -o eth0 -s 198.51.100.3 -j MASQUERADE

This is effectively a source NAT rule and can also be accomplished with:

POSTROUTING -o eth0 -s 198.51.100.3 -j SNAT --to 3.222.10.5

Or, starting with VNS3 5.3 you can also use variables:

POSTROUTING -o eth0 -s ${plugin_instance_1_ip} -j SNAT --to ${vns3_public_ip}

Routing through a plugin

ALPHA

Starting with VNS3 5.2.4 users can now create routes in multiple route tables, including the plugin route table. Each interface has it’s own route table and can be routed to explicitly with a firewall rule. This allows users to route traffic through a plugin where the return traffic will also back through the plugin. To properly route traffic through a plugin requires firewall rules for route table lookups and routes in those tables.

Firewall rules

Versions 5.2.4-

In version 5.2 you accomplish this routing scheme with ROUTING_CUST rules:

# Incoming overlay network traffic
ROUTING_CUST -i tun0 -s 100.127.255.192/26 -m mark --mark 0 -j LOOKUP --table plugin0
# Response traffic
ROUTING_CUST -i plugin0 -d 100.127.255.192/26 -m mark --mark 0 -j LOOKUP --table plugin0

This rule routes all inbound overlay network traffic (tun0 source 100.127.255.192/26) that to the plugin routing table. The mark is simply to make sure we are only routing previously un-marked traffic.

We will also need no-track rules as we are effectively “seeing this traffic twice”. So all the firewall rules will look like this:

# Send new connections from overlay clients to container
PREROUTING_CUST -i tun0 -s 100.127.255.192/26 -j NOTRACK
ROUTING_CUST -i tun0 -s 100.127.255.192/26 -m mark --mark 0 -j LOOKUP --table plugin0
# Response traffic after passing back through container
PREROUTING_CUST -i plugin0 -d 100.127.255.192/26 -j NOTRACK
ROUTING_CUST -i plugin0 -d 100.127.255.192/26 -m mark --mark 0 -j LOOKUP --table plugin0

In VNS3 5.3, the above is simplified to

PREROUTING -i plugin0 -d 100.127.255.192/26 -j LOOKUP --through --table plugin0

This rule will create the above no-track rules and the source and destination routing rules. the --through directive tells VNS3 that we want return traffic to also come through the plugin.

Routing rules

For this traffic to hit your plugin you need a the plugin route table to be properly configured. This includes 3 rules:

Plugin routes

Interface route for plugin network:

  • Table: plugin0
  • CIDR: 198.51.100.0/28
  • interface: plugin0

Route all traffic to your plugin:

  • Table: plugin0
  • CIDR: 0.0.0.0/0
  • interface: plugin0
  • Gateway: 198.51.100.3

Route overlay traffic back to overlay:

  • Table: plugin0
  • CIDR: 100.127.255.192/26
  • interface: tun0

Eth0 Routes

We also need to route traffic hitting VNS3 on eth0 correctly. This includes the following routes in the eth0 route table.

Interface route sending plugin network to plugin interface:

  • Table: eth0
  • CIDR: 198.51.100.0/28
  • interface: plugin0

Route overlay traffic to plugin interface gateway:

  • Table: eth0
  • CIDR: 100.127.255.192/26
  • interface: plugin0
  • Gateway: 198.51.100.3

Ensure eth0 vpc/vnet traffic stays on eth0:

  • Table: eth0
  • CIDR: vpc cidr
  • interface: eth0

Route eth0 traffic to internet:

  • Table: eth0
  • CIDR: 0.0.0.0/0
  • interface: eth0
  • Gateway: VNS3 subnet gateway e.g. 10.100.0.1

Copying to a plugin

Sometimes you simply want to temporarily duplicate traffic to a plugin for analysis or alerting purposes. This should be a temporary configuration as copying traffic is typically an expensive operation to run long term.

Versions 5.2 and earlier

This can be accomplished with the following rule:

MACRO_CUST -s 10.100.0.0/24 -j COPY --bidirectional --to 198.51.100.3 --from tun0

This bidirectional command will effectively creates 2 rules for you:

  1. It will copy inbound traffic at the prerouting step from tun0 to your plugin
  2. It will copy outbound traffic at the postrouting step from tun0 to your plugin

To just target inbound traffic, simply use --inbound. Similarly, to only target outbound traffic, use --outbound.

Notes

[1] For the iptables aficionados, you might notice we are calling iptables “chains” VNS3 tables. In VNS3 we abstract away chains and only have users worry about tables as we believe this is easier to work with. IPtables table and chain logic is handled by VNS3.