IPsec Configuration

This guide describes the IPsec protocol and how to configure IPsec VPNs with VNS3, exposing all tunnel parameters via the UI or API

What is IPSec?

IPsec is a set of protocols defined by the IETF, to provide IP security at the network layer. An IPsec based VPN is made up of two parts.

  • Internet Key Exchange protocol (IKE), underlying port UDP 500
  • IPsec protocol (ESP), underlying Protocol 50 or if using “nat-traversal” UDP 4500

Basically there is an initial brief interaction where one or each of the devices attempt to discover each other, via the Internet, they then trade Phase 1 (IKE) parameters and attempt to get a Phase 1 (sometimes called IKE or ISAKMP) connection which creates the keys used to encrypt Phase2. They then trade Phase 2 parameters and attempt to create an encrypted Phase 2 (sometimes called IPSec SA or ESP) tunnel connection.

They then transport data back and forth as well as maintain the connection with some additional administrative traffic.

IPSec with VNS3

VNS3 Config IPSec About diagram

IPsec tunnels negotiated with VNS3 include the following:

  • Policy-based VPN - encapsulates traffic between two sites as defined by a specific policy or ACL. This is used instead of a Route-based VPN that encapsulates traffic based on routes on both sides which can make it easier to administer but downgrades the security.
  • Route-based VPN - is a VPN with a single policy (usually 0.0.0.0/0 to 0.0.0.0/0, meaning any subnet can send to/from any other subnet on the other side of the connection). Route-based VPN configuration creates a system interface (usually VTI or GRE) which is then used to define routes and ACLs to allow specific subnet communication.
  • Encapsulating Security Packet (ESP) wire level protocol - encrypting and authenticating of the data flowing over the tunnel. This is used instead of Authentication Header (AH) which only authenticates.
  • Tunnel Mode - encapsulates the entire IP packet for communication over untrusted networks. This is used instead of Transport mode that only encapsulates the IP payload.
  • Internet Key Exchange (IKE) - protocol used to setup the shared security associations (SA) for the IPsec tunnel. This is used instead of manual key exchange.
  • Main Mode - used to setup the IPsec tunnel SAs using IKE. This is used instead of Aggressive mode that requires fewer messages to establish the SA but does so in a less secured manner.
  • Preshared Key (PSK) - used for authentication between two connecting parties. This is used instead of certificates.

Viewing IPSec Status

VNS3 Config IPSec page

Click IPsec under Connections left menu heading.

The IPsec page displays configuration settings needed to negotiate IPsec VPN tunnels, as well as currently configured endpoints and tunnels. Local Private IP The Local private IP address is a static and controllable IP that can be used for negotiation and identification purposes and increases interoperability. Some physical network routing and security appliances will need this IP included in their side of a IPsec tunnel configuration as the Peer or IKE ID (Watchguard, Juniper, SonicWall).

NOTE: Local Private IP conflates the need for a persistent virtual IPsec interface, making cloud migration easier, and the IPsec parameter of Local IKE ID. These will be “disentangled” as an update to the Interfaces and IPsec functions in an upcoming release.

View the IPSec log file by clicking Logging. Do not enable verbose logging unless directed to do so by Cohesive.

Policy-based VPN

To create an IPsec endpoint, click Define new remote endpoint.

Information entered on the Endpoint page should match the configuration of the remote endpoint you will connect to via IPsec. Enter the following information:

  • Name - enter a name for the endpoint.
  • Public IP - The public IP of the remote endpoint. If the endpoint does not have a public IP enter the public IP of the intermediate device that will be performing NAT.
  • NAT-Traversal - NAT-Traversal encapsulation is no longer a device wide setting. It can be enabled on a Endpoint-by-Endpoint basis. If checked tunnel traffic will be encapsulated in UDP 4500. If unchecked, Native IPsec will be used and tunnel traffic will move over ESP Protocol 50. This setting needs to match the remote Endpoint’s setting.
  • IKEV1/IKEV2 - This toggle chooses whether to use IKEV1 or IKEV2. This setting needs to match the remote device configuration.
  • Pre-Shared Key - enter a passphrase to use as a shared key and keep a record of that key to be entered on the remote IPsec device. We recommend using something simple during initial configuration to avoid any copy/paste errors
  • Remote Peer IKE ID - Usually defaults to the remote devices public ip if it has one, or its private LAN IP. It can be an address, a fully qualified domain name (@mynet.local for example), or a string/passphrase.
  • Enable PFS - Whether to use a negotiation element called “Perfect Forward Secrecy” which uses a configuration parameter of a “DH Group”.
  • Extra Config Parameters - We recommend connecting to the Controller with tunnels using AES256 encryption and SHA authentication for both IKE and ESP. Extra Parameters syntax can be found on the following page. Please contact Cohesive Networks Support for assistance.

VNS3 Config IPSec Policy create page

Click Create.

Creating Tunnel for Policy Based IPSec

Once an IPsec Endpoint has been defined, you will need to configure the tunnel(s) or VPN Policy(s) that will allow tunneled traffic between two endpoints. Note you will not have to do this for a route based VPN.

This will include what subnet will be advertised behind each side of the connection as well as the authentication and encryption settings used. These configurations must match between the two sides of the connection.

Click New Tunnel.

Enter the Local Subnet that will be advertised behind the VNS3 Controller. In this example we used the Overlay Network Subnet of 172.31.1.0/24. Enter the Remote Subnet that you wish to connect to your VNS3 Overlay Subnet. This will be provided by the party that is configuring the remote IPsec Endpoint.

Provide a name for the tunnel to allow for easy identification in more complex topologies (required).

External Ping provides a pinging functionality over the IPsec tunnel that can be used in addition to IPsec DPD and Keep Alive settings to ensure the tunnel remains up during low traffic periods.

Enter an IP address of a pingable server on the Remote Subnet specified. Set the time interval (in seconds) for the ping.

Click Create.

VNS3 Config IPSec Policy Tunnel 1

Connected Tunnel Status

Your VNS3 Controller IPsec setup is complete.

VNS3 Config IPSec Tunnel Connected

The next steps will be creating a complementary configuration on a connecting device. These steps are often completed by a 3rd party that owns and manages the physical IPsec device.

When negotiating an IPsec connection with a device that is not in your control making sure the IPsec Endpoint and Tunnel configurations match on both sides of the connection is critical to the successful negotiation of the tunnel.

Note the “Configuration Settings” values, you will need these to correctly configure your extranet device.

Use our Connection Checklist Spreadsheet to help organize the information to be shared between the two sides of the tunnel.

Use our setup tutorials for Cisco, Juniper, Fortinet and more from our Product Resource Page.

Once the IPsec connection is live, this guide will detail how to add clients to the created overlay network.

vns3-config-ipsec-tunnel-conn

Cryptographic Parameters

VNS3’s IPSec subsystem is good at autodiscovery on IKE and ESP choices with a wide range of boxes. We recommend being as specific as possible when entering tunnel parameters. Match the algorithm, hash and DiffieH group for your gateway settings by specifying them in the “Extra Configuration” text field.

We support the following:

  • combinations algorithms 3DES, AES128, AES256, AES256_GCM (AES256_CCM for phase2);
  • hashes SHA1, MD5, SHA2_256, or SHA2_512;
  • and DH groups 2, 5, 14, 15, 16, 17, 18, 19, 20, 21, 23.

Example entries for IKE (Phase 1) and ESP (Phase 2) in the extra params box:

phase1=aes128-sha1
phase1=aes256-sha2_256
phase1=3des-md5-dh2
phase1=aes256-sha2_512-dh5
phase2=aes256-sha1
phase2=3des-sha1

PFS Group Extra params entry for PFS Group is technically required only when it must be different from pfs group in phase1. If that is the case, then use

pfsgroup=dh2
pfsgroup=dh14

IKE and ESP Lifetimes

phase1-lifetime=3600s (default setting on VNS3)
phase2-lifetime=28800s (default setting on VNS3)

Dead Peer Detection - Disabled by default, to enable DPD to attempt to re-connect during periods of no response use the following:

dpdaction=restart (other options are “hold” meaning just wait, or “clear” meaning drop the security association)
dpddelay=30s
dpdtimeout=90s

Other, less frequently used options available are:

connection=bidirectional (default). Can also be “recieve”, meaning to not initiate connections, only receive them.

connection-rekey=yes (default). Can als be “no” meaning no Phase1 or Phase2 re-key operations are done.

local-peer-id=<an address, a fully qualified domain name, a simple text string with no special characters> VNS3 default to Local Private IP for this value, it is common to set local-peer-id=

mtu=<an integer> - MTU stands for “maximum transmission unit” For policy-based VPNs this is usually not specifically set. For route-based VPNs, if the connection is NOT going over the Internet but via a VPC/VNET peering link or a Direct Connect that supports “Jumbo Frames”, then “mtu” might be used to increase the size of the MTU (allowing higher throughput).

compat:some-text - This option should only be used at the instruction of Cohesive. It allows underlying parameters of the IPSec, BGP, Routing, Firewall, or SSL VPN subsystems to be passed straight into the environment with no parsing or validation. It is only used in a small fraction of interoperability situations.

Phase 1 Parameters

  • Allowed Algorithms: 3des, aes128, aes256, aes256_gcm, aes256_ccm (phase2 only)
  • aes256_gcm and aes256_ccm only work with IKEv2
  • Allowed Hashings: md5, sha1, sha2_256, sha2_512
  • Allowed DH Groups: 2, 5, 14, 15, 16, 17, 18, 19, 20, 21, 23, 24
  • Phase 1 Combinations

VNS3 Config IPSec Params 1

Phase 2 Parameters

  • Allowed Algorithms: 3des, aes128, aes256, aes256_gcm, aes256_ccm (phase2 only)
  • Allowed Hashings: md5, sha1, sha2_256, sha2_512
  • Perfect Forward Secrecy (PFS) DH Groups: 2, 5, 14, 15, 16, 17, 18, 19, 20, 21, 23, 24
  • Phase 2 Combinations:
    • 3des-sha1
    • 3des-md5
    • 3des-sha2_256
    • 3des-sha2_512
    • aes128-sha1
    • aes128-md5
    • aes128-sha2_256
    • aes128-sha2_512
    • aes256-sha1
    • aes256-md5
    • aes256-sha2_256
    • aes256-sha2_512
    • aes256_gcm (Phase 2 does NOT have a hash/integrity function setting such as sha1, sha2_256, etc.)
    • aes256_ccm (Phase2 only and there is no associated hash/integrity function such as sha1, sha2_256, etc.)

IPSec Runtime Status

Once there is a complementary connection on a connecting device, check the status of your IPsec connection from the VNS3 Controller click on Runtime Status.

Each tunnel will be displayed as a connected tunnel. Click the Tunnel ID link to see the Tunnel page that lists specific negotiation parameters, links to the IPsec log and provides simple controls (restart, refresh, and delete) for that tunnel.

If you do not see your tunnel listed, it is not correctly configured. Double check that you have entered all the information correctly in both the VNS3 Controller and your IPsec device. This document includes some basic IPsec troubleshooting information for additional detail see our IPsec Troubleshooting document.

VNS3 Config IPSec Status 1

Additional IPsec Endpoints and Tunnels can be added to your configuration depending on your license. Repeat the steps to connect to additional remote subnets.

NOTE: If you have 2 tunnels servicing the same datacenter subnet, these tunnels should not be up both at the same time (unless using route-based VPN with route metrics or BGP).

IPSec Tunnel Page

Each IPsec tunnel has a detail page. Refer to this page for all relevant information about a tunnel including the Tunnel Parameters used to negotiate and the tunnel history.

VNS3 Config IPSec Tunnel Page

Restart The tunnel detail page also allows you to restart the individual tunnel to force a renegotiation of without impacting any other tunnel connections.

Delete Remove the tunnel from the associated Endpoint Configuration.

Show Log The resulting pop-up window displays the log message for the specific tunnel.

Configure Logging Allows you to toggle between standard and verbose logging. Verbose Logging is disabled by default and should remain disabled during normal operations. Leaving Verbose Logging enabled over a extended period of time can fill the Controller instances virtual disk drive.  This causes the Controller to become inaccessible via the UI and requires our intervention to free up disk space.

DisableAllows you to disable a tunnel but keep the configuration. This is helpful for manual failover setups and initial IPsec connection buildout.

Route-based VPN

Route-based VPN is the form of VPN increasingly being requested by customers. The move to route-based has been driven by a number of factors including: ability to better manage routing, easier to debug packet flow, and to overcome traditional vendor problems in managing large numbers of policy-based VPN security associations.

  • Managing Routing: With route-based VPN, packet flow goes through explicit system interfaces, rather than through the system kernel. As a result you can more easily manage IP addressing overlaps between different connecting parties, or with your own infrastructure addressing scheme. Additionally, it makes route weighting techniques possible, like static route metrics and use of dynamic routing protocols like BGP.
  • Debug Packet Flow: On policy-based VPNs (in Linux) much of the traffic flow takes place in the system kernel and is policy controlled (of course) by the policy database vs. the functions being performed “normally” by the routing system and the system firewall. Using the interfaces created by the route-based VPN allows you to use commonplace system utilities to monitor and manage traffic flow.
  • Vendor Reliability Problems: Whilst “modern” metal vendors like Palo Alto, Fortinet, Sonicwall, and open source vendors like Openswan, FreeSwan, and Libreswan are able to manage dozens to thousands of phase2 security associations, some of the older, traditional “metal” vendors have had on-going, decade plus problems with bugs in their policy-based VPNs. When operating at scale, with a large number of connections, and/or a large number of subnet policies (more than one or two phase2 security associations) these vendors have significant failures. With each release of a new “firmware” version, new cryptographic algorithm, new hash technique, new Diffie Hellman group, or security patch, rekey errors are introduced and re-introduced causing VPNs to go out of protocol synchronization and require re-sets to renew traffic flow. Enter the “route-based” VPN which is built out of a specialized policy-based VPN which has only ONE phase2 / security association used to create interfaces across which traffic flows. FORTUNATELY, even the legacy metal network vendors can manage the “lookup” of a single policy per VPN in their software, thus significantly reducing errors industry wide.

VTI vs. GRE

  • When discussing route-based VPNs, Cohesive uses the distinction of “via VTI” vs. “over GRE”.
  • VTI (virtual tunnel interfaces) are system created interfaces, that are integrated to a single policy which in most cases allows “everything to everything”; meaning any address from one side of the VPN can potentially reach any address on the other side of the VPN. The only software protocol running between the two peers in a site-to-site connection is the IPsec protocol. Traffic going via the IPsec protocol is mediated through the locally created system VTI interface.
  • GRE (generic routing encapsulation) interface are created by the VPN hosts speaking a point to point tunneling protocol to each other via a single policy. The traffic over the single policy, layer 3, IPsec VPN tunnel is the GRE protocol (protocol 47) - which if successfully communicated between the two sides creates a Layer 2, GRE tunnel terminated on both ends via the single VPN policy.
  • Since no specific protocol is needed between the hosts with VTI, and since there is no additional tunnel besides the VPN tunnel, we describe the traffic flow as “via the VTI interfaces” as it is routed over the VPN via the VTI interface.
  • Since a separate GRE tunnel is created inside the IPsec VPN tunnel, and all subsequent traffic is routed THROUGH the GRE tunnel running inside the IPsec tunnel, we describe the traffic flow as “over the GRE tunnel”.

VTI

  • With VTI, the Cohesive implementation allows you to specify any local subnet - remote subnet pair which can be used to limit the scope of traffic allowed via the VTI interfaces (going over the IPsec tunnel). However, in connecting to most implementations in other IPsec devices it is expected that your “local subnet” is 0.0.0.0/0 (everything) and the “remote subnet” is 0.0.0.0/0 (everything).
  • After specifying the local/remote subnet pairs (most often 0.0.0.0/0 to 0.0.0.0/0) the Cohesive implementation expects an address for the VTI interface. Strictly speaking, VTI interfaces are not required to have an address, but for now in the Cohesive implementation it is required. This constraint may be relaxed in future.
  • The VTI interface address can be any non-overlapping addresses. There is no “right” address for the VTI interfaces between the two hosts, just they are non-overlapping and in the same CIDR. HOWEVER, if expecting to talk to other VPNs via BGP (dynamic routing) it is usually one of the two usable addresses in the middle of a /30 (4 address) network. One address used by one side of the VPN, the other used by the other side. Then BGP can be terminated on those shared network addresses. This technique is used to talk to Azure and Amazon AWS VPNs.

GRE

  • With GRE, the Cohesive implementation requires you to specify a /32 (single address) local subnet - remote subnet, the addresses of which will be used to terminate the GRE tunnel. Thereafter all subsequent traffic goes via the created GRE interfaces on each side and through the GRE tunnel which is then running over the IPsec tunnel.
  • As you might think, GRE over IPsec is more expensive in terms of compute overhead and packet encapsulation overhead because there are two tunneling encapsulations being made.
  • Like with VTI, even after specifying the /32 (single address) networks for both sides, the Cohesive implementation expects an address for the GRE interface. GRE interfaces are required to have an address.
  • The GRE interface addresses can be any non-overlapping addresses as long as they are in the same CIDR as defined by the GRE tunnel. When the addresses are defined they are defined using CIDR specification, thus allowing the two sides to be (for example) 192.168.10.10/24 and 192.168.10.90/24. Because they are defined in the same subnet (even though on the two hosts) they will be able to reach each other….and through this the encapsulated traffic can be routed. Most common will be a /30 (4 address subnet) where the middle two addresses are used respectively by the interfaces on each side. There is no “right” address for the GRE interfaces, just they are non-overlapping and in the same CIDR. HOWEVER, if expecting to talk to other VPNs via BGP (dynamic routing) it is usually one of the two usable addresses in the middle of the /30 network. Then BGP can be terminated on those shared network addresses. This technique is used to talk to Azure and Amazon AWS VPNs.

VTI Endpoint Configuration

This page assumes you have read about all of the parameters needed to establish an IPsec VPN connection in the previous pages discussing policy-based VPNs. If not, please do so before proceeding.

When “Enable Route-based VPN” is checked a section of the screen dynamically appears that lets you set the parameters needed.

The first choice is whether “Via VTI” or “Over GRE”. VTI is the most likely choice for most metal devices or cloud VPNs like AWS or Azure. GRE would be a choice for older Cisco or Juniper routers.

The VTI interface address is required. It can be any non-overlapping /32 address OR a /30 (4 address subnet) is used when planning to configure BGP. Local and Remote subnet will almost always be 0.0.0.0/0 - but there could be situations where your connecting party wants it to be less wide subnets than “everything to everything”.

Thats it!

VNS3 Config IPSec VTI Endpoint

After hitting “Create” - you will be taken to the IPsec page where you will see the single policy (in this case 0.0.0.0/0 to 0.0.0.0/0) used to create the VPN and you will see the VTI interface automatically created as a result of creating this endpoint configuration. The VTI interface will be named “ipsec_vtiX”, where X is the same ID as the endpoint ID being created (as seen in the URL after the endpoint is created).

Connected VTI Tunnel Page

Please read the Connected Tunnel page for Policy-based VPN for an explanation of the menu items on this page.

The only differences seen on this page versus a policy-based VPN are the “Tunnel Type” and “Tunnel Interface” display. In this case “Tunnel Interface” is “ipsec_vti5”. The number of the interface is system defined and is the same as the endpoint id (also system defined). The tunnel interface name and ID are not editable.

VNS3 Config IPSec VTI Tunnel Page

This same interface name will be displayed for use on the Routes and Interfaces pages.

Note: Route-based VPNs do not display tunnel “up/down” history. This will be added in a future release.

GRE in Endpoint Configuration

This page assumes you have read about all of the parameters needed to establish an IPsec VPN connection in the previous pages discussing policy-based VPNs. If not, please do so before proceeding.

When “Enable Route-based VPN” is checked a section of the screen dynamically appears that lets you set the parameters needed.

The second choice is whether “Via VTI” or “Over GRE”. VTI is the most likely choice for most metal devices or cloud VPNs like AWS or Azure. GRE would be a choice for older Cisco or Juniper routers.

The GRE interface address is required. It must be any non-overlapping address with the subnet mask bits after a “/“. For example here it is a /30, with this host using the “.1” address, the other host must be using the “.2” address. It could equally be 169.254.10.31/24 and the other side could be 169.254.10.90/24. The key is they must be in the same CIDR range when defined.

Unlike VTI, Local and Remote subnet CANNOT be 0.0.0.0/0. The local and remote subnet definitions must be single /32 addresses which can be used to terminate the GRE protocol connection.

Thats it!

VNS3 Config IPSec GRE

After hitting “Create” - you will be taken to the IPsec page where you will see the single policy (in this case18.207.185.140/32 to 192.0.10.254/32) used to create the VPN and you will see the GRE interface automatically created as a result of creating this endpoint configuration. The GRE interface will be named “greX”, where X is the same ID as the endpoint ID being created (as seen in the URL after the endpoint is created).

(In future the IPsec created GRE interfaces may be renamed to “ipsec_greX” to be more similar to the VTI naming convention.)

Connected GRE Tunnel Page

Please read the Connected Tunnel page for Policy-based VPN for an explanation of the menu items on this page.

The only differences seen on this page versus a policy-based VPN are the “Tunnel Type” and “Tunnel Interface” display. In this case it is “Tunnel Interface” is “gre7”. The number of the interface is system defined and is the same as the endpoint id (also system defined). The tunnel interface name and ID are not editable.

This same interface name will be displayed for use on the Routes and Interfaces pages.

VNS3 Config IPSec GRE Tunnel Page

Note: Route-based VPNs do not display tunnel “up/down” history. This will be added in a future release.

IPsec Configuration with Remote Device

  • The setup of the connecting device is outside the scope of this document.
  • The KEY to a successful, stable connection is to make sure both sides are configured identically and “EXPLICITLY”. By “explicitly” we mean that rather than relying on default values which might go out of synch over time across the VNS3 and connecting devices varied releases, declare all configuration parameters.
  • On the Cohesive Website the Product Resource page points to a variety of setup tutorials for Cisco, Fortinet, Juniper and more.