Troubleshooting Guide

Overview of IPsec

IPsec: Internet Protocol Security

  • IPsec is a set of protocols defined by the IETF, to provide IP security at the network layer.
  • An IPsec based VPN is made up of two parts.
    • Internet Key Exchange protocol (IKE), underlying port UDP 500
    • IPsec protocol (ESP), underlying Protocol 50 or if using “nat-traversal” UDP 4500
  • Basically there is an initial brief interaction where one or each of the devices attempt to discover each other, via the Internet, they then trade Phase 1 (IKE) parameters and attempt to get a Phase 1 (sometimes called IKE or ISAKMP) connection which creates the keys used to encrypt Phase2. They then trade Phase 2 parameters and attempt to create an encrypted Phase 2 (sometimes called IPSec SA or ESP) tunnel connection.
  • They then transport data back and forth as well as maintain the connection with some additional administrative traffic.

IPsec with VNS3

IPsec tunnels negotiated with VNS3 require the following:

  • Policy-based VPN - encapsulates traffic between two sites as defined by a specific policy or ACL. This is used instead of a Route-based VPN that encapsulates traffic based on routes on both sides which can make it easier to administer but downgrades the security.
  • Encapsulating Security Packet (ESP) wire level protocol - encrypting and authenticating of the data flowing over the tunnel. This is used instead of Authenication Header (AH) which only authenticates.
  • Tunnel Mode - encapsulates the entire IP packet for communication over untrusted networks. This is used instead of Transport mode that only encapsulates the IP payload.
  • Internet Key Exchange (IKE) - protocol used to setup the shared security associations (SA) for the IPsec tunnel. This is used instead of manual key exchange.
  • Main Mode - used to setup the IPsec tunnel SAs using IKE. This is used instead of Aggressive mode that requires fewer messages to establish the SA but does so in a less secured manner.
  • Preshared Key (PSK) - used for authentication between two connecting parties. This is used instead of certificates.

IPsec Parameters

IPsec Parameters

  • Endpoint IP Addresses - Of course you need to know the Public IP of your IPsec device and your partner/customers IPsec device.
  • Phase 1 and Phase 2 parameters
    • If your Phase 1 and Phase 2 parameters match exactly - there are almost no instances where you will not have a successful connection.
    • If they DO NOT match exactly - even if you get an initial connection - it is highly unlikely you will have a stable, long running connection.
  • Optional Parameters
    • Dead Peer Detection (DPD) which significantly enhance the stability and recovery of your connection in the event of problems in the underlying network connectivity (your LAN, Internet, Cloud Provider, etc..)
    • Compatibility parameters for un-usual situations, sometimes needed by VNS3
  • Other Parameters
    • There are some other, usually older or very vendor specific parameters not supported by VNS3

Phase 1 Parameters

  • NAT-Traversal enabled or not
  • Pre-shared Key (PSK)
  • IKE Peer ID (usually automatic, will cover examples of where not)
  • Phase 1 Algorithm
  • Phase 1 Hash
  • Phase 1 Diffie Hellman Group
  • Phase 1 Lifetime

Native IPsec or NAT-Traversal

  • Device wide setting in VNS3 v3, per Endpoint setting in v4+
  • Initial negotiations occur on UDP 500
  • NAT-T enabled communicates on UDP 4500
  • Native IPsec enabled communicates on Protocol 50 (not port 50)
  • NAT-T has NOTHING to do with nat-ing your traffic. It specifies whether the communication happens via UDP 4500 or Protocol 50

Pre-Shared Key or PSK

  • Pre-shared Key is the easiest form of interoperability.
  • The alternative is certificates for authentication; not interoperable, not supported by VNS3 today.
  • The PSK must be identical on both sides.
  • The number one reason for PSK issues is COPY/PASTE. It often introduces invisible characters. Ideally type them out.

IKE Peer ID

  • VNS3 has the concept of “Local Private IP”.
  • This is a persistent private IP used by NAT-Traversal.
  • VNS3 combines the Local Private IP with IKE Peer ID, and sends the Local Private IP setting as the IKE Peer ID as well.
  • The IKE Peer ID of your connection partner’s IPsec endpoint is the “REMOTE PEER ID”. (Of course your Local Private IP is your partner’s “REMOTE PEER ID”.)
  • You need to be clear on which point of view you are speaking from.

Phase 1

Algorithm

  • Phase 1 is sometimes called ISAKMP or IKE
  • Phase 1 algorithm is the encryption algorithm used for your phase 1
  • Choices are (in order of security):
    • AES256
    • AES128
    • 3DES

Hash

Phase 1 hash parameters in order of security:

  • sha2_512 (VNS3 3.5.1.13 and above)
  • sha2_256 (VNS3 3.5.1.13 and above)
  • sha1
  • md5

Diffie Hellman

Phase 1 Diffie Hellman groups in order of security:

  • 18 (VNS3 3.5.1.13 and above)
  • 17 (VNS3 3.5.1.13 and above)
  • 16 (VNS3 3.5.1.13 and above)
  • 15 (VNS3 3.5.1.13 and above)
  • 14 (VNS3 3.5.1.13 and above)
  • 5
  • 2

Lifetime

  • Phase 1 lifetime is most commonly done in number of seconds
  • Phase 1 lifetime MUST match connection partner to have a stable connection
  • Phase 1 common lifetimes are:
    • 3600s
    • 28800s
    • 86400s
  • Phase 1 and Phase 2 lifetimes CANNOT be identical on the same endpoint connection.

Phase 2

Parameters

  • Phase 2 Algorithm
  • Phase 2 Hash
  • Perfect Forward Secrecy (PFS) - optional
  • Diffie Hellman Group for PFS - use if PFS used
  • Phase 2 Lifetime
  • Subnet Offered
  • Subnet Requested

Algorithm

  • Phase 2 is sometimes called IPSEC or ESP
  • Phase 2 algorithm is the encryption algorithm used for your phase 2
  • Choices are (in order of security):
    • AES256
    • AES128
    • 3DES

Hash

  • Phase 2 hash parameters in order of security:
    • sha2_512 (VNS3 3.5.1.13 and above)
    • sha2_256 (VNS3 3.5.1.13 and above)
    • sha1
    • md5

Lifetime

  • Phase 2 lifetime is most commonly done in number of seconds
  • Phase 2 lifetime MUST match connection partner to have a stable connection
  • Phase 2 common lifetimes are:
    • 3600s
    • 28800s
    • 86400s
  • Phase 1 and Phase 2 lifetimes CANNOT be identical on the same endpoint connection.

Perfect Forward Secrecy (PFS)

  • PFS needs to be enabled, or disabled.
  • Both sides need to agree.
  • When enabled PFS uses a Diffie Hellman Key Exchange which needs an agreed upon group specification.

Diffie Hellman

  • PFS (Perfect Forward Secrecy) Diffie Hellman groups in order of security:
    • 18 (VNS3 3.5.1.13 and above)
    • 17 (VNS3 3.5.1.13 and above)
    • 16 (VNS3 3.5.1.13 and above)
    • 15 (VNS3 3.5.1.13 and above)
    • 14 (VNS3 3.5.1.13 and above)
  • 5

IPsec Parameters

  • Optional Parameters
    • Dead Peer Detection Enabled or Not
    • Dead Peer Detection Action
    • Dead Peer Detection “delay” or “retry” period
    • Dead Peer Detection “timeout” before declaring a peer dead and performing DPD action
  • Other Parameter choices unsupported by VPN3/VNS3
    • Main mode vs. Aggressive (VPN3/VNS3 is “Main” mode only)
    • Authentication Header (VPN3/VNS3 do not support Authentication Header or “AH”)
    • Tunnel vs. Transport Mode (VPN3/VNS3 is “Tunnel” mode only)
    • X509 Certificates vs. PSK (VPN3/VNS3 supports PSK authentication only)

Endpoint and IPsec Configuration

GUI Example Remote Endpoint and Tunnel Setup

The following sections show how to configure an example remote endpoint and IPsec tunnel in VNS3 via the GUI with page numbers referencing the configuration guide.

VNS3 IPsec

Click IPsec under Connections left menu heading.

The IPsec page displays configuration settings needed to negotiate IPsec VPN tunnels, as well as currently configured endpoints and tunnels.

Local Private IP

The Local private IP address is a static and controllable IP that can be used for negotiation and identification purposes and increases interoperability. Some physical network routing and security appliances will need this IP included in their side of a IPsec tunnel configuration as the Peer or IKE ID (Watchguard, Juniper, SonicWall).

Logging

Link to the IPsec log file. Do not enable verbose logging unless directed to do so by Cohesive.

VNS3 IPsec

IPsec Configuration: Define a New Remote Endpoint

To create an IPsec endpoint, click Define new remote endpoint.

Information entered on the Endpoint page should match the configuration of the remote endpoint you will connect to via IPsec.

Enter the following information:

  • Name - enter a name for the endpoint.
  • Public IP - The public IP of the remote endpoint. If the endpoint does not have a public IP enter the public IP of the intermediate device that will be performing NAT.
  • NAT-Traversal - NAT-Traversal encapsulation is no longer a device wide setting. It can be enabled on a Endpoint-by-Endpoint basis. If checked tunnel traffic will be encapsulated in UDP 4500. If unchecked, Native IPsec will be used and tunnel traffic will move over ESP Protocol 50. This setting needs to match the remote Endpoint’s setting.
  • IKEV1/IKEV2 - This toggle chooses whether to use IKEV1 or IKEV2. This setting needs to match the remote device configuration. If using IKEV2, leave NAT-Traversal unchecked as IKEV2 has a NAT auto detect feature.
  • Pre-Shared Key - enter a presider key and keep a record of that key to be entered on the remote IPsec device. We recommend using something simple during initial configuration to avoid any copy/paste errors.
  • NAT IP - if the remote device is behind NAT (there is an intermediary device between the endpoint and the public Internet), enter the remote endpoint’s private IP in this field. If the remote device is publicly accessible.
  • Extra Config Parameters - We recommend connecting to the Controller with tunnels using AES256 encryption and SHA authentication for both IKE and ESP. Extra Parameters syntax can be found on the following page. Please contact Cohesive Networks Support for assistance.

Click Create.

IPsec Configuration: Define a New Remote Endpoint

IPsec Configuration: Define a New Remote Endpoint

IPsec Configuration: Extra Parameters

VNS3’s IPSec subsystem is good at autodiscovery on IKE and ESP choices with a wide range of boxes. We recommend being as specific as possible when entering tunnel parameters. Match the algorithm, hash and DiffieH group for your gateway settings by specifying them in the “Extra Params” text field. We support combinations algorithms 3DES, AES128, or AES256; hashes SHA1, MD5, SHA2_256, or SHA2_512; and DH groups 2, 5, 14, 15, 16, 17, 18.

Example entries for IKE (Phase 1) and ESP (Phase 2) in the extra params box:

phase1=aes128-sha1
phase1=aes256-sha2_256
phase1=3des-md5-dh2
phase1=aes256-sha2_512-dh5
phase2=aes256-sha1
phase2=3des-sha1

PFS Group Extra params entry for PFS Group is technically required only when it must be different from pfs group in phase1. If that is the case, then use:

pfsgroup=dh2
pfsgroup=dh14

IKE and ESP Lifetimes

phase1-lifetime=3600s (default setting on VNS3)
phase2-lifetime=28800s (default setting on VNS3)

Dead Peer Detection - Disabled by default, to enable DPD to attempt to re-connect during periods of no response use the following:

dpdaction=restart (other options are “hold” meaning just wait, or “clear” meaning drop the security association)
dpddelay=30s
dpdtimeout=90s

Other, less frequently used options available are:

connection=bidirectional (default)
Other options are “receive” meaning to not initiate connections, only receive them.

connection-rekey=yes (default)
Other options are “no” meaning no Phase1 or Phase2 re-key operations are done.

compat:some-text This option should only be used at the instruction of Cohesive. It allows underlying parameters of the IPSec, BGP, Routing, Firewall, or SSL VPN subsystems to be passed straight into the environment with no parsing or validation. It is only used in a small fraction of interoperability situations.

IPsec Configuration: Setup a Tunnel

Once an IPsec Endpoint has been defined, you will need to configure the tunnel or VPN Policy that will allow tunneled traffic between two endpoints. This will include what subnet will be advertised behind each side of the connection as well as the authentication and encryption settings used. These configurations must match between the two sides of the connection.

Click New Tunnel.

Enter the Local Subnet that will be advertised behind the VNS3 Controller. In this example we used the Overlay Network Subnet of 172.31.1.0/24.

Enter the Remote Subnet that you wish to connect to your VNS3 Overlay Subnet. This will be provided by the party that is configuring the remote IPsec Endpoint.

Provide a name for the tunnel to allow for easy identification in more complex topologies (required).

External Ping provides a pinging functionality over the IPsec tunnel that can be used in addition to IPsec DPD and Keep Alive settings to ensure the tunnel remains up during low traffic periods.

Enter an IP address of a pingable server on the Remote Subnet specified.

Set the time interval (in seconds) for the ping.

Click Create.

![IPsec Configuration: Setup a Tunnel](https://cohesive-networks.s3.amazonaws.com/docs/images/troubleshooting-ipsec-config-setup-tunnel.png#medium ““IPsec Configuration: Setup a Tunnel)

IPsec Configuration: Connected Tunnel

Your VNS3 Controller IPsec setup is complete.

The next steps will detail setting the IPsec connection from a IPsec extranet device (In our example a Cisco ASA). These steps are often completed by a 3rd party that owns and manages the physical IPsec device.

When negotiating an IPsec connection with a device that is not in your control making sure the IPsec Endpoint and Tunnel configurations match on both sides of the connection is critical to the successful negotiation of the tunnel.

Note the “Configuration Settings” values, you will need these to correctly configure your extranet device.

Use our Connection Spreadsheet to help organize the information to be shared between the two sides of the tunnel.

Once the IPsec connection is live, this guide will detail how to add clients to the created overlay network.

IPsec Configuration: Connected Tunnel

IPsec configuration: VNS3 Application Programing Interface (API)

The following sections show how to configure an example remote endpoint and IPsec tunnel in VNS3 via the API with page numbers referencing the API guide.

VNS3 is fully programable and scriptable via the API, the instructions are available for download here:
http://www.cohesiveft.com/dnld/VNS3_API_v20120521.pdf

The VNS3 3.0 API Tool can be downloaded from here:
ZIP: http://www.cohesiveft.com/dnld/VNS3_API_Tool_20120704.zip
TAR: http://www.cohesiveft.com/dnld/VNS3_API_Tool_20120704.tar.gz

IPsec Configuration: Define a New Remote Endpoint

IPsec Configuration: Define a New Remote Endpoint

IPsec Configuration: Setup a Tunnel

IPsec Configuration: Setup a Tunnel

Some clarifications on VNS3 IPsec Parameters

Allowed Algorithms: 3des, aes128, aes256

Allowed Hashings: sha1, md5, sha2_256, sha2_512

Allowed DH Groups: 2, 5, 14, 15, 16, 17, 18

Phase 1 Combinations VNS3 Config IPSec Params 1

Phase 2 Combinations

  • 3des-sha1
  • 3des-md5
  • 3des-sha2_256
  • 3des-sha2_512
  • aes128-sha1
  • aes128-md5
  • aes128-sha2_256
  • aes128-sha2_512
  • aes256-sha1
  • aes256-md5
  • aes256-sha2_256
  • aes256-sha2_512
  • aes256_gcm (Phase 2 does NOT have a hash/integrity function setting such as sha1, sha2_256, etc.)
  • aes256_ccm (Phase2 only and there is no associated hash/integrity function such as sha1, sha2_256, etc.)

*AES256 with MD5 is not supported; it may appear to work but can be highly unstable. **AES256-SHA1 with DH=2 is not supported; it may appear to work but will be highly unstable.

Allowed pfsgroup settings:

  • The pfsgroup takes a Diffie Hellman group number as its setting; 2, 5, 14, 15, 16, 17, or 18.
  • Do not use DH=2 with a Phase 1 or Phase 2 Algorithm of AES256.
  • Avoid having a different DH group used for Phase 1 than the one used for pfsgroup setting.
    • Many endpoints do not properly shift from using one DH in Phase 1 to a different one in the PFS setting. It will not be documented as not supporting this, but our experience has taught us this is an unnecessary complexity to avoid.

Allowed lifetime settings

  • The lifetime settings for Phase 1 and Phase 2 CANNOT be identical to each other, this causes significant interoperability problems.
  • “Old school” approach is usually for keys to have longer lifetimes than tunnels
  • “New school” approach is many re-key events before a re-tunnel
  • Neither is more right than the other. In theory “new school” is more secure.
  • Lifetimes MUST MATCH IDENTICALLY between the two endpoints or tunnel can become unstable or hang.

Allowed DPD settings

  • DPD must be enabled explicitly in VNS3 or it will not operate.
  • The defaults shown in VNS3 documentation mean a tunnel could be down for 90 seconds before VNS3 will attempt to recover it.
  • Cohesive does not recommend using a dpddelay of less than 10 seconds, and dpdtimeout of less than 30 seconds. Please contact support@cohesive.net if you believe you have an applicable use case requiring shorter durations than this.

Less common settings:
connection, connection-rekey, and compat: are only used in rare circumstances. Please contact support@cohesive.net before attempting to use.

Troubleshooting via the logfiles

Troubleshooting Guide: IPsec No Negotiation

Log Symptom
Repeated occurrences of: Nov 1 14:06:02 vpncubed pluto[13770]: "cftconn_1_CFTOffice_1" #4: initiating Main Mode

Description
This indicates the VNS3 Controller is trying to connect to an endpoint but not receiving any responses to its most basic negotiation messages.

Likely Causes Endpoint IP not correct on VNS3 AWS Security Group or customer firewall not allowing traffic on ports:

  • UDP 500
  • UDP 4500 (NAT-T)
  • Protocol 50 (not NAT-T)

Troubleshooting Guide: Phase 1 Failure

Example 1

Log Symptom

Nov  1 14:43:21 vpncubed pluto[8949]: "cftconn_1_CFTOffice_1" #2: probable authentication failure (mismatch of preshared secrets? malformed payload in packet

Description
Pre-Shared Keys (PSK) are used to authenticate IPsec tunnel negotiations & are entered on both sides of the tunnel. If there is a mismatch due to incorrect entry or extra/invisible characters from copy/paste, or omission, Phase 1/IKE/ISAKMP will not succeed.

Likely Causes
PSK Mismatch. Verify the PSK is entered correctly. It is recommended something short & easily remembered like ‘test’ is used during initial tunnel configuration.

Example 2

Log Symptom

ignoring informational payload, type NO_PROPOSAL_CHOSEN msgid=00000000
preceded by transitions through STATE_MAIN_I1, STATE_MAIN_I2,STATE_MAIN_I3 but no STATE_MAIN_I4 ISAKMP SA established.
This pattern repeats.

Description
Use the same parameters on both devices for Phase 1 negotiation. VNS3 can use an auto-discovery feature that attempts to match the settings of a customer’s remote device but we recommend against it.

Likely Causes
Phase 1 Mismatch on one or more of:
- Algorithm
- Hash - DH Group (Both sides of the tunnel must use the same parameters for Phase1 negotiation)

Example 3

Log Symptom

Nov 27 20:38:59 vpncubed27usuli386 pluto[21655]: "cftconn_1_VPN_sd5lrc_3" #2: max number of retransmissions (2) reached STATE_MAIN_I3.  Possible authentication failure: no acceptable response to our first encrypted message
Nov 27 20:38:19 vpncubed27usuli386 pluto[21655]: "cftconn_1_VPN_sd5lrc_3" #2: sending notification PAYLOAD_MALFORMED to 72.159.151.2:4500
STATE_MAIN_I3: sent MI3, expecting MR3

Description
Less likely that this is caused by PSK mismatch even though it says possible authentication failure. This is more likely caused by mismatch of algorithm, hash, or DH.

Likely Causes
Phase 1 Mismatch on one or more of:
- Algorithm
- Hash - DH Group (Both sides of the tunnel must use the same parameters for Phase1 negotiation)

Example 4

Log Symptom

Nov 15 22:10:44 vpncubed pluto[2799]: "cftconn_3_Missouri_6" #8608: sending encrypted notification INVALID_ID_INFORMATION to 74.118.11.221:4500
Nov 15 22:10:44 vpncubed pluto[2799]: "cftconn_3_Missouri_6" #8608: we require peer to have ID '74.118.11.221', but peer declares '@missouricuVPNgw.mizzoucu.com'
Nov 15 22:10:44 vpncubed pluto[2799]: "cftconn_3_Missouri_6" #8608: Main mode peer ID is ID_FQDN: '@missouricuVPNgw.mizzoucu.com'
Nov 15 22:10:44 vpncubed pluto[2799]: "cftconn_3_Missouri_6" #8608: received Vendor ID payload [Dead Peer Detection]
Nov 15 22:10:44 vpncubed pluto[2799]: "cftconn_3_Missouri_6" #8608: STATE_MAIN_I3: sent MI3, expecting MR3
Nov 15 22:10:44 vpncubed pluto[2799]: "cftconn_3_Missouri_6" #8608: transition from state STATE_MAIN_I2 to state STATE_MAIN_I3

Description
INVALID_ID_INFORMATION followed by this explanation about a Peer ID being a FQDN (fully qualified domain name). Many implementations allow the IKE Peer ID to be a text string, a FQDN or a IP Address.
VNS3 only accepts and provides IP Address format for IKE Peer ID

Likely Causes
Other side provided an unacceptable IKE Peer ID

Example 5

Log Symptom

Nov  8 15:22:03 vpncubed27usuli386 pluto[14078]: "cftconn_1_gibbonspc_vpn_2" #1: we require peer to have ID '209.123.105.201', but peer declares '172.30.100.10'
Nov  8 15:22:03 vpncubed27usuli386 pluto[14078]: "cftconn_1_gibbonspc_vpn_2" #1: sending encrypted notification INVALID_ID_INFORMATION to 209.123.105.201:4500

AFTER STATE_MAIN_I3: sent MI3, expecting MR3

Description
INVALID_ID_INFORMATION followed by this explanation about the peer IP address. In this case it is likely that NAT-T is enabled and the device is behind an edge router so it is NAT-ed out from a LAN address. Here the box has declared its IKE Peer ID to be its LAN address, whereas we only know its Public IP address which we expect as its ID. The solution is to put the LAN address (in this case 172.30.100.10) into the IKE ID/ PEER ID box in VNS3 Endpoint Definition page.

Likely Causes
Other side provides its LAN id as its IKE Peer ID when VNS3 is expecting the Public IP

Troubleshooting Guide: Phase 1 Success!

Log Symptom

Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #1: STATE_MAIN_I4: ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY cipher=oakley_3des_cbc_192 prf=oakley_sha group=modp1024}
Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #1: transition from state STATE_MAIN_I3 to state STATE_MAIN_I4
Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #1: Main mode peer ID is ID_IPV4_ADDR: '72.159.151.2'
Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #1: received Vendor ID payload [Dead Peer Detection]
Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #1: STATE_MAIN_I3: sent MI3, expecting MR3
Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #1: transition from state STATE_MAIN_I2 to state STATE_MAIN_I3

Description
Progression from STATE_MAIN_I1 to STATE_MAIN_I4, followed by ISAKMP SA established

Likely Causes
All parameters match leading to successful phase 1 negotiation and clear status of: ISAKMP SA established !

Troubleshooting Guide: Phase 2 Failure

Example 1

Log Symptom

Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #1: received Delete SA payload: deleting ISAKMP State #1
Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #1: received and ignored informational message
Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #1: ignoring informational payload, type NO_PROPOSAL_CHOSEN msgid=00000000
Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #2: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP+IKEv2ALLOW {using isakmp#1 msgid:84d5e7fe proposal=AES(12)_256-SHA1(2)_160 pfsgroup=OAKLEY_GROUP_MODP1024}
Nov 29 16:45:46 vpncubed27usuli386 pluto[9495]: "cftconn_1_VPN_sd5lrc_5" #1: STATE_MAIN_I4: ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY cipher=oakley_3des_cbc_192 prf=oakley_sha group=modp1024}

Description
Reading from the bottom, first there is a successful phase 1 negotiation as seen via the “ISAKMP SA established”. This is followed by an informational message from the other side saying “NO PROPOSAL CHOSEN” Since no agreement could be reached the other side says “let’s start over by deleting the ISAKMP/Phase 1/IKE connection, and beginning phase 1 over.

Likely Causes
Mismatch on one or more of:

  • Algorithm
  • Hash
  • Subnet requested/offered
  • if Cisco ASA, need traffic initiated from Cisco side (interesting traffic)
Troubleshooting Guide: Phase 2 Failure Detail

NO_PROPOSAL_CHOSEN for phase-2 usually one of following:

  • Mismatch on the algorithm & hash used on the two ends.
    ACTION: Verify settings for phase2alg=aes256-sha (VPN3) or phase2=aes256-sha1 (VNS3) or similar, based on agreement with remote gateway administrator. Also confirm if they are using PFS & what Diffie Hellman group they are using for PFS if they are using it.
  • Subnet mismatch.
    ACTION: Confirm what subnet CIDR you are requesting & they are offering, those have to be an exact match.
  • “Interesting Traffic”. Cisco has the concept of interesting traffic, meaning traffic bound for the tunnel. Without this, they will not complete the tunnel.
    ACTION: Have a ping attempted to one of the servers on your end of the tunnel to see if it causes the tunnel to complete.

Example 2

Log Symptom

vpncubed25uswdconnectfreehci386 pluto[14486]: "cftconn_1_LabCorp_5" #13: ignoring informational payload, type INVALID_ID_INFORMATION msgid=00000000
Sep  7 04:27:43 vpncubed25uswdconnectfreehci386 pluto[14486]: "cftconn_1_LabCorp_5" #14: initiating Quick Mode PSK+ENCRYPT+TUNNEL+UP+IKEv2ALLOW {using isakmp#13 msgid:fe94918c proposal=3DES(3)_192-SHA1(2)_160 pfsgroup=no-pfs}
Sep  7 04:27:43 vpncubed25uswdconnectfreehci386 pluto[14486]: "cftconn_1_LabCorp_5" #13: Dead Peer Detection (RFC 3706): enabled
Sep  7 04:27:43 vpncubed25uswdconnectfreehci386 pluto[14486]: "cftconn_1_LabCorp_5" #13: STATE_MAIN_I4: ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY cipher=oakley_3des_cbc_192 prf=oakley_sha group=modp1024}
Sep  7 04:27:43 vpncubed25uswdconnectfreehci386 pluto[14486]: "cftconn_1_LabCorp_5" #13: transition from state STATE_MAIN_I3 to state STATE_MAIN_I4

Description
INVALID_ID_INFORMATION after a successful Phase 1 negotiation as seen in ISAKMP SA established.
See the “initiating Quick Mode” - which is the beginning of Phase 2 negotiations.
This is usually a mismatched subnet specification between the two side.

Likely Causes
Mismatched Subnets

Troubleshooting Guide: Phase 2 Success!

Log Symptom

Aug 26 06:37:58 vpncubed200usdconnectsmei386 pluto[2806]: "cftconn_27_WRGOutlink_55" #179291: STATE_QUICK_R2: IPsec SA established tunnel mode {ESP/NAT=>0xc1139640 <0x2a27a8f0 xfrm=AES_256-HMAC_SHA1 NATOA=none NATD=216.116.84.42:4500 DPD=none}

Aug 26 06:37:58 vpncubed200usdconnectsmei386 pluto[2806]: "cftconn_27_WRGOutlink_55" #179291: transition from state STATE_QUICK_R1 to state STATE_QUICK_R2

Aug 26 06:37:58 vpncubed200usdconnectsmei386 pluto[2806]: "cftconn_27_WRGOutlink_55" #179291: STATE_QUICK_R1: sent QR1, inbound IPsec SA installed, expecting QI2

Description
IPSec SA Established. After an ISAKMP SA established, then a progression through STATE_QUICK_R1 and STATE_QUICK_R2.

Likely Causes
All parameters match leading to successful phase 2 negotiation and clear status of: IPsec SA established !

Troubleshooting Guide: Flapping Tunnel

Log Symptom
IPSEC SA established - but it keeps repeating significantly faster than the SALIFETIME or phase2-lifetime settings.

Description
IPsec tunnels connect via public Internet. As a result these connections can experience problems due to problems with intervening vendors, public cloud’s edge, user’s edge and/or user’s internal routing. HOWEVER, if the IPsec SA is being re-established more frequently than every ten minutes seek help from Cohesive Networks support immediately.

Likely Causes
Underlying network problems. Phase 1 or Phase 2 lifetime settings mismatch.

Troubleshooting Guide: Flapping Tunnel Detail

Tunnels “flapping” (unexpected up/down events)

Two major reasons:

  • Significant packet loss on the intervening Internet route between the IPsec peers. Rare in USA/Europe, much more likely in Singapore and South America.
  • Successfully connected but you do not have IDENTICAL configurations. IPSec interoperability is tricky. With vendor interoperability you need to be even more precise.
    • IKE lifetime or SA/IPsec lifetime are not set to the same values on each end of the tunnel respectively. These should be set to the same values for both ends to ensure coordinated re-key events.
    • The IKE and SA lifetimes used have identical values for IKE and SA lifetime. For example both IKE lifetime and SA lifetime are set to 28800 seconds. This can create a situation where IKE and SA re-key events happen simultaneously resulting in a hung tunnel where each side is using a different key set. They need to be different values.
    • There is a slight mismatch of the cryptographic algorithm family. While the initial negotiation of a tunnel using values like aes128 on one side and aes192 on the other appears successful, the resulting tunnel is unstable.
    • The remote IPsec device is End-of-Lifed and does not follow the most recent IPsec standard. If the IPsec device is 5 years old or NEWER - there should be no issue. If the IPsec device has been End-of-Lifed by its vendor or is on a very old software version a completely stable connection might not be possible. This is not a VNS3 issue, a stable connection would likely not be possible to any vendor.

Troubleshooting Guide: Traffic Problems

Example 1

Log Symptom

STATE_QUICK_I2: sent QI2, IPsec SA established tunnel mode {ESP/NAT=>0x3f2d7d97 <0xc720352d xfrm=AES_256-HMCA_SHA1 NATOA=none NATD=none DPD=none}

Description
It will appear that there has been a successful Phase 2 connection shown with “IPsec SA established”

However, NAT-Traversal encapsulation is required when connecting IPsec tunnels to some Cloud Edge devices (AWS EC2 is the most common environment with this requirement). A tunnel can be successfully negotiated without NAT-Travesal to EC2 but the EC2 edge will block all traffic. As a result the tunnel will show Connected but nothing will pass through the tunnel.

VNS3, when NAT-Traversal is enabled will appear to negotiate given the terms of the remote gateway. If the remote gateway DOES NOT offer a connection with NAT-Traversal, then VNS3 will accept the connection without it.

The solution depends on if VNS3 is running in a cloud that will accept native IPsec (Protocol 50) traffic. If it does then the solution is to disable NAT-T on the VNS3 Controller. If it does not, then the solution is to make the remote gateway use NAT-T.

In the case where VNS3 is running with NAT-T disabled such as can be done when using AWS VPC feature, then VNS3 will NOT enable NAT-T even if the other side requests it.

The solution depends on if VNS3 is running in a cloud that will accept native IPsec (Protocol 50) traffic. If it does then the solution is to disable NAT-T on the VNS3 manager. If it does not, then the solution is to make the remote gateway use NAT-T.

Likely Causes
NAT-Traversal not enabled on remote gateway but is enabled on VNS3 NAT-Traversal disabled on VNS3, but remote gateway has it enabled.

Example 2

Log Symptom

STATE_QUICK_I2: sent QI2, IPsec SA established tunnel mode {ESP/NAT=>0x3f2d7d97 <0xc720352d xfrm=AES_256-HMCA_SHA1 NATOA=none NATD=107.23.96.102 DPD=none}

Notice that the NATD= field has the public ip address of the remote gateway the connection is with. 

Description Here you see a Phase 2 connection success, and it appears that NAT-T was successfully negotiated. Yet, traffic is not flowing.

VNS3 clients running the OpenVPN agent receive route pushes when they connect to the VNS3 manager. If the tunnel came up AFTER this initial connection, then no route push will be received until the OpenVPN agent has been restarted. This will prevent traffic from being received by the VNS3 client machine.

This can be seen using the Network Sniffer to see traffic being sent to the VNS3 client - but no response coming back. Restart the OpenVPN agent on the VNS3 client and re-test. Or ask Cohesive Networks Support for info on the VNS3 routing agent.

Also, the VNS3 clients have had their OpenVPN agent restarted and are showing a route (netstat -nr will show this) to the remote gateway’s subnets.
Using the network sniffer on TUN0 interface will show no traffic. Using the network sniffer on ETH0 interface will show no data traffic, just negotiation traffic.
Firewall (possibly on remote gateway, but sometimes in the network path to the remote subnets) needs to be modified.
OR VNS3 Firewall has blocking entries. Check these on the Network Firewall page in the Web UI.

Likely Causes
Connected clients do not have route to the connected datacenter.
Firewall on remote gateway or VNS3 firewall not allowing traffic.

Troubleshooting Guide: Other Boxes

  • Cisco ASA lack of “interesting traffic” means no tunnel completion
  • Cisco ASA idle connection timeout defaults to closing tunnel every 30 minutes
  • Cisco ASA 8.4(2) - 8.4(4) have many bugs that cause interoperability problems
  • Firebox Watchguard needs VNS3 static LAN IP as IKE Peer ID (usually 192.0.2.254 or something like that)
  • Checkpoint does not follow NAT-T standards must be used in VPC with Native IPsec / Protocol 50/ESP
  • General care and attention to version age & EOL status
    • With EOL’d software, there can be interop issues due to the age of the IPsec standards used.