The VNS3 HA Plugin enhances the high availability of VNS3 appliances in cloud environments by automatically updating cloud provider route tables (and optionally advertised VNS3 routes) to redirect traffic away from a failed controller, keeping network services available without manual intervention.

Key Features

Automatic failover: Monitors the health of the HA pair and updates cloud route tables to redirect traffic to the healthy controller, without manual intervention.
Multi-cloud: Supports AWS and Azure, using each provider’s API and IAM/managed-identity permissions.
Robust health checks: Combines ICMP ping and VNS3 API checks (AND logic, with a configurable failure threshold) to avoid false-positive failovers.
Flexible topologies: Supports primary/secondary, hybrid, public, and private HA pair configurations.
Optional VNS3 route failover: Can additionally fail over VNS3-advertised routes between peers — see VNS3 Route Failover (Optional).

How It Works

VNS3 controllers are assigned cloud API permissions (IAM roles or a managed identity for AWS and Azure respectively) with least privilege policies, giving them access to cloud route information and the ability to update it. Each controller continuously checks the health of its peer using both ICMP ping and a VNS3 API (TCP 8000) check — both must succeed for the peer to be considered healthy, and both must fail for failure_threshold consecutive checks before the peer is declared unhealthy. Requiring both checks, and consecutive failures rather than a single one, keeps transient network blips from triggering an unnecessary failover.

When a peer is declared unhealthy, the surviving controller updates the cloud route tables to point to itself (and optionally adds any advertised VNS3 routes from the unhealthy peer). When the peer becomes reachable again, the surviving controller doesn’t restore routes immediately — a peer’s VNS3 API or ping responding doesn’t by itself confirm its IPsec tunnels or BGP sessions have finished renegotiating. Instead it waits until the peer has been continuously reachable for recovery_delay seconds (default 90); any drop in reachability during that window resets the timer. Once the delay elapses, routes are restored to their original owner (and any advertised VNS3 routes added to the healthy controller are removed).

To tell planned route changes apart from failure-driven ones, each controller keeps state files recording the initial route table configuration (stashed while both controllers are healthy) and the latest observed configuration. Changes observed while the topology is healthy are treated as intentional and simply recorded; changes observed while the topology is unhealthy are treated as failover artifacts and corrected.

Optional health check: if peer_overlay_ip is configured, the plugin also checks the peer’s overlay-network IP (TCP 8000 and ping) for additional visibility into cases where overlay connectivity specifically is impacted.

VNS3 Route Failover (Optional)

By default, the plugin only manages cloud provider route tables — VNS3 itself owns and manages VNS3 routes, and no VNS3 API credentials are needed. Setting manage_vns3_routes: true adds an optional second failover layer, independent of and running alongside the cloud-route failover described above:

While the peer is reachable, each controller periodically asks the peer’s VNS3 API for its currently advertised routes and keeps a local snapshot.
When the peer becomes unreachable, the controller adds any of those advertised routes it doesn’t already have (matched by exact CIDR) to its own VNS3 route table, tagging each one’s description with [HA-failover: borrowed from peer <peer_underlay_ip>].
Once the peer has been continuously reachable for recovery_delay seconds — the same delay and timer the cloud-route layer uses — every route carrying that tag is removed.
Tunnel-backed routes are never failed over. A route’s tunnel ID is specific to the controller that created it and has no equivalent on the peer, so tunnel routes are skipped entirely.
Only advertised routes are in scope. Non-advertised routes on either controller are never touched.
Collisions are skipped, not resolved. If a borrowed route would collide with an existing, differently-sourced local route of the same CIDR, the plugin logs it and leaves the existing route alone. Avoid configuring overlapping advertised CIDRs between the pair if you rely on this feature.
Manual changes are never fought. If someone deletes or edits a route this layer added while it’s still borrowed, that’s logged once and then left alone for the rest of that failover episode — it is not re-added, and an edit is not reverted.
Restart-safe without a state file. If the plugin process restarts while routes are still borrowed, it re-derives “I’m currently in a failed-over state” directly from the health.state file.
Isolated failure domain. This layer logs under its own VNS3 route failover: prefix and keeps its own state file (peer_vns3_routes_state-latest.json); it never writes to the health.state or the cloud-route state files, and a failure here (e.g. bad credentials) is logged but never blocks cloud-route failover or peer health checks.

Credentials: one or both controllers need the other controller’s VNS3 API credential configured locally (peer_vns3_api_password or peer_vns3_api_token) so they can read its advertised routes. The plugin also needs its own local controller’s VNS3 API credentials (my_vns3_api_password or my_vns3_api_token) so it can make updates to the local VNS3 routing configuration. The toggle and credentials are per-controller, so one side of the pair can enable this feature independently of the other.

Version 4 Requirements and Specifications

Supported Clouds

Version 4 of the HA Plugin supports both AWS and Azure. Google Cloud and OCI support may be offered in later versions.

Supported VNS3 Versions

The plugin is compatible with VNS3 versions 5.x and later.

Prerequisites

VNS3 HA pair Controllers Setup

Same VPC/VNET: The HA pair controllers must be deployed within the same Virtual Private Cloud (VPC) or Virtual Network (VNET), since cloud route tables are scoped to a VPC/VNET.
VNS3 Routes configured: VNS3 Routes should be properly configured to support your desired HA pair topology. Refer to the VNS3 documentation for your specific topology.
Cloud metadata access: Each controller needs access to the cloud provider’s metadata service to retrieve instance, network interface, and other relevant details.
API/IAM permissions: Controllers need the appropriate API/IAM permissions to update cloud route tables — see Configuration below.
Inbound access for health checks: Controllers must allow inbound ICMP (ping) and TCP 8000 (VNS3 API) from each other.
Network connectivity to managed subnets: Controllers need reliable connectivity to each other and to the subnets/address ranges they manage.
VNS3 API credentials (optional): Only required if enabling VNS3 Route Failover — not needed for the default cloud-route-only behavior.

Architecture / Use-cases

Active-Active Monitoring Configuration

Both controllers run the HA plugin in active mode, continuously monitoring each other. Cloud route tables typically point all relevant routes to one designated controller under normal operations.

Normal Operations: Both controllers run identical configurations and monitor each other. One controller is the primary route target in cloud route tables.
Failover: If a controller becomes unresponsive, the healthy controller updates the route tables to redirect traffic to itself and marks the topology “unhealthy.”
Recovery: Once the recovered controller has been continuously reachable for recovery_delay seconds (see How It Works), routes are restored to their original owner and the topology is marked “healthy.”

Distributed Routing Configuration

The HA pair is set up so cloud route tables distribute routes between both controllers — some routes point to one controller, others to its peer.

Normal Operations: Routes are distributed across both controllers by design (e.g. subnet locality, AZ optimization), so traffic can use the “local” controller for reduced latency and cost.
Failover: If a controller becomes unresponsive, its routes are redirected to the healthy peer.
Recovery: Once recovered, routes are restored to the original distributed configuration.

Public or Private

The plugin supports both public and private HA pair configurations. In a public setup, it monitors controllers' public IPs and updates the public route tables. In a private setup, it monitors private IPs and updates the private route tables, keeping internal traffic redirected to the healthy controller.

Configuration

Step 1: VNS3 Instance/VM Permissions

AWS permissions

VNS3 Controllers running in AWS need an IAM Role (AWS Documentation) with a policy granting permission to update VPC route tables. Example policies below: Allow-All, VPC-ID Limited, Route Table-ID Limited, and Tag Limited.

NOTE: ec2:ReplaceRoute can be scoped by condition (vpc-id, route-table-ids, or a tag key:value), but the ec2:Describe* actions cannot be limited by resource ARN or condition.

Allow-All

No edits required — copy/paste into a policy, attach to a role, and assign the role to the controller running the HA Plugin.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeRouteTables",
                "ec2:ReplaceRoute"
            ],
            "Resource": "*"
        }
    ]
}

VPC-ID Limited

Replace REGION, ACCOUNT-ID, and VPC-ID with the relevant details for your deployment.

 {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "VisualEditor1",
             "Effect": "Allow",
             "Action": [
 				"ec2:DescribeRouteTables",
 				"ec2:DescribeInstances"
 			],
             "Resource": "*"
         },
         {
             "Sid": "VisualEditor0",
             "Effect": "Allow",
             "Action": "ec2:ReplaceRoute",
             "Resource": "arn:aws:ec2:REGION:ACCOUNT-ID:route-table/*",
             "Condition": {
                 "ArnEquals": {
                     "ec2:Vpc": "arn:aws:ec2:REGION:ACCOUNT-ID:vpc/VPC-ID"
                 }
             }
         }
     ]
 }

Route Table-ID Limited

Replace REGION and ACCOUNT-ID, and ROUTE-TABLE-IDs with the rtb-ids associated with the VNS3 controllers in your deployment.

 {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "VisualEditor1",
             "Effect": "Allow",
             "Action": [
 				"ec2:DescribeRouteTables",
 				"ec2:DescribeInstances"
 			],
             "Resource": "*"
		 },
         {
             "Sid": "VisualEditor0",
             "Effect": "Allow",
             "Action": "ec2:ReplaceRoute",
             "Resource": "arn:aws:ec2:REGION:ACCOUNT-ID:route-table/*",
             "Condition": {
                 "ForAnyValue:StringEquals": {
                     "ec2:RouteTableID": [
                         "ROUTE-TABLE-1-ID",
                         "ROUTE-TABLE-2-ID"
                     ]
                 }
             }
         }
     ]
 }

Tag Limited

Replace TAG-KEY and TAG-VALUE with the tag key/value applied to the rtb-ids associated with the VNS3 controllers in your deployment.

 {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "VisualEditor1",
             "Effect": "Allow",
             "Action": [
 				"ec2:DescribeRouteTables",
 				"ec2:DescribeInstances"
 			],
             "Resource": "*"
         },
         {
             "Sid": "VisualEditor0",
             "Effect": "Allow",
             "Action": "ec2:ReplaceRoute",
             "Resource": "arn:aws:ec2:REGION:ACCOUNT-ID:route-table/*",
             "Condition": {
                 "StringEquals": {
                     "ec2:ResourceTag/TAG-KEY": "TAG-VALUE"
                 }
             }
         }
     ]
 }

Azure permissions

Permissions/Actions:

Microsoft.Compute/virtualMachines/read
Microsoft.Network/networkInterfaces/read
Microsoft.Network/virtualNetworks/read
Microsoft.Network/publicIPAddresses/read
Microsoft.Network/routeTables/read
Microsoft.Network/routeTables/routes/read
Microsoft.Network/routeTables/routes/write

Scope:

Resource group for the VNS3 controllers

Step 2: Plugin Network Access

The HA Plugin needs network access to reach the cloud API and to communicate with the HA pair peer via VNS3 API (TCP 8000) and ICMP ping.

2.1: Cloud Network Security Groups

Configure the Cloud Provider Network Security Groups to allow the following between the VNS3 controllers in the HA Pair.

Required Traffic

TCP traffic on port 8000 (VNS3 API) via Public IPs or Private IPs depending on your architecture.
ICMP traffic via Private IPs

NOTE: Optional health checks via overlay network IPs are allowed by default when VNS3 controllers are Peered.

2.2: VNS3 Firewall Rules

Add the following rules to both VNS3 Controllers in the HA pair to give the HA Plugin the appropriate network access. These assume the HA Plugin is allocated the 198.51.100.2 Plugin Network IP:

POSTROUTING -o eth0 -s 198.51.100.2/32 -j MASQUERADE-ONCE
FORWARD -s 198.51.100.2/32 -j ACCEPT
FORWARD -d 198.51.100.2/32 -j ACCEPT
INPUT -i plugin0 -d 198.51.100.2/32 -j ACCEPT
OUTPUT -o plugin0 -s 198.51.100.2/32 -j ACCEPT
#Optional VNS3 Route Failover for local API Access
INPUT -s 198.51.100.2/32 -j ACCEPT
OUTPUT -d 198.51.100.2/32 -j ACCEPT

Step 3: Upload and Allocate HA Plugin

The HA Plugin is available via the Plugin Catalog on all version 5 and later VNS3 Controllers with network access to the publicly available Cohesive Networks Plugin storage site.

Install the HA Plugin via the Plugin Catalog. Once installed and available, allocate an instance of the HA Plugin.

Step 4: HA Plugin Configuration File

Configure the HA plugin via a configuration file or environment variables. Environment variables are prefixed with HAENV_; the configuration file uses plain yaml keys (no prefix).

Variables

Required Variables:

cloud: aws or azure (google and oci support may be offered in later versions)
my_underlay_ip: Private IP of this VNS3 controller
peer_underlay_ip: Private IP of the peer/pair VNS3 controller

Optional Variables:

sleep_time: Seconds to wait between health checks to the peer controller. Default is 15.
log_level: Level of detail in the plugin’s logs. Values: info, debug, error. Default is info.
failure_threshold: Number of consecutive health check failures before triggering failover. Default is 3.
recovery_delay: Seconds the peer must be continuously reachable again before routes are restored to it after a failover. Default is 90. Any drop in reachability during this window resets the wait, since the peer’s VNS3 API/ping responding doesn’t by itself confirm its IPsec tunnels or BGP sessions have finished renegotiating. This same delay governs both the cloud-route layer and the optional VNS3 route failover layer below.
peer_overlay_ip: Overlay IP of the peer/pair controller (enables the optional overlay network health check)
peer_public_ip: Public IP of the peer/pair VNS3 controller (alternative to underlay IP for health checks)
my_public_ip: Public IP of this VNS3 controller (alternative to underlay IP for health checks)
vpc_endpoint_url: URL of a private VPC interface endpoint for the EC2 API (AWS only). Set this when the VNS3 controller runs in a subnet with no outbound path to the public AWS endpoints (e.g. no NAT gateway/internet gateway) — the plugin needs to reach the EC2 API to read and update route tables, and this setting points it at your VPC’s private EC2 endpoint instead of the public one. Not needed if your EC2 interface endpoint was created with private DNS names enabled (the “Enable DNS name” option, on by default for EC2) — in that case, the standard AWS SDK calls already resolve privately within the VPC automatically, with no plugin configuration required. Only needed when private DNS names are disabled for the endpoint, which requires explicitly pointing the SDK at the endpoint’s own DNS hostname.
dry_run: Test mode - true or false. When true, no actual route changes are made (AWS only). Default is false.
manage_vns3_routes: true or false. Enables the optional VNS3 Route Failover layer. Default is false.
my_vns3_api_password / my_vns3_api_token: This controller’s own VNS3 API credential. Required (exactly one of the two) only if manage_vns3_routes is enabled.
peer_vns3_api_password / peer_vns3_api_token: The peer controller’s VNS3 API credential. Required (exactly one of the two) only if manage_vns3_routes is enabled.

NOTE: You only need to specify host and peer/pair public IPs OR underlay/private IPs depending on the deployment architecture. For most deployments, use underlay/private IPs.

Configuring via Config file

Configure the HA Plugin via Configuration File by accessing the Plugin directly (via SSH and a VNS3 firewall port-forward rule) OR via the Plugin Manager available on VNS3 version 5 and later (recommended).

The configuration file is located at /opt/hacontainer/conf/vars.yml and should be in yaml format. Examples for both controllers:

AWS Example (both controllers use identical config with different IPs):

Controller A:

cloud: aws
my_underlay_ip: 10.0.0.253
peer_underlay_ip: 10.0.1.253
sleep_time: 15
log_level: info
failure_threshold: 3

Controller B:

cloud: aws
my_underlay_ip: 10.0.1.253
peer_underlay_ip: 10.0.0.253
sleep_time: 15
log_level: info
failure_threshold: 3

Azure Example:

cloud: azure
my_underlay_ip: 10.1.0.10
peer_underlay_ip: 10.1.0.11
sleep_time: 15
log_level: info

AWS Example with VNS3 Route Failover enabled:

cloud: aws
my_underlay_ip: 10.0.0.253
peer_underlay_ip: 10.0.1.253
sleep_time: 15
log_level: info
failure_threshold: 3
manage_vns3_routes: true
my_vns3_api_token: <this-controllers-vns3-api-token>
peer_vns3_api_token: <peer-controllers-vns3-api-token>

Configuring via the Environment

Configure the HA plugin via the environment by capitalizing the variable and prepending HAENV_:

HAENV_CLOUD=aws
HAENV_MY_UNDERLAY_IP=10.0.0.253
HAENV_PEER_UNDERLAY_IP=10.0.1.253
HAENV_SLEEP_TIME=15
HAENV_LOG_LEVEL=info
HAENV_FAILURE_THRESHOLD=3

Tip: For more verbose logging you can set log level to debug with the variable log_level. Or via the environment with HAENV_LOG_LEVEL=debug

Step 5: Restart the HA Plugin

Once Steps 1-4 are complete, restart the HA Plugin process via the Plugin Manager executables actions, or restart the plugin directly, so the updated configuration file is picked up.

Monitoring and Logging

The HA Plugin provides comprehensive logging for troubleshooting and monitoring:

Log Files:

/mnt/logs/plugins/ha/process.log - Main process log with health check results and failover actions
/mnt/logs/plugins/ha/my_route_tables_state-initial.json - Initial state of cloud route tables (captured when both controllers are healthy)
/mnt/logs/plugins/ha/my_route_tables_state-latest.json - Current state of cloud route tables
/mnt/logs/plugins/ha/health.state - Current health state (true for healthy, false for unhealthy)
/mnt/logs/plugins/ha/peer_vns3_routes_state-latest.json - Latest snapshot of the peer’s advertised VNS3 routes (only present if manage_vns3_routes is enabled)

Log Verbosity:

Peer-accessibility status logs at info only when it changes (e.g. accessible → down) or once an hour as a heartbeat, so a steady, healthy pair doesn’t flood the log with a repeated status line on every health check. When the peer is unreachable, the log line names which check(s) failed (ping, underlay API, overlay API) to speed up triage. The optional VNS3 route failover layer logs separately under a VNS3 route failover: prefix.

What to Monitor:

Health check results (ICMP and TCP 8000 checks)
Failover events (route table updates)
Recovery events (restoration of original routes)
Route change detection (planned vs. failure-driven changes)
If manage_vns3_routes is enabled: borrowed/returned VNS3 routes and any skipped collisions (VNS3 route failover: log lines)

Log Levels:

error - Only errors and critical issues
info - Normal operations, health checks, failover events (recommended)
debug - Detailed information including route comparisons and state changes

Troubleshooting

Common Issues:

Plugin fails to start:

Verify /mnt/logs mount is available and writable
Check configuration file syntax at /opt/hacontainer/conf/vars.yml
Review /mnt/logs/plugins/ha/process.log for specific errors
A transient failure at startup (e.g. a brief cloud API error) is retried automatically — if it keeps failing, the error in process.log reflects the real underlying cause

Failover not triggering:

Verify ICMP and TCP 8000 are allowed between controllers
Check failure_threshold setting (must have consecutive failures)
Confirm cloud permissions are correctly configured
Review health check logs in process.log

Permission errors (AWS):

Verify IAM role is attached to the EC2 instance
Confirm IAM policy includes all required actions
Check VPC endpoint configuration if using private endpoints

Permission errors (Azure):

Verify managed identity is configured and assigned
Confirm role assignment includes all required permissions
Check resource group scope

Routes not restoring after recovery:

Check whether recovery_delay has elapsed yet — the peer must be continuously reachable for the full recovery_delay window before routes are restored, and any drop in reachability resets that wait
Review state files (initial vs. latest route tables)
Check health.state file status
Verify topology was marked as healthy before failure

VNS3 Route Failover not borrowing/returning routes:

Confirm manage_vns3_routes is true and both my_vns3_api_* and peer_vns3_api_* credentials are set
Remember only advertised routes are eligible — tunnel-backed routes are never failed over
A route logged as skipped due to a CIDR collision with an existing, differently-sourced route needs that overlap resolved manually
Check the VNS3 route failover:-prefixed lines in process.log and peer_vns3_routes_state-latest.json

Automating Your Configuration

And that’s it! But you should only ever have to do this once. This configuration can be totally automated when deploying new network segments.

The API endpoints you would use are the following:

Update your VNS3 firewall allowing plugin network access with POST /firewall/rules
Update your VNS3 route table with POST /routes
Upload the Plugin Image with POST api/container_system/images
Start the Plugin with POST /api/container_system/containers (can pass environment variables)

Legacy Documentation (v2.x/v3.x)

Note: The information below applies to versions 2.x and 3.x of the HA Plugin, which used primary/secondary (and selfish/selfless) operational modes. Version 4.x simplified this to a single “active” mode run identically by both controllers.

Version 2.x/3.x Plugin Modes

While it is recommended you use the latest version 4.x or later HA plugin, versions 2.x and 3.x required different configuration. The HA Plugin in v2.x had two modes with the following operations:

Primary Mode (v2.x/v3.x only):

During normal operation an HA Plugin in primary mode captures the Cloud Provider Route Table settings and stores any routes where the Primary VNS3 controller is the “target”/“next hop ip”.
When Primary recovers after an outage, the HA Plugin looks at the routes stored during normal operations and replaces any of those routes on the Cloud Provider Route Table that are pointing at the Secondary VNS3 Controller.

Secondary Mode (v2.x/v3.x only):

During normal operation an HA Plugin in secondary mode:
- Captures the Cloud Provider Route Table settings and stores any routes where the Primary VNS3 controller is the “target”/“next hop ip”.
- Captures the Primary VNS3 controller’s VNS3 routes and stashes them every N seconds depending on configuration settings.
- Sends heartbeats on a specific and configurable periodicity to the Primary VNS3 Controller to detect if it is down/unresponsive.

Migration from v2.x/v3.x to v4.x

To migrate from version 2.x/3.x to 4.x:

Remove deprecated configuration variables:

Remove mode variable (was primary or secondary)
Remove HAFLAVOR environment variable (was selfish-aws, selfish-azure, or selfless)
Remove my_vns3_id and peer_vns3_id variables

Remove legacy VNS3 API credential variables:

Remove my_api_password variable
Remove peer_api_password variable
These were specific to the removed v2.x/v3.x route-stashing behavior. If you want the modern equivalent, see VNS3 Route Failover (Optional), which uses its own my_vns3_api_password/my_vns3_api_token and peer_vns3_api_password/peer_vns3_api_token variables and is off by default.

Update to active mode configuration:

Both controllers now use identical configurations
Only swap my_underlay_ip and peer_underlay_ip between the two controllers

Add new optional variables (if needed):

failure_threshold for tuning sensitivity
dry_run for testing (AWS only)
manage_vns3_routes (and its credential variables) if you need the optional VNS3-level route failover back

Have any questions? Contact Cohesive Networks support. We take pride in our speedy and high quality support.

Updated on 01 Oct 2026